IXP425 high traffic error "Budget exhausted after napi rescheduled"

Hello,

I've owned a few Titan Wireless TW5334 (target: ixp4xx ) boards for quite some time. My main issue is that these boards (along with other routers, i.e. WRT54G) would crash or reboot under heavy traffic load or P2P traffic. This made the boards unstable for my intended usage. The crash/reboot issue was resolved around OpenWRT version 12 (Attitude Adjustment).

https://wiki.openwrt.org/oldwiki/openwrtdocs/hardware/zinwell/zw4400

I have now attempted to re-flash the board with 17.01.4 and have had success booting. Now, when performing a speedtest, the device doesn't crash, but the bandwidth slows (especially during ingress/download). I receive this message in the logs for the WAN port (eth1):

Tue Feb 27 15:32:32 2018 kern.warn kernel: [ 133.632913] eth1: Budget exhausted after napi rescheduled

I never made a bug report years ago because others noted similar issues, and I never had a log result. Does anyone have any idea what's taking place?

https://wiki.linuxfoundation.org/networking/napi

It seems like what's going on is that the cpu is unable to process the packets fast enough, the message about "Budget exhausted" seems relevant to that.

Looking at wikidevi on the chipset series they're 266-666 mhz arm single core processors from the 2005 era with 66 mhz ddr1/2 RAM interfaces. https://wikidevi.com/wiki/Intel/Intel_XScale_Processors

Most likely they are simply too weak for modern network speeds.

OK, I did read that site before I posted. That makes sense.

When I tested years ago, the device would crash near its max, that was normal in the old firmware. On the WRT54G, that max was about 30 down/30 up, I don't recall my max speed with the TW-533-4.

I'm receiving about 82 down/87 up (from LuCI GUI) with the TW-533-4 today, and I don't see the error log message again until after reboot.

Thanks @dlakelan!

I did notice this on the Linux site:

Issues
IRQ race a.k.a rotting packet
The are two common race issues that a driver may have to deal with. These are cases where it is possible to cause the receiver to stop because of hardware and logic interaction.

Yeah, that's a lot of bandwidth for a 2005 era device. I'm pretty sure at the time I was hooked up to a less than 2 megabit ADSL and 10mbit cable connections were pretty fast.

It's particularly telling that the issue happens on ingress, because on ingress without napi you're going to get a single interrupt per packet, which at 1500 byte packets and say 85 mbit/s is 7083 interrupts per second minimum (for 500 byte packets, 3x as much 21000 interrupts per second). So somewhere around 50 to 150 microseconds between interrupts. If the interrupt itself takes 10 microseconds to do a context switch back and forth, you might just spend 20% of your time just context switching. So, NAPI should help by letting it shut down the interrupts and spend its time actually processing the packets. It's possible that NAPI tunes itself, so you see this overrun, and then it increases its budget and it doesn't overrun anymore. I dont' know.

1 Like

I found this at: http://lwn.net/2002/0321/a/napi-howto.php3

budget is the remaining number of packets the network subsystem on the current CPU can send up the stack before yielding to other system tasks.

I also found this at: ftp://ftp.dei.uc.pt/pub/linux/kernel/people/hadi/docs/UKUUG2005.pdf

If at the end of its run the network interface has no more packets on its ring, it is moved to the poll_off state and interrupts are re-enabled (an IO write). If, on the other hand, it still had packets left, it is moved back to the poll_on state and as a result rescheduled for a future poll.

So I checked the budget:

root@LEDE:~# sysctl net.core.netdev_budget
net.core.netdev_budget = 300

I increased the budget by 100:

root@LEDE:~# sysctl -w net.core.netdev_budget=400
net.core.netdev_budget = 400

I saw the error again...so I increased it to 500...and something interesting happened...My bandwidth increased to: 92 down/ 98 up (the GUI shows peaks to 103 down/110 up)!!!

I'll monitor and mark this solved soon. Thanks again!

awesome. I did something similar on my router, which is running Debian and 4.14 kernel. You should also look at:

net.core.netdev_budget_usecs

which only exists on recent kernels, so if you're running snapshot. If you run out of time you can wind up rescheduling as well. On my router, a Debian based x86 box, I have budget = 800 and usecs = 5000 on a gigabit fiber connection. Seems to avoid problems I was having earlier during high load.

I agree. I may upgrade to a snapshot in order to test. I've gotten some success changing the following:

sysctl -w net.core.dev_weight=128
sysctl -w net.core.netdev_budget=900
sysctl -w net.core.netdev_max_backlog=100000

100k packets is like 15 or 20 seconds of backlog at full 85 mbit speed. I suspect you would do well to actually set that buffer to about 0.1 seconds worth... which is around 700 to 1000 packets. That means you'll start to drop packets earlier when you receive a packet storm.

1 Like

I agree...I believe the only reasonable option to tweak would be:

Perhaps napi wasn't added until circa-Attude Adjustment (which is why my former devices used to crash under high load). I may test with snapshot, in order to be ready for production with the future release of OpenWRT 17.04.5 (or however it may be named).

I also recalled one more thing...LuCI!!! The IXP425 is a SoC (system on chip) and network CPU.

When logged out, I received much higher numbers. Using the same averages, ~100/100 as was displayed on the LuCI GUI.

I'll monitor (and prepare to re-flash)...