I can't leverage 1Gbps, high softirq cpu

[carofe]

Hi,
I activated 1Gps internet and I can only get around 500Mbps download speed. I think the bottleneck is the router. When I run a test with speedtest.net, this is what sysstat shows:

04:23:09     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
04:23:10     all    1.00    0.00    5.50    0.00    0.00   93.00    0.00    0.00    0.00    0.50
04:23:10       0    2.00    0.00    8.00    0.00    0.00   90.00    0.00    0.00    0.00    0.00
04:23:10       1    0.00    0.00    3.00    0.00    0.00   96.00    0.00    0.00    0.00    1.00

04:24:35        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
...
04:24:36         eth2  52483.17  52211.88  63753.56  63348.59      0.00      0.00      0.00     52.23
...
04:24:36         lan4   9910.89  42241.58    526.84  62415.52      0.00      0.00      0.00     51.13
04:24:36          wan  42346.53   9920.79  61977.65    662.71      0.00      0.00      0.00     50.77

04:25:36        IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
...
04:25:37         eth2      0.00      0.00      0.00    301.00      0.00      0.00      0.00      0.00      0.00
...
04:25:37         lan4      0.00      0.00      0.00    106.00      0.00      0.00      0.00      0.00      0.00
04:25:37          wan      0.00      0.00      0.00    445.00      0.00      0.00      0.00      0.00      0.00
...
04:25:37       br-lan      0.00      0.00      0.00     75.00      0.00      0.00      0.00      0.00      0.00

All CPU cores get 100% utilization on softirq, probably handling net_rx ones. Wan dev utilization is only 50% as expected. RX drops goes pretty high, probably because of the CPU saturation.

I have already played with the eth dev features with ethtool but no improvement.

My next option is playing round with the kernel tuneables with sysctl for the TCP stack, but I was wondering if there are some specific things I should look at, or just buy a more powerful router.

My router is a Netgear R8000 https://openwrt.org/toh/netgear/r8000

Thank you.

[NPeca75]

Hi

out of curiosity ... if you want to give it a try,
19.07.10 has swconfig driven network
maybe you could try to downgrade and make speed test
this way it will be clear, it is pure CPU problem, or if it is DSA, then there are a hope that one day things maybe sorted out

edit:
according to @slh

bcm53xx has only been migrated to the DSA based b53 switch driver in late October (master), 21.02.x is still using swconfig (and will continue for its remaining release cycle).

so maybe 21.02 if you wish

[carofe]

I would like not to downgrade. I just upgraded to 22.03.5 like a month ago.

[carofe]

@NPeca75 , is there a way if the dsa-based b53 switch driver is available and enabled?

[carofe]

OK, I found this: https://openwrt.org/docs/guide-user/network/dsa/converting-to-dsa
But I don't see any VLAN setup in my router.

I have a switch in my office that connects to the router, which is that lan4 you see. My internet test go from the switch to the router to the internet. The speed between computers connected to the switch reaches close to 1Gbps.

[carofe]

These are some kernel logs showing some DSA message:

[    4.684027] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    4.697233] 8021q: 802.1Q VLAN Support v1.8
[    4.701529] Registering SWP/SWPB emulation handler
[    4.709607] b53-srab-switch 18007000.ethernet-switch: found switch: BCM53012, rev 5
[    4.835063] b53-srab-switch 18007000.ethernet-switch lan1 (uninitialized): PHY [dsa-0.0:00] driver [Generic PHY] (irq=POLL)
[    4.847553] b53-srab-switch 18007000.ethernet-switch lan2 (uninitialized): PHY [dsa-0.0:01] driver [Generic PHY] (irq=POLL)
[    4.860039] b53-srab-switch 18007000.ethernet-switch lan3 (uninitialized): PHY [dsa-0.0:02] driver [Generic PHY] (irq=POLL)
[    4.872550] b53-srab-switch 18007000.ethernet-switch lan4 (uninitialized): PHY [dsa-0.0:03] driver [Generic PHY] (irq=POLL)
[    4.885103] b53-srab-switch 18007000.ethernet-switch wan (uninitialized): PHY [dsa-0.0:04] driver [Generic PHY] (irq=POLL)
[    4.896977] b53-srab-switch 18007000.ethernet-switch: Using legacy PHYLIB callbacks. Please migrate to PHYLINK!
[    4.907902] DSA: tree 0 setup

[egc]

Your router has a dual core Armv7 (A9) running at 1 GHz.
I do not think it is capable of doing LAN<>WAN at 1 GB/s.

Broadcom has proprietary hardware offloading called CTF (Cut Through Forwarding) which will get you 1 GB/s (note this is incompatible with SQM) but I doubt that is available on OpenWRT.

[slh]

CTF is not available, and these SOCs are a bit old as well - I wouldn't have expected more than ~200 MBit/s with OpenWrt.

[carofe]

@egc @slh
Thank you for the feedback.
Do you recommend some router hardware that can do 1Gbps with OpenWRT?

[egc]

There are a few, I do not know them all so just my personal take:
Netgear R7800:

Techdata: NETGEAR R7800

But only if you use the NSS builds which are not regular builds.

More modern and very powerful:

Techdata: Dynalink DL-WRX36

Something which maybe is not permitted to share in this forum but other third party firmwares might use the Broadcom CTF module so as a stop gap you can try those.

I administer some Broadcom Netgear R6400 and R7000 which are using another third party firmware (because they are not well supported on OpenWRT due to the Broadcom closed source code) and those can do around 900 Mb/s LAN<>WAN with CTF&FA enabled.

However I cannot guarantee that works for your R8000 with its tri band configuration it is something of an odd duck.

[slh]

The r7800 is certainly a top device, but it's no longer in production and 'only' 802.11ac (and modern 802.11ax successors can be even cheaper than their 802.11ac predecessors). Without NSS, ipq8065 is good for (plain-) routing up to 500-550 MBit/s (which will reduce a tad with DSA in the medium term future), less with PPPoE, SQM or VPN on top. Personally I would not suggest buying any device with NSS (which is very, very unlikely to 'ever' be merged into OpenWrt/main) in mind, if you already own it and can extend its lifetime that way, fine - but just don't rely on its presence as a buying decision.

Simple 1 GBit/s routing (at wirespeed) should be possible with mt7622bv+mt7915 or filogic 830 based devices (sadly not quite with ipq807x, for the same NSS rooted reasons, otherwise these would be great alternatives as well), but if you want more of a margin, you end up at.

So you have 500Mbps-1Gbps fiber and need a router READ THIS FIRST

It's such a frequently asked question over the last few years that we really need a post we can point people to. So your cable company rolled out 500Mbps or 1Gbps download speeds, or you have 1Gbps symmetric fiber (GPON/EPON) from ATT or Centurylink or Orange or whoever. You realize your old all in one router from 2009 is not up to the task of handling this, but hey by now stuff should be cheap and available that will handle your new fiber connection right? So you want advice about a router for…

OpenWrt on x86_64 and a simple (OpenWrt supported-) 802.11ax AP makes sense these days, see e.g. these for low-cost entry-level devices

Tips for getting cheap used x86-based firewall with full Gbit NAT (a PC Engines APU) if you are in the US

While I was watching for old stuff I could play with on ebay I noticed that a few sellers are offering a few "cloud managed routers" (i.e. if you don't pay a subscription they are paperweights) from a company called "simplewan" that looked oddly familiar. here some pictures (from auctions, not mine) they are bright orange. [simplewan 3] [simplewan2] [simplewan 1] The images are a dead giveaway, these devices seem to just be a branded version of PCEngines APU1 (the ones with black USB 2.0 p…

Other options (beyond mt7622bv+mt7916 or filogic 830) include

• RaspberryPi 4, with a second rtl815x based USB3 ethernet card, sadly still scarce and way overpriced
• NanoPi r4s and similar (supported-) rockchip based devices
• various 4-port x86_64 mini-PCs (starting new around 180 EUR)

BananaPi BPi3/ BPi4 are a bit more diy devices, but also quite promising.

[carofe]

@slh, @egc , thank you for your recommendations! I'll be looking into those.

[carofe]

I'm actually liking the option of a Mini PC for general ethernet, in combination with R8000 connected to it to give wifi access.

Thank you, again, for the guidance. I'm glad OpenWRT exists, and it is so active. It is in my list of opensource projects I financially donate to.

[carofe]

I ended up buying a Protectli mini pc. I'm getting now the 1Gbps over ethernet :). I'm using the R8000 as an access point for wifi.

Thank you!