Is there evidence of CPU saturation?
Running a test on the device is showing:
CPU: 0% usr 12% sys 0% nic 52% idle 0% io 0% irq 34% sirq
Running from outside:
CPU: 0% usr 7% sys 0% nic 31% idle 0% io 0% irq 60% sirq
So maybe? It's certainly higher.
@bmork I think I have tenacity - I really do not like giving up - but lack the requisite skills and knowhow here. Are there any other promising angles? It seems so frustrating and such a shame to think there is something holding back OpenWrt performance that is as of yet unknown.
I believe the fact that we see similar behaviour between QMI using 1500 byte USB transactions and MBIM using 16k or 32k USB transactions points toward a problem between our USB driver and the ethernet/DSA port. The direct test on the NR7101 is interesting. There is no reason this should be faster than forwarding out the switch. I can't explain that at all.
Got a feeling that I should have known more about modern Linux networking with "NAPI" and all that. But I don't.
1 Like
Could it be a memory issue or could receive packet steering help:
https://lkml.kernel.org/netdev/65d97336-d3c8-0fe7-659a-88ddecd1b13e@hartkopp.net/T/
I built an image with performance events and counters enabled for the kernel to see if I could identify where all the CPU time is going.
It does seem like the CPU is overloading whether the traffic is local or forwarded, but not needing to forward is leaving a little bit more for the WAN to go faster.
Looking at the call graph, I think the main load is somewhere in usbnet, but interrupts seem to be disabled during the load, so perf can't identify exactly where. Though I'm really not very familiar with how any of this works.
1 Like
When upgrading the OEM firmware from within the OEM firmware using the web interface, does it write to Kernel or Kernel2, or both?
Is it worth me trying the OEM firmware for a bit to see how I get on? Is the benefit of the latter restricted to high-bandwidth cases?
I seem to recall a fear about upgrading OEM firmware in terms of the device getting locked down to prevent non-OEM firmware from being installed. But I may be misremembering?
Never both, I believe. But it could be either one depending on firmware version, currently active partition and moon phase. Or something like that.
In most cases it will write to the inactive partition.
Any other insights?
Did the link in my post above about receive packet steering offer any fruit?
I see that sirq is much higher in the latter case.
I’m also still wondering how the IP passthrough is implemented by Zyxel and whether that really has no bearing. I came across these posts here:
Do we know how the Zyxel OEM firmware sets up the bridge mode?
I see we can request firmware under GPL here:
Haven't had much time to look into this more. An ISP also installed gigabit G.fast here, so I don't really need this high speed cellular connection anymore.
The device has a 2 core, 4 thread CPU, but the processing for the WAN seems to be entirely single threaded currently, so if packet steering would spread it around more, that could help. I think there might be excessive processing being done in some interrupt handler for the modem, which is causing it to stall to some degree, but I don't really know how to debug that stuff.
AFAIK there's no difference in routing with the IP passthrough enabled, it just makes the DHCP assign the address for the WAN to one client on the LAN side and forwards all incoming ports to it. Performance is the same whether it's enabled or not with the OEM firmware.
1 Like
Has anyone tried ROOTer on this? It should contain LTE specific patches https://www.ofmodemsandmen.com/download.html
1 Like
I have tried that, needs MBIM mode or the modem crashes after a speed threshold. Performance is worse than the original Zyxel firmware
1 Like
But so is OpenWrt apparently. So which is less worse?
And they’ve figured out how to implement pass through IP, right?
The Zyxel firmware is less worse; I'm getting 400mbps down / 120mbps up and ip passthrough where on rooter I get 120 mbps down / 40 mbps up. I haven't tried stock openwrt as there are no complete instructions on how to get it up and running and on my previous attempts I needed to manually install several packages.
I just requested a copy of the firmware code using:
Maybe I can identify something special there?
@bmork have you tried that? Any ideas where to look?
I haven't requested the source code for the NR7101. It's a MT7621 device with a Quectel modem. Didn't think there would be anything new in there.
My best guess is that we're missing something related to ethernet <-> xhci in the MT7621 SoC. Maybe there's a more efficient way we can move buffers between those IP blocks? If there is, then that should affect any MT7621 device with a USB3 port.
Can anyone please test again NR7101 with OpenWrt with the following:
- enable flow offloading and hw flow offloading with bridger installed; and
- install irqbalance and enable packet steering.
We could also perhaps consider a kernel patch to test overclocking the CPU to 1200Mhz.
what is a "bridger installed". What packet it is? I can test everything to get this equipment to function like it does on the original Zyxel software. And on OFW it reaches 600mbis/s on the N78 band in Poland.
I am not an expert on the above techniques, which might help improve throughput, but there are many posts about these on this forum. I understand that for hardware flow offloading to work, the bridger package needs to be installed.
What throughput do you see when using OpenWrt on this device without these techniques?
About 200 -250 mbit/s