All this time.. lol - Oh well.. I didn't clue in on that either. Thanks so much for your help troubleshooting all of this!!
So now I can simply do iperf3 tests between these 2 networks right? Which will make everything go through both LAN and WAN ports of my Rpi4 right?
If so, how can they communicate between each other? My linuxbox on my main LAN can't see my laptop on my GuestLAN.
it will go through both ethernet devices yes. But you have now two LANs, "testlan" which is just port 5 on your switch... and whatever existing lan you had.
if the lan and testlan interfaces are in the same firewall zone, you should be able to see each other, pinging from the machine on port 5 to a machine on your regular lan should work for example.
ok, so FROM the testlan laptop (192.168.20.121) I can ping ALL my devices on my regular LAN (192.168.10.x). From my Rpi4 I can ping the laptop on the testlan (192.168.20.121), however none of my regular LAN clients/devices can ping the testlan laptop (192.168.20.121)
If this is accurate and my traffic is in fact going through both my WAN (USB3 UE300 eth1) and LAN (onboad nic eth0) ports of my Rpi4 then everything looks pretty good..
note: My regular LAN ip range is back to 192.168.76.x (which it was before I ran the other tests)
Some breaking news (the first one is good! The second one is bringing a lot of information).
The rev 1.4 board (with 8GB) is able to perform like the 1.1 board (with 2GB) if you put the following line into config.txt: total_mem=2048 which is hopefully clearly enough for OpenWrt.
Putting the kernel 5.4.95 from r15696 (dtb, overlays, kernel8.img from boot part, and modules folder 5.4.95 into /lib/modules/ from rootfs) into the old OpenWrt r11631, does not reduces the performance. Old OpenWrt r11631 is performing as good with old and new kernels: 1000 Mbps WAN to LAN or LAN to WAN with ax88179 without regression!
(This time I used my old OpenWrt installation (r11631) for testing my kernels between 4.19 and 5.4, instead of using Raspbian)
Everything is now becoming clear. Here is a summary of the information spread in the thread:
On the old OpenWrt version with 4.19.86 kernel, I had no ksoftirqd/0 high usage (both on integrated NIC and USB NIC)
On the new OpenWrt version with 5.4 kernel, I had high ksoftirqd/0 usage (both on integrated NIC and USB NIC), bringing big performance regression. I supposed it has something in common.
When I saw that modern Raspbian using 5.4 kernel had the same high usage on integrated NIC (while I hadn't this high usage on my old OpenWrt 4.19), I supposed the kernel update was guilty.
When I tested raspbian with 4.19 kernel the high usage on integrated NIC was still here so I didn't understand, and supposed OpenWrt had a patch or a different kernel.
irqbalance and software offloading are giving a little bit of help but not enough to fully compensate the performance loss
New kernels (like 5.10) is performing better on integrated NIC. But it does nothing more than before on USB NIC!
rtl8153 driver is eating a little bit less ressources. On new 1.4 boards with 8GB of RAM at least, it performs better than ax88179 but still is impacted by the performance regression
however, with 2GB RAM, things are performing better (and the max bandwidth regression is not visible anymore for rtl8153).
Finally, no matter which kernel you are using, all of them are able to have both low of high ksoftirqd/0 usage depending on the OS settings. I still haven't found which OS settings (/etc/config/files show not differences)
OpenWrt upgrade is finally causing the trouble, not the kernel!
Just as a reminder, I'm using the following configuration:
eth0 is integrated NIC, used for the LAN (with a DHCP server enabled on it), connected to the house using dumb switches
eth1 is USB NIC, used for the WAN (as DHCP and DHCPv6 client), directly connected to the fiber "ONT" adapter (GPON to GbE)
no more specific configuration
@Cylac with 939 Mbps displayed into iperf you have no performance issue anymore I didn't follow everything but I hope what you did with the help @dlakelan made your RPi4 working full Gigabit
@fosix I don't know if it's related, but as I'm going to soon use OpenWrt again (probably with my rtl8153 adapter) I'll keep you informed if it goes from "working fine" to "performance drops" after few hours of usage. As the CPU usage is quite high recently, even if able to route 1000 Mbps in both way (LAN to WAN or WAN to LAN), be sure your CPU isn't reaching thermal throttling temperature after few hours! it could be a reason why performance drops. Mine has a little fan on it so unless the fan stops, it won't reach thermal throttling
Conclusion:
There is just 1 question left, and then, everything will be answered about this issue. What is the difference between r11631 and r15696 which is increasing the ksoftirqd/0 usage?
I just tested, with 4096 it's running a little bit slower (in both way, iperf3 is showing ~830 Mbps instead of 933 Mbps with 2048).
(I don't know what is causing this performance drop when rising memory amount).
(By the way It would be interesting to lower at 1024 to see if the performance is increasing again )
I placed the rootfs files of OpenWrt r11631 (old) and r15696 (new) into those links so that anybody can try to find what is (or are?) the difference(s) making the old one performing better than the new one (when using the same kernel and same boards)
A lot of others things changed too, but the thing that disappeared (smp tune, "smp" stands for Symmetric multiprocessing) is now "SMP packet steering" and is a new functionality (disabled by default) that is available on the same tab that the one that is used for changing LAN IPv6 prefix
(believing my test, a reboot is needed after activation, to take the activation into account)
Also, when using boards with 4GB or (even worst) 8GB, the performance decreases compared to 2GB one. So, also adding total_mem=2048 into config.txt file, will make current snapshot to perform as good as the old one, even with ax88179, even with new boards, for 1000 Mbps WAN to LAN (or 1000 Mbps LAN to WAN) with no CPU bottleneck
Thanks to everyone who participated into this thread
(I'll probably use the rtl8153 anyway now that I have one, just to test some full duplex routing test, which means running 2000 Mbps on each NIC. The ax88179 is able to handle full duplex, when connected to a big computer, but on the RPi4 it's struggling a little bit - rtl8153 will probably struggle too, but maybe a little bit less).
(New 5.10 kernel is using less CPU ressources for integrated NIC compared to 5.4 so for extreme performance tuning, new kernel, software offload, and irqbalance added to total_mem=2048 and packet steering may do a great job).
I'm playing at full duplex through the Pi4 with OpenWrt SNAPSHOT r15696-17fa01bb79 with kernel "5.4.95"
~ 940 Mbps both ways (upload + download at the same time).
In order to avoid flooding the Internet I placed my actual LAN on the WAN side (eth1, rtl8153) and a computer on the Pi4 LAN side (creating a LAN behind another LAN). Traffic between the 2 LAN's is running trough the RPi4 as it would for Internet traffic.
I'll keep you informed if it's slowing down during the following hours. But for now it's performing really great
Let's leave the speed out of the equation for the moment.
The bigger issue is that I'm getting 1.0 load on the cpu without the router doing anything.
For now the only remedy is a reboot, then it's working as usual and after a few hours this 1.0 load appears out of nowhere and stays there untill I reboot the device.
CPU load counter stays at 1.00 untill the rest (5min, 15 min) catch up.
Performance drops after few hours (400-500mbps), after restart everything's back to normal (800-900mbps)