See the test results I just posted with my own builds. You should be able to hit gigabit with SQM on. If not, maybe the bottleneck is the server you're testing against? I initially had some lower than expected results and it turned out my iperf3 server on the other end was a bottleneck. Re-tested against my workstation and got the expected results.
1 Like
Yeah, after installing the build you gave me (the OC one) I managed to get gigabit with SQM easily 
1 Like
I'd like to give your build a try as I am having IPv6 issues with a different build from github.
I see a kmod-r8169 in the root of your share along with another llvm file. Are these needed, or just the ext4 sysupgrade file?
Thanks.
You only need ext4/squashfs sysupgrade file (also packages folder if you want to install something which requires kmod packages). I uploaded whole builder output folder.
kmod-r8168 is alternative lan nic driver. There is already kmod-r8169 included in the build.
1 Like
Question, did you see any ping spikes when you did irq on the a53 cores and queues on the a72 cores? Also, are two a53 cores enough for the irqs? Leaving like 2 a53 cores for some docker application + disk IO?
Amazing tests btw! Clarifies so much!
For clarity, do you mind resharing the code to accomplish this (placed in /etc/rc.local I guess). You posted above but want to be crystal clear based on your last round of tests. Thanks!
Yeah, putting the commands in rc.local should work, assuming the IRQ numbers are somewhat static, which they seem to be. You may just want to double check the IRQ numbers in /proc/interrupts and then do a reboot and check again to make sure they stay the same. I haven't seen mine change yet, but they could possibly change with future builds/kernels.
#eth0 IRQ on A53 cores
echo f > /proc/irq/35/smp_affinity
#eth1 IRQ on A53 cores
echo f > /proc/irq/87/smp_affinity
#eth0 queue on A72 cores
echo 30 > /sys/class/net/eth0/queues/rx-0/rps_cpus
#eth1 queue on A72 cores
echo 30 > /sys/class/net/eth1/queues/rx-0/rps_cpus
If you want to get more fancy, you could get the IRQ with a command instead of hard coding it:
#eth0 IRQ
echo f > /proc/irq/`grep eth0 /proc/interrupts|awk -F ':' '{print $1}'|xargs`/smp_affinity
#eth1 IRQ
echo f > /proc/irq/`grep eth1 /proc/interrupts|awk -F ':' '{print $1}'|xargs`/smp_affinity
The grep command prints the line from /proc/interrupts, awk prints the first column which is the IRQ number, and xargs trims the whitespace.
2 Likes
I didn't run any ping tests during the iperf tests above and docker was not running. I just wanted to find out which cpu configuration would provide the highest SQM throughput for a networking-only load. I am currently running another round of tests to try to find out which configuration provides the best pings with docker running on various cores.
Not sure if anyone is following the soft reboot issue, but see: https://github.com/openwrt/openwrt/pull/4860#issuecomment-1021594298
I do not have the hardware yet. Anyone willing to test?
Just tested a torrent with unlimited connections with cake on a 400 mbit connection and pings are stable...
Looks like that two a53 cores are enough for the irq.
1 Like
I'm still seeing an odd issue where I get one or two high pings (80-150ms) while the bufferbloat test is warming up.
Background info:
- Actual tested line speed is 110 / 8Mb
- Cake / piece_of_cake is set to 97 / 7.2Mb
- Normal average ping time to google is 25ms
- I start pinging google.com from the router, then run waveform bufferbloat test
- About 50-75% of the time while the download portion is "warming up", I will see one result of 80-150ms on the background google ping
- Bufferbloat test result is still A+ every time
- Every once in a while, bufferbloat test records one or two 80-100 pings. But most of the time I only see it on the background google ping
I have tried the following:
- stopped docker and unplugged the USB drive, still happens
- moved IRQs and queues around to different cores, still happens
- set performance governor on all cores, still happens
- switched from cake to fq_codel, still happens
- dropped cake bandwidth to 90Mb, still happens
- tried other speed tests (wifiman, speedtest.net, fast.com)... still happens but not as bad. I still see 50-60 pings sometimes right when the test starts.
The only other thing I can think of to try is seeing if I can reproduce on a different router. Or maybe it could be my switch. Maybe I should just not worry about it since I'm still getting an A+ bufferbloat, but my understanding was that pings of 100+ shouldn't really be possible with cake turned on and set to a reasonable bandwidth, and I can reproduce it consistently. Will update if I figure anything out.
Note: When I first got the R4S I was having tons of high pings all over the place, frequently 250, 500, 1000ms. That went away when I stopped using a 2.5" magnetic USB drive. However the current issue doesn't seem related since it happens with no USB drive connected.
I have compiled OpenWRT with this patch. Works for me! 
Here is latest master branch of OpenWRT with new reboot fix patch (disables problematic UHS for uSD) and reverted firewall4 commit (still uses old firewall). Also has luci included and 1024m rootfs.
ext4:
https://drive.google.com/file/d/1hBTOIfSidnWPGafyVyyl6vcKYDeXU559/view?usp=sharing
squashfs:
https://drive.google.com/file/d/1V_RO2RyqLkq1m_X-4MNZisvUI4GpnhGF/view?usp=sharing
kmod packages (also includes r8168 lan driver kmod, but image uses default openwrt r8169 driver) and public key for "local repo":
https://drive.google.com/drive/folders/1PxlDwV62Ln9IiVFzd-y2RxU-cv4iAHRs?usp=sharing
I did a test using a torrent with unlimited connections...
First without SQM:
And after with SQM:
Its not perfect since there were a few ping spikes here and there(not constant, as you can see in the avr ping), but i think its pretty good!
3 Likes
Thanks for testing, that does look pretty good.
I tried a few more things on my end:
- Tried r8168 driver, still happens
- Disabled banip and adblock, still happens
- Tried a NanoPi R2C, still happens
- Tried plugging my workstation directly into the R4S lan port to bypass the switch, still happens
- Tried a completely different platform (mt7621 running stock 21.02.1), still happened but even worse.
So I came to the conclusion that it has nothing to do with the R4S... it seems to be a more general issue. Maybe it's just an edge case that SQM doesn't handle well. Nobody in the house is complaining though, so I'm going to call it good. It's at least nice to know that it's not an R4S issue.
Knowing that, the good news is I can probably push my docker workload a little higher (by increasing security camera fps). For now since my internet is slow, I'm going to leave all the networking on the A53 cores and run docker on the A72 cores. If my ISP ever upgrades to gigabit, I'll put docker on two A53 cores, IRQs on two A53 cores, and queues on the A72 cores.
2 Likes
It seems that soon we will have 100% working official Openwrt https://github.com/openwrt/openwrt/pull/4860#issuecomment-1022511047 
2 Likes
I hope so, snapshots are far from ideal for daily stable driving for a router that you want to setup and forget.
Only issue that current snapshot had was reboot issue. I run my R4S as my main router with git main branch and only reboot patches applied.
So, is really enough to put it on rc.local? Cause i did the commands manually on the shell and did not restart the router at all on the past 3 days, but when i checked the cpu usage before doing those torrent tests, only the a72 cores were being used at all! Meaning that the irq went back to the a72 cores since all a53 cores were idle when doing the torrent testing but after running the commands again 2 a53 cores got 50% use when doing the torrent with sqm.
What i mean is that rc.local will only execute the commands once after a reboot right? So its just like executing the commands manually on the shell, but it seems that the modification did not stick at all after some time! But will monitor to confirm it, in other words, if the modification is going to get undone again.
Dropped that into the wiki for others. https://openwrt.org/toh/friendlyarm/nanopi_r4s_v1#cpu_management
added this too