#eth0 interrupts on core 0
echo 1 > /proc/irq/35/smp_affinity
#eth0 queue on core 1
echo 2 > /sys/class/net/eth0/queues/rx-0/rps_cpus
#eth1 interrupts on core 2
echo 4 > /proc/irq/87/smp_affinity
#eth1 queue on core 3
echo 8 > /sys/class/net/eth1/queues/rx-0/rps_cpus
Your IRQ numbers might be different... you can find them with "grep eth /proc/interrupts". First column is the IRQ, so replace those in the commands above. The number you echo into the file defines which CPUs to use. It's basically a binary number where each bit represents one CPU, then convert that to hex. Rightmost bit is CPU 0. For example:
00000001 > hex 1 > cpu 0
00000010 > hex 2 > cpu 1
00001111 > hex F > cpu 0,1,2,3
00110000 > hex 30 > cpu 4,5
A great deal of the excitement associated with people sending money to Friendly ELEC for this small wonder is the fact that it runs OpenWrt FOSS. The OpenWrt maintainers surely know their part in this.
Seeing as how you enablers just convinced me to order an R4S, I sure hope that particular idiot was not affiliated with Friendly ELEC. Only the most altruistic of maintainers would help someone that shoots flames back at them for their trouble earn more money from their free labor.
Hi all,
I've a new NanoPi R4S for sale in Europe. I ordered two a few months back and my network has been working well with one single device, so I sale the second device.
Drop me a mail if someone interested.
Thank you,
Peter
I swapped the connections and tested the unloaded lag and the whether the WAN is connect to the UE300 or to the internal NIC, the lag is the same, 18-20 ms. My thought was I might get better not going the USB but that isn't the case.
Did some SQM testing with iperf3 using 3 different builds:
All tests done with cake / piece_of_cake set to 1Gb up/down
Stock build: 1.8 / 1.4ghz, r8169 driver
OC build: 2.2 / 1.8ghz, r8169 driver
r8168 build: 2.2 / 1.8ghz, r8168 driver
//////// Default IRQ and queue affinity //////////
Stock build:
egress: 935
ingress: 917
bidirectional: 898 / 750
OC + r8169 build:
egress: 940
ingress: 916
bidirectional: 890 / 800
OC + r8168 build:
egress: 935
ingress: 917
bidirectional: 890 / 790
//////// IRQs and queues on A53 cores only //////////
Stock build:
egress: 940
ingress: 819
bidirectional: 884 / 710
OC + r8169 build:
egress: 940
ingress: 920
bidirectional: 890 / 690
OC + r8168 build:
egress: 941
ingress: 920
bidirectional: 877 / 715
//////// IRQs and queues on A72 cores only //////////
Stock build:
egress: 936
ingress: 920
bidirectional: 902 / 776
OC + r8169 build:
egress: 934
ingress: 920
bidirectional: 910 / 835
OC + r8168 build:
egress: 932
ingress: 920
bidirectional: 910 / 858
//////// IRQs on A72 cores, queues on A53 cores //////////
Stock build:
egress: 936
ingress: 885
bidirectional: 882 / 655
OC + r8169 build:
egress: 934
ingress: 920
bidirectional: 884 / 680
OC + r8168 build:
egress: 932
ingress: 910
bidirectional: 874 / 680
//////// IRQs on A53 cores, queues on A72 cores //////////
Stock build:
egress: 940
ingress: 920
bidirectional: 875 / 899
OC + r8169 build:
egress: 940
ingress: 920
bidirectional: 887 / 888
OC + r8168 build:
egress: 939
ingress: 920
bidirectional: 843 / 902
////// TAKEAWAYS
Any of the cores seem to be able to handle gigabit in either direction. However when you put the queues on A53 cores and run a bidirectional test, the A53 cores struggle to keep up and drop to around 880 / 700 Mb
When you put queues on the A72 cores, they can almost keep up with a full gigabit bidirectional load (almost 900 both ways)
Overall the best result I got was by putting the IRQs on the A53 cores and queues on the A72 cores.
The overclock and r8168 driver didn't seem to matter a whole lot. Most of the results on all the builds were within the variability between runs.
I would say the overclock is not worth it unless you need it for docker / other stuff.
The ingress speeds on all tests were somewhat unstable. Speeds were around 940Mb for the most part but would drop to around 750 about every 5 seconds, bringing the average down to around 920Mb.
I would call this device borderline for a symmetrical gigabit connection. I had to drop SQM to 850Mb up/down to get stable bidirectional performance. Asymmetrical is more doable and I managed to get stable speeds at 1000 / 700.
I am curious if using friendly arm own implementation of openwrt if it makes any difference? Just a shot in the dark, but we might see more performance there? Otherwise we should question them i think, cause this device is suppose to achieve gigabit sqm.
Just realized the limiting factor was that the iperf3 server was running on the R2C. I thought if the R2C had SQM turned off it could handle gigabit no problem. But looking at the CPU load during the bidirectional test, 2 of the cores on the R2C are maxed out. So I'm re-testing with my workstation as the server and will update the results, but it's looking better so far.
Edit: updated results and added comparison of stock build and OC build with r8168.
See the test results I just posted with my own builds. You should be able to hit gigabit with SQM on. If not, maybe the bottleneck is the server you're testing against? I initially had some lower than expected results and it turned out my iperf3 server on the other end was a bottleneck. Re-tested against my workstation and got the expected results.
I'd like to give your build a try as I am having IPv6 issues with a different build from github.
I see a kmod-r8169 in the root of your share along with another llvm file. Are these needed, or just the ext4 sysupgrade file?
Thanks.
You only need ext4/squashfs sysupgrade file (also packages folder if you want to install something which requires kmod packages). I uploaded whole builder output folder.
kmod-r8168 is alternative lan nic driver. There is already kmod-r8169 included in the build.
Question, did you see any ping spikes when you did irq on the a53 cores and queues on the a72 cores? Also, are two a53 cores enough for the irqs? Leaving like 2 a53 cores for some docker application + disk IO?
For clarity, do you mind resharing the code to accomplish this (placed in /etc/rc.local I guess). You posted above but want to be crystal clear based on your last round of tests. Thanks!
Yeah, putting the commands in rc.local should work, assuming the IRQ numbers are somewhat static, which they seem to be. You may just want to double check the IRQ numbers in /proc/interrupts and then do a reboot and check again to make sure they stay the same. I haven't seen mine change yet, but they could possibly change with future builds/kernels.
#eth0 IRQ on A53 cores
echo f > /proc/irq/35/smp_affinity #eth1 IRQ on A53 cores
echo f > /proc/irq/87/smp_affinity #eth0 queue on A72 cores
echo 30 > /sys/class/net/eth0/queues/rx-0/rps_cpus #eth1 queue on A72 cores
echo 30 > /sys/class/net/eth1/queues/rx-0/rps_cpus
If you want to get more fancy, you could get the IRQ with a command instead of hard coding it:
I didn't run any ping tests during the iperf tests above and docker was not running. I just wanted to find out which cpu configuration would provide the highest SQM throughput for a networking-only load. I am currently running another round of tests to try to find out which configuration provides the best pings with docker running on various cores.
I'm still seeing an odd issue where I get one or two high pings (80-150ms) while the bufferbloat test is warming up.
Background info:
Actual tested line speed is 110 / 8Mb
Cake / piece_of_cake is set to 97 / 7.2Mb
Normal average ping time to google is 25ms
I start pinging google.com from the router, then run waveform bufferbloat test
About 50-75% of the time while the download portion is "warming up", I will see one result of 80-150ms on the background google ping
Bufferbloat test result is still A+ every time
Every once in a while, bufferbloat test records one or two 80-100 pings. But most of the time I only see it on the background google ping
I have tried the following:
stopped docker and unplugged the USB drive, still happens
moved IRQs and queues around to different cores, still happens
set performance governor on all cores, still happens
switched from cake to fq_codel, still happens
dropped cake bandwidth to 90Mb, still happens
tried other speed tests (wifiman, speedtest.net, fast.com)... still happens but not as bad. I still see 50-60 pings sometimes right when the test starts.
The only other thing I can think of to try is seeing if I can reproduce on a different router. Or maybe it could be my switch. Maybe I should just not worry about it since I'm still getting an A+ bufferbloat, but my understanding was that pings of 100+ shouldn't really be possible with cake turned on and set to a reasonable bandwidth, and I can reproduce it consistently. Will update if I figure anything out.
Note: When I first got the R4S I was having tons of high pings all over the place, frequently 250, 500, 1000ms. That went away when I stopped using a 2.5" magnetic USB drive. However the current issue doesn't seem related since it happens with no USB drive connected.
