Netgear R7800 exploration (IPQ8065, QCA9984)

quarky · July 30, 2019, 1:23am

@Pedro @fantom-x did both of you include the shortcut-fe flow-offload module in your builds? Without that, AFAIK, only the x86 CPUs can handle gigabit line speed. The netfilter stack is really CPU intensive. Without some form of flow-offload, I don't think any of the existing router SoCs can handle gigabit line speed. Adding to that the qdisc shaping, the ipq8065 will be grasping for 'air' trying to keep up.

Pedro · July 30, 2019, 1:24am

yep. it's there and enabled.

fantom-x · July 30, 2019, 1:25am

No, I did not include that. But without SQM and with software offload the router can reach 800..900Mbps already.

Pedro · July 30, 2019, 1:26am

I mean, you mean the kmod-ipt-offload? that is the one that I enabled. I'm running master without any patches except for the mib one that fantom-x just confirmed is not needed anymore.

Pedro · July 30, 2019, 1:32am

just to make sure, this is my .config file:

https://pastebin.com/nh7hcktq

quarky · July 30, 2019, 1:48am

No, that's not it. If you want, you can try the following patches (it's for 4.4, but should work with 4.14/4.19 with minor tweaks):

https://github.com/quarkysg/openwrt/blob/lede-17.01-ipq806x-nss/target/linux/generic/patches-4.4/999-01-shortcut-fe-support.patch
https://github.com/quarkysg/openwrt/tree/lede-17.01-ipq806x-nss/package/kernel/shortcut-fe
https://github.com/quarkysg/openwrt/tree/lede-17.01-ipq806x-nss/package/kernel/fast-classifier

Use the fast-classifier Connection Manager and disable the Shortcut-FE Connection Manager.

AFAIK, only 4.19 has software flow-offload and it's not as stable as the QCA shortcut-fe. IMHO, existing router SoCs really need some form of flow-offload as they just don't have enough grunt to handle Linux's networking stacks.

My fast-classifier CM code has some issues I've fixed. Do try if you're interested.

The Krait CPU would be near 100% trying to keep that going. With WiFi and SQM all trying to have a slice of the CPU pie, bandwidth will suffer. The ipq8065 as it stands really need flow-offload, the NSS core or both.

Do note that with the shortcut-fe enabled, only egress qdisc will work. Ingress qdisc will be bypassed. I think this is true for all forms of flow-offload, even with the NSS core. I have yet to experiment with the NSS qdisc ingress tho.

For me I applied codel qdisc (offloaded to the NSS cores) on all my interfaces, i.e. WiFi, WAN, ethernet, WireGuard tunnels. I'm able to get A rating with DSLreport for my 500/100mbps internet link using LAN cable, with 32 threads.

lantis1008 · July 30, 2019, 1:56pm

Will get back to you this weekend

Chief · July 30, 2019, 3:02pm

Pedro:

I'm running latest master with kernel 4.14. I can do codel at 200mbps but not cake. cake tops out at around 140.
And codel at 200 is also near its limit. I think one of the cpu's ( the one dealing with the codel thread ) runs at around 90 % when shaping a 200/100 connection.

The only change I did to the master code was disabling the switch pooling stuff that was making ksoftirqd eat all the cpu, and I also run this on rc local:
echo 2 > /proc/irq/31/smp_affinity
echo 2 > /proc/irq/32/smp_affinity


echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor
echo 800000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_max_freq
echo 800000 > /sys/devices/system/cpu/cpufreq/policy1/scaling_max_freq
sleep 1
echo 1750000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_max_freq
echo 1750000 > /sys/devices/system/cpu/cpufreq/policy1/scaling_max_freq
ethtool -C eth0 tx-usecs 0
ethtool -C eth1 tx-usecs 0
ethtool -C eth0 rx-usecs 31
ethtool -C eth1 rx-usecs 31


exit 0

140Mbps with cake is the result of two bugs in openwrt bsp for IPQ. Fix the two and this rate will be doubled. It is really funny, how you guys spent months with tweaking and unstable nss code etc.

Pedro · July 30, 2019, 3:04pm

what bugs? can you elaborate?

Pedro · July 30, 2019, 3:49pm

It appears that you have fixed the problem, at least by this comment you made on another thread:

any reason why you are not sharing the fix?

quarky · July 31, 2019, 12:57am

The stability issue with the NSS drivers have been resolved, sort of. My R7800 has been stable for more than 3 weeks with the NSS cores offloading routing tasks. With the NSS cores offloads, my R7800 can push close to gigabit line speed (the last time I tested, it achieved 940mbps) with less than 5% Krait CPU utilisation. So it leaves all the CPU cycles of the Krait to WiFi and other tasks, mainly VPN. Without the offload, the Krait CPU would spend most of it's CPU cycles processing netfilter and routing rules at high bandwidth, leaving very little for other tasks.

IMHO, getting the NSS cores to work would be most beneficial to the ipq806x based routers. All other tweaks as it stands will not provide much benefit.

Videopac · July 31, 2019, 5:22am

That is good news. But this NSS driver stability issue resolution isn't in snapshot builds yet, right?

Chief · July 31, 2019, 5:53am

quarky:

Chief:

It is really funny, how you guys spent months with tweaking and unstable nss code etc.

The stability issue with the NSS drivers have been resolved, sort of. My R7800 has been stable for more than 3 weeks with the NSS cores offloading routing tasks. With the NSS cores offloads, my R7800 can push close to gigabit line speed (the last time I tested, it achieved 940mbps) with less than 5% Krait CPU utilisation. So it leaves all the CPU cycles of the Krait to WiFi and other tasks, mainly VPN. Without the offload, the Krait CPU would spend most of it's CPU cycles processing netfilter and routing rules at high bandwidth, leaving very little for other tasks.

IMHO, getting the NSS cores to work would be most beneficial to the ipq806x based routers. All other tweaks as it stands will not provide much benefit.

That's just based on your limited knowledge. If the bugs I know off are fixed, then it will do nat without software offload at only around 20% cpu load. If it is stable only in one environment I wouldn't call that stable for all. And what about all the other things newer kernel support, sqm etc. , unless these issues are fixed it will never go into mainline.

OEM with nss cores cannot achieve the same sqm throughput compared to my fixed build

ratking · July 31, 2019, 6:50am

What is it you "fixed"?

lantis1008 · July 31, 2019, 7:14am

And do you intend on sharing what it is you've fixed?

Videopac · July 31, 2019, 7:58am

The power of open source is sharing, so: share what you know or have built.

Pedro · July 31, 2019, 9:11am

I think these are wonderful news. Having another way of extracting more performance from these routers can only be seen as positive, and on top of that if made on a easy non invasive way then it's even better. so the news you are sharing regarding those two bugs on bsp code and their performance effect are amazing.

I truly hope that you can somehow assist on fixing them in trunk. I'm sure that if you haven't yet provided a patch it is because either your code is not yet up to standards and you are working on it or maybe you have some NDA commitments or even some other contractual obligations. Anyway, I hope you can find a way of helping out.

having said that, I have to be honest and tell you that, well, maybe because of my lack of english knowledge, I didn't quite appreciate how you exposed the situation. It almost seems as if you are trolling everyone, and a bit of lack of respect to the OpenWRT developers and actual users like me. A bit like showing a delicious lolipop to a baby and then trowing it away to the garbage. This for sure must be a fault on my english skills.

fantom-x · July 31, 2019, 12:10pm

Your English is great, but pls stop feeding the troll. He got nothing to show.

Ansuel · July 31, 2019, 2:47pm

Hey i fixed my kernel my router is now more powerful than my i7

fantom-x · July 31, 2019, 3:41pm

Well, at least you did not create an account a few days ago