Netgear R7800 exploration (IPQ8065, QCA9984)

Pedro · July 30, 2019, 1:32am

just to make sure, this is my .config file:

quarky · July 30, 2019, 1:48am

No, that's not it. If you want, you can try the following patches (it's for 4.4, but should work with 4.14/4.19 with minor tweaks):

https://github.com/quarkysg/openwrt/blob/lede-17.01-ipq806x-nss/target/linux/generic/patches-4.4/999-01-shortcut-fe-support.patch
https://github.com/quarkysg/openwrt/tree/lede-17.01-ipq806x-nss/package/kernel/shortcut-fe
https://github.com/quarkysg/openwrt/tree/lede-17.01-ipq806x-nss/package/kernel/fast-classifier

Use the fast-classifier Connection Manager and disable the Shortcut-FE Connection Manager.

AFAIK, only 4.19 has software flow-offload and it's not as stable as the QCA shortcut-fe. IMHO, existing router SoCs really need some form of flow-offload as they just don't have enough grunt to handle Linux's networking stacks.

My fast-classifier CM code has some issues I've fixed. Do try if you're interested.

The Krait CPU would be near 100% trying to keep that going. With WiFi and SQM all trying to have a slice of the CPU pie, bandwidth will suffer. The ipq8065 as it stands really need flow-offload, the NSS core or both.

Do note that with the shortcut-fe enabled, only egress qdisc will work. Ingress qdisc will be bypassed. I think this is true for all forms of flow-offload, even with the NSS core. I have yet to experiment with the NSS qdisc ingress tho.

For me I applied codel qdisc (offloaded to the NSS cores) on all my interfaces, i.e. WiFi, WAN, ethernet, WireGuard tunnels. I'm able to get A rating with DSLreport for my 500/100mbps internet link using LAN cable, with 32 threads.

lantis1008 · July 30, 2019, 1:56pm

Will get back to you this weekend

Chief · July 30, 2019, 3:02pm

Pedro:

I'm running latest master with kernel 4.14. I can do codel at 200mbps but not cake. cake tops out at around 140.
And codel at 200 is also near its limit. I think one of the cpu's ( the one dealing with the codel thread ) runs at around 90 % when shaping a 200/100 connection.

The only change I did to the master code was disabling the switch pooling stuff that was making ksoftirqd eat all the cpu, and I also run this on rc local:
echo 2 > /proc/irq/31/smp_affinity
echo 2 > /proc/irq/32/smp_affinity


echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor
echo 800000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_max_freq
echo 800000 > /sys/devices/system/cpu/cpufreq/policy1/scaling_max_freq
sleep 1
echo 1750000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_max_freq
echo 1750000 > /sys/devices/system/cpu/cpufreq/policy1/scaling_max_freq
ethtool -C eth0 tx-usecs 0
ethtool -C eth1 tx-usecs 0
ethtool -C eth0 rx-usecs 31
ethtool -C eth1 rx-usecs 31


exit 0

140Mbps with cake is the result of two bugs in openwrt bsp for IPQ. Fix the two and this rate will be doubled. It is really funny, how you guys spent months with tweaking and unstable nss code etc.

Pedro · July 30, 2019, 3:04pm

what bugs? can you elaborate?

Pedro · July 30, 2019, 3:49pm

It appears that you have fixed the problem, at least by this comment you made on another thread:

any reason why you are not sharing the fix?

quarky · July 31, 2019, 12:57am

The stability issue with the NSS drivers have been resolved, sort of. My R7800 has been stable for more than 3 weeks with the NSS cores offloading routing tasks. With the NSS cores offloads, my R7800 can push close to gigabit line speed (the last time I tested, it achieved 940mbps) with less than 5% Krait CPU utilisation. So it leaves all the CPU cycles of the Krait to WiFi and other tasks, mainly VPN. Without the offload, the Krait CPU would spend most of it's CPU cycles processing netfilter and routing rules at high bandwidth, leaving very little for other tasks.

IMHO, getting the NSS cores to work would be most beneficial to the ipq806x based routers. All other tweaks as it stands will not provide much benefit.

Videopac · July 31, 2019, 5:22am

That is good news. But this NSS driver stability issue resolution isn't in snapshot builds yet, right?

Chief · July 31, 2019, 5:53am

quarky:

Chief:

It is really funny, how you guys spent months with tweaking and unstable nss code etc.

The stability issue with the NSS drivers have been resolved, sort of. My R7800 has been stable for more than 3 weeks with the NSS cores offloading routing tasks. With the NSS cores offloads, my R7800 can push close to gigabit line speed (the last time I tested, it achieved 940mbps) with less than 5% Krait CPU utilisation. So it leaves all the CPU cycles of the Krait to WiFi and other tasks, mainly VPN. Without the offload, the Krait CPU would spend most of it's CPU cycles processing netfilter and routing rules at high bandwidth, leaving very little for other tasks.

IMHO, getting the NSS cores to work would be most beneficial to the ipq806x based routers. All other tweaks as it stands will not provide much benefit.

That's just based on your limited knowledge. If the bugs I know off are fixed, then it will do nat without software offload at only around 20% cpu load. If it is stable only in one environment I wouldn't call that stable for all. And what about all the other things newer kernel support, sqm etc. , unless these issues are fixed it will never go into mainline.

OEM with nss cores cannot achieve the same sqm throughput compared to my fixed build

ratking · July 31, 2019, 6:50am

What is it you "fixed"?

lantis1008 · July 31, 2019, 7:14am

And do you intend on sharing what it is you've fixed?

Videopac · July 31, 2019, 7:58am

The power of open source is sharing, so: share what you know or have built.

Pedro · July 31, 2019, 9:11am

I think these are wonderful news. Having another way of extracting more performance from these routers can only be seen as positive, and on top of that if made on a easy non invasive way then it's even better. so the news you are sharing regarding those two bugs on bsp code and their performance effect are amazing.

I truly hope that you can somehow assist on fixing them in trunk. I'm sure that if you haven't yet provided a patch it is because either your code is not yet up to standards and you are working on it or maybe you have some NDA commitments or even some other contractual obligations. Anyway, I hope you can find a way of helping out.

having said that, I have to be honest and tell you that, well, maybe because of my lack of english knowledge, I didn't quite appreciate how you exposed the situation. It almost seems as if you are trolling everyone, and a bit of lack of respect to the OpenWRT developers and actual users like me. A bit like showing a delicious lolipop to a baby and then trowing it away to the garbage. This for sure must be a fault on my english skills.

fantom-x · July 31, 2019, 12:10pm

Your English is great, but pls stop feeding the troll. He got nothing to show.

Ansuel · July 31, 2019, 2:47pm

Hey i fixed my kernel my router is now more powerful than my i7

fantom-x · July 31, 2019, 3:41pm

Well, at least you did not create an account a few days ago

Pedro · July 31, 2019, 5:21pm

I see no good reason for someone that has an interest in the project to come here and troll people, as such, I prefer to believe that there is an underlying reason that is preventing him to share his knowledge.
I mean, there are really bad people in the world, but you mostly see them in youtube comment boxes, where they have nothing to lose, not on project forums.
Anyway, we will soon find out.

quarky · August 1, 2019, 1:16am

At the moment, we're still at the testing stage. Also, we're at kernel v4.4.x. The OpenWRT devs will be abandoning the 4.4.x kernel after the next 17-01.x release.

To have any chance of even getting the devs to consider including the NSS drivers for the ipq806x, we would need to move to v4.14 at least. There're also extensive changes required to the netfilter stacks, which IMO will most likely not be accepted into mainline. Furthermore there's also the issue of bundling proprietary firmware blobs. So I would think it will be very challenging to have OpenWRT include the NSS drivers into mainline.

lantis1008 · August 3, 2019, 6:54am

Turns out my 160MHz capable client (Intel 9260) is not capable of 80+80 client mode, only 160.
I tested it at 80MHz width (unable to start a contiguous 160 channel at this stage) and got 797mbps with iperf tcp.

abenz · August 3, 2019, 9:58pm

Thanks for the update. Unable to start a contiguous channel due to DFS or another reason? Note it might take the interface a minute or so before it comes up (ie after a scan it will bring the interface up if no radar/etc found).

AFAIK, only one person here was able to broadcast a 160Mhz channel with OpenWrt using a QCA9994 (same as 9984 in R7800 but with higher operating temp), on an x86 router, and was able to break 100MBps. I'm curious to see what ipq806x (with cpu freq statically set at max) can do over wifi with 160mhz width or with 3/4 stream clients.

Unfortunately my tplink c2600 has the older qca9980 chipset, it does not support 160Mhz channels. Its seems to be abandoned; QCA didn't release a new firmware for it for 4 years, while the 9984 received many updates since release.
CT firmwares have been released for it though.. I wasn't able to get mesh working with stock ath10k, but ath10k-ct with accompanying firmware enabled mesh.