Netgear R7800 exploration (IPQ8065, QCA9984)

slh · 2018-03-01T02:14:33.742Z

@luaraneda, by the way, it seems as if the ipq806x target would be split into independent ones for ipq40xx and ipq806x (see blogic's staging tree), which could help getting the kernel just a tad smaller again.

luaraneda · 2018-03-01T02:35:04.589Z

Yes, I was just noticing that. If I remember correctly, on 2017 there was an attempt to do that by a member of Codeaurora, but it was dropped/rejected (don't remember which).
Maybe the kernel size issue triggered the change. I couldn't find any information on the mailing list...

On other news, yesterday I tried to compile kernel 4.14 using GCC 7.3, but the resulting image is approximately 4 kB bigger that the generated by GCC 5, which is not good for the affected devices.
Some notes about GCC 7.3:

uboot-fritz4040 is not compiling. I deselected it and continue my tests.
I only compiled the images. This time I didn't test them as usual on my Asus RT-AC58U.

slh · 2018-03-01T03:46:10.673Z

I've been testing gcc 7.3 for arm with kernel 4.9 for a while now (nbg6817, so 4 MB kernel partition), I haven't noticed any problems so far (but obviously I have no size restrictions yet).

--- a/toolchain/gcc/Config.in
+++ b/toolchain/gcc/Config.in
@@ -3,7 +3,7 @@
 choice
 	prompt "GCC compiler Version" if TOOLCHAINOPTS
 	default GCC_USE_VERSION_7_1_ARC if arc
-	default GCC_USE_VERSION_7 if x86_64 || i386
+	default GCC_USE_VERSION_7 if x86_64 || i386 || arm
 	default GCC_USE_VERSION_5
 	help
 	  Select the version of gcc you wish to use.
--- a/toolchain/gcc/Config.version
+++ b/toolchain/gcc/Config.version
@@ -6,6 +6,7 @@ config GCC_VERSION_7
 	default y if GCC_USE_VERSION_7
 	default y if (!TOOLCHAINOPTS && x86_64)
 	default y if (!TOOLCHAINOPTS && i386)
+	default y if (!TOOLCHAINOPTS && arm)
 	bool
 
 config GCC_VERSION

gcc 7.3 will become particularly interesting for spectre mitigation (retpoline), but the required ARMv7 specific patches haven't been merged mainline yet.

dissent1 · 2018-03-01T09:18:53.831Z

@luaraneda all pcie patches that are missing in my tree are upstreamed
Btw ipq target is being split apart into ipq40xx and ipq806x. Check blogic's and mkresin's staging trees. This should decrease the size of the kernel image by removing ipq40xx specific config options.

hnyman · 2018-03-01T11:00:58.243Z

slh:

I haven’t gotten it to work (last test with v9, which didn’t work at all - although /sys/class/led/ & friends were present)

Sounds similar as my own result with v9 or later.

So far v8 is the best.

fantom-x · 2018-03-03T21:39:09.666Z

A question to the experts here. For a while now I have been experiencing significant latency spikes that affect VoIP and page loading. I tested by pinging 8.8.8.8 when the router (Netgear R7800) is pretty much idle. I would start 6..8 concurrent ping sessions and every minute I would see two or three spikes to 50..100 ms like below and they would most of the time happen synchronously in multiple sessions.

2018-03-03 15:26:07 64 bytes from 8.8.8.8: icmp_seq=525 ttl=60 time=58.013 ms
2018-03-03 15:26:35 64 bytes from 8.8.8.8: icmp_seq=553 ttl=60 time=37.198 ms
2018-03-03 15:27:07 64 bytes from 8.8.8.8: icmp_seq=585 ttl=60 time=76.856 ms
2018-03-03 15:28:06 64 bytes from 8.8.8.8: icmp_seq=643 ttl=60 time=60.067 ms

As suggested in this thread, I tried moving IRQ's to CPU1/0 in different permutations, tried both 17.01 and master, etc and all with no success. Then I noticed that while the IRQ's are running on their CPU exclusively, all the other processes (kernel workers, hostapd, dnsmasq, etc) are constantly jumping back and forth between the CPU's.

I used @hnyman's build env and built a firmware with (and without) isolcpus=1 (based on master code): the only differences from the original here are an additional boot param and a few more recent commits. The rest remained the same.

Then I tested four permutations below (using wired connection and the same source code):

No isolcpus=1 and IRQ's on CPU0
No isolcpus=1 and IRQ's on CPU1
isolcpus=1 and IRQ's on CPU0
isolcpus=1 and IRQ's on CPU1

The first three yielded no difference, but the last one dropped the size of the spikes to ~20ms and they are now 10+ minutes apart vs several each minute.

2018-03-03 15:34:03 PING 8.8.8.8 (8.8.8.8): 56 data bytes
2018-03-03 15:37:49 64 bytes from 8.8.8.8: icmp_seq=226 ttl=60 time=21.615 ms
2018-03-03 15:50:47 
2018-03-03 15:50:47 --- 8.8.8.8 ping statistics ---
2018-03-03 15:50:47 1000 packets transmitted, 1000 packets received, 0.0% packet loss
2018-03-03 15:50:47 round-trip min/avg/max/stddev = 11.082/11.963/21.615/0.475 ms
2018-03-03 15:50:47 PING 8.8.8.8 (8.8.8.8): 56 data bytes
2018-03-03 15:57:12 64 bytes from 8.8.8.8: icmp_seq=382 ttl=60 time=22.734 ms
2018-03-03 16:07:33 
2018-03-03 16:07:33 --- 8.8.8.8 ping statistics ---
2018-03-03 16:07:33 1000 packets transmitted, 1000 packets received, 0.0% packet loss
2018-03-03 16:07:33 round-trip min/avg/max/stddev = 10.913/11.839/22.734/0.492 ms
2018-03-03 16:07:33 PING 8.8.8.8 (8.8.8.8): 56 data bytes
2018-03-03 16:24:15 
2018-03-03 16:24:15 --- 8.8.8.8 ping statistics ---
2018-03-03 16:24:15 1000 packets transmitted, 1000 packets received, 0.0% packet loss
2018-03-03 16:24:15 round-trip min/avg/max/stddev = 11.192/11.921/15.246/0.326 ms

So CPU1 is now only for servicing IRQ's for eth0, eth1, wifi0, and wifi1 while everything else is running on CPU0.

Does this make sense or I am seeing things? I am not quite sure I can explain why there is such a difference.

draigun · 2018-03-04T01:28:42.582Z

I've suffered ping spikes as well when running OpenWRT/LEDE. Using stock firmware, they go away. My original post is located here: Build for Netgear R7800

Though my testing is absolutely not all encompassing, as compared to yours.

For now, I had to move back to stock firmware, since nlbwmon has shown itself to be unstable and unpredictable in behavior. Constant issues with displaying the data from the GUI; it randomly consumed 100% CPU for a period of at least 16 hours. That, and the noticeable increase in latency, as you mentioned.

There have been some PCIe fixes recently for this platform, and they do mitigate some of the issues with the relationship between OpenWRT/LEDE<->ipq806x.

fantom-x · 2018-03-04T01:35:03.085Z

Yes, stock firmware runs with no spikes and introduces almost unnoticeable addition latency overhead while LEDE adds around 1..2 ms. That I could live with, but the spikes were atrocious. I am running the latest the greatest, so should have all the patches in.

draigun · 2018-03-04T01:36:22.687Z

Yup. Personally—I only noticed the huge latency spikes. Modem is an SB8200 fwiw, and it's Broadcom based, so no PUMA issue.

quarky · 2018-03-04T06:13:20.595Z

I managed to compile and install the qca-nss-gmac driver. The nss-gmac driver replaces the standard Linux stmmac driver. The nss-gmac driver seems to be working properly on my R7800. I'm not sure if the driver actually uses the additional 2 NSS CPU cores tho. It doesn't seem like it is using it.

I've uploaded the codes needed to compile and run with the nss-gmac driver here:

GitHub

quarkysg/openwrt

This repository is a mirror of https://git.openwrt.org/openwrt/openwrt.git It is for reference only and is not active for check-ins or for reporting issues. We will continue to accept Pull Request...

Let me know if you would like a pre-build binary instead for testing. My image is based on the lede-17.01 branch, with my custom changes as well. For those experiencing latency spikes, do assess if you can try out the qca-nss-gmac driver and see if that resolves your issue.

fantom-x · 2018-03-04T12:07:28.566Z

quarky:

Let me know if you would like a pre-build binary instead for testing. My image is based on the lede-17.01 branch, with my custom changes as well. For those experiencing latency spikes, do assess if you can try out the qca-nss-gmac driver and see if that resolves your issue.

I will try a prebuilt image.

quarky · 2018-03-04T15:00:24.935Z

fantom-x:

I will try a prebuilt image.

Details of my builds can be found in the community build thread for R7800.

Build for Netgear R7800 Community Builds, Projects & Packages

I've uploaded my build for R7800 based on lede-17.01 branch here: https://app.box.com/s/oxaht7753bskakihl5xqq0r159wt76gi It has two sysupgrade image; one with the default stmmac ethernet driver while another image is built with the nss-gmac ethernet driver. For those who experienced high latency for your R7800, appreciate if you can help test the image and see if it solves the high latency issue. You can also build the image yourself using the GitHub repository here: stmmac driver: nss…

After re-looking into the source code, looks like the NSS cores are not used, as it requires the NSS firmware to be uploaded. But the nss-gmac driver is required for further work on the nss-drv to be done. I'll see if I can get the nss-drv successfully loaded. That would be the one to drive the NSS cores and perform the offload magic.

hnyman · 2018-03-04T15:10:25.854Z

quarky:

Details of my builds can be found in the community build thread for R7800.

Might be actually better here, or you might start an own thread about that nss-gmac test build.

philjohn · 2018-03-04T15:31:44.488Z

I was under the impression that using the NSS cores needed SSDK and various incantations to offload packet processing ... I may be wide of the mark though.

I managed to get most of the NSS stuff building and running successfully, but I couldn't resolve the patches required to get SSDK to work on the current 17.01 kernel, there might be updated stuff in the upstream codeaurora tree now though.

bouwew · 2018-03-04T15:57:50.424Z

Came across this:

GitHub

frederic/netgear-R7800-GPL

netgear-R7800-GPL - Netgear R7800 GPL package

Seems more up-to-date than the github of Paul Chambers...

bouwew · 2018-03-04T16:07:18.492Z

GitHub

SVoxel/R7800

Custom firmware build for R7800

This guy provides improved factory firmwares.
@quarky This could provide more info on the nss-stuff, see /package/nss*

dissent1 · 2018-03-04T16:19:16.408Z

@quarky your changes do not enable nss-gmac driver as your dts entries are missing the corresponding compatibility.
I've been trying to enable it but failed, smth in the driver itself leads to a boot loop.
Here's he DTS patch
https://pastebin.com/1SQEhsBc

fantom-x · 2018-03-04T16:27:13.744Z

fantom-x:

Let me know if you would like a pre-build binary instead for testing. My image is based on the lede-17.01 branch, with my custom changes as well. For those experiencing latency spikes, do assess if you can try out the qca-nss-gmac driver and see if that resolves your issue.

@quarky I have just tried your build (nss) it looks like the latency overhead introduced by the router has decrease to 0.5ms or less (down from 1+ ms), but I cannot confirm for sure as I only had 15 minutes or so. The latency spikes remained unfortunately.
Two more observations: 1) I could not get an IP from a LAN port until I removed my VLAN's and then it all worked. 2) The router went into a reboot loop once I sysupgrad'ed to my firmware. Had to recover the router via TFTP.

philjohn · 2018-03-04T16:34:04.173Z

Not only that, looking at the official firmware (and the codeaurora stuff, along with descriptions of how the NSS is initialized and works) you'll need the following:

kmod-qca-nss-connmgr
kmod-qca-nss-drv
kmod-qca-nss-gmac
kmod-qca-ssdk-nohnat

Connmgr monitors conntrack events and creates rules that offload onto the NSS cores (see here: https://lkml.org/lkml/2015/1/21/804)

quarky · 2018-03-04T21:15:09.635Z

dissent1:

@quarky your changes do not enable nss-gmac driver as your dts entries are missing the corresponding compatibility.

I’ve been trying to enable it but failed, smth in the driver itself leads to a boot loop.

My changes includes those DTS entry for the GMACs devices. The nss-gmac drivers is running in my build. Next step is to get the nss-drv driver working. I’ve disabled the stmmac Ethernet driver, so the only Ethernet driver has to be the nss-gmac.