Ipq806x NSS build (Netgear R7800 / TP-Link C2600 / Linksys EA8500)

xeonpj · 2021-04-01T19:43:52.996Z

If you want to do the stability test fast, load the router with torrent, and transfer large files via LAN, it will restart soon, in a matter of hours. we were talking with nss active, without nss the stability is complete, perfect. Right now I have disabled torrent and waiting to be able to test the scaling in nss on my r7800. If my network holds without restarting, it is 100% stable for all XD

viulian · 2021-04-01T21:03:59.337Z

Thank you for the clarification!

I've applied the changes from here: Ipq806x NSS build (Netgear R7800 / TP-Link C2600 / Linksys EA8500) - #510 by Ansuel but that had no effect on my router, NSS cores were still running at 800MHz, thus I was wondering if maybe only the stability of the EA8500 was evaluated ... ?

But since I wanted to help testing stability for R7800 with NSS scaling too, I discovered that the frequencies are stored in a file called qcom-ipq8065.dtsi and came up with this little patch file (watch out, as the source file has some TABs indentation, and when I copy/pasted the diff output, they got converted to spaces and patched failed):

--- a/arch/arm/boot/dts/qcom-ipq8065.dtsi
+++ b/arch/arm/boot/dts/qcom-ipq8065.dtsi
@@ -157,13 +157,13 @@
 };

 &nss0 {
-       qcom,low-frequency = <800000000>;
-       qcom,mid-frequency = <800000000>;
+       qcom,low-frequency = <110000000>;
+       qcom,mid-frequency = <550000000>;
        qcom,max-frequency = <800000000>;
 };

 &nss1 {
-       qcom,low-frequency = <800000000>;
-       qcom,mid-frequency = <800000000>;
+       qcom,low-frequency = <110000000>;
+       qcom,mid-frequency = <550000000>;
        qcom,max-frequency = <800000000>;
 };

Now the NSS driver shows the three frequencies, and I can see that it idles at 110MHz and jumps to 800Mhz when needed.

For now it only crashed once, maybe 5-10 mins after first restart. After that it was stable for normal internet traffic. I also stressed it with 2 iperfs (different port numbers) on the router itself and trying to test both Ethernet and WiFi at the same time but except becoming a little hotter (with 4C), all was fine. I'll try the @xeonpj suggestion also.

I think I have also successfully enabled the ramoops driver and triggered a kernel panic, just to see if I can catch the reason why it may reboot and I saw the dump in the /sys/fs/pstore. Thus, now waiting for the new reboot

If you have other ideas to test, I'll try them

Thank you again for all the work!

xeonpj · 2021-04-01T21:59:34.946Z

Thank you for your work. I did not compile, but I did crash the router. LOL. If you want, I can test your image, and you tell me how to read the possible panic kernel.

viulian · 2021-04-01T22:26:39.570Z

Sorry @xeonpj I don't have yet the infrastructure setup (nor repo on github dedicated to this) - just trying to help the persons that already are on it

BTW: I also crashed the router too with your idea, I started a 4.5Gb Fedora torrent, and after 15-20 minutes router already rebooted. Trouble is there was no panic listed in console-ramoops-0 file, just the dmesg output of the previous run. No 'dmesg-ramoops-0` created, although, although when I manually triggered the kernel panic, it did create that file. Not now though ...

I'll keep trying, maybe is indeed a voltage change issue ?

(also, to read the kernel panic, I don't think is possible with @ACwifidude image, I used his branch and the ramoops configuration from IPQ806x NSS Drivers - #2008 by Gram for R7800).

xeonpj · 2021-04-01T22:44:13.929Z

Actually, it has also already been about the voltage issue, and reduction from MHz to 600, and not much. there is nothing to dump on either.

viulian · 2021-04-01T23:23:41.128Z

R7800 rebooted again while laptop (via WiFi) was downloading the Fedora torrent - and this time I got the error message. Will continue to fish for core dumps ... Hope this helps!

<1>[ 5486.098270] NSS core 0 signal COREDUMP COMPLETE 4000
<1>[ 5486.098320]
<1>[ 5486.098320] 66123ac4: Starting NSS-FW logbuffer dump for core 0
<1>[ 5486.102309] 66123ac4: Warn: trap[522]: TRAP async error: DC A9FFE2FC IC 8E9D9AEC
<1>[ 5486.109784] 66123ac4: Warn: trap[542]: 15: access FABRICs for additional info.
<1>[ 5486.117149] 66123ac4: Warn: trap[288]: 01400000: fabric err addr reg 00000000 last bad addr 00000000: FAB_APPS test_if 00000005 slave stat 00000000
<1>[ 5486.124196] 66123ac4: Warn: trap[288]: 00D00000: fabric err addr reg 00000000 last bad addr 00000000: FAB_NSS0 test_if 00000002 slave stat 00000000
<1>[ 5486.137285] 66123ac4: Warn: trap[313]: FAB_NSS0 stat0-3 00006000 00000200 00000834 00000083
<1>[ 5486.150468] 66123ac4: Warn: trap[288]: 01900000: fabric err addr reg 00000000 last bad addr 00000000: FAB_NSS1 test_if 0000007C slave stat 00000000
<1>[ 5486.158809] 66123ac4: Warn: trap[288]: 01300000: fabric err addr reg 00000000 last bad addr 00000000: FAB_SYSTEM test_if 00007BF8 slave stat 00000000
<1>[ 5486.171992] 66123ac4: Warn: trap[298]: NULL access
<1>[ 5486.185550] 66123ac4: Warn: trap[300]: FABRIC can't tell where access from, so examine your code to see where could do such access
<1>[ 5486.190161] 66123ac4: Warn: trap[301]: if this address consistently got trapped, use DCAPT to catch the writer
<3>[ 5486.203490] wlan0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[ 5486.211948] NSS core 1 signal COREDUMP COMPLETE 4000
<1>[ 5486.211959]
<1>[ 5486.211959] f08c8d90: Starting NSS-FW logbuffer dump for core 1
<0>[ 5486.211967] Kernel panic - not syncing: NSS FW coredump: bringing system down
<2>[ 5486.212075] CPU1: stopping
<4>[ 5486.212099] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.108 #0
<4>[ 5486.212109] Hardware name: Generic DT based system
<4>[ 5486.212151] [<c030f954>] (unwind_backtrace) from [<c030b968>] (show_stack+0x14/0x20)
<4>[ 5486.212178] [<c030b968>] (show_stack) from [<c0901ee0>] (dump_stack+0x94/0xa8)
<4>[ 5486.212203] [<c0901ee0>] (dump_stack) from [<c030eb80>] (handle_IPI+0x184/0x1b8)
<4>[ 5486.212243] [<c030eb80>] (handle_IPI) from [<c05e17c8>] (gic_handle_irq+0xb4/0xb8)
<4>[ 5486.212273] [<c05e17c8>] (gic_handle_irq) from [<c0301a8c>] (__irq_svc+0x6c/0x90)
<4>[ 5486.212286] Exception stack(0xd846bf48 to 0xd846bf90)
<4>[ 5486.212321] bf40:                   00000000 20000093 0007e9db ffffffff d8c51690 00000001
<4>[ 5486.212363] bf60: 00000000 00000002 5af5ec47 000004fd c0c0cd10 d87eb400 c0c67680 d846bf98
<4>[ 5486.212389] bf80: c0388538 c039a664 a0000013 ffffffff
<4>[ 5486.212430] [<c0301a8c>] (__irq_svc) from [<c039a664>] (tick_nohz_idle_exit+0x64/0x148)
<4>[ 5486.212461] [<c039a664>] (tick_nohz_idle_exit) from [<c034ac14>] (do_idle+0x110/0x240)
<4>[ 5486.212484] [<c034ac14>] (do_idle) from [<c034afec>] (cpu_startup_entry+0x1c/0x20)
<4>[ 5486.212507] [<c034afec>] (cpu_startup_entry) from [<423024cc>] (0x423024cc)

xeonpj · 2021-04-02T00:30:07.190Z

I don't understand anything, but it seems that for some reason the nss kernel crashes, and it gives an error because it is not ready, the same thing that creates the kernel panic. but why does the nss kernel drop?

viulian · 2021-04-02T09:29:16.498Z

I said ... why should Fedora be the one that causes damage, when Ubuntu can do the same. 10-15mins into downloading the Ubuntu torrent I captured the problem below - which seems different than the first one that was complaining about NULL access. This one complains about bad address access at 3F02FAA0 :

<1>[   22.326819] memory: 0 0 (avl 406224896) items 2 active_cores 2
<1>[   22.336108] 71d303b4: nss core 1 booted successfully
<6>[   22.337700] Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
<1>[   22.348118] NSS core 0 signal COREDUMP COMPLETE 4000
<1>[   22.353457]
<1>[   22.353457] 7becaa80: Starting NSS-FW logbuffer dump for core 0
<1>[   22.358342] 7becaa80: Warn: trap[522]: TRAP async error: DC A9FFE2FC IC 8E9D9AEC
<1>[   22.365759] 7becaa80: Warn: trap[542]: 16: access FABRICs for additional info.
<1>[   22.373133] 7becaa80: Warn: trap[288]: 01400000: fabric err addr reg 00000000 last bad addr 00000000: FAB_APPS test_if 00000005 slave stat 00000000
<1>[   22.380127] 7becaa80: Warn: trap[288]: 00D00000: fabric err addr reg 00000000 last bad addr 3F02FA80: FAB_NSS0 test_if 00000002 slave stat 00000000
<1>[   22.393283] 7becaa80: Warn: trap[295]: bad address access at 3F02FAA0
<1>[   22.406466] 7becaa80: Warn: trap[313]: FAB_NSS0 stat0-3 00006000 00000200 00000834 00000083
<1>[   22.413065] 7becaa80: Warn: trap[288]: 01900000: fabric err addr reg 3F02FAA0 last bad addr 00000000: FAB_NSS1 test_if 0000007C slave stat 00000000
<1>[   22.421181] 7becaa80: Warn: trap[288]: 01300000: fabric err addr reg 3F02FAA0 last bad addr 00000000: FAB_SYSTEM test_if 00007BF8 slave stat 00000000
<1>[   22.434426] 7becaa80: Warn: trap[300]: FABRIC can't tell where access from, so examine your code to see where could do such access
<1>[   22.447986] 7becaa80: Warn: trap[301]: if this address consistently got trapped, use DCAPT to catch the writer
<1>[   22.462620] NSS core 1 signal COREDUMP COMPLETE 4000
<1>[   22.469537]
<1>[   22.469537] 71d303b4: Starting NSS-FW logbuffer dump for core 1
<0>[   22.474681] Kernel panic - not syncing: NSS FW coredump: bringing system down
<2>[   22.482017] CPU1: stopping
<4>[   22.489044] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.108 #0
<4>[   22.491645] Hardware name: Generic DT based system
<4>[   22.497829] [<c030f954>] (unwind_backtrace) from [<c030b968>] (show_stack+0x14/0x20)
<4>[   22.502423] [<c030b968>] (show_stack) from [<c0901ee0>] (dump_stack+0x94/0xa8)
<4>[   22.510320] [<c0901ee0>] (dump_stack) from [<c030eb80>] (handle_IPI+0x184/0x1b8)
<4>[   22.517362] [<c030eb80>] (handle_IPI) from [<c05e17c8>] (gic_handle_irq+0xb4/0xb8)
<4>[   22.524907] [<c05e17c8>] (gic_handle_irq) from [<c0301a8c>] (__irq_svc+0x6c/0x90)
<4>[   22.532274] Exception stack(0xd846bf18 to 0xd846bf60)
<4>[   22.539826] bf00:                                                       00000000 00000005
<4>[   22.544880] bf20: 18112000 d8c52a80 d87f3400 00000000 d8c51e30 00000005 00000005 00000000
<4>[   22.553040] bf40: 3c083540 3bfdb200 00000015 d846bf68 c0732e50 c0732e54 00000013 ffffffff
<4>[   22.561199] [<c0301a8c>] (__irq_svc) from [<c0732e54>] (cpuidle_enter_state+0x94/0x498)
<4>[   22.569352] [<c0732e54>] (cpuidle_enter_state) from [<c073329c>] (cpuidle_enter+0x30/0x4c)
<4>[   22.577164] [<c073329c>] (cpuidle_enter) from [<c034acdc>] (do_idle+0x1d8/0x240)
<4>[   22.585493] [<c034acdc>] (do_idle) from [<c034afec>] (cpu_startup_entry+0x1c/0x20)
<4>[   22.593046] [<c034afec>] (cpu_startup_entry) from [<423024cc>] (0x423024cc)

supermarket · 2021-04-02T21:04:31.220Z

If someone knows how to fix the iproute2 patch

What was wrong with iproute2? I didn’t see a problem.

10ffefe60299 landed upstream a couple of weeks ago and had a build fix for iproute2. Could that be the fix for what you saw?

KONG · 2021-04-02T21:16:04.960Z

NSS modifies iproute2, I just updated 400-add-nss-qdisc.patch to work with the new iproute2 version, will push it to my branch after I finished compiling/testing.

In order to make it easier for me to update my code I deleted my clone from acwifidude, since he also pushed packages etc. to his repo. Thus I did a fresh clone from openwrt repo cherry picked the changes for nss and added my changes on top.

KONG · 2021-04-02T23:57:47.694Z

Fixes pushed to: https://github.com/ricsc/openwrt/tree/kernel5.4-nss-qsdk10.0

nashford · 2021-04-03T08:52:38.592Z

I did an opkg update prior, i am trying to install any of the kernel modules listed here:

neither work. Am on a current build as of today. Any ideas? The reason I did this is that my 5GHY never came up on 160mhz auto or 100 (in the US) at all..... ?

following this:

pkg update
opkg remove ath10k-firmware-qca9984-ct kmod-ath10k-ct
opkg install ath10k-firmware-qca9984 kmod-ath10k

It has made me unable to add any KMOD now....

root@OpenWrt:~# opkg install ath10k-firmware-qca9984-ct kmod-ath10k-ct
Package ath10k-firmware-qca9984-ct (2020-11-08-1) installed in root is up to date.
Unknown package 'kmod-ath10k-ct'.
Collected errors:

pkg_hash_fetch_best_installation_candidate: Packages for kmod-ath10k-ct found, but incompatible with the architectures configured
opkg_install_cmd: Cannot install package kmod-ath10k-ct.

Thoughts on both issues?

viulian · 2021-04-03T14:25:21.605Z

Check the reply of this post: Ipq806x NSS build (Netgear R7800 / TP-Link C2600 / Linksys EA8500) - #442 by sppmaster

nashford · 2021-04-03T19:20:42.097Z

Ah - OK & Currently 160mhz DFS is not working right ? @kong - Is there a build where DFS 160 MHZ + GIG WAN Speed works?

ACwifidude · 2021-04-03T19:28:25.044Z

You can either force install using opkg or you can build fresh with the ath10k drivers (-ct does not support 160mhz currently, hope they fix that). opkg will complain about any kmods additions because custom builds add an additional hash to the end.

I just updated the master branch with @KONG patch (works awesome, appreciate it!) and cleaned up all the commits.

nashford · 2021-04-03T23:03:51.653Z

Ah - if i don't use -ct , will 160mhz . DFS work again? I assume Kongs patch only covers the slower CPU 8500 - nothing in it for the 7800 right ?

nashford · 2021-04-03T23:05:55.587Z

I am also wondering as i have a 1G Symetric link, should I use x86 ITX HW for this, would I see any issues with speed still assuming adequate CPU etc. Would the NAT offload be an issue with fast enough gear still or would line speed and radio speed be a given ?

nashford · 2021-04-03T23:16:26.263Z

I tried to force install via over-ride for checksum, reinstall etc. - none work. which flag do you use ? I am on an r7800, your 21.x build, one day old.

root@OpenWrt:~# opkg install kmod-ath10k --force-depends --force-checksum
Unknown package 'kmod-ath10k'.
Collected errors:

pkg_hash_fetch_best_installation_candidate: Packages for kmod-ath10k found, but incompatible with the architectures configured
opkg_install_cmd: Cannot install package kmod-ath10k.

xeonpj · 2021-04-04T00:45:08.892Z

I just installed it, it reaches 600mb by wifi in speed test, the maximum of my line, amazing. The NSS patch is not yet for r7800, right? I keep seeing 800,800,800 mhz.

thank you all for everything.

ACwifidude · 2021-04-04T01:36:57.399Z

NSS is still locked at the maximum speed on the r7800 (and all other devices). Haven’t played with the r7800 NSS frequencies to get it back to on demand. I don’t torrent so it is interesting above that the crashes people are seeing are possibly load related.

There are some good crash logs up above that I hope make sense to one of our skilled developers.

Latest master build has Kong’s ea8500 settings and new iproute2 patch.