Qualcommax NSS Build

I included it today in my nss-wif build today.

root@QNAP:~# htop
root@QNAP:~# dmesg | grep -Ei "psto|ramo"
[    0.000000] OF: reserved mem: 0x0000000051200000..0x00000000512fffff (1024 KiB) map non-reusable ramoops@51200000
[    2.276163] pstore: Using crash dump compression: deflate
[    2.276949] pstore: Registered ramoops as persistent store backend
[    2.282474] ramoops: using 0x100000@0x51200000, ecc: 0

I've just triggered kernel panic and after a reboot I found this in pstore.

Summary
Panic#1 Part1
<6>[   12.233042] remoteproc remoteproc0: remote processor cd00000.q6v5_wcss is now up
<6>[   12.284500] ath11k c000000.wifi: chip_id 0x0 chip_family 0x0 board_id 0xff soc_id 0xffffffff
<6>[   12.284554] ath11k c000000.wifi: fw_version 0x290604a5 fw_build_timestamp 2023-10-12 02:06 fw_build_id WLAN.HK.2.9.0.1-01977-QCAHKSWPL_SILICONZ-1
<6>[   12.425164] remoteproc remoteproc0: stopped remote processor cd00000.q6v5_wcss
<6>[   12.425208] remoteproc remoteproc0: powering up cd00000.q6v5_wcss
<6>[   12.431384] remoteproc remoteproc0: Booting fw image IPQ8074/q6_fw.mdt, size 668
<6>[   12.781801] remoteproc remoteproc0: remote processor cd00000.q6v5_wcss is now up
<14>[   12.783945] kmodloader: done loading kernel modules from /etc/modules.d/*
<6>[   12.833231] ath11k c000000.wifi: chip_id 0x0 chip_family 0x0 board_id 0xff soc_id 0xffffffff
<6>[   12.833280] ath11k c000000.wifi: fw_version 0x290604a5 fw_build_timestamp 2023-10-12 02:06 fw_build_id WLAN.HK.2.9.0.1-01977-QCAHKSWPL_SILICONZ-1
<4>[   13.031650] ath11k c000000.wifi: htt event 48 not handled
<6>[   13.035127] ath11k c000000.wifi: nss init soc nss if_num 203 userpd_id 0
<5>[   15.017527] F2FS-fs (sda1): Mounted with checkpoint version = 16ae637
<6>[   17.300892] ECM init
<6>[   17.300972] ECM database jhash random seed: 0xed7de475
<6>[   17.320408] ECM init complete
<6>[   17.611323] br-lan: port 1(10g-2) entered blocking state
<6>[   17.611369] br-lan: port 1(10g-2) entered disabled state
<6>[   17.615883] nss-dp 3a007000.dp6-syn 10g-2: entered allmulticast mode
<6>[   17.621560] nss-dp 3a007000.dp6-syn 10g-2: entered promiscuous mode
<6>[   17.638768] br-lan: port 2(lan1) entered blocking state
<6>[   17.638860] br-lan: port 2(lan1) entered disabled state
<6>[   17.643100] nss-dp 3a001600.dp4 lan1: entered allmulticast mode
<6>[   17.648721] nss-dp 3a001600.dp4 lan1: entered promiscuous mode
<6>[   17.661532] br-lan: port 3(lan2) entered blocking state
<6>[   17.661619] br-lan: port 3(lan2) entered disabled state
<6>[   17.665793] nss-dp 3a001400.dp3 lan2: entered allmulticast mode
<6>[   17.671374] nss-dp 3a001400.dp3 lan2: entered promiscuous mode
<6>[   17.681635] br-lan: port 4(lan3) entered blocking state
<6>[   17.682628] br-lan: port 4(lan3) entered disabled state
<6>[   17.687832] nss-dp 3a001200.dp2 lan3: entered allmulticast mode
<6>[   17.693451] nss-dp 3a001200.dp2 lan3: entered promiscuous mode
<6>[   17.710917] br-vlan: port 1(lan4) entered blocking state
<6>[   17.710980] br-vlan: port 1(lan4) entered disabled state
<6>[   17.715474] nss-dp 3a001000.dp1 lan4: entered allmulticast mode
<6>[   17.721280] nss-dp 3a001000.dp1 lan4: entered promiscuous mode
<6>[   18.367327] configuring additional NSS pbufs
<6>[   18.398100] additional pbufs of size 10002432 got added to NSS
<6>[   19.640534] br-lan: port 5(phy0-ap0) entered blocking state
<6>[   19.640589] br-lan: port 5(phy0-ap0) entered disabled state
<6>[   19.645662] ath11k c000000.wifi phy0-ap0: entered allmulticast mode
<6>[   19.651811] ath11k c000000.wifi phy0-ap0: entered promiscuous mode
<6>[   19.658958] br-lan: port 5(phy0-ap0) entered blocking state
<6>[   19.663016] br-lan: port 5(phy0-ap0) entered forwarding state
<6>[   19.668995] br-lan: port 5(phy0-ap0) entered disabled state
<6>[   19.681743] ath11k c000000.wifi phy0-ap0: left allmulticast mode
<6>[   19.681813] ath11k c000000.wifi phy0-ap0: left promiscuous mode
<6>[   19.687696] br-lan: port 5(phy0-ap0) entered disabled state
<6>[   19.704875] br-lan: port 5(phy0-ap0) entered blocking state
<6>[   19.704956] br-lan: port 5(phy0-ap0) entered disabled state
<6>[   19.709801] ath11k c000000.wifi phy0-ap0: entered allmulticast mode
<6>[   19.715840] ath11k c000000.wifi phy0-ap0: entered promiscuous mode
<6>[   20.279745] br-lan: port 5(phy0-ap0) entered blocking state
<6>[   20.279790] br-lan: port 5(phy0-ap0) entered forwarding state
<6>[   20.636072] br-lan: port 6(phy1-ap0) entered blocking state
<6>[   20.636140] br-lan: port 6(phy1-ap0) entered disabled state
<6>[   20.640577] ath11k c000000.wifi phy1-ap0: entered allmulticast mode
<6>[   20.646434] ath11k c000000.wifi phy1-ap0: entered promiscuous mode
<6>[   20.652485] br-lan: port 6(phy1-ap0) entered blocking state
<6>[   20.658542] br-lan: port 6(phy1-ap0) entered forwarding state
<6>[   20.669242] br-lan: port 6(phy1-ap0) entered disabled state
<6>[   20.704512] nss-dp 3a001600.dp4 lan1: PHY Link up speed: 1000
<6>[   20.704724] br-lan: port 2(lan1) entered blocking state
<6>[   20.709288] br-lan: port 2(lan1) entered forwarding state
<6>[   20.830919] br-lan: port 6(phy1-ap0) entered blocking state
<6>[   20.830965] br-lan: port 6(phy1-ap0) entered forwarding state
<6>[   21.792437] nss-dp 3a001400.dp3 lan2: PHY Link up speed: 1000
<6>[   21.792550] br-lan: port 3(lan2) entered blocking state
<6>[   21.797188] br-lan: port 3(lan2) entered forwarding state
<6>[   21.856556] nss-dp 3a001800.dp5 10g-1: PHY Link up speed: 1000
<4>[   22.816388] Qualcomm QCA8075 90000.mdio-1:10: Downshift occurred from negotiated speed 1Gbps to actual speed 100Mbps, check cabling!
<6>[   22.816466] nss-dp 3a001000.dp1 lan4: PHY Link up speed: 100
<6>[   22.827562] br-vlan: port 1(lan4) entered blocking state
<6>[   22.833023] br-vlan: port 1(lan4) entered forwarding state
<6>[ 7142.276534] usb 2-1: USB disconnect, device number 2
<5>[ 7142.292062] sd 0:0:0:0: [sda] Synchronizing SCSI cache
<6>[ 7142.414055] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=DRIVER_OK
<6>[ 9179.431275] sysrq: Trigger a crash
<0>[ 9179.431312] Kernel panic - not syncing: sysrq triggered crash
<4>[ 9179.433576] CPU: 1 PID: 26499 Comm: sh Tainted: G           O       6.6.32 #0
<4>[ 9179.439394] Hardware name: QNAP 301w (DT)
<4>[ 9179.446507] Call trace:
<4>[ 9179.450495]  dump_backtrace+0xb0/0x10c
<4>[ 9179.452754]  show_stack+0x18/0x24
<4>[ 9179.456573]  dump_stack_lvl+0x48/0x60
<4>[ 9179.459959]  dump_stack+0x18/0x24
<4>[ 9179.463605]  panic+0x2d0/0x31c
<4>[ 9179.466903]  sysrq_handle_moom+0x0/0x30
<4>[ 9179.469856]  __handle_sysrq+0xe4/0x1e0
<4>[ 9179.473587]  write_sysrq_trigger+0xbc/0xfc
<4>[ 9179.477408]  proc_reg_write+0xb0/0xf8
<4>[ 9179.481487]  vfs_write+0xa4/0x298
<4>[ 9179.485218]  ksys_write+0x5c/0xe0
<4>[ 9179.488518]  __arm64_sys_write+0x1c/0x28
<4>[ 9179.491817]  invoke_syscall.constprop.0+0x5c/0x100
<4>[ 9179.495812]  do_el0_svc+0x40/0xc8
<4>[ 9179.500410]  el0_svc+0x30/0xb8
<4>[ 9179.503794]  el0t_64_sync_handler+0x120/0x12c
<4>[ 9179.506748]  el0t_64_sync+0x178/0x17c
<2>[ 9179.511177] SMP: stopping secondary CPUs
<0>[ 9179.514826] Kernel Offset: disabled
<0>[ 9179.518812] CPU features: 0x0,00000000,00000000,0000400b
<0>[ 9179.522029] Memory Limit: none

1 Like

Maybe a dump question but how/why can I see in tcpdump (on the bridge with wlans) EAPOL frames?

It does not happen with the stock build.

Yup! It's actually the reason I went digging to to get pstore working. As I'm dealing with the dreaded ath11k firmware bug. and here on my NSS builds.

@sppmaster , @Spacebar are you still using 2.7.0.1? I'm seeing the issue when I have 802.11r FT enabled on my DL-WRX36, and clients transitioning from the other 2 MX5300s. It doesn't happen on the MX5300, since I'm using the WiFi 6 radio as backhaul, and qca9984 for the clients.

It looks like it helps, but I'm seeing other issues, mostly lower throughout and firmware being super sensitive to any wlan config changes.

The ath11k driver by default will run in frame_mode=2, which is to encapsulate native WiFi to Ethernet frames. Which is also required for NSS offloading. Are you running in monitor mode?

Can't really comment on stock, since primary driver qca_ol is proprietary, among a whole stack of other qca_* modules.

1 Like

I've always used fw_build_id WLAN.HK.2.9.0.1-01977-QCAHKSWPL_SILICONZ-1 since it was out.

1 Like

Sorry, meant to ask @Spacebar, typo in autofill :smile:

1 Like

Hello.

Yes, I've been using WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1 on WAX620+WAX630. Low latency, stable and I do not have any errors/issues as before.
I don't use FT or anything on my setup. I figured out that the simple wpad-basic-mbedtls was enough too.

In terms for latency, I'm going to do some crusader tests between the WLAN.HK versions like I did earlier with Flent. 2.6 = 50ms, 2.7 = 55ms, 2.9 = 100ms.

I've been also testing the AQL patch for IPQ807x. But crashes as soon as a client connect to the wifi. :confused:
https://patchwork.kernel.org/project/linux-wireless/patch/20230501130725.7171-1-quic_tamizhr@quicinc.com/

1 Like

Ah sure, now when you mentioned that frame_mode I do recall reading about it in the changelog. And it was in the AP and AP/VLAN mode. I was asking in the regards of testing that VLAN feature which was not succesful, sadly. More about it here:

On the official openwrt build I'm using very similiar patch to the one you have in your openwrt-ipq repo and it works there, so I'm guessing it must have something to do with the offloading itself...

1 Like

For me it still fails with DL-WRX36 with your patch.
Sometimes it complaints about found existing contents in memory:

root@router5:~# dmesg | grep -E "ramo|psto"
[    0.000000] OF: reserved mem: 0x0000000051200000..0x00000000512fffff (1024 KiB) map non-reusable ramoops@51200000
[    2.957910] ramoops: found existing invalid buffer, size 2656, start 268436261
[    2.960284] ramoops: found existing invalid buffer, size 2144, start 16886
[    2.967029] pstore: Using crash dump compression: deflate
[    2.973819] pstore: Registered ramoops as persistent store backend
[    2.979262] ramoops: using 0x100000@0x51200000, ecc: 0

After triggering the sysrq, the pstore directory remains empty. /sys/fs/pstore

@qosmio @sppmaster
Were your successes with DL-WRX36?
(the log in Github commit shows Linksys MX5300)

I'm on QNAP 301w.

@qosmio

I am a bit late to the party, but here is the output of my Ax3600 (512mb ram).

root@OpenWrt:~# dmesg | grep -E "ramo|psto"
[    0.000000] OF: reserved mem: 0x000000004e900000..0x000000004e9fffff (1024 KiB) nomap non-reusable ramoops@51200000
[    1.147890] 0x000000900000-0x000000a00000 : "pstore"

For the people using my builds

I have preconfigured BanIP with the recommendations from the readme.
I also added some improvements to the wifi config:

I also removed nss crypto support as suggested by @qosmio and made some configuration changes to the ssh algorithms used.

@qosmio I also resorted my default config so you can use the NSS related part for the readme of yours. I tried to keep up with the recent development in this forum and reflect the changes

3 Likes

I am a bit late to the party, but here is the output of my Ax6 (512mb ram).

root@OpenWrt:~# dmesg | grep -E "ramo|psto"
[    3.201677] pstore: Using crash dump compression: deflate
[    3.201714] pstore: Registered ramoops as persistent store backend
[    3.206096] ramoops: using 0x100000@0x4e900000, ecc: 0

Shoot, you're right. I think I got a bit overzealous. It worked on my MX5300s, the DL-WRX36 would not. I checked the stock DTS for that board, and saw that they're allocating space for wifi_dump and wigi_dump and may not like those regions being used.

		m3_dump@51000000 {
			no-map;
			reg = <0x00 0x51000000 0x00 0x100000>;
		};

		wifi_dump@51100000 {
			no-map;
			reg = <0x00 0x51100000 0x00 0x600000>;
		};

		wigig_dump@51300000 {
			no-map;
			reg = <0x00 0x51300000 0x00 0x400000>;
		};

Tried bumping it to 0x51800000 but no luck unfortunately.

		ramoops_region: ramoops@51800000 {
			compatible = "ramoops";
			reg = <0x0 0x51800000 0x0 0x100000>;
			no-map;
			record-size = <0x1000>;
		};

Really strange it's throwing an error -28 (No Space Left). Unless the feature needs to be available in U-boot? Not sure. Back to using an RPI4 + UART + screen...

@JuliusBairaktaris , @zxlhhyccc

Have you tested if you're able to actually capture kernel panics?

You can trigger a kernel panic with:

sh -c 'echo 10 > /proc/sys/kernel/panic; echo c > /proc/sysrq-trigger'

After the router reboots, mount pstore to see if anything was captured.

mkdir -p /tmp/ramoops
mount -t pstore pstore /tmp/ramoops
➤ ls -ltr /tmp/ramoops
drwxr-x---    2 root     root           0 Jun  9 23:02 .
drwxr-xr-x    8 root     root        4.0K Jun  9 23:23 ..
-r--r--r--    1 root     root        6.6K Jun  9 23:21 dmesg-ramoops-0
root@OpenWrt:~# mkdir -p /tmp/ramoops
root@OpenWrt:~# mount -t pstore pstore /tmp/ramoops
root@OpenWrt:~# ls -ltr /tmp/ramoops
-r--r--r--    1 root     root          9035 Jun 11 23:20 dmesg-ramoops-0
root@OpenWrt:~#
Panic#1 Part1
<6>[   32.220507] device lan2 entered promiscuous mode
<6>[   32.224219] l11: disabling
<6>[   32.235184] br-lan: port 3(lan3) entered blocking state
<6>[   32.235248] br-lan: port 3(lan3) entered disabled state
<6>[   32.240325] device lan3 entered promiscuous mode
<6>[   32.920210] configuring additional NSS pbufs
<6>[   32.931516] additional pbufs of size 3100672 got added to NSS
<6>[   33.149615] br-lan: port 4(phy1-ap0) entered blocking state
<6>[   33.149677] br-lan: port 4(phy1-ap0) entered disabled state
<6>[   33.156232] device phy1-ap0 entered promiscuous mode
<6>[   33.161169] br-lan: port 4(phy1-ap0) entered blocking state
<6>[   33.164828] br-lan: port 4(phy1-ap0) entered forwarding state
<6>[   33.172818] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
<6>[   33.179451] br-lan: port 4(phy1-ap0) entered disabled state
<6>[   33.336631] IPv6: ADDRCONF(NETDEV_CHANGE): phy1-ap0: link becomes ready
<6>[   33.336864] br-lan: port 4(phy1-ap0) entered blocking state
<6>[   33.342108] br-lan: port 4(phy1-ap0) entered forwarding state
<6>[   33.764830] br-lan: port 5(phy0-ap0) entered blocking state
<6>[   33.764878] br-lan: port 5(phy0-ap0) entered disabled state
<6>[   33.769759] device phy0-ap0 entered promiscuous mode
<6>[   33.775056] br-lan: port 5(phy0-ap0) entered blocking state
<6>[   33.780040] br-lan: port 5(phy0-ap0) entered forwarding state
<6>[   34.216219] br-lan: port 5(phy0-ap0) entered disabled state
<6>[   34.530557] IPv6: ADDRCONF(NETDEV_CHANGE): phy0-ap0: link becomes ready
<6>[   34.530805] br-lan: port 5(phy0-ap0) entered blocking state
<6>[   34.536062] br-lan: port 5(phy0-ap0) entered forwarding state
<6>[   36.384293] nss-dp 3a001200.dp2 wan: PHY Link up speed: 1000
<6>[   36.384374] IPv6: ADDRCONF(NETDEV_CHANGE): wan: link becomes ready
<6>[   37.344400] nss-dp 3a001600.dp4 lan2: PHY Link up speed: 1000
<6>[   37.344519] br-lan: port 2(lan2) entered blocking state
<6>[   37.349179] br-lan: port 2(lan2) entered forwarding state
<6>[  153.199439] sh (13803): drop_caches: 3
<6>[  184.386993] device phy0-ap0 left promiscuous mode
<6>[  184.387197] br-lan: port 5(phy0-ap0) entered disabled state
<6>[  184.396927] device phy1-ap0 left promiscuous mode
<6>[  184.397197] br-lan: port 4(phy1-ap0) entered disabled state
<6>[  186.490510] br-lan: port 4(phy1-ap0) entered blocking state
<6>[  186.490550] br-lan: port 4(phy1-ap0) entered disabled state
<6>[  186.495306] device phy1-ap0 entered promiscuous mode
<6>[  186.500587] br-lan: port 4(phy1-ap0) entered blocking state
<6>[  186.505668] br-lan: port 4(phy1-ap0) entered forwarding state
<6>[  186.607159] br-lan: port 5(phy0-ap0) entered blocking state
<6>[  186.607204] br-lan: port 5(phy0-ap0) entered disabled state
<6>[  186.612163] device phy0-ap0 entered promiscuous mode
<6>[  186.617461] br-lan: port 5(phy0-ap0) entered blocking state
<6>[  186.622346] br-lan: port 5(phy0-ap0) entered forwarding state
<6>[  187.164236] br-lan: port 4(phy1-ap0) entered disabled state
<6>[  187.164601] br-lan: port 5(phy0-ap0) entered disabled state
<6>[  187.364409] IPv6: ADDRCONF(NETDEV_CHANGE): phy0-ap0: link becomes ready
<6>[  187.364719] br-lan: port 5(phy0-ap0) entered blocking state
<6>[  187.369906] br-lan: port 5(phy0-ap0) entered forwarding state
<6>[  188.871091] IPv6: ADDRCONF(NETDEV_CHANGE): phy1-ap0: link becomes ready
<6>[  188.871359] br-lan: port 4(phy1-ap0) entered blocking state
<6>[  188.876546] br-lan: port 4(phy1-ap0) entered forwarding state
<6>[  200.922032] bash (15111): drop_caches: 3
<6>[  501.507760] bash (18115): drop_caches: 3
<6>[ 3801.547653] bash (25842): drop_caches: 3
<6>[ 4101.543599] bash (26500): drop_caches: 3
<6>[ 5233.115235] nss-dp 3a001600.dp4 lan2: PHY Link is down
<6>[ 5233.115688] br-lan: port 2(lan2) entered disabled state
<6>[ 7401.554064] bash (1312): drop_caches: 3
<6>[ 7701.559635] bash (1981): drop_caches: 3
<6>[11001.571315] bash (9280): drop_caches: 3
<6>[11301.558824] bash (9940): drop_caches: 3
<6>[14601.582629] bash (17252): drop_caches: 3
<6>[14901.553857] bash (20665): drop_caches: 3
<6>[18201.087959] bash (28402): drop_caches: 3
<6>[18501.133998] bash (29070): drop_caches: 3
<6>[21801.145288] bash (3897): drop_caches: 3
<6>[22101.092031] bash (4584): drop_caches: 3
<6>[25401.172370] bash (11849): drop_caches: 3
<6>[25700.917097] bash (12996): drop_caches: 3
<6>[29000.983371] bash (20282): drop_caches: 3
<6>[29300.984379] bash (20942): drop_caches: 3
<6>[32601.005079] bash (28201): drop_caches: 3
<6>[32900.999501] bash (28876): drop_caches: 3
<6>[35997.089130] nss-dp 3a001600.dp4 lan2: PHY Link up speed: 10
<6>[35997.089240] br-lan: port 2(lan2) entered blocking state
<6>[35997.093543] br-lan: port 2(lan2) entered forwarding state
<6>[36003.233061] nss-dp 3a001600.dp4 lan2: PHY Link is down
<6>[36003.233479] br-lan: port 2(lan2) entered disabled state
<6>[36006.305114] nss-dp 3a001600.dp4 lan2: PHY Link up speed: 1000
<6>[36006.305234] br-lan: port 2(lan2) entered blocking state
<6>[36006.309878] br-lan: port 2(lan2) entered forwarding state
<6>[36012.449037] nss-dp 3a001600.dp4 lan2: PHY Link is down
<6>[36012.449448] br-lan: port 2(lan2) entered disabled state
<6>[36015.521102] nss-dp 3a001600.dp4 lan2: PHY Link up speed: 1000
<6>[36015.521235] br-lan: port 2(lan2) entered blocking state
<6>[36015.525864] br-lan: port 2(lan2) entered forwarding state
<6>[36201.002550] bash (3740): drop_caches: 3
<6>[36415.904223] nss-dp 3a001600.dp4 lan2: PHY Link is down
<6>[36415.904638] br-lan: port 2(lan2) entered disabled state
<6>[36501.024080] bash (4409): drop_caches: 3
<6>[39801.027126] bash (11686): drop_caches: 3
<6>[40101.032678] bash (12341): drop_caches: 3
<6>[43401.046213] bash (19636): drop_caches: 3
<6>[43701.058119] bash (20319): drop_caches: 3
<6>[47001.055473] bash (27580): drop_caches: 3
<6>[47301.068544] bash (28239): drop_caches: 3
<6>[50601.074027] bash (3052): drop_caches: 3
<6>[50901.079013] bash (3735): drop_caches: 3
<6>[54201.033160] bash (11031): drop_caches: 3
<6>[54501.099687] bash (11698): drop_caches: 3
<6>[57801.123786] bash (18979): drop_caches: 3
<6>[58101.110557] bash (19649): drop_caches: 3
<6>[61401.123376] bash (26919): drop_caches: 3
<6>[61701.136861] bash (27584): drop_caches: 3
<6>[65001.150371] bash (2405): drop_caches: 3
<6>[65301.140890] bash (3068): drop_caches: 3
<6>[68601.155332] bash (10358): drop_caches: 3
<6>[68901.159704] bash (11021): drop_caches: 3
<6>[72201.167059] bash (18299): drop_caches: 3
<6>[72501.170718] bash (18970): drop_caches: 3
<6>[75801.179956] bash (26244): drop_caches: 3
<6>[76101.192804] bash (26902): drop_caches: 3
<6>[79401.192424] bash (1739): drop_caches: 3
<6>[79701.196696] bash (2429): drop_caches: 3
<6>[83001.204820] bash (10053): drop_caches: 3
<6>[83301.205459] bash (10751): drop_caches: 3
<6>[85066.053041] nss-dp 3a001600.dp4 lan2: PHY Link up speed: 10
<6>[85066.053151] br-lan: port 2(lan2) entered blocking state
<6>[85066.057453] br-lan: port 2(lan2) entered forwarding state
<6>[85072.196990] nss-dp 3a001600.dp4 lan2: PHY Link is down
<6>[85072.197433] br-lan: port 2(lan2) entered disabled state
<6>[85075.269051] nss-dp 3a001600.dp4 lan2: PHY Link up speed: 1000
<6>[85075.269162] br-lan: port 2(lan2) entered blocking state
<6>[85075.273815] br-lan: port 2(lan2) entered forwarding state
<6>[85082.437002] nss-dp 3a001600.dp4 lan2: PHY Link is down
<6>[85082.437314] br-lan: port 2(lan2) entered disabled state
<6>[85084.485065] nss-dp 3a001600.dp4 lan2: PHY Link up speed: 1000
<6>[85084.485226] br-lan: port 2(lan2) entered blocking state
<6>[85084.489828] br-lan: port 2(lan2) entered forwarding state
<6>[86601.232735] bash (18237): drop_caches: 3
<6>[86901.233294] bash (18895): drop_caches: 3
<6>[90201.203943] bash (26371): drop_caches: 3
<6>[90501.256956] bash (27048): drop_caches: 3
<6>[91578.932903] sysrq: Trigger a crash
<0>[91578.932942] Kernel panic - not syncing: sysrq triggered crash
<4>[91578.935207] CPU: 0 PID: 29456 Comm: sh Not tainted 6.1.92 #0
<4>[91578.941018] Hardware name: Redmi AX6 (DT)
<4>[91578.946747] Call trace:
<4>[91578.950648]  dump_backtrace.part.0+0xbc/0xd0
<4>[91578.952909]  show_stack+0x18/0x30
<4>[91578.957419]  dump_stack_lvl+0x6c/0x88
<4>[91578.960633]  dump_stack+0x18/0x34
<4>[91578.964278]  panic+0x158/0x304
<4>[91578.967575]  sysrq_handle_crash+0x1c/0x20
<4>[91578.970529]  __handle_sysrq+0x8c/0x1a0
<4>[91578.974609]  write_sysrq_trigger+0xc0/0x120
<4>[91578.978255]  proc_reg_write+0xb0/0x100
<4>[91578.982333]  vfs_write+0xa4/0x280
<4>[91578.986153]  ksys_write+0x5c/0xe0
<4>[91578.989539]  __arm64_sys_write+0x1c/0x30
<4>[91578.992838]  invoke_syscall.constprop.0+0x5c/0x110
<4>[91578.996833]  do_el0_svc+0x58/0x170
<4>[91579.001431]  el0_svc+0x18/0x54
<4>[91579.004815]  el0t_64_sync_handler+0x114/0x120
<4>[91579.007856]  el0t_64_sync+0x174/0x178
<2>[91579.012284] SMP: stopping secondary CPUs
<0>[91579.015935] Kernel Offset: disabled
<0>[91579.019920] CPU features: 0x00000,00000000,0000400b
<0>[91579.023136] Memory Limit: none
1 Like

It seems to work on nbg7815

BusyBox v1.36.1 (2024-06-11 16:26:59 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r26582+69-10a1e9b44a
 -----------------------------------------------------
root@NBG7815:~# dmesg | grep -E "ramo|psto"
[    0.000000] OF: reserved mem: 0x0000000051200000..0x00000000512fffff (1024 KiB) nomap non-reusable ramoops@51200000
[    2.258657] pstore: Using crash dump compression: deflate
[    2.259833] pstore: Registered ramoops as persistent store backend
[    2.265259] ramoops: using 0x100000@0x51200000, ecc: 0
root@NBG7815:~# ls -ltr /tmp/ramoops
-r--r--r--    1 root     root          6793 Jun 11 18:52 dmesg-ramoops-0
root@NBG7815:~# cat /tmp/ramoops/dmesg-ramoops-0
Panic#1 Part1
<6>[   18.397039] br-lan: port 2(lan2) entered blocking state
<6>[   18.397102] br-lan: port 2(lan2) entered disabled state
<6>[   18.401216] nss-dp 3a001200.dp2 lan2: entered allmulticast mode
<6>[   18.407269] nss-dp 3a001200.dp2 lan2: entered promiscuous mode
<6>[   18.421380] br-lan: port 3(lan3) entered blocking state
<6>[   18.421440] br-lan: port 3(lan3) entered disabled state
<6>[   18.425495] nss-dp 3a001400.dp3 lan3: entered allmulticast mode
<6>[   18.431060] nss-dp 3a001400.dp3 lan3: entered promiscuous mode
<6>[   18.442186] br-lan: port 4(lan4) entered blocking state
<6>[   18.442455] br-lan: port 4(lan4) entered disabled state
<6>[   18.447837] nss-dp 3a001600.dp4 lan4: entered allmulticast mode
<6>[   18.453477] nss-dp 3a001600.dp4 lan4: entered promiscuous mode
<6>[   18.464363] br-lan: port 5(wan) entered blocking state
<6>[   18.464591] br-lan: port 5(wan) entered disabled state
<6>[   18.469935] nss-dp 3a001800.dp5 wan: entered allmulticast mode
<6>[   18.475460] nss-dp 3a001800.dp5 wan: entered promiscuous mode
<6>[   18.484368] br-home: port 2(lan2.90) entered blocking state
<6>[   18.486491] br-home: port 2(lan2.90) entered disabled state
<6>[   18.491929] lan2.90: entered allmulticast mode
<6>[   18.498072] lan2.90: entered promiscuous mode
<6>[   18.505845] br-IoT: port 2(lan2.91) entered blocking state
<6>[   18.506370] br-IoT: port 2(lan2.91) entered disabled state
<6>[   18.511779] lan2.91: entered allmulticast mode
<6>[   18.517867] lan2.91: entered promiscuous mode
<6>[   18.525512] br-home: port 3(lan3.90) entered blocking state
<6>[   18.526052] br-home: port 3(lan3.90) entered disabled state
<6>[   18.531511] lan3.90: entered allmulticast mode
<6>[   18.537642] lan3.90: entered promiscuous mode
<6>[   18.546055] br-IoT: port 3(lan3.91) entered blocking state
<6>[   18.546113] br-IoT: port 3(lan3.91) entered disabled state
<6>[   18.551446] lan3.91: entered allmulticast mode
<6>[   18.557436] lan3.91: entered promiscuous mode
<6>[   19.598669] configuring additional NSS pbufs
<6>[   19.611421] additional pbufs of size 3100672 got added to NSS
<6>[   20.530864] br-home: port 4(phy2-ap0) entered blocking state
<6>[   20.530913] br-home: port 4(phy2-ap0) entered disabled state
<6>[   20.537706] ath11k c000000.wifi phy2-ap0: entered allmulticast mode
<6>[   20.541813] ath11k c000000.wifi phy2-ap0: entered promiscuous mode
<6>[   20.679122] br-lan: port 6(phy0-ap0) entered blocking state
<6>[   20.679236] br-lan: port 6(phy0-ap0) entered disabled state
<6>[   20.684713] ath11k c000000.wifi phy0-ap0: entered allmulticast mode
<6>[   20.689768] ath11k c000000.wifi phy0-ap0: entered promiscuous mode
<6>[   20.696104] br-lan: port 6(phy0-ap0) entered blocking state
<6>[   20.701666] br-lan: port 6(phy0-ap0) entered forwarding state
<6>[   20.757926] br-lan: port 7(phy1-ap0) entered blocking state
<6>[   20.757984] br-lan: port 7(phy1-ap0) entered disabled state
<6>[   20.763987] ath11k c000000.wifi phy1-ap0: entered allmulticast mode
<6>[   20.768880] ath11k c000000.wifi phy1-ap0: entered promiscuous mode
<6>[   20.774392] br-lan: port 7(phy1-ap0) entered blocking state
<6>[   20.780393] br-lan: port 7(phy1-ap0) entered forwarding state
<6>[   21.154475] br-IoT: port 4(phy1-ap1) entered blocking state
<6>[   21.154522] br-IoT: port 4(phy1-ap1) entered disabled state
<6>[   21.160009] ath11k c000000.wifi phy1-ap1: entered allmulticast mode
<6>[   21.165855] ath11k c000000.wifi phy1-ap1: entered promiscuous mode
<6>[   21.174141] br-IoT: port 4(phy1-ap1) entered blocking state
<6>[   21.176978] br-IoT: port 4(phy1-ap1) entered forwarding state
<6>[   21.236769] ath11k c000000.wifi phy1-ap1: left allmulticast mode
<6>[   21.236836] ath11k c000000.wifi phy1-ap1: left promiscuous mode
<6>[   21.243139] br-IoT: port 4(phy1-ap1) entered disabled state
<6>[   21.255847] br-IoT: port 4(phy1-ap1) entered blocking state
<6>[   21.255899] br-IoT: port 4(phy1-ap1) entered disabled state
<6>[   21.260407] ath11k c000000.wifi phy1-ap1: entered allmulticast mode
<6>[   21.266747] ath11k c000000.wifi phy1-ap1: entered promiscuous mode
<6>[   21.272786] br-IoT: port 4(phy1-ap1) entered blocking state
<6>[   21.278325] br-IoT: port 4(phy1-ap1) entered forwarding state
<6>[   21.472745] nss-dp 3a001400.dp3 lan3: PHY Link up speed: 1000
<6>[   21.472829] nss-dp 3a001200.dp2 lan2: PHY Link up speed: 1000
<6>[   21.477945] br-lan: port 2(lan2) entered blocking state
<6>[   21.483262] br-lan: port 2(lan2) entered forwarding state
<6>[   21.489220] br-lan: port 3(lan3) entered blocking state
<6>[   21.493844] br-lan: port 3(lan3) entered forwarding state
<6>[   21.499687] br-home: port 2(lan2.90) entered blocking state
<6>[   21.504412] br-home: port 2(lan2.90) entered forwarding state
<6>[   21.510413] br-IoT: port 2(lan2.91) entered blocking state
<6>[   21.515705] br-IoT: port 2(lan2.91) entered forwarding state
<6>[   21.521580] br-home: port 3(lan3.90) entered blocking state
<6>[   21.526918] br-home: port 3(lan3.90) entered forwarding state
<6>[   21.533028] br-IoT: port 3(lan3.91) entered blocking state
<6>[   21.538104] br-IoT: port 3(lan3.91) entered forwarding state
<6>[   21.540323] nss-dp 3a001600.dp4 lan4: PHY Link up speed: 1000
<6>[   21.552704] br-lan: port 4(lan4) entered blocking state
<6>[   21.554941] br-lan: port 4(lan4) entered forwarding state
<6>[   21.665028] nss-dp 3a001000.dp1 lan1: PHY Link up speed: 1000
<6>[   82.974528] br-home: port 4(phy2-ap0) entered blocking state
<6>[   82.974578] br-home: port 4(phy2-ap0) entered forwarding state
<6>[  444.986220] sysrq: Trigger a crash
<0>[  444.986258] Kernel panic - not syncing: sysrq triggered crash
<4>[  444.988523] CPU: 3 PID: 11991 Comm: sh Tainted: G           O       6.6.32 #0
<4>[  444.994340] Hardware name: Zyxel NBG7815 (DT)
<4>[  445.001452] Call trace:
<4>[  445.005788]  dump_backtrace+0xa0/0xd8
<4>[  445.008047]  show_stack+0x18/0x24
<4>[  445.011865]  dump_stack_lvl+0x48/0x60
<4>[  445.015164]  dump_stack+0x18/0x24
<4>[  445.018809]  panic+0x2d0/0x31c
<4>[  445.022109]  sysrq_handle_moom+0x0/0x30
<4>[  445.025061]  __handle_sysrq+0xe4/0x1e0
<4>[  445.028793]  write_sysrq_trigger+0xbc/0xfc
<4>[  445.032614]  proc_reg_write+0xb0/0xf8
<4>[  445.036693]  vfs_write+0xa4/0x298
<4>[  445.040424]  ksys_write+0x5c/0xe0
<4>[  445.043723]  __arm64_sys_write+0x1c/0x28
<4>[  445.047023]  invoke_syscall.constprop.0+0x5c/0x100
<4>[  445.051017]  do_el0_svc+0x40/0xc4
<4>[  445.055615]  el0_svc+0x30/0xb8
<4>[  445.059000]  el0t_64_sync_handler+0x120/0x12c
<4>[  445.061954]  el0t_64_sync+0x178/0x17c
<2>[  445.066381] SMP: stopping secondary CPUs
<0>[  445.070033] Kernel Offset: disabled
<0>[  445.074019] CPU features: 0x0,00000000,00000000,0000400b
<0>[  445.077235] Memory Limit: none

1 Like

Looks like a few more boards are working with the ramoops patch.

So far we have:

Model Size Working
8devices Mango-DVK 1G ?
Arcadyan AW1000 1G ?
Arcadyan MF269 1G ?
Buffalo WXR-5950AX12 1G ?
CMCC RM2-6 512M ?
Compex WPQ873 1G ?
Edgecore EAP102 1G ?
Edimax CAX1800 512M ?
Linksys MX4200v1 512M ?
Linksys MX4200v2 1G ?
Linksys MX8500 1G ?
Netgear RAX120v2 1G ?
Netgear SXR80 1G ?
Netgear SXS80 1G ?
Netgear WAX218 512M ?
Netgear WAX620 1G ?
Netgear WAX630 1G ?
prpl Foundation Haze 1G ?
Redmi AX6 512M Y
Spectrum SAX1V1K 1G ?
Xiaomi AX9000 1G ?
Yuncore AX880 1G ?
Yuncore FAP650 512M ?
Zbtlink ZBT-Z800AX 1G ?
ZTE MF269 512M ?

AX3600 Xiaomi|512M|Y
Dynalink DL-WRX36|1G|N
Linksys MX5300|1G|Y
QNAP 301w|1G|Y
Zyxel NBG7815|1G|Y

Unverified

Model Size Working
8devices Mango-DVK 1G ?
Arcadyan AW1000 1G ?
Arcadyan MF269 1G ?
Buffalo WXR-5950AX12 1G ?
CMCC RM2-6 512M ?
Compex WPQ873 1G ?
Edgecore EAP102 1G ?
Edimax CAX1800 512M ?
Linksys MX4200v1 512M ?
Linksys MX4200v2 1G ?
Linksys MX8500 1G ?
Netgear RAX120v2 1G ?
Netgear SXR80 1G ?
Netgear SXS80 1G ?
Netgear WAX218 512M ?
Netgear WAX620 1G ?
Netgear WAX630 1G ?
prpl Foundation Haze 1G ?
Redmi AX6 512M ?
Spectrum SAX1V1K 1G ?
Xiaomi AX9000 1G ?
Yuncore AX880 1G ?
Yuncore FAP650 512M ?
Zbtlink ZBT-Z800AX 1G ?
ZTE MF269 512M ?

I wonder if upstream would accept it even if it doesn't function on the DL-WRX36 or verified on the remaining boards. Technically it's not harming anything, but you'd lose 1MB of RAM.

2 Likes

I would suggest making a pull if you are able to gather feedback from a couple more devices. 1mb is nothing for qualcommax targets. Maybe someone else is able to fix the dynalink

I can test on RAX120. Got a link to your fork?

You can build against main branch with this PR:

I have merge into my NSS Offload branch too:

My NSS defconfig

3 Likes

How could this party forget IPQ6018?
In Kernel Log:

[    3.733545] pstore: Using crash dump compression: deflate
[    3.733799] pstore: Registered ramoops as persistent store backend
[    3.739259] ramoops: using 0x100000@0x4e200000, ecc: 0

After router reboot, I got those output in /tmp/ramoops/dmesg-ramoops-0:

<4>[  114.380257] CPU: 0 PID: 10411 Comm: sh Tainted: G           O       6.6.32 #0
<4>[  114.386073] Hardware name: GL Technologies, Inc. AXT1800 (DT)
<4>[  114.393189] Call trace:
<4>[  114.398912]  dump_backtrace+0xb0/0x114
<4>[  114.401170]  show_stack+0x18/0x24
<4>[  114.404990]  dump_stack_lvl+0x48/0x60
<4>[  114.408376]  dump_stack+0x18/0x24
<4>[  114.412021]  panic+0x2dc/0x330
<4>[  114.415318]  sysrq_handle_moom+0x0/0x30
<4>[  114.418271]  __handle_sysrq+0xe4/0x1cc
<4>[  114.422003]  write_sysrq_trigger+0xbc/0xfc
<4>[  114.425824]  proc_reg_write+0xb0/0xf8
<4>[  114.429902]  vfs_write+0xa4/0x2a0
<4>[  114.433634]  ksys_write+0x5c/0xe0
<4>[  114.436934]  __arm64_sys_write+0x1c/0x28
<4>[  114.440233]  invoke_syscall.constprop.0+0x5c/0x108
<4>[  114.444227]  do_el0_svc+0x40/0xc8
<4>[  114.448826]  el0_svc+0x30/0xb8
<4>[  114.452211]  el0t_64_sync_handler+0x120/0x12c
<4>[  114.455165]  el0t_64_sync+0x178/0x17c
<2>[  114.459592] SMP: stopping secondary CPUs
<0>[  114.463243] Kernel Offset: disabled

Also, is there anyway to fix this?

2 Likes