Qualcommax NSS Build

Are you able to grab kernel panic logs?

Sorry im a little beginner. May I know how to get it

Just curious, on your non-NSS build are able to get stats to show up for you wlan interfaces? I have a patch, but wondering if it's the same on vanilla builds.

phy0-ap0  Link encap:Ethernet  HWaddr 
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:841 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

phy1-ap0  Link encap:Ethernet  HWaddr 
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
1 Like

Check and see if you have any ramoops records in pstore.

ls /sys/fs/pstore/

Then cat to see what it captured.

cat /sys/fs/pstore/*

Good morning @qosmio,

I am going to prepare a new build, do you need me to prove something special?

Regards, Agustin

1 Like

Y, this is empty. There are no phy stats collected. But AFAIR it was never there in the past at any time resp. it was discussed in the past already (I think there is sth. about within the AX3600 thread). It should be shown within cat /proc/net/dev. But there are only LAN/WAN ports filled. I was always wondering how LuCI is getting the realtime stats. But I've observed that LuCI is loosing this ability if the nss firmware changes e. g. for 11.4 LuCI is showing realtime stats while for 12.5 it is not showing anything. I didn't dig further.

1 Like

I've make those iperf3 tests after almost every update of my 301w firmware.
My setup is 301w (in router mode running iperf3 server) and Moto Fusion Smartphone (160MHz 5GHz WLAN client).
I get these results.

Server listening on 5201 (test #3)
-----------------------------------------------------------
Accepted connection from 192.168.1.6, port 44110
[  5] local 192.168.1.1 port 5201 connected to 192.168.1.6 port 44118
[  8] local 192.168.1.1 port 5201 connected to 192.168.1.6 port 44122
[ 10] local 192.168.1.1 port 5201 connected to 192.168.1.6 port 44124
[ 12] local 192.168.1.1 port 5201 connected to 192.168.1.6 port 44138
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  37.6 MBytes   315 Mbits/sec    0   1.43 MBytes
[  8]   0.00-1.00   sec  11.6 MBytes  97.4 Mbits/sec    0    283 KBytes
[ 10]   0.00-1.00   sec  18.9 MBytes   158 Mbits/sec    0    520 KBytes
[ 12]   0.00-1.00   sec  21.2 MBytes   178 Mbits/sec    0    553 KBytes
[SUM]   0.00-1.00   sec  89.4 MBytes   749 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  37.5 MBytes   314 Mbits/sec    0   1.43 MBytes
[  8]   1.00-2.00   sec  26.6 MBytes   223 Mbits/sec    0    635 KBytes
[ 10]   1.00-2.00   sec  29.8 MBytes   249 Mbits/sec    0    720 KBytes
[ 12]   1.00-2.00   sec  31.2 MBytes   261 Mbits/sec    0    706 KBytes
[SUM]   1.00-2.00   sec   125 MBytes  1.05 Gbits/sec    0

- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  26.00-27.00  sec  35.0 MBytes   294 Mbits/sec    0   2.29 MBytes
[  8]  26.00-27.00  sec  33.4 MBytes   280 Mbits/sec    0   2.10 MBytes
[ 10]  26.00-27.00  sec  40.6 MBytes   341 Mbits/sec    0   2.44 MBytes
[ 12]  26.00-27.00  sec  37.6 MBytes   316 Mbits/sec    0   2.36 MBytes
[SUM]  26.00-27.00  sec   147 MBytes  1.23 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  27.00-28.00  sec  34.8 MBytes   292 Mbits/sec    0   2.29 MBytes
[  8]  27.00-28.00  sec  32.4 MBytes   272 Mbits/sec    0   2.10 MBytes
[ 10]  27.00-28.00  sec  37.0 MBytes   310 Mbits/sec    0   2.44 MBytes
[ 12]  27.00-28.00  sec  36.6 MBytes   307 Mbits/sec    0   2.36 MBytes
[SUM]  27.00-28.00  sec   141 MBytes  1.18 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  28.00-29.00  sec  36.6 MBytes   307 Mbits/sec    0   2.29 MBytes
[  8]  28.00-29.00  sec  36.9 MBytes   309 Mbits/sec    0   2.10 MBytes
[ 10]  28.00-29.00  sec  37.6 MBytes   315 Mbits/sec    0   2.44 MBytes
[ 12]  28.00-29.00  sec  38.2 MBytes   321 Mbits/sec    0   2.36 MBytes
[SUM]  28.00-29.00  sec   149 MBytes  1.25 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  29.00-30.00  sec  36.4 MBytes   305 Mbits/sec    0   2.29 MBytes
[  8]  29.00-30.00  sec  33.8 MBytes   283 Mbits/sec    0   2.10 MBytes
[ 10]  29.00-30.00  sec  37.4 MBytes   314 Mbits/sec    0   2.44 MBytes
[ 12]  29.00-30.00  sec  39.9 MBytes   335 Mbits/sec    0   2.36 MBytes
[SUM]  29.00-30.00  sec   147 MBytes  1.24 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.01  sec  1.06 GBytes   304 Mbits/sec    0             sender
[  8]   0.00-30.01  sec   953 MBytes   266 Mbits/sec    0             sender
[ 10]   0.00-30.01  sec   930 MBytes   260 Mbits/sec    0             sender
[ 12]   0.00-30.01  sec   956 MBytes   267 Mbits/sec    0             sender
[SUM]   0.00-30.01  sec  3.84 GBytes  1.10 Gbits/sec    0             sender

I get the same results in both directions - sending from server (-R) and from client. Currently I'm on NSS 12.2 + ath11k 2.9.
I've never been able to get past 1.3-1.4 Gbits/sec with the Moto but it's the only 160MHz capable device I have in my hands.
Do you use latest iperf3 on Windows. I remember that there was a bug in the older Windows iperf3 versions that gave wrong (low) results in some cases.
Currently I use this build.

PS E:\iperf3-win64> .\iperf3 -v
iperf 3.17.1 (cJSON 1.7.15)
CYGWIN_NT-10.0-22631 s-100915 3.5.3-1.x86_64 2024-04-03 17:25 UTC x86_64
Optional features available: CPU affinity setting, support IPv4 don't fragment, POSIX threads

Later I will try iprf3 from my Win 11 Laptop (2.5G cable)<-->(10G LAN port) 301w (WLAN 5G)<-->Moto (5G 160MHz) and I will update the post.
Now I have another job caused by @qosmio's latest commits :wink: and the kid has even more important job with the Laptop :slight_smile: .

@asvio @anom3 @rmandrad
Probably you'll (and all having nbg7815 or 301w) be interested in this important PR.

1 Like

Hey All,

Currently have 5 active branches, the only key difference being backports version, ath11k firmware, and the still pending PR for MX4300/LN301.

Devices supported upstream

branch backports ath11k
qualcommax-6.x-nss-wifi 6.9.9 2.9
qualcommax-6.x-nss-wifi-6.11 6.11 2.9
qualcommax-6.x-nss-ath-2.12 6.9.9 2.12
Pending Devices (MX4300/LN301)
branch backports ath11k
qualcommax-6.x-nss-mx4300 6.9.9 2.12
qualcommax-6.x-nss-mx4300-6.11 6.11 2.12

I pushed quite a few updates earlier, but some of the highlight commits to look for.

date commit desc
2024-09-30 f8e3b55066 kernel: Fix UDP hanging in unbound
2024-09-30 0e215ddc6a ath11k_nss: Fix iface stats showing up as 0
2024-09-29 857fe848f6 ath11k_nss: fix WDS instability by disabling NAWDS mode
2024-09-29 1261cf826c ath11k_nss: Fix kernel panic on free_netdev during reboot
2024-09-27 e1cac247f3 ath11k_nss: fix peer ast not found warnings

I know the number of branches looks a bit crazy, but wanted to ensure I separate them out to make rebasing with upstream easier. Eventually everything will be merged back in to qualcommax-6.x-nss-wifi once I can confirm 6.11 is stable for everyone. For know keeping qualcommax-6.x-nss-wifi on 6.9.9.

Long story short:

Build off branch: mx4300-6.11 :slight_smile:

3 Likes

My latest commits should now show proper stats in ifconfig and /proc/net/dev.

Running backports 6.11 + NSS 11.4 + ATH11K 2.12 on (dl-wrx36, mx5300, and mx4300)

4 Likes

Hey,

Can't comment for everyone, but I've been shuffling different builds (with different nss and ath11k fw versions for testing) based on @asvio repos in a last few days and weeks frequently - config was always preserved and haven't encountered any issues. My build is a bit different than asvio's default - I've replaced dnsmasq with full odhcpd, installed unbound with TLS and added adblocknto it, as well as changed wpad to openssl-full and added some random packages to taste. No issues with config preservation when reflashing back and forth.

Oh, and I'm on nbg7815.

1 Like

Maybe it's better to comment on the issue on the PR page just to not spam here but as this is really important for our devices I put a notice here.
Reading the PR I understand that the issue itself isn't that the settings are not preserved during sysupgrade rather than they are written in the wrong place of the emmc.

1 Like

@qosmio

root@OpenWrt:~# ls /sys/fs/pstore/
dmesg-ramoops-0
root@OpenWrt:~# cat /sys/fs/pstore/*
Panic#1 Part1
root@OpenWrt:~# cat /sys/fs/pstore/*
Panic#1 Part1
<5>[    2.013814] ubi0: background thread "ubi_bgt0d" started, PID 558
<6>[    2.023678] block ubiblock0_1: created from ubi0:1(rootfs)
<6>[    2.029292] clk: Disabling unused clocks
<6>[    2.038191] VFS: Mounted root (squashfs filesystem) readonly on device 254                                                                                                                     :0.
<6>[    2.038653] Freeing unused kernel memory: 896K
<6>[    2.044416] Run /sbin/init as init process
<7>[    2.048662]   with arguments:
<7>[    2.048665]     /sbin/init
<7>[    2.048668]   with environment:
<7>[    2.048670]     HOME=/
<7>[    2.048673]     TERM=linux
<14>[    2.262769] init: Console is alive
<14>[    2.262911] init: - watchdog -
<5>[    2.995447] random: crng init done
<14>[    3.303565] kmodloader: loading kernel modules from /etc/modules-boot.d/*
<4>[    3.338239] gpio_button_hotplug: loading out-of-tree module taints kernel.
<6>[    3.343954] pstore: Using crash dump compression: deflate
<6>[    3.344016] pstore: Registered ramoops as persistent store backend
<6>[    3.349584] ramoops: using 0x100000@0x4e900000, ecc: 0
<4>[    3.399464] ssdk_dt_parse_interrupt[941]:INFO:intr-gpio does not exist
<4>[    3.821573] regi_init[2525]:INFO:Initializing HPPE Done!!
<4>[    3.821698] regi_init[2574]:INFO:qca-ssdk module init succeeded!
<6>[    3.828693] EDMA ver 1 hw init
<6>[    3.832263] EDMA HW Reset completed succesfully
<6>[    3.834933] Num rings - TxDesc:1 (23-23) TxCmpl:1 (7-7)
<6>[    3.839373] RxDesc:1 (15-15) RxFill:1 (7-7)
<6>[    3.845000] dp2: ppe offload disabled: 0 for macid 2
<6>[    3.848718] dp2: Switch attached to macid 2 status: 0
<6>[    4.046240] Qualcomm QCA8075 90000.mdio-1:01: attached PHY driver (mii_bus                                                                                                                     :phy_addr=90000.mdio-1:01, irq=POLL)
<6>[    4.047287] dp3: ppe offload disabled: 0 for macid 3
<6>[    4.055130] dp3: Switch attached to macid 3 status: 0
<6>[    4.145800] Qualcomm QCA8075 90000.mdio-1:02: attached PHY driver (mii_bus                                                                                                                     :phy_addr=90000.mdio-1:02, irq=POLL)
<6>[    4.146893] dp4: ppe offload disabled: 0 for macid 4
<6>[    4.154694] dp4: Switch attached to macid 4 status: 0
<6>[    4.235796] Qualcomm QCA8075 90000.mdio-1:03: attached PHY driver (mii_bus                                                                                                                     :phy_addr=90000.mdio-1:03, irq=POLL)
<6>[    4.236833] dp5: ppe offload disabled: 0 for macid 5
<6>[    4.244687] dp5: Switch attached to macid 5 status: 0
<6>[    4.325786] Qualcomm QCA8075 90000.mdio-1:04: attached PHY driver (mii_bus                                                                                                                     :phy_addr=90000.mdio-1:04, irq=POLL)
<6>[    4.326793] **********************************************************
<6>[    4.334679] * NSS Data Plane driver
<6>[    4.341208] **********************************************************
<14>[    4.347167] kmodloader: done loading kernel modules from /etc/modules-boo                                                                                                                     t.d/*
<14>[    4.361270] init: - preinit -
<5>[    7.341163] UBIFS (ubi0:2): default file-system created
<5>[    7.341805] UBIFS (ubi0:2): Mounting in unauthenticated mode
<5>[    7.345322] UBIFS (ubi0:2): background thread "ubifs_bgt0_2" started, PID                                                                                                                      728
<5>[    7.406451] UBIFS (ubi0:2): UBIFS: mounted UBI device 0, volume 2, name "r                                                                                                                     ootfs_data"
<5>[    7.406487] UBIFS (ubi0:2): LEB size: 126976 bytes (124 KiB), min./max. I/                                                                                                                     O unit sizes: 2048 bytes/2048 bytes
<5>[    7.413267] UBIFS (ubi0:2): FS size: 24887296 bytes (23 MiB, 196 LEBs), ma                                                                                                                     x 206 LEBs, journal size 1269760 bytes (1 MiB, 10 LEBs)
<5>[    7.423185] UBIFS (ubi0:2): reserved for root: 1175487 bytes (1147 KiB)
<5>[    7.434798] UBIFS (ubi0:2): media format: w5/r0 (latest is w5/r0), UUID D0                                                                                                                     31711E-5FC7-4E13-AECA-8C7613491D7F, small LPT model
<14>[    7.443001] mount_root: overlay filesystem has not been fully initialized                                                                                                                      yet
<14>[    7.453015] mount_root: switching to ubifs overlay
<4>[    7.462498] overlayfs: null uuid detected in lower fs '/', falling back to                                                                                                                      xino=off,index=off,nfs_export=off.
<12>[    7.468969] urandom-seed: Seed file not found (/etc/urandom.seed)
<14>[    7.545789] procd: - early -
<14>[    7.545906] procd: - watchdog -
<14>[    8.126709] procd: - watchdog -
<14>[    8.154817] procd: - ubus -
<14>[    8.315789] procd: - init -
<14>[    8.880605] kmodloader: loading kernel modules from /etc/modules.d/*
<14>[    9.025363] urngd: v1.0.2 started.
<6>[    9.032802] NET: Registered PF_ALG protocol family
<6>[    9.042669] tun: Universal TUN/TAP device driver, 1.6
<6>[    9.053619] ipip: IPv4 and MPLS over IPv4 tunneling driver
<13>[    9.303871] hotplug: symlinking qca-nss0.bin to /lib/firmware/qca-nss0-re                                                                                                                     tail.bin
<6>[    9.432347] qca-nss 39000000.nss: NSS FW Version: NSS.HK.11.4.0.5-6-R
<6>[    9.432400] qca-nss 39000000.nss: fw of size 835960 bytes copied to addr:                                                                                                                      40000000, nss_id: 0
<1>[    9.439801]
<6>[    9.485441] qca-nss 39000000.nss: NSS core 0 booted successfully
<1>[    9.491776] NSS core 0 signal COREDUMP COMPLETE 4000
<1>[    9.491803] ffffffc07907f580: Starting NSS-FW logbuffer dump for core 0
<1>[    9.495809] ffffffc07907f580: Warn: trap[813]: Trap on CHIP ID 00050002
<1>[    9.502137] ffffffc07907f580: Warn: trap[620]: Trapped: TRAP_TD(00000200)                                                                                                                      DCAPT(3C000080)
<1>[    9.508741] ffffffc07907f580: Warn: trap[645]: Trapped: Thread: 9, reason:                                                                                                                      00001000, PC: 3F0039DC, previous PC: 3F0039D8
<1>[    9.517082] ffffffc07907f580: Warn: trap[594]: A0_3: 3F012850 3F02DC4C 3F0                                                                                                                     127A8 00000000
<1>[    9.528013] ffffffc07907f580: Warn: trap[594]: A4_7: 3F02DB9C 3F0127E0 3F0                                                                                                                     2DA4C 3F00D718
<1>[    9.536085] ffffffc07907f580: Warn: trap[599]: D0_3: 00000000 00000020 000                                                                                                                     00024 00003FFF
<1>[    9.544153] ffffffc07907f580: Warn: trap[599]: D4_7: 40000000 00000000 000                                                                                                                     01000 00000000
<1>[    9.552231] ffffffc07907f580: Warn: trap[599]: D8_11: 00000000 00000000 00                                                                                                                     000000 00000000
<1>[    9.560304] ffffffc07907f580: Warn: trap[599]: D12_15: 00000000 00000000 0                                                                                                                     0000000 00000000
<1>[    9.568378] ffffffc07907f580: Warn: trap[649]: Thread_9 has non-recoverabl                                                                                                                     e trap
<0>[    9.576538] Kernel panic - not syncing: NSS FW coredump: bringing system d                                                                                                                     own
<4>[    9.584084] CPU: 0 PID: 1052 Comm: napi/nss-13 Tainted: G           O                                                                                                                            6.6.52 #0
<4>[    9.591115] Hardware name: Redmi AX6 (DT)
<4>[    9.598920] Call trace:
<4>[    9.602910]  dump_backtrace+0xa0/0xd8
<4>[    9.605168]  show_stack+0x18/0x24
<4>[    9.608988]  dump_stack_lvl+0x48/0x60
<4>[    9.612286]  dump_stack+0x18/0x24
<4>[    9.615932]  panic+0x2c8/0x314
<4>[    9.619229]  nss_fw_coredump_notify+0x1e4/0x3dc [qca_nss_drv]
<4>[    9.622187]  nss_core_handle_napi_emergency+0x38/0x4c [qca_nss_drv]
<4>[    9.628002]  __napi_poll+0x38/0x178
<4>[    9.634075]  napi_threaded_poll_loop+0x80/0x1bc
<4>[    9.637548]  napi_threaded_poll+0x78/0x84
<4>[    9.642061]  kthread+0x10c/0x110
<4>[    9.646226]  ret_from_fork+0x10/0x20
<2>[    9.649527] SMP: stopping secondary CPUs
<0>[    9.653091] Kernel Offset: disabled
<0>[    9.656991] CPU features: 0x0,00000000,00000000,0000400b
<0>[    9.660208] Memory Limit: none
root@OpenWrt:~#

I don't remember having any problems updating nbg7815 while keeping the setting.
Anyway, I implemented it on one of my branches without nss support to check if there was any difference. I didn't really notice the difference.
I will wait for the PR to be approved or in case I notice any problems with the sysupgrade.

1 Like

@asvio
I haven't noticed any issues either and wrote a comment on the PR.
@qosmio
I am testing a build with backports 6.11 + NSS 12.2 + ATH11K 2.9 based on your qualcommax-6.x-nss-wifi-6.11 branch.
You are not environmentally friendly with all those branches :slight_smile:. I have to compile twice now or maybe I just don't know how to use what you recommended. And I expect sometime in the future two MX4300/LN1301 from USA.

I have two dmesg-ramoops files after reboot.

Oops#1 Part1
<6>[    4.906710] xhci-hcd xhci-hcd.2.auto: xHCI Host Controller
<6>[    4.910444] xhci-hcd xhci-hcd.2.auto: new USB bus registered, assigned bus number 3
<6>[    4.915865] xhci-hcd xhci-hcd.2.auto: hcc params 0x0220fe65 hci version 0x110 quirks 0x0000008002000010
<6>[    4.923379] xhci-hcd xhci-hcd.2.auto: irq 43, io mem 0x08c00000
<6>[    4.932762] xhci-hcd xhci-hcd.2.auto: xHCI Host Controller
<6>[    4.938549] xhci-hcd xhci-hcd.2.auto: new USB bus registered, assigned bus number 4
<6>[    4.944113] xhci-hcd xhci-hcd.2.auto: Host supports USB 3.0 SuperSpeed
<6>[    4.951957] hub 3-0:1.0: USB hub found
<6>[    4.958259] hub 3-0:1.0: 1 port detected
<6>[    4.962206] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[    4.966300] hub 4-0:1.0: USB hub found
<6>[    4.974153] hub 4-0:1.0: 1 port detected
<6>[    4.982102] usbcore: registered new interface driver usb-storage
<6>[    4.983122] usbcore: registered new interface driver uas
<14>[    4.988021] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
<14>[    5.003471] init: - preinit -
<6>[    6.008907] usb 2-1: new SuperSpeed USB device number 2 using xhci-hcd
<6>[    6.043513] scsi host0: uas
<5>[    7.113824] scsi 0:0:0:0: Direct-Access     SanDisk  Extreme 55AE     3008 PQ: 0 ANSI: 6
<5>[    7.116766] scsi 0:0:0:1: Enclosure         SanDisk  SES Device       3008 PQ: 0 ANSI: 6
<5>[    7.117139] sd 0:0:0:0: [sda] 3906963617 512-byte logical blocks: (2.00 TB/1.82 TiB)
<5>[    7.129403] sd 0:0:0:0: [sda] Write Protect is off
<7>[    7.136781] sd 0:0:0:0: [sda] Mode Sense: 37 00 10 00
<5>[    7.137001] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
<6>[    7.259033] sd 0:0:0:0: [sda] Preferred minimum I/O size 512 bytes
<6>[    7.259055] sd 0:0:0:0: [sda] Optimal transfer size 1048576 bytes
<6>[    7.268192]  sda: sda1
<5>[    7.270735] sd 0:0:0:0: [sda] Attached SCSI disk
<5>[   13.128869] random: crng init done
<14>[   15.602134] mount_root: loading kmods from internal overlay
<14>[   15.614204] kmodloader: loading kernel modules from //etc/modules-boot.d/*
<14>[   15.615516] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
<14>[   16.036894] block: attempting to load /tmp/overlay/upper/etc/config/fstab
<11>[   16.037033] block: unable to load configuration (fstab: Entry not found)
<14>[   16.042801] block: attempting to load /tmp/overlay/etc/config/fstab
<11>[   16.049532] block: unable to load configuration (fstab: Entry not found)
<14>[   16.055490] block: attempting to load /etc/config/fstab
<11>[   16.065707] block: unable to load configuration (fstab: Entry not found)
<11>[   16.067389] block: no usable configuration
<14>[   16.074592] block: attempting to load /etc/config/fstab
<11>[   16.078248] block: unable to load configuration (fstab: Entry not found)
<11>[   16.083408] block: no usable configuration
<6>[   16.091221] loop0: detected capacity change from 0 to 1048576
<6>[   16.128883] loop0: detected capacity change from 1048576 to 1009920
<6>[   16.305783] EXT4-fs (loop0): recovery complete
<6>[   16.305946] EXT4-fs (loop0): mounted filesystem cexxxxa-40xx-5xx0-xxx-58xxxxxcaf r/w with ordered data mode. Quota mode: disabled.
<6>[   16.489066] nss-dp 3a001600.dp4 lan1: PHY Link up speed: 1000
<1>[   16.517642] Unable to handle kernel read from unreadable memory at virtual address 0000000000000090
<1>[   16.517689] Mem abort info:
<1>[   16.525507]   ESR = 0x0000000096000005
<1>[   16.528273]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[   16.532118]   SET = 0, FnV = 0
<1>[   16.537560]   EA = 0, S1PTW = 0
<1>[   16.540433]   FSC = 0x05: level 1 translation fault
<1>[   16.543466] Data abort info:
<1>[   16.548323]   ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
<1>[   16.551463]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
<1>[   16.556756]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
<1>[   16.561879] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000042fe4000
<1>[   16.567254] [0000000000000090] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
<0>[   16.573605] Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP
<4>[   16.582098] Modules linked in: uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom qca_nss_dp(O) qca_ssdk(O) ramoops reed_solomon pstore gpio_button_hotplug(O) vfat fat f2fs ext4 mbcache jbd2 aquantia hwmon crc_ccitt crc32c_generic crc32_generic
<4>[   16.589723] CPU: 0 PID: 954 Comm: block Tainted: G           O       6.6.52 #0
<4>[   16.611959] Hardware name: QNAP 301w (DT)
<4>[   16.619159] pstate: 20400005 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
<4>[   16.623244] pc : edma_napi+0x2c0/0x4c4 [qca_nss_dp]
<4>[   16.630011] lr : edma_napi+0x290/0x4c4 [qca_nss_dp]
<4>[   16.634871] sp : ffffffc080003e00
<4>[   16.639729] x29: ffffffc080003e00 x28: 0000000000000040 x27: ffffff800287c000
<4>[   16.643211] x26: ffffffc080003f50 x25: ffffff803fd9e480 x24: ffffffc080b7d1c0
<4>[   16.650327] x23: ffffffc080003f40 x22: ffffffc078e27ee0 x21: 0000000000000001
<4>[   16.657446] x20: ffffff8003088900 x19: 0000000000000001 x18: 0000000000000014
<4>[   16.664564] x17: 000000005c83cc57 x16: 00000000ccaae286 x15: 00000000027925ee
<4>[   16.671683] x14: 00000000198e17db x13: 00000000310db0ac x12: 0000000067d52c77
<4>[   16.678801] x11: 00000000818466c0 x10: 000000007a248d2f x9 : 0000000000000000
<4>[   16.685918] x8 : ffffff80028cb6b4 x7 : 0000000000000000 x6 : 0000000000000000
<4>[   16.693036] x5 : fffffffe00000000 x4 : ffffffc078e28098 x3 : 0000000000000000
<4>[   16.700154] x2 : 0000000000000000 x1 : ffffff80046a6000 x0 : ffffffc078e28088
<4>[   16.707274] Call trace:
<4>[   16.714382]  edma_napi+0x2c0/0x4c4 [qca_nss_dp]
<4>[   16.716643]  __napi_poll+0x38/0x178
<4>[   16.721153]  net_rx_action+0x134/0x2b0
<4>[   16.724627]  handle_softirqs+0xf8/0x22c
<4>[   16.728447]  __do_softirq+0x14/0x20
<4>[   16.732177]  ____do_softirq+0x10/0x1c
<4>[   16.735651]  call_on_irq_stack+0x24/0x4c
<4>[   16.739470]  do_softirq_own_stack+0x1c/0x44
<4>[   16.743464]  irq_exit_rcu+0x90/0xc8
<4>[   16.747369]  el1_interrupt+0x38/0x68
<4>[   16.750842]  el1h_64_irq_handler+0x18/0x24
<4>[   16.754662]  el1h_64_irq+0x68/0x6c
<4>[   16.758567]  mas_store_gfp+0x0/0x8c
<4>[   16.761953]  do_vmi_munmap+0x80/0xd8
<4>[   16.765339]  __vm_munmap+0x84/0x100
<4>[   16.769156]  __arm64_sys_munmap+0x20/0x30
<4>[   16.772370]  invoke_syscall.constprop.0+0x5c/0x100
<4>[   16.776539]  do_el0_svc+0x40/0xcc
<4>[   16.781223]  el0_svc+0x30/0xb8
<4>[   16.784608]  el0t_64_sync_handler+0x120/0x12c
<4>[   16.787563]  el0t_64_sync+0x178/0x17c
<0>[   16.791992] Code: 54000240 f9400001 b4ffff81 f941e022 (f9404843) 
<4>[   16.795639] ---[ end trace 0000000000000000 ]---

Panic#2 Part1
<6>[    4.932762] xhci-hcd xhci-hcd.2.auto: xHCI Host Controller
<6>[    4.938549] xhci-hcd xhci-hcd.2.auto: new USB bus registered, assigned bus number 4
<6>[    4.944113] xhci-hcd xhci-hcd.2.auto: Host supports USB 3.0 SuperSpeed
<6>[    4.951957] hub 3-0:1.0: USB hub found
<6>[    4.958259] hub 3-0:1.0: 1 port detected
<6>[    4.962206] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[    4.966300] hub 4-0:1.0: USB hub found
<6>[    4.974153] hub 4-0:1.0: 1 port detected
<6>[    4.982102] usbcore: registered new interface driver usb-storage
<6>[    4.983122] usbcore: registered new interface driver uas
<14>[    4.988021] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
<14>[    5.003471] init: - preinit -
<6>[    6.008907] usb 2-1: new SuperSpeed USB device number 2 using xhci-hcd
<6>[    6.043513] scsi host0: uas
<5>[    7.113824] scsi 0:0:0:0: Direct-Access     SanDisk  Extreme 55AE     3008 PQ: 0 ANSI: 6
<5>[    7.116766] scsi 0:0:0:1: Enclosure         SanDisk  SES Device       3008 PQ: 0 ANSI: 6
<5>[    7.117139] sd 0:0:0:0: [sda] 3906963617 512-byte logical blocks: (2.00 TB/1.82 TiB)
<5>[    7.129403] sd 0:0:0:0: [sda] Write Protect is off
<7>[    7.136781] sd 0:0:0:0: [sda] Mode Sense: 37 00 10 00
<5>[    7.137001] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
<6>[    7.259033] sd 0:0:0:0: [sda] Preferred minimum I/O size 512 bytes
<6>[    7.259055] sd 0:0:0:0: [sda] Optimal transfer size 1048576 bytes
<6>[    7.268192]  sda: sda1
<5>[    7.270735] sd 0:0:0:0: [sda] Attached SCSI disk
<5>[   13.128869] random: crng init done
<14>[   15.602134] mount_root: loading kmods from internal overlay
<14>[   15.614204] kmodloader: loading kernel modules from //etc/modules-boot.d/*
<14>[   15.615516] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
<14>[   16.036894] block: attempting to load /tmp/overlay/upper/etc/config/fstab
<11>[   16.037033] block: unable to load configuration (fstab: Entry not found)
<14>[   16.042801] block: attempting to load /tmp/overlay/etc/config/fstab
<11>[   16.049532] block: unable to load configuration (fstab: Entry not found)
<14>[   16.055490] block: attempting to load /etc/config/fstab
<11>[   16.065707] block: unable to load configuration (fstab: Entry not found)
<11>[   16.067389] block: no usable configuration
<14>[   16.074592] block: attempting to load /etc/config/fstab
<11>[   16.078248] block: unable to load configuration (fstab: Entry not found)
<11>[   16.083408] block: no usable configuration
<6>[   16.091221] loop0: detected capacity change from 0 to 1048576
<6>[   16.128883] loop0: detected capacity change from 1048576 to 1009920
<6>[   16.305783] EXT4-fs (loop0): recovery complete
<6>[   16.305946] EXT4-fs (loop0): mounted filesystem xxxx-xxxxx-xxxx-xx r/w with ordered data mode. Quota mode: disabled.
<6>[   16.489066] nss-dp 3a001600.dp4 lan1: PHY Link up speed: 1000
<1>[   16.517642] Unable to handle kernel read from unreadable memory at virtual address 0000000000000090
<1>[   16.517689] Mem abort info:
<1>[   16.525507]   ESR = 0x0000000096000005
<1>[   16.528273]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[   16.532118]   SET = 0, FnV = 0
<1>[   16.537560]   EA = 0, S1PTW = 0
<1>[   16.540433]   FSC = 0x05: level 1 translation fault
<1>[   16.543466] Data abort info:
<1>[   16.548323]   ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
<1>[   16.551463]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
<1>[   16.556756]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
<1>[   16.561879] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000042fe4000
<1>[   16.567254] [0000000000000090] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
<0>[   16.573605] Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP
<4>[   16.582098] Modules linked in: uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom qca_nss_dp(O) qca_ssdk(O) ramoops reed_solomon pstore gpio_button_hotplug(O) vfat fat f2fs ext4 mbcache jbd2 aquantia hwmon crc_ccitt crc32c_generic crc32_generic
<4>[   16.589723] CPU: 0 PID: 954 Comm: block Tainted: G           O       6.6.52 #0
<4>[   16.611959] Hardware name: QNAP 301w (DT)
<4>[   16.619159] pstate: 20400005 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
<4>[   16.623244] pc : edma_napi+0x2c0/0x4c4 [qca_nss_dp]
<4>[   16.630011] lr : edma_napi+0x290/0x4c4 [qca_nss_dp]
<4>[   16.634871] sp : ffffffc080003e00
<4>[   16.639729] x29: ffffffc080003e00 x28: 0000000000000040 x27: ffffff800287c000
<4>[   16.643211] x26: ffffffc080003f50 x25: ffffff803fd9e480 x24: ffffffc080b7d1c0
<4>[   16.650327] x23: ffffffc080003f40 x22: ffffffc078e27ee0 x21: 0000000000000001
<4>[   16.657446] x20: ffffff8003088900 x19: 0000000000000001 x18: 0000000000000014
<4>[   16.664564] x17: 000000005c83cc57 x16: 00000000ccaae286 x15: 00000000027925ee
<4>[   16.671683] x14: 00000000198e17db x13: 00000000310db0ac x12: 0000000067d52c77
<4>[   16.678801] x11: 00000000818466c0 x10: 000000007a248d2f x9 : 0000000000000000
<4>[   16.685918] x8 : ffffff80028cb6b4 x7 : 0000000000000000 x6 : 0000000000000000
<4>[   16.693036] x5 : fffffffe00000000 x4 : ffffffc078e28098 x3 : 0000000000000000
<4>[   16.700154] x2 : 0000000000000000 x1 : ffffff80046a6000 x0 : ffffffc078e28088
<4>[   16.707274] Call trace:
<4>[   16.714382]  edma_napi+0x2c0/0x4c4 [qca_nss_dp]
<4>[   16.716643]  __napi_poll+0x38/0x178
<4>[   16.721153]  net_rx_action+0x134/0x2b0
<4>[   16.724627]  handle_softirqs+0xf8/0x22c
<4>[   16.728447]  __do_softirq+0x14/0x20
<4>[   16.732177]  ____do_softirq+0x10/0x1c
<4>[   16.735651]  call_on_irq_stack+0x24/0x4c
<4>[   16.739470]  do_softirq_own_stack+0x1c/0x44
<4>[   16.743464]  irq_exit_rcu+0x90/0xc8
<4>[   16.747369]  el1_interrupt+0x38/0x68
<4>[   16.750842]  el1h_64_irq_handler+0x18/0x24
<4>[   16.754662]  el1h_64_irq+0x68/0x6c
<4>[   16.758567]  mas_store_gfp+0x0/0x8c
<4>[   16.761953]  do_vmi_munmap+0x80/0xd8
<4>[   16.765339]  __vm_munmap+0x84/0x100
<4>[   16.769156]  __arm64_sys_munmap+0x20/0x30
<4>[   16.772370]  invoke_syscall.constprop.0+0x5c/0x100
<4>[   16.776539]  do_el0_svc+0x40/0xcc
<4>[   16.781223]  el0_svc+0x30/0xb8
<4>[   16.784608]  el0t_64_sync_handler+0x120/0x12c
<4>[   16.787563]  el0t_64_sync+0x178/0x17c
<0>[   16.791992] Code: 54000240 f9400001 b4ffff81 f941e022 (f9404843) 
<4>[   16.795639] ---[ end trace 0000000000000000 ]---
<3>[   16.807748] pstore: backend (ramoops) writing error (-28)
<0>[   16.807772] Kernel panic - not syncing: Oops: Fatal exception in interrupt
<2>[   16.812119] SMP: stopping secondary CPUs
<0>[   16.818891] Kernel Offset: disabled
<0>[   16.822961] CPU features: 0x0,00000000,00000000,0000400b
<0>[   16.826178] Memory Limit: none

But seems this error after reboot affects other devices too as noted by lots of other users.
Maybe there is a solution already by @robimarko and PR.

1 Like

@sppmaster , @zxlhhyccc These particular crashes are related to the NSS-12.5-K6.x-NAPI branch of nss-packages. Due to the buggy nature of those patches, with no real performance gains, I've removed it altogether.

@sppmaster the PR is unrelated to this particular crash. You would see something like this:

[  327.592248] Call trace:
[  327.599355]  free_netdev+0x164/0x1a0
[  327.601615]  0xffffffc0798183a8
[  327.605433]  0xffffffc079818478
[  327.608298]  platform_shutdown+0x24/0x34
[  327.611424]  device_shutdown+0x160/0x268
[  327.615591]  kernel_restart+0x40/0xc0
[  327.619497]  __do_sys_reboot+0x104/0x220
[  327.623057]  __arm64_sys_reboot+0x24/0x30
[  327.627049]  invoke_syscall.constprop.0+0x5c/0x100
[  327.630956]  do_el0_svc+0x40/0xc4
[  327.635642]  el0_svc+0x30/0xb8
[  327.639026]  el0t_64_sync_handler+0x120/0x12c
[  327.641981]  el0t_64_sync+0x178/0x17c

Key difference here it's free_netdev+0x164/0x1a0. If you see anything with nss and napi, very likely you might be using the NAPI branch.

1 Like

I'm a little ashamed because of the noise I made with my last post. Maybe in a hurry reading for that crash on reboot I overlooked the dmesg-ramoops files were simply left from the time I've tried NAPI threaded build. I've deleted them manually. And I really didn't cold reboot the router for a really long time (I can't even remember).
Really sorry and as always thanks for your invaluable help.

1 Like

No worries! Just glad to hear it wasn't another new NSS bug to chase down.

This is a good habit. I tend to manually move over ramoops dumps to a USB with timestamps so I know when exactly it occurred.

1 Like

may i ask a question
i prefer my builds with gro disabled also i delete 600-net-enable-fraglist-GRO-by-default.patch

this hack does negatifly impact or overcome disable_gro option
I didn't encounter any noticeable problems but I'm still not sure.

1 Like

You have the option to have it disabled before ECM NSS offloading comes up.

uci set ecm.general.disable_gro=1
uci commit ecm

You can verify with the following

for i in br-lan /sys/class/net/br-lan/brif/*; do
  i=${i##*\/}
  ethtool -k ${i} | \
  awk -v i=$i '/generic-receive-offload/{printf "%8s %s\n", i,$2}'
done

Before

  br-lan on
    lan1 on
    lan2 on
    lan3 on
    lan4 on
phy0-ap0 on
phy1-ap0 on

After

$ service qca-nss-ecm restart
  br-lan off
    lan1 off
    lan2 off
    lan3 off
    lan4 off
phy0-ap0 off
phy1-ap0 off
2 Likes

Sorry to bother you again. I've compiled a build with latest commits.
Flashed it and after reboot this time I have dmesg-ramoops files with current date.

<4>[   44.840940] CPU: 3 PID: 1627 Comm: napi/ath11k_ahb Tainted: G           O       6.6.52 #0
<4>[   44.863179] Hardware name: QNAP 301w (DT)
<4>[   44.871504] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
<4>[   44.875502] pc : ath11k_dp_rx_process_mon_status+0xcc4/0x14c4 [ath11k]
<4>[   44.882273] lr : ath11k_dp_rx_process_mon_status+0xcac/0x14c4 [ath11k]
<4>[   44.888868] sp : ffffffc082dc3be0
<4>[   44.895373] x29: ffffffc082dc3be0 x28: ffffffc082dc3cb0 x27: 0000000000000000
<4>[   44.898767] x26: ffffff80232f2540 x25: 0000000000000001 x24: ffffff80082bbf48
<4>[   44.905885] x23: ffffff802170a000 x22: ffffff80232f2000 x21: ffffff80232f2438
<4>[   44.913003] x20: ffffff8026121800 x19: ffffff80082b0000 x18: 0000000000000000
<4>[   44.920121] x17: 0000000000000002 x16: 0000000000000000 x15: 0000000000000000
<4>[   44.927238] x14: 0000000000000060 x13: 0000000000000140 x12: 0000000000000002
<4>[   44.934357] x11: 00000000000000c9 x10: 0000000000000000 x9 : 0000000000000002
<4>[   44.941475] x8 : 0000000000000081 x7 : 0000000000000001 x6 : 0000000000000000
<4>[   44.948593] x5 : 0000000001020091 x4 : ffffff80232f2438 x3 : ffffff80082b0000
<4>[   44.955710] x2 : 0000000000000fd3 x1 : 0000000000000000 x0 : 0000000000000001
<4>[   44.962830] Call trace:
<4>[   44.969938]  ath11k_dp_rx_process_mon_status+0xcc4/0x14c4 [ath11k]
<4>[   44.972203]  ath11k_dp_rx_process_mon_rings+0x2c/0x38 [ath11k]
<4>[   44.978452]  ath11k_dp_service_srng+0xb8/0x530 [ath11k]
<4>[   44.984266]  0xffffffc07987f288
<4>[   44.989383]  __napi_poll+0x38/0x178
<4>[   44.992508]  napi_threaded_poll_loop+0x80/0x1b0
<4>[   44.995983]  napi_threaded_poll+0x78/0x84
<4>[   45.000496]  kthread+0x10c/0x110
<4>[   45.004662]  ret_from_fork+0x10/0x20
<0>[   45.007964] Code: 91401060 39718000 b5ffe921 36001040 (f9400822) 
<4>[   45.011527] ---[ end trace 0000000000000000 ]---

<4>[   44.840940] CPU: 3 PID: 1627 Comm: napi/ath11k_ahb Tainted: G           O       6.6.52 #0
<4>[   44.863179] Hardware name: QNAP 301w (DT)
<4>[   44.871504] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
<4>[   44.875502] pc : ath11k_dp_rx_process_mon_status+0xcc4/0x14c4 [ath11k]
<4>[   44.882273] lr : ath11k_dp_rx_process_mon_status+0xcac/0x14c4 [ath11k]
<4>[   44.888868] sp : ffffffc082dc3be0
<4>[   44.895373] x29: ffffffc082dc3be0 x28: ffffffc082dc3cb0 x27: 0000000000000000
<4>[   44.898767] x26: ffffff80232f2540 x25: 0000000000000001 x24: ffffff80082bbf48
<4>[   44.905885] x23: ffffff802170a000 x22: ffffff80232f2000 x21: ffffff80232f2438
<4>[   44.913003] x20: ffffff8026121800 x19: ffffff80082b0000 x18: 0000000000000000
<4>[   44.920121] x17: 0000000000000002 x16: 0000000000000000 x15: 0000000000000000
<4>[   44.927238] x14: 0000000000000060 x13: 0000000000000140 x12: 0000000000000002
<4>[   44.934357] x11: 00000000000000c9 x10: 0000000000000000 x9 : 0000000000000002
<4>[   44.941475] x8 : 0000000000000081 x7 : 0000000000000001 x6 : 0000000000000000
<4>[   44.948593] x5 : 0000000001020091 x4 : ffffff80232f2438 x3 : ffffff80082b0000
<4>[   44.955710] x2 : 0000000000000fd3 x1 : 0000000000000000 x0 : 0000000000000001
<4>[   44.962830] Call trace:
<4>[   44.969938]  ath11k_dp_rx_process_mon_status+0xcc4/0x14c4 [ath11k]
<4>[   44.972203]  ath11k_dp_rx_process_mon_rings+0x2c/0x38 [ath11k]
<4>[   44.978452]  ath11k_dp_service_srng+0xb8/0x530 [ath11k]
<4>[   44.984266]  0xffffffc07987f288
<4>[   44.989383]  __napi_poll+0x38/0x178
<4>[   44.992508]  napi_threaded_poll_loop+0x80/0x1b0
<4>[   44.995983]  napi_threaded_poll+0x78/0x84
<4>[   45.000496]  kthread+0x10c/0x110
<4>[   45.004662]  ret_from_fork+0x10/0x20
<0>[   45.007964] Code: 91401060 39718000 b5ffe921 36001040 (f9400822) 
<4>[   45.011527] ---[ end trace 0000000000000000 ]---
<3>[   45.021831] pstore: backend (ramoops) writing error (-28)
<0>[   45.022202] Kernel panic - not syncing: Oops: Fatal exception in interrupt
<2>[   45.027500] SMP: stopping secondary CPUs
<0>[   45.034269] Kernel Offset: disabled
<0>[   45.038341] CPU features: 0x0,00000000,00000000,0000400b
<0>[   45.041558] Memory Limit: none

I wonder how could be that possible I may use the NAPI branch.
In feeds.conf.default I have NSS-12.5-K6.x. Am I right or I have to check something else?

src-git packages https://git.openwrt.org/feed/packages.git
src-git luci https://git.openwrt.org/project/luci.git
src-git routing https://git.openwrt.org/feed/routing.git
src-git telephony https://git.openwrt.org/feed/telephony.git
src-git nss_packages https://github.com/qosmio/nss-packages.git;NSS-12.5-K6.x
src-git sqm_scripts_nss https://github.com/qosmio/sqm-scripts-nss.git
#src-git video https://github.com/openwrt/video.git
#src-git targets https://github.com/openwrt/targets.git
#src-git oldpackages http://git.openwrt.org/packages.git
#src-link custom /usr/src/openwrt/custom-feed