Netgear R7800 exploration (IPQ8065, QCA9984)

What image? Any repository for it.
Running @ACwifidude master from two months ago with ondemand governor and kernel 5.10, it's over 80 days uptime.

@sppmaster with nss offload the cpu won't scale as much...(so the problem may be less frequent) but it's suspicious that with nss the system is more stable... i'm starting to believe we are missing a fundamental clock and with nss this clock is enabled...

I'm using the CXO clock that is a separate clock oscillator used for debug. (on the system you have PXO and CXO) CXO is used to hw measure clock of different devices... for some reason L2 clock can be correctly detected at 1GHz on a default fw but on our firmware it's always at something like 800mhz or something like that... this thing is not even bugged (it's not 100% precise but it does work most of the time) as you can measure also other clock and they report correct hz than the expected one...

for now the only concrete thing i notice is that we are missing tssc definition in gcc... but you guys also don't have that defined... wonder if defining nss pll cause the clk to stabilize for L2 ?

2 Likes

@hnyman I'm still messing with custom fw but I notice the situation with instability got worse so i would like to introduce 2 patch that should improve the situation.... (while i still search the cause of the wrong L2 cache)

1 Like

Sure, i can test it if you issue the PR.

1 Like

I have successfully bricked a R7800

Little bit of info:

The router originally had the stock firmware but the LAN ports were not working.

I decided to flash OpenWRT and maybe use the device as a wifi client AP

I flashed the latest OpenWRT from the stock web interface and got a boot loop.

I opened the case and connected via serial, and if I held the reset button and interrupted via serial TTY the device entered OpenWRT successfully.

I wanted to restore to the stock firmware (since OpenWRT goes into a bootloop) but since I cannot use TFTP (since the LAN ports do not work) I tried the USB debrick method with a powered USB HUB.

Upon entering u-boot command “usb start” I get this:

(IPQ) # usb start

(Re)start USB...

USB0: BUG: failure at xhci-mem.c:288/xhci_ring_alloc()!

BUG!

resetting ...

Resetting with watch dog!

U-Boot 2012.07 [local,local] (Sep 03 2015 - 17:33:28)

U-boot 2012.07 dni1 V0.4 for DNI HW ID: 29764958 NOR flash 0MB; NAND flash 128MB; RAM 512MB; 1st Radio 4x4; 2nd Radio 4x4; Cascade

smem ram ptable found: ver: 0 len: 5

DRAM: 491 MiB

NAND: SF: Unsupported manufacturer 00

ipq_spi: SPI Flash not found (bus/cs/speed/mode) = (0/0/48000000/0)

128 MiB

MMC:

*** Warning - bad CRC, using default environment

PCI0 Link Intialized

PCI1 Link Intialized

In: serial

Out: serial

Err: serial

131072 bytes read: OK

MMC Device 0 not found

cdp: get part failed for 0:HLOS

Net: MAC1 addr:dc:ef:9:ef:f0:98

Switch inits failed

Board Net Initialization Failed

No ethernet found.

Restore to factory default

Erasing at 0x7e60000 -- 100% complete. Cleanmarker written at 0x7e60000.

nand erase OK

Rebooting...

resetting ...

I could not fix the bootloop so stupidly ran a command that I read on the ddwrt forum which goes like this:

nand erase 0x3480000 0x4480000
Yeah I know I'm an idiot.

As a result now I get a kernel panic when starting OpenWRT.

No ethernet found.

Firmware recovering from TFTP server is stopped or failed! :(
Hit any key to stop autoboot:  0
No ethernet found.

 nmrp server is stopped or failed !

Loading from device 0: nand0 (offset 0x1480000)

** check kernel image **
   Verifying Checksum ... OK

** check rootfs image **
   Verifying Checksum ... OK
MMC Device 0 not found

Loading from nand0, offset 0x1480000
   Image Name:   ARM OpenWrt Linux-5.10.152
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    3097550 Bytes = 3 MiB
   Load Address: 42208000
   Entry Point:  42208000
Automatic boot of image at addr 0x44000000 ...
   Image Name:   ARM OpenWrt Linux-5.10.152
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    3097550 Bytes = 3 MiB
   Load Address: 42208000
   Entry Point:  42208000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK
## Can't malloc 15 bytes
mtdparts variable not set, see 'help mtdparts'
out of memory
info: "mtdparts" not set
Using machid 0x136c from environment

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.10.152 (perus@ub2210) (arm-openwrt-linux-muslgnueabi-gcc (OpenWrt GCC 11.2.0 r19812-5c0c01d226) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Fri Nov 11 14:55:40 2022
[    0.000000] CPU: ARMv7 Processor [512f04d0] revision 0 (ARMv7), cr=10c5787d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[    0.000000] OF: fdt: Machine model: Netgear Nighthawk X4S R7800
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000042000000-0x000000005fffffff]
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000042000000-0x000000005fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000042000000-0x000000005fffffff]
[    0.000000] percpu: Embedded 15 pages/cpu s30924 r8192 d22324 u61440
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 121800
[    0.000000] Kernel command line:
[    0.000000] Bootloader command line (ignored): console=ttyHSL1,115200n8
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 473132K/491520K available (6849K kernel code, 614K rwdata, 1668K rodata, 1024K init, 233K bss, 18388K reserved, 0K cma-reserved, 0K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] clocksource: dg_timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 305801671480 ns
[    0.000008] sched_clock: 32 bits at 6MHz, resolution 160ns, wraps every 343597383600ns
[    0.000026] Switching to timer-based delay loop, resolution 160ns
[    0.000330] Calibrating delay loop (skipped), value calculated using timer frequency.. 12.50 BogoMIPS (lpj=62500)
[    0.000362] pid_max: default: 32768 minimum: 301
[    0.000541] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.000565] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.001654] CPU: Testing write buffer coherency: ok
[    0.001976] qcom_scm: convention: smc legacy
[    0.003038] Setting up static identity map for 0x42300000 - 0x42300060
[    0.003235] rcu: Hierarchical SRCU implementation.
[    0.003556] dyndbg: Ignore empty _ddebug table in a CONFIG_DYNAMIC_DEBUG_CORE build
[    0.003834] smp: Bringing up secondary CPUs ...
[    0.005872] smp: Brought up 1 node, 2 CPUs
[    0.005893] SMP: Total of 2 processors activated (25.00 BogoMIPS).
[    0.005907] CPU: All CPU(s) started in SVC mode.
[    0.016497] VFP support v0.3: implementor 51 architecture 64 part 4d variant 2 rev 0
[    0.016672] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.016706] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.016848] pinctrl core: initialized pinctrl subsystem
[    0.018609] NET: Registered protocol family 16
[    0.018935] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.020260] thermal_sys: Registered thermal governor 'step_wise'
[    0.022477] cpuidle: using governor ladder
[    0.022554] cpuidle: using governor menu
[    0.069840] usbcore: registered new interface driver usbfs
[    0.069926] usbcore: registered new interface driver hub
[    0.069996] usbcore: registered new device driver usb
[    0.070057] pps_core: LinuxPPS API ver. 1 registered
[    0.070074] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.070111] PTP clock support registered
[    0.072344] clocksource: Switched to clocksource dg_timer
[    0.073598] NET: Registered protocol family 2
[    0.073832] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.074670] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[    0.074796] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.074844] TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.074901] TCP: Hash tables configured (established 4096 bind 4096)
[    0.075008] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.075044] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.075320] NET: Registered protocol family 1
[    0.075375] PCI: CLS 0 bytes, default 64
[    0.076866] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[    0.082507] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.082526] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.191415] qcom-pcie 1b500000.pci: supply vdda not found, using dummy regulator
[    0.191642] qcom-pcie 1b500000.pci: supply vdda_phy not found, using dummy regulator
[    0.191756] qcom-pcie 1b500000.pci: supply vdda_refclk not found, using dummy regulator
[    0.192037] qcom-pcie 1b500000.pci: host bridge /soc/pci@1b500000 ranges:
[    0.192123] qcom-pcie 1b500000.pci:       IO 0x000fe00000..0x000fe0ffff -> 0x000fe00000
[    0.192167] qcom-pcie 1b500000.pci:      MEM 0x0008000000..0x000fdfffff -> 0x0008000000
[    0.418253] qcom-pcie 1b500000.pci: Link up
[    0.418433] qcom-pcie 1b500000.pci: PCI host bridge to bus 0000:00
[    0.418461] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.418485] pci_bus 0000:00: root bus resource [io  0x0000-0xffff] (bus address [0xfe00000-0xfe0ffff])
[    0.418507] pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fdfffff]
[    0.418591] pci 0000:00:00.0: [17cb:0101] type 01 class 0x060400
[    0.418751] pci 0000:00:00.0: supports D1
[    0.418768] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    0.423598] PCI: bus0: Fast back to back transfers disabled
[    0.424122] pci 0000:01:00.0: [168c:0046] type 00 class 0x028000
[    0.424420] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
[    0.425794] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    0.430747] PCI: bus1: Fast back to back transfers disabled
[    0.430845] pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x081fffff]
[    0.430876] pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x081fffff 64bit]
[    0.431032] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    0.431060] pci 0000:00:00.0:   bridge window [mem 0x08000000-0x081fffff]
[    0.432413] pcieport 0000:00:00.0: AER: enabled with IRQ 44
[    0.433587] qcom-pcie 1b700000.pci: supply vdda not found, using dummy regulator
[    0.433842] qcom-pcie 1b700000.pci: supply vdda_phy not found, using dummy regulator
[    0.433963] qcom-pcie 1b700000.pci: supply vdda_refclk not found, using dummy regulator
[    0.434238] qcom-pcie 1b700000.pci: host bridge /soc/pci@1b700000 ranges:
[    0.434319] qcom-pcie 1b700000.pci:       IO 0x0031e00000..0x0031e0ffff -> 0x0031e00000
[    0.434361] qcom-pcie 1b700000.pci:      MEM 0x002e000000..0x0031dfffff -> 0x002e000000
[    0.659803] qcom-pcie 1b700000.pci: Link up
[    0.659960] qcom-pcie 1b700000.pci: PCI host bridge to bus 0001:00
[    0.659985] pci_bus 0001:00: root bus resource [bus 00-ff]
[    0.660007] pci_bus 0001:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x31e00000-0x31e0ffff])
[    0.660029] pci_bus 0001:00: root bus resource [mem 0x2e000000-0x31dfffff]
[    0.660110] pci 0001:00:00.0: [17cb:0101] type 01 class 0x060400
[    0.660264] pci 0001:00:00.0: supports D1
[    0.660282] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
[    0.664854] PCI: bus0: Fast back to back transfers disabled
[    0.665383] pci 0001:01:00.0: [168c:0046] type 00 class 0x028000
[    0.665696] pci 0001:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
[    0.667169] pci 0001:01:00.0: PME# supported from D0 D3hot D3cold
[    0.667629] pci 0001:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0001:00:00.0 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
[    0.672071] PCI: bus1: Fast back to back transfers disabled
[    0.672160] pci 0001:00:00.0: BAR 8: assigned [mem 0x2e000000-0x2e1fffff]
[    0.672188] pci 0001:01:00.0: BAR 0: assigned [mem 0x2e000000-0x2e1fffff 64bit]
[    0.672350] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    0.672374] pci 0001:00:00.0:   bridge window [mem 0x2e000000-0x2e1fffff]
[    0.673672] pcieport 0001:00:00.0: AER: enabled with IRQ 46
[    0.676828] L2 @ QSB rate. Forcing new rate.
[    0.677056] L2 @ 384000 KHz
[    0.677249] CPU0 @ 800000 KHz
[    0.677267] CPU1 @ QSB rate. Forcing new rate.
[    0.677398] CPU1 @ 384000 KHz
[    0.681684] gsbi 16300000.gsbi: GSBI port protocol: 6 crci: 0
[    0.684250] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
[    0.685086] msm_serial 16340000.serial: msm_serial: detected port #0
[    0.685141] msm_serial 16340000.serial: uartclk = 7372800
[    0.685223] 16340000.serial: ttyMSM0 at MMIO 0x16340000 (irq = 47, base_baud = 460800) is a MSM
[    0.685265] msm_serial: console setup on port #0
[    1.511093] printk: console [ttyMSM0] enabled
[    1.516375] msm_serial: driver initialized
[    1.525701] loop: module loaded
[    1.527375] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xa1
[    1.527661] nand: Micron MT29F1G08ABBEAH4
[    1.534362] nand: 128 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
[    1.538273] 8 fixed-partitions partitions found on MTD device qcom_nand.0
[    1.545781] OF: Bad cell count for /soc/nand-controller@1ac00000/nand@0/partitions
[    1.552539] OF: Bad cell count for /soc/nand-controller@1ac00000/nand@0/partitions
[    1.561177] Creating 8 MTD partitions on "qcom_nand.0":
[    1.567623] 0x000000000000-0x000000c80000 : "qcadata"
[    1.593972] 0x000000c80000-0x000001180000 : "APPSBL"
[    1.602977] 0x000001180000-0x000001200000 : "APPSBLENV"
[    1.604366] 0x000001200000-0x000001340000 : "art"
[    1.609671] 0x000001340000-0x000001480000 : "artbak"
[    1.614526] 0x000001480000-0x000001880000 : "kernel"
[    1.624085] 0x000001880000-0x000007900000 : "ubi"
[    1.787564] 0x000007900000-0x000008000000 : "reserve"
[    1.813513] mdio_bus 37000000.mdio-mii: MDIO device at address 0 is missing.
[    1.814350] mdio_bus 37000000.mdio-mii: MDIO device at address 4 is missing.
[    1.821296] ipq806x-gmac-dwmac 37200000.ethernet: IRQ eth_wake_irq not found
[    1.826849] ipq806x-gmac-dwmac 37200000.ethernet: IRQ eth_lpi not found
[    1.834937] ipq806x-gmac-dwmac 37200000.ethernet: PTP uses main clock
[    1.841159] ipq806x-gmac-dwmac 37200000.ethernet: User ID: 0x10, Synopsys ID: 0x37
[    1.846794] ipq806x-gmac-dwmac 37200000.ethernet:    DWMAC1000
[    1.854206] ipq806x-gmac-dwmac 37200000.ethernet: DMA HW capability register supported
[    1.859936] ipq806x-gmac-dwmac 37200000.ethernet: RX Checksum Offload Engine supported
[    1.867761] ipq806x-gmac-dwmac 37200000.ethernet: COE Type 2
[    1.875634] ipq806x-gmac-dwmac 37200000.ethernet: TX Checksum insertion supported
[    1.881374] ipq806x-gmac-dwmac 37200000.ethernet: Wake-Up On Lan supported
[    1.888868] ipq806x-gmac-dwmac 37200000.ethernet: Enhanced/Alternate descriptors
[    1.895593] ipq806x-gmac-dwmac 37200000.ethernet: Enabled extended descriptors
[    1.903155] ipq806x-gmac-dwmac 37200000.ethernet: Ring mode enabled
[    1.910105] ipq806x-gmac-dwmac 37200000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    1.919399] ipq806x-gmac-dwmac 37400000.ethernet: IRQ eth_wake_irq not found
[    1.925057] ipq806x-gmac-dwmac 37400000.ethernet: IRQ eth_lpi not found
[    1.932396] ipq806x-gmac-dwmac 37400000.ethernet: PTP uses main clock
[    1.939085] ipq806x-gmac-dwmac 37400000.ethernet: User ID: 0x10, Synopsys ID: 0x37
[    1.945070] ipq806x-gmac-dwmac 37400000.ethernet:    DWMAC1000
[    1.952379] ipq806x-gmac-dwmac 37400000.ethernet: DMA HW capability register supported
[    1.958307] ipq806x-gmac-dwmac 37400000.ethernet: RX Checksum Offload Engine supported
[    1.966010] ipq806x-gmac-dwmac 37400000.ethernet: COE Type 2
[    1.973912] ipq806x-gmac-dwmac 37400000.ethernet: TX Checksum insertion supported
[    1.979637] ipq806x-gmac-dwmac 37400000.ethernet: Wake-Up On Lan supported
[    1.987096] ipq806x-gmac-dwmac 37400000.ethernet: Enhanced/Alternate descriptors
[    1.993874] ipq806x-gmac-dwmac 37400000.ethernet: Enabled extended descriptors
[    2.001338] ipq806x-gmac-dwmac 37400000.ethernet: Ring mode enabled
[    2.008480] ipq806x-gmac-dwmac 37400000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    2.017776] i2c /dev entries driver
[    2.030273] sdhci: Secure Digital Host Controller Interface driver
[    2.030324] sdhci: Copyright(c) Pierre Ossman
[    2.035490] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.043961] NET: Registered protocol family 10
[    2.047751] Segment Routing with IPv6
[    2.049864] NET: Registered protocol family 17
[    2.053894] 8021q: 802.1Q VLAN Support v1.8
[    2.057883] Registering SWP/SWPB emulation handler
[    2.088811] qcom_rpm 108000.rpm: RPM firmware 3.0.16777364
[    2.103236] s1a: Bringing 0uV into 1050000-1050000uV
[    2.103513] s1a: supplied by regulator-dummy
[    2.107427] s1b: Bringing 0uV into 1050000-1050000uV
[    2.111733] s1b: supplied by regulator-dummy
[    2.116692] s2a: Bringing 0uV into 775000-775000uV
[    2.120941] s2a: supplied by regulator-dummy
[    2.125522] s2b: Bringing 0uV into 775000-775000uV
[    2.129977] s2b: supplied by regulator-dummy
[    2.142663] thermal thermal_zone0: failed to read out thermal zone (-110)
[    2.157061] UBI: auto-attach mtd6
[    2.157112] ubi0: attaching mtd6
[    2.302715] random: crng init done
[    2.775585] ubi0: scanning is finished
[    2.775665] ubi0 error: ubi_read_volume_table: the layout volume was not found
[    2.778547] ubi0 error: ubi_attach_mtd_dev: failed to attach mtd6, error -2ю[    2.793572] /dev/root: Can't open blockdev
[    2.793603] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[    2.796560] Please append a correct "root=" boot option; here are the available partitions:
[    2.804225] 1f00           12800 mtdblock0
[    2.804232]  (driver?)
[    2.816553] 1f01            5120 mtdblock1
[    2.816561]  (driver?)
[    2.823011] 1f02             512 mtdblock2
[    2.823018]  (driver?)
[    2.829455] 1f03            1280 mtdblock3
[    2.829462]  (driver?)
[    2.836035] 1f04            1280 mtdblock4
[    2.836042]  (driver?)
[    2.842476] 1f05            4096 mtdblock5
[    2.842484]  (driver?)
[    2.848989] 1f06           98816 mtdblock6
[    2.848997]  (driver?)
[    2.855565] 1f07            7168 mtdblock7
[    2.855573]  (driver?)
[    2.862008] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    2.864467] CPU1: stopping
[    2.872702] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.10.152 #0
[    2.875384] Hardware name: Generic DT based system
[    2.881588] [<c030e4ac>] (unwind_backtrace) from [<c030a22c>] (show_stack+0x14/0x20)
[    2.886260] [<c030a22c>] (show_stack) from [<c062f228>] (dump_stack+0x94/0xa8)
[    2.894156] [<c062f228>] (dump_stack) from [<c030d1d0>] (do_handle_IPI+0x140/0x184)
[    2.901184] [<c030d1d0>] (do_handle_IPI) from [<c030d230>] (ipi_handler+0x1c/0x2c)
[    2.908739] [<c030d230>] (ipi_handler) from [<c037178c>] (__handle_domain_irq+0x90/0xf4)
[    2.916391] [<c037178c>] (__handle_domain_irq) from [<c06485bc>] (gic_handle_irq+0x90/0xb8)
[    2.924631] [<c06485bc>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[    2.932682] Exception stack(0xc146df18 to 0xc146df60)
[    2.940327] df00:                                                       00000000 00000000
[    2.945393] df20: 1cd5a000 dd9a0d40 00000000 aaa434e0 c1cb3040 00000000 dd9a0030 00000000
[    2.953554] df40: 00000000 00000000 0004e0c0 c146df68 c07b63a4 c07b63c4 60000013 ffffffff
[    2.961706] [<c0300b8c>] (__irq_svc) from [<c07b63c4>] (cpuidle_enter_state+0x180/0x380)
[    2.969858] [<c07b63c4>] (cpuidle_enter_state) from [<c07b6614>] (cpuidle_enter+0x3c/0x5c)
[    2.978012] [<c07b6614>] (cpuidle_enter) from [<c034e6b0>] (do_idle+0x208/0x2a4)
[    2.986081] [<c034e6b0>] (do_idle) from [<c034ea08>] (cpu_startup_entry+0x1c/0x20)
[    2.993631] [<c034ea08>] (cpu_startup_entry) from [<423015ac>] (0x423015ac)
[    3.001011] Rebooting in 1 seconds..


The only thing I can do is enter uboot.

It’s a longshot but ive seen on other devices that you can send OpenWRT from a serial console directly. Like on the BT HH5.

cat lede-lantiq-bthomehubv5a_ram-u-boot.asc > /dev/ttyUSB0

And then flash OpenWRT again from a wifi connection.

So basically my question is are there other methods of flashing this router besides TFTP and USB?

Thank you for your time

A post was split to a new topic: EA8500 problems

Since your lan ports are dead, you can't tftpboot an initramfs image; however, perhaps there is a way to load an openwrt initramfs image using the serial? (Look for a loadb or loads command in uboot from the serial console.)

Unfortunately, I have never done this so I'd wait for some advice from others before attempting anything (and digging yourself in deeper). Once you have an initramfs image running, you then will have to figure out what needs repairing/re-flashing and if that can be done.

My experience using initramfs images is that they do not have wifi. Hence if the solution is to flash something, this will mean transferring the image to be flashed over the serial console.

By the time you've figured all of this out, it likely will become clear that it is time to use this device as a door stop and move on to something else. Life is short, move on.

1 Like

Ignore this post. The problem just disappeared seconds after I posted this (after an hour of trying variations on the same fix).

The original Fw needs some time to setup from a reset conditions if it's not connected to internet... Stupid scripts that needs to timeout

I've just got an unpleasent reboot during a Zoom meeting. I'm using a master NSS build from @ACwifidude and I added only samba4.

 OpenWrt SNAPSHOT, r21340+13-c34eab6fd1
 -----------------------------------------------------
root@R7800:~# uname -a
Linux R7800 5.15.79 #0 SMP Wed Nov 23 23:01:05 2022 armv7l GNU/Linux

Here are the dmesg files. What is the reason for the reboot?

<6>[   61.222661] br-lan: port 3(wlan1) entered forwarding state
<6>[  289.839172] usb 4-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
<6>[  289.886394] scsi host1: uas
<5>[  289.893928] scsi 1:0:0:0: Direct-Access     StoreJet                  0    PQ: 0 ANSI: 6
<5>[  289.895406] sd 1:0:0:0: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
<5>[  289.901498] sd 1:0:0:0: [sda] Write Protect is off
<7>[  289.908635] sd 1:0:0:0: [sda] Mode Sense: 43 00 00 00
<5>[  289.913475] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
<6>[  289.918827] sd 1:0:0:0: [sda] Optimal transfer size 33553920 bytes
<6>[  289.928838]  sda: sda1
<5>[  289.934902] sd 1:0:0:0: [sda] Attached SCSI disk
<1>[ 1575.662907] 8<--- cut here ---
<1>[ 1575.662933] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
<1>[ 1575.665005] pgd = 8a3a5825
<1>[ 1575.673162] [0000000c] *pgd=00000000
<0>[ 1575.675616] Internal error: Oops: 17 [#1] SMP ARM
<4>[ 1575.679345] Modules linked in: ksmbd asn1_decoder ecm nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath wireguard nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack mac80211 libchacha20poly1305 curve25519_neon cfg80211 ppp_async poly1305_arm nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcurve25519_generic libcrc32c crc_ccitt compat chacha_neon fuse qca_nss_qdisc qca_nss_pppoe pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev nfsv4 nfsv3 nfs nfs_ssc msdos bonding ip6_udp_tunnel udp_tunnel sit qca_nss_drv qca_nss_gmac oid_registry tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo lockd sunrpc grace hfsplus hfs cdrom dns_resolver
<4>[ 1575.679660]  nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic twofish_common tgr192 tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha512_generic sha1_generic seqiv md5 md4 echainiv ecb des_generic libdes cmac authencesn authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug vfat fat f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys trusted tpm
<4>[ 1575.822832] CPU: 1 PID: 7318 Comm: kworker/1:103 Not tainted 5.10.139 #0
<4>[ 1575.845059] Hardware name: Generic DT based system
<4>[ 1575.851870] Workqueue: ksmbd-io handle_ksmbd_work [ksmbd]
<4>[ 1575.856437] PC is at idr_remove+0x0/0x10
<4>[ 1575.861913] LR is at session_fd_check+0x3b4/0x6e8 [ksmbd]
<4>[ 1575.865892] pc : [<c0633544>]    lr : [<bf982190>]    psr: 40000013
<4>[ 1575.871187] sp : cd347e88  ip : dd047720  fp : 00000000
<4>[ 1575.877261] r10: bf9bb240  r9 : 9156cbcd  r8 : bf9bb27c
<4>[ 1575.882471] r7 : c311ff00  r6 : 00000000  r5 : c311ff00  r4 : c4f799d4
<4>[ 1575.887681] r3 : 80000000  r2 : 00000000  r1 : 00000077  r0 : 00000000
<4>[ 1575.894279] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[ 1575.900795] Control: 10c5787d  Table: 477c006a  DAC: 00000051
<0>[ 1575.907996] Process kworker/1:103 (pid: 7318, stack limit = 0xebad35c3)
<0>[ 1575.913723] Stack: (0xcd347e88 to 0xcd348000)
<0>[ 1575.920154] 7e80:                   c4b79240 00000000 ced51a04 c311ff00 bf9bb27c 9156cbcd
<0>[ 1575.924676] 7ea0: bf9bb27c bf982498 c4b79240 bf99a7b4 00000128 00000000 c4b79240 c0645d48
<0>[ 1575.932833] 7ec0: c4f7995c 00000001 00010000 dc4a7850 00000000 00000000 c4b792a4 d0bed400
<0>[ 1575.940991] 7ee0: c4b79240 c4b792a4 bf9ba8bc 00000044 00000011 bf9bb27c bf9b2df8 bf984b80
<0>[ 1575.949153] 7f00: c0c46d80 bf9b2dcc cd347f44 c4b792a4 c274ab00 dd99f980 ff7f6f00 00000000
<0>[ 1575.957313] 7f20: 00000040 00000000 cd346000 c0338940 00000008 dd99f998 c274ab00 c274ab14
<0>[ 1575.965472] 7f40: dd99f980 00000008 dd99f998 c0d03d00 dd99fb40 c0338c28 c0d9bab8 c0d0c168
<0>[ 1575.973633] 7f60: c274ab00 c26dd240 c7764380 00000000 cd346000 c0338bb4 c274ab00 cd345ec4
<0>[ 1575.981792] 7f80: c26dd264 c033eb0c 00000000 c7764380 c033e9b0 00000000 00000000 00000000
<0>[ 1575.989952] 7fa0: 00000000 00000000 00000000 c0300148 00000000 00000000 00000000 00000000
<0>[ 1575.998110] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[ 1576.006269] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[ 1576.014463] [<c0633544>] (idr_remove) from [<bf982190>] (session_fd_check+0x3b4/0x6e8 [ksmbd])
<0>[ 1576.022654] [<bf982190>] (session_fd_check [ksmbd]) from [<bf982498>] (session_fd_check+0x6bc/0x6e8 [ksmbd])
<0>[ 1576.031122] [<bf982498>] (session_fd_check [ksmbd]) from [<bf99a7b4>] (smb2_set_info+0x190/0xf14 [ksmbd])
<0>[ 1576.041110] [<bf99a7b4>] (smb2_set_info [ksmbd]) from [<bf984b80>] (handle_ksmbd_work+0x198/0x49c [ksmbd])
<0>[ 1576.050562] [<bf984b80>] (handle_ksmbd_work [ksmbd]) from [<c0338940>] (process_one_work+0x1fc/0x470)
<0>[ 1576.060086] [<c0338940>] (process_one_work) from [<c0338c28>] (worker_thread+0x74/0x5d4)
<0>[ 1576.069375] [<c0338c28>] (worker_thread) from [<c033eb0c>] (kthread+0x15c/0x160)
<0>[ 1576.077532] [<c033eb0c>] (kthread) from [<c0300148>] (ret_from_fork+0x14/0x2c)
<0>[ 1576.084905] Exception stack(0xcd347fb0 to 0xcd347ff8)
<0>[ 1576.091940] 7fa0:                                     00000000 00000000 00000000 00000000
<0>[ 1576.097067] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[ 1576.105227] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[ 1576.113385] Code: e1a00005 e58d600c ebffff75 eaffffe9 (e590300c) 
<4>[ 1576.119953] ---[ end trace 1af3b11863bca57f ]---
<6>[  289.839172] usb 4-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
<6>[  289.886394] scsi host1: uas
<5>[  289.893928] scsi 1:0:0:0: Direct-Access     StoreJet                  0    PQ: 0 ANSI: 6
<5>[  289.895406] sd 1:0:0:0: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
<5>[  289.901498] sd 1:0:0:0: [sda] Write Protect is off
<7>[  289.908635] sd 1:0:0:0: [sda] Mode Sense: 43 00 00 00
<5>[  289.913475] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
<6>[  289.918827] sd 1:0:0:0: [sda] Optimal transfer size 33553920 bytes
<6>[  289.928838]  sda: sda1
<5>[  289.934902] sd 1:0:0:0: [sda] Attached SCSI disk
<1>[ 1575.662907] 8<--- cut here ---
<1>[ 1575.662933] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
<1>[ 1575.665005] pgd = 8a3a5825
<1>[ 1575.673162] [0000000c] *pgd=00000000
<0>[ 1575.675616] Internal error: Oops: 17 [#1] SMP ARM
<4>[ 1575.679345] Modules linked in: ksmbd asn1_decoder ecm nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath wireguard nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack mac80211 libchacha20poly1305 curve25519_neon cfg80211 ppp_async poly1305_arm nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcurve25519_generic libcrc32c crc_ccitt compat chacha_neon fuse qca_nss_qdisc qca_nss_pppoe pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev nfsv4 nfsv3 nfs nfs_ssc msdos bonding ip6_udp_tunnel udp_tunnel sit qca_nss_drv qca_nss_gmac oid_registry tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo lockd sunrpc grace hfsplus hfs cdrom dns_resolver
<4>[ 1575.679660]  nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic twofish_common tgr192 tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha512_generic sha1_generic seqiv md5 md4 echainiv ecb des_generic libdes cmac authencesn authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug vfat fat f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys trusted tpm
<4>[ 1575.822832] CPU: 1 PID: 7318 Comm: kworker/1:103 Not tainted 5.10.139 #0
<4>[ 1575.845059] Hardware name: Generic DT based system
<4>[ 1575.851870] Workqueue: ksmbd-io handle_ksmbd_work [ksmbd]
<4>[ 1575.856437] PC is at idr_remove+0x0/0x10
<4>[ 1575.861913] LR is at session_fd_check+0x3b4/0x6e8 [ksmbd]
<4>[ 1575.865892] pc : [<c0633544>]    lr : [<bf982190>]    psr: 40000013
<4>[ 1575.871187] sp : cd347e88  ip : dd047720  fp : 00000000
<4>[ 1575.877261] r10: bf9bb240  r9 : 9156cbcd  r8 : bf9bb27c
<4>[ 1575.882471] r7 : c311ff00  r6 : 00000000  r5 : c311ff00  r4 : c4f799d4
<4>[ 1575.887681] r3 : 80000000  r2 : 00000000  r1 : 00000077  r0 : 00000000
<4>[ 1575.894279] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[ 1575.900795] Control: 10c5787d  Table: 477c006a  DAC: 00000051
<0>[ 1575.907996] Process kworker/1:103 (pid: 7318, stack limit = 0xebad35c3)
<0>[ 1575.913723] Stack: (0xcd347e88 to 0xcd348000)
<0>[ 1575.920154] 7e80:                   c4b79240 00000000 ced51a04 c311ff00 bf9bb27c 9156cbcd
<0>[ 1575.924676] 7ea0: bf9bb27c bf982498 c4b79240 bf99a7b4 00000128 00000000 c4b79240 c0645d48
<0>[ 1575.932833] 7ec0: c4f7995c 00000001 00010000 dc4a7850 00000000 00000000 c4b792a4 d0bed400
<0>[ 1575.940991] 7ee0: c4b79240 c4b792a4 bf9ba8bc 00000044 00000011 bf9bb27c bf9b2df8 bf984b80
<0>[ 1575.949153] 7f00: c0c46d80 bf9b2dcc cd347f44 c4b792a4 c274ab00 dd99f980 ff7f6f00 00000000
<0>[ 1575.957313] 7f20: 00000040 00000000 cd346000 c0338940 00000008 dd99f998 c274ab00 c274ab14
<0>[ 1575.965472] 7f40: dd99f980 00000008 dd99f998 c0d03d00 dd99fb40 c0338c28 c0d9bab8 c0d0c168
<0>[ 1575.973633] 7f60: c274ab00 c26dd240 c7764380 00000000 cd346000 c0338bb4 c274ab00 cd345ec4
<0>[ 1575.981792] 7f80: c26dd264 c033eb0c 00000000 c7764380 c033e9b0 00000000 00000000 00000000
<0>[ 1575.989952] 7fa0: 00000000 00000000 00000000 c0300148 00000000 00000000 00000000 00000000
<0>[ 1575.998110] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[ 1576.006269] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[ 1576.014463] [<c0633544>] (idr_remove) from [<bf982190>] (session_fd_check+0x3b4/0x6e8 [ksmbd])
<0>[ 1576.022654] [<bf982190>] (session_fd_check [ksmbd]) from [<bf982498>] (session_fd_check+0x6bc/0x6e8 [ksmbd])
<0>[ 1576.031122] [<bf982498>] (session_fd_check [ksmbd]) from [<bf99a7b4>] (smb2_set_info+0x190/0xf14 [ksmbd])
<0>[ 1576.041110] [<bf99a7b4>] (smb2_set_info [ksmbd]) from [<bf984b80>] (handle_ksmbd_work+0x198/0x49c [ksmbd])
<0>[ 1576.050562] [<bf984b80>] (handle_ksmbd_work [ksmbd]) from [<c0338940>] (process_one_work+0x1fc/0x470)
<0>[ 1576.060086] [<c0338940>] (process_one_work) from [<c0338c28>] (worker_thread+0x74/0x5d4)
<0>[ 1576.069375] [<c0338c28>] (worker_thread) from [<c033eb0c>] (kthread+0x15c/0x160)
<0>[ 1576.077532] [<c033eb0c>] (kthread) from [<c0300148>] (ret_from_fork+0x14/0x2c)
<0>[ 1576.084905] Exception stack(0xcd347fb0 to 0xcd347ff8)
<0>[ 1576.091940] 7fa0:                                     00000000 00000000 00000000 00000000
<0>[ 1576.097067] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[ 1576.105227] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[ 1576.113385] Code: e1a00005 e58d600c ebffff75 eaffffe9 (e590300c) 
<4>[ 1576.119953] ---[ end trace 1af3b11863bca57f ]---
<0>[ 1576.132592] Kernel panic - not syncing: Fatal exception
<2>[ 1576.132619] CPU0: stopping
<4>[ 1576.136612] CPU: 0 PID: 7239 Comm: ksmbd:1727 Tainted: G      D           5.10.139 #0
<4>[ 1576.139392] Hardware name: Generic DT based system
<4>[ 1576.147299] [<c030e46c>] (unwind_backtrace) from [<c030a204>] (show_stack+0x14/0x20)
<4>[ 1576.151985] [<c030a204>] (show_stack) from [<c062ef48>] (dump_stack+0x94/0xa8)
<4>[ 1576.159884] [<c062ef48>] (dump_stack) from [<c030d190>] (do_handle_IPI+0x140/0x184)
<4>[ 1576.166913] [<c030d190>] (do_handle_IPI) from [<c030d1f0>] (ipi_handler+0x1c/0x2c)
<4>[ 1576.174468] [<c030d1f0>] (ipi_handler) from [<c037174c>] (__handle_domain_irq+0x90/0xf4)
<4>[ 1576.182111] [<c037174c>] (__handle_domain_irq) from [<c06482e0>] (gic_handle_irq+0x90/0xb8)
<4>[ 1576.190355] [<c06482e0>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
<4>[ 1576.198421] Exception stack(0xcbc69e98 to 0xcbc69ee0)
<4>[ 1576.206059] 9e80:                                                       00000000 cbc68000
<4>[ 1576.211104] 9ea0: 1cd4a000 00005213 ca5305c0 dd990d80 ffffe000 ca5305c0 c0c46d80 bf9b2c3c
<4>[ 1576.219264] 9ec0: ca530954 cbc69f1c 00000000 cbc69ee8 c09c60cc c09c63f8 60000013 ffffffff
<4>[ 1576.227421] [<c0300b8c>] (__irq_svc) from [<c09c63f8>] (__schedule+0x434/0x508)
<4>[ 1576.235575] [<c09c63f8>] (__schedule) from [<c09c6534>] (schedule+0x68/0x110)
<4>[ 1576.242696] [<c09c6534>] (schedule) from [<c09ca104>] (schedule_timeout+0x74/0xd8)
<4>[ 1576.250012] [<c09ca104>] (schedule_timeout) from [<bf983bd4>] (ksmbd_conn_handler_loop+0x130/0x268 [ksmbd])
<4>[ 1576.257471] [<bf983bd4>] (ksmbd_conn_handler_loop [ksmbd]) from [<c033eb0c>] (kthread+0x15c/0x160)
<4>[ 1576.267088] [<c033eb0c>] (kthread) from [<c0300148>] (ret_from_fork+0x14/0x2c)
<4>[ 1576.276110] Exception stack(0xcbc69fb0 to 0xcbc69ff8)
<4>[ 1576.283317] 9fa0:                                     00000000 00000000 00000000 00000000
<4>[ 1576.288447] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[ 1576.296604] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000

Hard disk had a stroke and samba made the system crash

1 Like

I didn't have any device connected to the USB ports for more than a day when this crash happened.
Before it a SSD was connected for over a week and all was OK. Although it seems to me that samba4 was eating too much RAM.

Some more data from the last crash. Latest master NSS from @ACwifidude.

<1>[563194.845040] 8<--- cut here ---
<1>[563194.845078] Unable to handle kernel paging request at virtual address ca1ee28e
<1>[563194.846999] pgd = b24e1e43
<1>[563194.854285] [ca1ee28e] *pgd=4c01141e(bad)
<0>[563194.857069] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[563194.861234] Modules linked in: ecm nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_compat nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables usblp sch_cake ppp_async nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pptp pptp qca_nss_pppoe
<4>[563194.861915]  pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev nfsv4 nfsv3 nfs msdos ip_gre gre ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel tun lockd sunrpc grace hfsplus hfs cdrom dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic twofish_common tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha1_generic seqiv ecb cmac authencesn authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd qca_nss_drv qca_nss_gmac ramoops reed_solomon pstore gpio_button_hotplug vfat fat f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys
<4>[563194.932121]  trusted tpm oid_registry asn1_encoder asn1_decoder
<4>[563195.040924] CPU: 1 PID: 7146 Comm: kworker/1:1 Not tainted 5.15.88 #0
<4>[563195.046912] Hardware name: Generic DT based system
<4>[563195.053248] Workqueue: events dbs_work_handler
<4>[563195.058019] PC is at 0xca1ee28e
<4>[563195.062533] LR is at krait_mux_set_parent+0xd4/0x11c
<4>[563195.066007] pc : [<ca1ee28e>]    lr : [<c06b5a1c>]    psr: 600000b3
<4>[563195.070956] sp : c61b9d68  ip : 00000000  fp : c1f1d600
<4>[563195.077378] r10: c1c17318  r9 : 00000000  r8 : c61b9db4
<4>[563195.082673] r7 : 20000013  r6 : 00000101  r5 : 00000001  r4 : c15ba35c
<4>[563195.087971] r3 : ca1ee28f  r2 : c0d9b300  r1 : 20000093  r0 : 000346dc
<4>[563195.094308] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA Thumb  Segment none
<4>[563195.100908] Control: 10c5787d  Table: 4c35406a  DAC: 00000051
<1>[563195.108544] Register r0 information: non-paged memory
<1>[563195.114270] Register r1 information: non-paged memory
<1>[563195.119392] Register r2 information: non-slab/vmalloc memory
<1>[563195.124514] Register r3 information: non-slab/vmalloc memory
<1>[563195.130330] Register r4 information: slab kmalloc-128 start c15ba300 pointer offset 92 size 128
<1>[563195.136063] Register r5 information: non-paged memory
<1>[563195.144913] Register r6 information: non-paged memory
<1>[563195.149862] Register r7 information: non-paged memory
<1>[563195.154982] Register r8 information: non-slab/vmalloc memory
<1>[563195.160105] Register r9 information: NULL pointer
<1>[563195.165918] Register r10 information: slab kmalloc-256 start c1c17300 pointer offset 24 size 256
<1>[563195.170614] Register r11 information: slab kmalloc-128 start c1f1d600 pointer offset 0 size 128
<1>[563195.179553] Register r12 information: NULL pointer
<0>[563195.188402] Process kworker/1:1 (pid: 7146, stack limit = 0x203ad98d)
<0>[563195.193006] Stack: (0xc61b9d68 to 0xc61ba000)
<0>[563195.199605] 9d60:                   c15ba368 c0d8099c ffffffff 00000002 c61b9db4 c06b70f4
<0>[563195.204038] 9d80: 00000000 c0d8099c ffffffff c0342cbc c1c17300 c0d8099c c1665d80 00000002
<0>[563195.212284] 9da0: 23c34600 2faf0800 00000000 c06a4484 c158ba00 c1c18100 23c34600 2faf0800
<0>[563195.220530] 9dc0: c158ba00 2faf0800 c1665d80 00000000 c15af0c0 c06a8a9c c1665c68 c15af0c0
<0>[563195.228776] 9de0: c06b5f1c c140b180 23c34600 2faf0800 00000000 c06a8ad4 c1665d80 2faf0800
<0>[563195.237024] 9e00: c15af0c0 c1f1d780 dd999010 2faf0800 00000000 c06a8d0c 000f387c 2faf0800
<0>[563195.245271] 9e20: 00000000 ffffffff 2faf0800 c0d8366c c1f1d840 c1f1e5c0 2faf0800 00000000
<0>[563195.253516] 9e40: c1f1d780 dd999010 2faf0800 c06a8ec0 c1d11400 c1f1d800 00000000 c1f1d780
<0>[563195.261763] 9e60: dd999010 c07de400 c1665d80 c1f1d780 c0d05038 2faf0800 00000000 c1d11400
<0>[563195.270008] 9e80: dd999010 00000006 2faf0800 c1f1d800 00000000 000c3500 00000000 c07de810
<0>[563195.278256] 9ea0: 00000000 c0342cbc c1d11e00 2faf0800 c1d11e00 c0dd1038 00000000 00000002
<0>[563195.286503] 9ec0: 000927c0 c07e30e8 c1d11e00 000927c0 000c3500 00000024 d2445a00 c1d11e00
<0>[563195.294749] 9ee0: c1f1db80 c1f1db00 c1f1db80 c1f1e740 c1f1db00 dd99f305 c613c8c0 c07e6474
<0>[563195.302996] 9f00: c1f1dbb8 00000000 c1f1db84 c0d90974 00000000 00000040 dd99f305 c07e71fc
<0>[563195.311244] 9f20: c1f1dbb8 c613c880 dd99c040 dd99f300 00000000 c0339484 c61b8000 dd99c040
<0>[563195.319489] 9f40: c0d03d00 c613c880 dd99c040 c613c898 c0d03d00 dd99c058 c61b8000 dd99c200
<0>[563195.327735] 9f60: 00000000 c033a0dc d20b3ed8 c6da5b40 c6c14080 c033a07c c613c880 c61b8000
<0>[563195.335983] 9f80: d20b3ed8 c6c140a0 00000000 c0341374 c6da5b40 c034121c 00000000 00000000
<0>[563195.344227] 9fa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[563195.352474] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.360722] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[563195.368969] [<c06b5a1c>] (krait_mux_set_parent) from [<c06b70f4>] (krait_notifier_cb+0x58/0xb8)
<0>[563195.377212] [<c06b70f4>] (krait_notifier_cb) from [<c0342cbc>] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[563195.386156] [<c0342cbc>] (srcu_notifier_call_chain) from [<c06a4484>] (__clk_notify+0x70/0x94)
<0>[563195.395269] [<c06a4484>] (__clk_notify) from [<c06a8a9c>] (clk_change_rate+0xfc/0x2b8)
<0>[563195.403688] [<c06a8a9c>] (clk_change_rate) from [<c06a8ad4>] (clk_change_rate+0x134/0x2b8)
<0>[563195.411673] [<c06a8ad4>] (clk_change_rate) from [<c06a8d0c>] (clk_core_set_rate_nolock+0xb4/0x22c)
<0>[563195.420007] [<c06a8d0c>] (clk_core_set_rate_nolock) from [<c06a8ec0>] (clk_set_rate+0x3c/0x170)
<0>[563195.429036] [<c06a8ec0>] (clk_set_rate) from [<c07de400>] (_set_opp+0x1fc/0x524)
<0>[563195.438061] [<c07de400>] (_set_opp) from [<c07de810>] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[563195.445440] [<c07de810>] (dev_pm_opp_set_rate) from [<c07e30e8>] (__cpufreq_driver_target+0xe0/0x240)
<0>[563195.453254] [<c07e30e8>] (__cpufreq_driver_target) from [<c07e6474>] (od_dbs_update+0xcc/0x1a0)
<0>[563195.462629] [<c07e6474>] (od_dbs_update) from [<c07e71fc>] (dbs_work_handler+0x38/0x74)
<0>[563195.471570] [<c07e71fc>] (dbs_work_handler) from [<c0339484>] (process_one_work+0x234/0x4b0)
<0>[563195.479643] [<c0339484>] (process_one_work) from [<c033a0dc>] (worker_thread+0x60/0x5c4)
<0>[563195.488063] [<c033a0dc>] (worker_thread) from [<c0341374>] (kthread+0x158/0x174)
<0>[563195.496223] [<c0341374>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<0>[563195.503684] Exception stack(0xc61b9fb0 to 0xc61b9ff8)
<0>[563195.510802] 9fa0:                                     00000000 00000000 00000000 00000000
<0>[563195.516016] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.524263] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[563195.532512] Code: f8fb 3c3b 1324 2170 (8d8b) e7c1 
<4>[563195.539363] ---[ end trace 313bacd35760440f ]---
<1>[563194.845040] 8<--- cut here ---
<1>[563194.845078] Unable to handle kernel paging request at virtual address ca1ee28e
<1>[563194.846999] pgd = b24e1e43
<1>[563194.854285] [ca1ee28e] *pgd=4c01141e(bad)
<0>[563194.857069] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[563194.861234] Modules linked in: ecm nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_compat nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables usblp sch_cake ppp_async nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pptp pptp qca_nss_pppoe
<4>[563194.861915]  pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev nfsv4 nfsv3 nfs msdos ip_gre gre ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel tun lockd sunrpc grace hfsplus hfs cdrom dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic twofish_common tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha1_generic seqiv ecb cmac authencesn authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd qca_nss_drv qca_nss_gmac ramoops reed_solomon pstore gpio_button_hotplug vfat fat f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys
<4>[563194.932121]  trusted tpm oid_registry asn1_encoder asn1_decoder
<4>[563195.040924] CPU: 1 PID: 7146 Comm: kworker/1:1 Not tainted 5.15.88 #0
<4>[563195.046912] Hardware name: Generic DT based system
<4>[563195.053248] Workqueue: events dbs_work_handler
<4>[563195.058019] PC is at 0xca1ee28e
<4>[563195.062533] LR is at krait_mux_set_parent+0xd4/0x11c
<4>[563195.066007] pc : [<ca1ee28e>]    lr : [<c06b5a1c>]    psr: 600000b3
<4>[563195.070956] sp : c61b9d68  ip : 00000000  fp : c1f1d600
<4>[563195.077378] r10: c1c17318  r9 : 00000000  r8 : c61b9db4
<4>[563195.082673] r7 : 20000013  r6 : 00000101  r5 : 00000001  r4 : c15ba35c
<4>[563195.087971] r3 : ca1ee28f  r2 : c0d9b300  r1 : 20000093  r0 : 000346dc
<4>[563195.094308] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA Thumb  Segment none
<4>[563195.100908] Control: 10c5787d  Table: 4c35406a  DAC: 00000051
<1>[563195.108544] Register r0 information: non-paged memory
<1>[563195.114270] Register r1 information: non-paged memory
<1>[563195.119392] Register r2 information: non-slab/vmalloc memory
<1>[563195.124514] Register r3 information: non-slab/vmalloc memory
<1>[563195.130330] Register r4 information: slab kmalloc-128 start c15ba300 pointer offset 92 size 128
<1>[563195.136063] Register r5 information: non-paged memory
<1>[563195.144913] Register r6 information: non-paged memory
<1>[563195.149862] Register r7 information: non-paged memory
<1>[563195.154982] Register r8 information: non-slab/vmalloc memory
<1>[563195.160105] Register r9 information: NULL pointer
<1>[563195.165918] Register r10 information: slab kmalloc-256 start c1c17300 pointer offset 24 size 256
<1>[563195.170614] Register r11 information: slab kmalloc-128 start c1f1d600 pointer offset 0 size 128
<1>[563195.179553] Register r12 information: NULL pointer
<0>[563195.188402] Process kworker/1:1 (pid: 7146, stack limit = 0x203ad98d)
<0>[563195.193006] Stack: (0xc61b9d68 to 0xc61ba000)
<0>[563195.199605] 9d60:                   c15ba368 c0d8099c ffffffff 00000002 c61b9db4 c06b70f4
<0>[563195.204038] 9d80: 00000000 c0d8099c ffffffff c0342cbc c1c17300 c0d8099c c1665d80 00000002
<0>[563195.212284] 9da0: 23c34600 2faf0800 00000000 c06a4484 c158ba00 c1c18100 23c34600 2faf0800
<0>[563195.220530] 9dc0: c158ba00 2faf0800 c1665d80 00000000 c15af0c0 c06a8a9c c1665c68 c15af0c0
<0>[563195.228776] 9de0: c06b5f1c c140b180 23c34600 2faf0800 00000000 c06a8ad4 c1665d80 2faf0800
<0>[563195.237024] 9e00: c15af0c0 c1f1d780 dd999010 2faf0800 00000000 c06a8d0c 000f387c 2faf0800
<0>[563195.245271] 9e20: 00000000 ffffffff 2faf0800 c0d8366c c1f1d840 c1f1e5c0 2faf0800 00000000
<0>[563195.253516] 9e40: c1f1d780 dd999010 2faf0800 c06a8ec0 c1d11400 c1f1d800 00000000 c1f1d780
<0>[563195.261763] 9e60: dd999010 c07de400 c1665d80 c1f1d780 c0d05038 2faf0800 00000000 c1d11400
<0>[563195.270008] 9e80: dd999010 00000006 2faf0800 c1f1d800 00000000 000c3500 00000000 c07de810
<0>[563195.278256] 9ea0: 00000000 c0342cbc c1d11e00 2faf0800 c1d11e00 c0dd1038 00000000 00000002
<0>[563195.286503] 9ec0: 000927c0 c07e30e8 c1d11e00 000927c0 000c3500 00000024 d2445a00 c1d11e00
<0>[563195.294749] 9ee0: c1f1db80 c1f1db00 c1f1db80 c1f1e740 c1f1db00 dd99f305 c613c8c0 c07e6474
<0>[563195.302996] 9f00: c1f1dbb8 00000000 c1f1db84 c0d90974 00000000 00000040 dd99f305 c07e71fc
<0>[563195.311244] 9f20: c1f1dbb8 c613c880 dd99c040 dd99f300 00000000 c0339484 c61b8000 dd99c040
<0>[563195.319489] 9f40: c0d03d00 c613c880 dd99c040 c613c898 c0d03d00 dd99c058 c61b8000 dd99c200
<0>[563195.327735] 9f60: 00000000 c033a0dc d20b3ed8 c6da5b40 c6c14080 c033a07c c613c880 c61b8000
<0>[563195.335983] 9f80: d20b3ed8 c6c140a0 00000000 c0341374 c6da5b40 c034121c 00000000 00000000
<0>[563195.344227] 9fa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[563195.352474] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.360722] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[563195.368969] [<c06b5a1c>] (krait_mux_set_parent) from [<c06b70f4>] (krait_notifier_cb+0x58/0xb8)
<0>[563195.377212] [<c06b70f4>] (krait_notifier_cb) from [<c0342cbc>] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[563195.386156] [<c0342cbc>] (srcu_notifier_call_chain) from [<c06a4484>] (__clk_notify+0x70/0x94)
<0>[563195.395269] [<c06a4484>] (__clk_notify) from [<c06a8a9c>] (clk_change_rate+0xfc/0x2b8)
<0>[563195.403688] [<c06a8a9c>] (clk_change_rate) from [<c06a8ad4>] (clk_change_rate+0x134/0x2b8)
<0>[563195.411673] [<c06a8ad4>] (clk_change_rate) from [<c06a8d0c>] (clk_core_set_rate_nolock+0xb4/0x22c)
<0>[563195.420007] [<c06a8d0c>] (clk_core_set_rate_nolock) from [<c06a8ec0>] (clk_set_rate+0x3c/0x170)
<0>[563195.429036] [<c06a8ec0>] (clk_set_rate) from [<c07de400>] (_set_opp+0x1fc/0x524)
<0>[563195.438061] [<c07de400>] (_set_opp) from [<c07de810>] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[563195.445440] [<c07de810>] (dev_pm_opp_set_rate) from [<c07e30e8>] (__cpufreq_driver_target+0xe0/0x240)
<0>[563195.453254] [<c07e30e8>] (__cpufreq_driver_target) from [<c07e6474>] (od_dbs_update+0xcc/0x1a0)
<0>[563195.462629] [<c07e6474>] (od_dbs_update) from [<c07e71fc>] (dbs_work_handler+0x38/0x74)
<0>[563195.471570] [<c07e71fc>] (dbs_work_handler) from [<c0339484>] (process_one_work+0x234/0x4b0)
<0>[563195.479643] [<c0339484>] (process_one_work) from [<c033a0dc>] (worker_thread+0x60/0x5c4)
<0>[563195.488063] [<c033a0dc>] (worker_thread) from [<c0341374>] (kthread+0x158/0x174)
<0>[563195.496223] [<c0341374>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<0>[563195.503684] Exception stack(0xc61b9fb0 to 0xc61b9ff8)
<0>[563195.510802] 9fa0:                                     00000000 00000000 00000000 00000000
<0>[563195.516016] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.524263] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[563195.532512] Code: f8fb 3c3b 1324 2170 (8d8b) e7c1 
<4>[563195.539363] ---[ end trace 313bacd35760440f ]---
<0>[563195.563539] Kernel panic - not syncing: Fatal exception
<2>[563195.563579] CPU0: stopping
<4>[563195.567916] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D           5.15.88 #0
<4>[563195.570438] Hardware name: Generic DT based system
<4>[563195.578079] [<c030e560>] (unwind_backtrace) from [<c030a17c>] (show_stack+0x14/0x20)
<4>[563195.582768] [<c030a17c>] (show_stack) from [<c0649ca0>] (dump_stack_lvl+0x40/0x4c)
<4>[563195.590753] [<c0649ca0>] (dump_stack_lvl) from [<c030cfbc>] (do_handle_IPI+0x144/0x184)
<4>[563195.598217] [<c030cfbc>] (do_handle_IPI) from [<c030d018>] (ipi_handler+0x1c/0x2c)
<4>[563195.606550] [<c030d018>] (ipi_handler) from [<c037a3dc>] (handle_percpu_devid_irq+0x84/0x16c)
<4>[563195.613932] [<c037a3dc>] (handle_percpu_devid_irq) from [<c0373c6c>] (handle_domain_irq+0x68/0x94)
<4>[563195.622612] [<c0373c6c>] (handle_domain_irq) from [<c0663bac>] (gic_handle_irq+0x8c/0xb4)
<4>[563195.631552] [<c0663bac>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[563195.639883] Exception stack(0xc0d01f10 to 0xc0d01f58)
<4>[563195.647434] 1f00:                                     20f9acf8 00000000 00000001 c03147e0
<4>[563195.652560] 1f20: 00000000 c0d04f24 c0d00000 00000000 00000000 ffffe000 c0d04ec8 c0d04f58
<4>[563195.660804] 1f40: c0dc0e80 c0d01f60 c030761c c0307620 60000013 ffffffff
<4>[563195.669048] [<c0300b7c>] (__irq_svc) from [<c0307620>] (arch_cpu_idle+0x44/0x50)
<4>[563195.675908] [<c0307620>] (arch_cpu_idle) from [<c03519cc>] (do_idle+0x258/0x2b8)
<4>[563195.683286] [<c03519cc>] (do_idle) from [<c0351d50>] (cpu_startup_entry+0x1c/0x20)
<4>[563195.690751] [<c0351d50>] (cpu_startup_entry) from [<c0c0119c>] (start_kernel+0x6b8/0x6c8)

@hnyman btw in all this time how is the stability? did you notice improvement with the cache scaling disabled?

2 Likes

It has been quite stable. I have not seen a crash since we last discussed.

1 Like

I've had several crashes using NSS builds since cache scaling was disabled. I've put the data here and here2.
As I have several R7800s I can say that two other R7800s using the same NSS master build as mentioned above are pretty stable with more than 20-30 days uptime.
At the same time I left another R7800 running on older NSS master build (with kernel 5.10.100) and most probably without cache scaling disabled. It ran for more than 93 days without any crashes.

That is actually likely with the old CPU freq and old cache scaling driver, and that old cache scaling enabled. The problems materialised in the move to a new scaling driver along 5.15.

(But this is offtopic in my non-NSS build's discussion thread... Might be better to continue in the R7800 exploration... I will move this bit there.)

EDIT: moved.

1 Like

I've looked at the /sys/fs/pstore of all the R7800 routers under my control and found this.
First R7800 with currently over 17 days uptime. I've got the following.

<1>[563194.845040] 8<--- cut here ---
<1>[563194.845078] Unable to handle kernel paging request at virtual address ca1ee28e
<1>[563194.846999] pgd = b24e1e43
<1>[563194.854285] [ca1ee28e] *pgd=4c01141e(bad)
<0>[563194.857069] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[563194.861234] Modules linked in: ecm nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_compat nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables usblp sch_cake ppp_async nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pptp pptp qca_nss_pppoe
<4>[563194.861915]  pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev nfsv4 nfsv3 nfs msdos ip_gre gre ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel tun lockd sunrpc grace hfsplus hfs cdrom dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic twofish_common tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha1_generic seqiv ecb cmac authencesn authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd qca_nss_drv qca_nss_gmac ramoops reed_solomon pstore gpio_button_hotplug vfat fat f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys
<4>[563194.932121]  trusted tpm oid_registry asn1_encoder asn1_decoder
<4>[563195.040924] CPU: 1 PID: 7146 Comm: kworker/1:1 Not tainted 5.15.88 #0
<4>[563195.046912] Hardware name: Generic DT based system
<4>[563195.053248] Workqueue: events dbs_work_handler
<4>[563195.058019] PC is at 0xca1ee28e
<4>[563195.062533] LR is at krait_mux_set_parent+0xd4/0x11c
<4>[563195.066007] pc : [<ca1ee28e>]    lr : [<c06b5a1c>]    psr: 600000b3
<4>[563195.070956] sp : c61b9d68  ip : 00000000  fp : c1f1d600
<4>[563195.077378] r10: c1c17318  r9 : 00000000  r8 : c61b9db4
<4>[563195.082673] r7 : 20000013  r6 : 00000101  r5 : 00000001  r4 : c15ba35c
<4>[563195.087971] r3 : ca1ee28f  r2 : c0d9b300  r1 : 20000093  r0 : 000346dc
<4>[563195.094308] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA Thumb  Segment none
<4>[563195.100908] Control: 10c5787d  Table: 4c35406a  DAC: 00000051
<1>[563195.108544] Register r0 information: non-paged memory
<1>[563195.114270] Register r1 information: non-paged memory
<1>[563195.119392] Register r2 information: non-slab/vmalloc memory
<1>[563195.124514] Register r3 information: non-slab/vmalloc memory
<1>[563195.130330] Register r4 information: slab kmalloc-128 start c15ba300 pointer offset 92 size 128
<1>[563195.136063] Register r5 information: non-paged memory
<1>[563195.144913] Register r6 information: non-paged memory
<1>[563195.149862] Register r7 information: non-paged memory
<1>[563195.154982] Register r8 information: non-slab/vmalloc memory
<1>[563195.160105] Register r9 information: NULL pointer
<1>[563195.165918] Register r10 information: slab kmalloc-256 start c1c17300 pointer offset 24 size 256
<1>[563195.170614] Register r11 information: slab kmalloc-128 start c1f1d600 pointer offset 0 size 128
<1>[563195.179553] Register r12 information: NULL pointer
<0>[563195.188402] Process kworker/1:1 (pid: 7146, stack limit = 0x203ad98d)
<0>[563195.193006] Stack: (0xc61b9d68 to 0xc61ba000)
<0>[563195.199605] 9d60:                   c15ba368 c0d8099c ffffffff 00000002 c61b9db4 c06b70f4
<0>[563195.204038] 9d80: 00000000 c0d8099c ffffffff c0342cbc c1c17300 c0d8099c c1665d80 00000002
<0>[563195.212284] 9da0: 23c34600 2faf0800 00000000 c06a4484 c158ba00 c1c18100 23c34600 2faf0800
<0>[563195.220530] 9dc0: c158ba00 2faf0800 c1665d80 00000000 c15af0c0 c06a8a9c c1665c68 c15af0c0
<0>[563195.228776] 9de0: c06b5f1c c140b180 23c34600 2faf0800 00000000 c06a8ad4 c1665d80 2faf0800
<0>[563195.237024] 9e00: c15af0c0 c1f1d780 dd999010 2faf0800 00000000 c06a8d0c 000f387c 2faf0800
<0>[563195.245271] 9e20: 00000000 ffffffff 2faf0800 c0d8366c c1f1d840 c1f1e5c0 2faf0800 00000000
<0>[563195.253516] 9e40: c1f1d780 dd999010 2faf0800 c06a8ec0 c1d11400 c1f1d800 00000000 c1f1d780
<0>[563195.261763] 9e60: dd999010 c07de400 c1665d80 c1f1d780 c0d05038 2faf0800 00000000 c1d11400
<0>[563195.270008] 9e80: dd999010 00000006 2faf0800 c1f1d800 00000000 000c3500 00000000 c07de810
<0>[563195.278256] 9ea0: 00000000 c0342cbc c1d11e00 2faf0800 c1d11e00 c0dd1038 00000000 00000002
<0>[563195.286503] 9ec0: 000927c0 c07e30e8 c1d11e00 000927c0 000c3500 00000024 d2445a00 c1d11e00
<0>[563195.294749] 9ee0: c1f1db80 c1f1db00 c1f1db80 c1f1e740 c1f1db00 dd99f305 c613c8c0 c07e6474
<0>[563195.302996] 9f00: c1f1dbb8 00000000 c1f1db84 c0d90974 00000000 00000040 dd99f305 c07e71fc
<0>[563195.311244] 9f20: c1f1dbb8 c613c880 dd99c040 dd99f300 00000000 c0339484 c61b8000 dd99c040
<0>[563195.319489] 9f40: c0d03d00 c613c880 dd99c040 c613c898 c0d03d00 dd99c058 c61b8000 dd99c200
<0>[563195.327735] 9f60: 00000000 c033a0dc d20b3ed8 c6da5b40 c6c14080 c033a07c c613c880 c61b8000
<0>[563195.335983] 9f80: d20b3ed8 c6c140a0 00000000 c0341374 c6da5b40 c034121c 00000000 00000000
<0>[563195.344227] 9fa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[563195.352474] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.360722] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[563195.368969] [<c06b5a1c>] (krait_mux_set_parent) from [<c06b70f4>] (krait_notifier_cb+0x58/0xb8)
<0>[563195.377212] [<c06b70f4>] (krait_notifier_cb) from [<c0342cbc>] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[563195.386156] [<c0342cbc>] (srcu_notifier_call_chain) from [<c06a4484>] (__clk_notify+0x70/0x94)
<0>[563195.395269] [<c06a4484>] (__clk_notify) from [<c06a8a9c>] (clk_change_rate+0xfc/0x2b8)
<0>[563195.403688] [<c06a8a9c>] (clk_change_rate) from [<c06a8ad4>] (clk_change_rate+0x134/0x2b8)
<0>[563195.411673] [<c06a8ad4>] (clk_change_rate) from [<c06a8d0c>] (clk_core_set_rate_nolock+0xb4/0x22c)
<0>[563195.420007] [<c06a8d0c>] (clk_core_set_rate_nolock) from [<c06a8ec0>] (clk_set_rate+0x3c/0x170)
<0>[563195.429036] [<c06a8ec0>] (clk_set_rate) from [<c07de400>] (_set_opp+0x1fc/0x524)
<0>[563195.438061] [<c07de400>] (_set_opp) from [<c07de810>] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[563195.445440] [<c07de810>] (dev_pm_opp_set_rate) from [<c07e30e8>] (__cpufreq_driver_target+0xe0/0x240)
<0>[563195.453254] [<c07e30e8>] (__cpufreq_driver_target) from [<c07e6474>] (od_dbs_update+0xcc/0x1a0)
<0>[563195.462629] [<c07e6474>] (od_dbs_update) from [<c07e71fc>] (dbs_work_handler+0x38/0x74)
<0>[563195.471570] [<c07e71fc>] (dbs_work_handler) from [<c0339484>] (process_one_work+0x234/0x4b0)
<0>[563195.479643] [<c0339484>] (process_one_work) from [<c033a0dc>] (worker_thread+0x60/0x5c4)
<0>[563195.488063] [<c033a0dc>] (worker_thread) from [<c0341374>] (kthread+0x158/0x174)
<0>[563195.496223] [<c0341374>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<0>[563195.503684] Exception stack(0xc61b9fb0 to 0xc61b9ff8)
<0>[563195.510802] 9fa0:                                     00000000 00000000 00000000 00000000
<0>[563195.516016] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.524263] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[563195.532512] Code: f8fb 3c3b 1324 2170 (8d8b) e7c1 
<4>[563195.539363] ---[ end trace 313bacd35760440f ]---
<0>[563195.563539] Kernel panic - not syncing: Fatal exception
<2>[563195.563579] CPU0: stopping
<4>[563195.567916] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D           5.15.88 #0
<4>[563195.570438] Hardware name: Generic DT based system
<4>[563195.578079] [<c030e560>] (unwind_backtrace) from [<c030a17c>] (show_stack+0x14/0x20)
<4>[563195.582768] [<c030a17c>] (show_stack) from [<c0649ca0>] (dump_stack_lvl+0x40/0x4c)
<4>[563195.590753] [<c0649ca0>] (dump_stack_lvl) from [<c030cfbc>] (do_handle_IPI+0x144/0x184)
<4>[563195.598217] [<c030cfbc>] (do_handle_IPI) from [<c030d018>] (ipi_handler+0x1c/0x2c)
<4>[563195.606550] [<c030d018>] (ipi_handler) from [<c037a3dc>] (handle_percpu_devid_irq+0x84/0x16c)
<4>[563195.613932] [<c037a3dc>] (handle_percpu_devid_irq) from [<c0373c6c>] (handle_domain_irq+0x68/0x94)
<4>[563195.622612] [<c0373c6c>] (handle_domain_irq) from [<c0663bac>] (gic_handle_irq+0x8c/0xb4)
<4>[563195.631552] [<c0663bac>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[563195.639883] Exception stack(0xc0d01f10 to 0xc0d01f58)
<4>[563195.647434] 1f00:                                     20f9acf8 00000000 00000001 c03147e0
<4>[563195.652560] 1f20: 00000000 c0d04f24 c0d00000 00000000 00000000 ffffe000 c0d04ec8 c0d04f58
<4>[563195.660804] 1f40: c0dc0e80 c0d01f60 c030761c c0307620 60000013 ffffffff
<4>[563195.669048] [<c0300b7c>] (__irq_svc) from [<c0307620>] (arch_cpu_idle+0x44/0x50)
<4>[563195.675908] [<c0307620>] (arch_cpu_idle) from [<c03519cc>] (do_idle+0x258/0x2b8)
<4>[563195.683286] [<c03519cc>] (do_idle) from [<c0351d50>] (cpu_startup_entry+0x1c/0x20)
<4>[563195.690751] [<c0351d50>] (cpu_startup_entry) from [<c0c0119c>] (start_kernel+0x6b8/0x6c8)
<1>[563194.845040] 8<--- cut here ---
<1>[563194.845078] Unable to handle kernel paging request at virtual address ca1ee28e
<1>[563194.846999] pgd = b24e1e43
<1>[563194.854285] [ca1ee28e] *pgd=4c01141e(bad)
<0>[563194.857069] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[563194.861234] Modules linked in: ecm nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_compat nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables usblp sch_cake ppp_async nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pptp pptp qca_nss_pppoe
<4>[563194.861915]  pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev nfsv4 nfsv3 nfs msdos ip_gre gre ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel tun lockd sunrpc grace hfsplus hfs cdrom dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic twofish_common tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha1_generic seqiv ecb cmac authencesn authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd qca_nss_drv qca_nss_gmac ramoops reed_solomon pstore gpio_button_hotplug vfat fat f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys
<4>[563194.932121]  trusted tpm oid_registry asn1_encoder asn1_decoder
<4>[563195.040924] CPU: 1 PID: 7146 Comm: kworker/1:1 Not tainted 5.15.88 #0
<4>[563195.046912] Hardware name: Generic DT based system
<4>[563195.053248] Workqueue: events dbs_work_handler
<4>[563195.058019] PC is at 0xca1ee28e
<4>[563195.062533] LR is at krait_mux_set_parent+0xd4/0x11c
<4>[563195.066007] pc : [<ca1ee28e>]    lr : [<c06b5a1c>]    psr: 600000b3
<4>[563195.070956] sp : c61b9d68  ip : 00000000  fp : c1f1d600
<4>[563195.077378] r10: c1c17318  r9 : 00000000  r8 : c61b9db4
<4>[563195.082673] r7 : 20000013  r6 : 00000101  r5 : 00000001  r4 : c15ba35c
<4>[563195.087971] r3 : ca1ee28f  r2 : c0d9b300  r1 : 20000093  r0 : 000346dc
<4>[563195.094308] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA Thumb  Segment none
<4>[563195.100908] Control: 10c5787d  Table: 4c35406a  DAC: 00000051
<1>[563195.108544] Register r0 information: non-paged memory
<1>[563195.114270] Register r1 information: non-paged memory
<1>[563195.119392] Register r2 information: non-slab/vmalloc memory
<1>[563195.124514] Register r3 information: non-slab/vmalloc memory
<1>[563195.130330] Register r4 information: slab kmalloc-128 start c15ba300 pointer offset 92 size 128
<1>[563195.136063] Register r5 information: non-paged memory
<1>[563195.144913] Register r6 information: non-paged memory
<1>[563195.149862] Register r7 information: non-paged memory
<1>[563195.154982] Register r8 information: non-slab/vmalloc memory
<1>[563195.160105] Register r9 information: NULL pointer
<1>[563195.165918] Register r10 information: slab kmalloc-256 start c1c17300 pointer offset 24 size 256
<1>[563195.170614] Register r11 information: slab kmalloc-128 start c1f1d600 pointer offset 0 size 128
<1>[563195.179553] Register r12 information: NULL pointer
<0>[563195.188402] Process kworker/1:1 (pid: 7146, stack limit = 0x203ad98d)
<0>[563195.193006] Stack: (0xc61b9d68 to 0xc61ba000)
<0>[563195.199605] 9d60:                   c15ba368 c0d8099c ffffffff 00000002 c61b9db4 c06b70f4
<0>[563195.204038] 9d80: 00000000 c0d8099c ffffffff c0342cbc c1c17300 c0d8099c c1665d80 00000002
<0>[563195.212284] 9da0: 23c34600 2faf0800 00000000 c06a4484 c158ba00 c1c18100 23c34600 2faf0800
<0>[563195.220530] 9dc0: c158ba00 2faf0800 c1665d80 00000000 c15af0c0 c06a8a9c c1665c68 c15af0c0
<0>[563195.228776] 9de0: c06b5f1c c140b180 23c34600 2faf0800 00000000 c06a8ad4 c1665d80 2faf0800
<0>[563195.237024] 9e00: c15af0c0 c1f1d780 dd999010 2faf0800 00000000 c06a8d0c 000f387c 2faf0800
<0>[563195.245271] 9e20: 00000000 ffffffff 2faf0800 c0d8366c c1f1d840 c1f1e5c0 2faf0800 00000000
<0>[563195.253516] 9e40: c1f1d780 dd999010 2faf0800 c06a8ec0 c1d11400 c1f1d800 00000000 c1f1d780
<0>[563195.261763] 9e60: dd999010 c07de400 c1665d80 c1f1d780 c0d05038 2faf0800 00000000 c1d11400
<0>[563195.270008] 9e80: dd999010 00000006 2faf0800 c1f1d800 00000000 000c3500 00000000 c07de810
<0>[563195.278256] 9ea0: 00000000 c0342cbc c1d11e00 2faf0800 c1d11e00 c0dd1038 00000000 00000002
<0>[563195.286503] 9ec0: 000927c0 c07e30e8 c1d11e00 000927c0 000c3500 00000024 d2445a00 c1d11e00
<0>[563195.294749] 9ee0: c1f1db80 c1f1db00 c1f1db80 c1f1e740 c1f1db00 dd99f305 c613c8c0 c07e6474
<0>[563195.302996] 9f00: c1f1dbb8 00000000 c1f1db84 c0d90974 00000000 00000040 dd99f305 c07e71fc
<0>[563195.311244] 9f20: c1f1dbb8 c613c880 dd99c040 dd99f300 00000000 c0339484 c61b8000 dd99c040
<0>[563195.319489] 9f40: c0d03d00 c613c880 dd99c040 c613c898 c0d03d00 dd99c058 c61b8000 dd99c200
<0>[563195.327735] 9f60: 00000000 c033a0dc d20b3ed8 c6da5b40 c6c14080 c033a07c c613c880 c61b8000
<0>[563195.335983] 9f80: d20b3ed8 c6c140a0 00000000 c0341374 c6da5b40 c034121c 00000000 00000000
<0>[563195.344227] 9fa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[563195.352474] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.360722] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[563195.368969] [<c06b5a1c>] (krait_mux_set_parent) from [<c06b70f4>] (krait_notifier_cb+0x58/0xb8)
<0>[563195.377212] [<c06b70f4>] (krait_notifier_cb) from [<c0342cbc>] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[563195.386156] [<c0342cbc>] (srcu_notifier_call_chain) from [<c06a4484>] (__clk_notify+0x70/0x94)
<0>[563195.395269] [<c06a4484>] (__clk_notify) from [<c06a8a9c>] (clk_change_rate+0xfc/0x2b8)
<0>[563195.403688] [<c06a8a9c>] (clk_change_rate) from [<c06a8ad4>] (clk_change_rate+0x134/0x2b8)
<0>[563195.411673] [<c06a8ad4>] (clk_change_rate) from [<c06a8d0c>] (clk_core_set_rate_nolock+0xb4/0x22c)
<0>[563195.420007] [<c06a8d0c>] (clk_core_set_rate_nolock) from [<c06a8ec0>] (clk_set_rate+0x3c/0x170)
<0>[563195.429036] [<c06a8ec0>] (clk_set_rate) from [<c07de400>] (_set_opp+0x1fc/0x524)
<0>[563195.438061] [<c07de400>] (_set_opp) from [<c07de810>] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[563195.445440] [<c07de810>] (dev_pm_opp_set_rate) from [<c07e30e8>] (__cpufreq_driver_target+0xe0/0x240)
<0>[563195.453254] [<c07e30e8>] (__cpufreq_driver_target) from [<c07e6474>] (od_dbs_update+0xcc/0x1a0)
<0>[563195.462629] [<c07e6474>] (od_dbs_update) from [<c07e71fc>] (dbs_work_handler+0x38/0x74)
<0>[563195.471570] [<c07e71fc>] (dbs_work_handler) from [<c0339484>] (process_one_work+0x234/0x4b0)
<0>[563195.479643] [<c0339484>] (process_one_work) from [<c033a0dc>] (worker_thread+0x60/0x5c4)
<0>[563195.488063] [<c033a0dc>] (worker_thread) from [<c0341374>] (kthread+0x158/0x174)
<0>[563195.496223] [<c0341374>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<0>[563195.503684] Exception stack(0xc61b9fb0 to 0xc61b9ff8)
<0>[563195.510802] 9fa0:                                     00000000 00000000 00000000 00000000
<0>[563195.516016] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.524263] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[563195.532512] Code: f8fb 3c3b 1324 2170 (8d8b) e7c1 
<4>[563195.539363] ---[ end trace 313bacd35760440f ]---

Second R7800 with currently over 25 days uptime. I've got nothing in pstore
Third R7800 was updated and restarted only 3 days ago and there was nothing in pstore.
I'll monitor all R7800s I have and when I see a restart I'll check the ptsore and put the data here.
All R7800s use the NSS master build default ondemand optimized settings.

echo 600000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
echo 600000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq
echo 25 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor

As this topic was widely discussed here on the forum. Isn't there a software tool which can be used to test for this cache issue or to test for other hardware faults? Something like MemTestx86 software.

@hnyman
I follow this issue for quite some time now and I can say that there was a point before the kernel 5.10.100 that this crash issue was really symptomatic for the builds at that time and the crashes were really frequent every few days. These builds haven't had pstore enabled yet and it was impossible to catch the real reason for a restart. Later the pstore was enabled for the master builds.
At one point (I cannot say the exact build) the crashes simply stopped (or at least they were very rare).
That's why I left one R7800 on this older build just to make sure this crash doesn't repeat again.
Then the master switched to kernel 5.15 (obviously using newer cache scaling driver which was disabled by @Ansuel later) and crashes still happened from time to time (at least on my R7800s). I cannot surely say if they were rarer or more frequent with cache scaling driver disabled.
I cannot be more exact about the crashes when using performance mode because I don't use it. I didn't switch to performance mode deliberately in order to be able to test/catch if the cache scaling crash would happen.
If the crashes continue frequently I'll set performance mode on one R7800 just to be completely sure that this cache scaling crash won't happen once again.

I saw other people reported crashes lately but I hope they read this topic and can say more by themselves.

@Ansuel , @quarky , @hnyman
I'm currently testing a NSS master build compiled by me based on Master acwififdude's 15 commits + @rickkz0r 's 13 commits with sqm nss script master/Kernel5.15. After approx. 30 hours I had a crash probably connected to the NSS firmware. Maybe it's not the cache scaling issue.

Panic#1 Part1
<6>[    7.897397] (unnamed net_device) (uninitialized): nss_gmac_clk_ctl_dev_init: ctx->clk_ctl_base(0xde880000) + CLK_HALT_NSSFAB0_NSSFAB1_STATEA(0x3c20): 0x5b00
<6>[    7.910775] (unnamed net_device) (uninitialized): nss_gmac_clk_ctl_dev_init: ctx->clk_ctl_base(0xde880000) + GMAC_COREn_CLK_CTL(2)(0x3cf4): 0x50
<6>[    7.924834] (unnamed net_device) (uninitialized): nss_gmac_dev_init: nss_base(0xde960000) + NSS_GMACn_CTL(2)(0x38): 0x80c0c
<6>[    7.937759] (unnamed net_device) (uninitialized): nss_gmac_dev_init: nss_base(0xde960000) + NSS_ETH_CLK_DIV0(0xc): 0x100
<6>[    7.948600] (unnamed net_device) (uninitialized): nss_gmac_qsgmii_dev_init: QSGMII_PHY_SGMII_1_CTL(0x13c) - 0xc09c408f
<6>[    7.959721] (unnamed net_device) (uninitialized): nss_gmac_qsgmii_dev_init: NSS_QSGMII_CLK_CTL(0x2c) - 0x0
<6>[    7.970217] (unnamed net_device) (uninitialized): SGMII Specific Init for GMAC2 Done!
<6>[    7.979889] (unnamed net_device) (uninitialized): ioremap OK. Size 0x4000. reg_base 0x37400000. mac_base 0x7bd0fd7a.
<6>[    7.987817] (unnamed net_device) (uninitialized): mdio bus '37000000.mdio-mii' OK.
<6>[    8.005838] ipq8064-mdio 37000000.mdio (unnamed net_device) (uninitialized): nss_gmac_ipc_offload_init: enable Rx checksum
<6>[    8.006392] ipq8064-mdio 37000000.mdio eth1: Initialized NSS GMAC2 interface eth1: (base = 0x37400000, irq = 40, PhyId = 0, PollLink = 0)
<1>[    8.102562] nss_driver - fw of size 544712  bytes copied to load addr: 40000000, nss_id : 0
<1>[    8.103254] nss_driver - Turbo Support 1
<1>[    8.109805] Supported Frequencies - 
<1>[    8.109811] 800Mhz 
<1>[    8.113891] 800Mhz 
<1>[    8.117513] 800Mhz 
<1>[    8.119271] 
<1>[    8.123696] bf281400: meminfo init succeed
<3>[    8.125228] (unnamed net_device) (dummy): netif_napi_add() called with weight 64
<1>[    8.150868] node size 1 # items 2
<1>[    8.150897] memory: 0 0 (avl 496586752) items 2 active_cores 2
<1>[    8.153181] addr/size storage words 1 1 # words 2 in DTS, ddr size 1000000
<1>[    8.158980] bf281400: nss core 0 booted successfully
<1>[    8.186258] nss_driver - fw of size 218860  bytes copied to load addr: 40800000, nss_id : 1
<1>[    8.187487] bf285b00: meminfo init succeed
<3>[    8.193611] debugfs: Directory 'dynamic_if' with parent 'stats' already present!
<3>[    8.197754] debugfs: File 'n2h' in directory 'strings' already present!
<3>[    8.205184] debugfs: File 'drv' in directory 'strings' already present!
<1>[    8.214335] node size 1 # items 2
<6>[    8.214445] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
<1>[    8.218116] memory: 0 0 (avl 496721920) items 2 active_cores 2
<1>[    8.218179] addr/size storage words 1 1 # words 2 in DTS, ddr size 1000000
<1>[    8.218206] bf285b00: nss core 1 booted successfully
<5>[    8.245916] SCSI subsystem initialized
<7>[    8.251348] libata version 3.00 loaded.
<4>[    8.254173] ahci 29000000.sata: supply ahci not found, using dummy regulator
<4>[    8.254361] ahci 29000000.sata: supply phy not found, using dummy regulator
<4>[    8.260538] ahci 29000000.sata: supply target not found, using dummy regulator
<6>[    8.267918] ahci 29000000.sata: forcing port_map 0x0 -> 0x1
<6>[    8.274297] ahci 29000000.sata: AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl platform mode
<6>[    8.279821] ahci 29000000.sata: flags: ncq sntf pm led clo only pmp pio slum part ccc apst 
<6>[    8.289203] scsi host0: ahci
<6>[    8.296879] ata1: SATA max UDMA/133 mmio [mem 0x29000000-0x2900017f] port 0x100 irq 32
<6>[    8.628366] ata1: SATA link down (SStatus 0 SControl 300)
<6>[    8.633340] fsl-ehci: Freescale EHCI Host controller driver
<6>[    8.634933] ehci-platform: EHCI generic platform driver
<6>[    8.649025] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
<6>[    8.650575] ohci-platform: OHCI generic platform driver
<6>[    8.795032] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
<6>[    8.795113] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
<6>[    8.799747] xhci-hcd xhci-hcd.0.auto: hcc params 0x0228f065 hci version 0x100 quirks 0x0000000002010010
<6>[    8.807157] xhci-hcd xhci-hcd.0.auto: irq 55, io mem 0x10000000
<6>[    8.816721] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
<6>[    8.822268] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
<6>[    8.827892] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
<6>[    8.836733] hub 1-0:1.0: USB hub found
<6>[    8.842139] hub 1-0:1.0: 1 port detected
<6>[    8.846765] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[    8.850302] hub 2-0:1.0: USB hub found
<6>[    8.857987] hub 2-0:1.0: 1 port detected
<6>[    8.861961] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
<6>[    8.865538] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 3
<6>[    8.871027] xhci-hcd xhci-hcd.1.auto: hcc params 0x0228f065 hci version 0x100 quirks 0x0000000002010010
<6>[    8.878435] xhci-hcd xhci-hcd.1.auto: irq 56, io mem 0x11000000
<6>[    8.887988] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
<6>[    8.893616] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 4
<6>[    8.899240] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
<6>[    8.907585] hub 3-0:1.0: USB hub found
<6>[    8.913384] hub 3-0:1.0: 1 port detected
<6>[    8.917533] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[    8.921638] hub 4-0:1.0: USB hub found
<6>[    8.929682] hub 4-0:1.0: 1 port detected
<6>[    8.937574] usbcore: registered new interface driver usb-storage
<6>[    8.938844] usbcore: registered new interface driver uas
<14>[    8.943087] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
<14>[    8.956666] init: - preinit -
<6>[   12.032172] ipq8064-mdio 37000000.mdio eth1: nss_gmac_start_up: Force link up
<6>[   12.032229] ipq8064-mdio 37000000.mdio eth1: 1000 Mbps Full Duplex
<6>[   12.038398] ipq8064-mdio 37000000.mdio eth1: nss_gmac_dev_set_speed:NSS_ETH_CLK_DIV0(0xc) - 0x100
<6>[   12.044386] ipq8064-mdio 37000000.mdio eth1: nss_gmac_dev_set_speed: qsgmii_base(0xde980000) + PCS_MODE_CTL(0x68): 0x80008082
<6>[   12.053432] ipq8064-mdio 37000000.mdio eth1: nss_gmac_reset: eth1 resetting...
<6>[   13.119105] ipq8064-mdio 37000000.mdio eth1: GMAC reset completed in 106 jiffies; dma_bus_mode - 0x20100
<6>[   13.119520] ipq8064-mdio 37000000.mdio eth1: nss_gmac_change_mtu: Enabling Normal Frame MTU (Requested MTU [1500])
<6>[   13.127790] ipq8064-mdio 37000000.mdio eth1: nss_gmac_change_mtu: Enabling Jumbo Frame MTU for eth1 (Requested MTU [1500])
<6>[   13.137917] ipq8064-mdio 37000000.mdio eth1: Set addr a0:04:60:09:e4:13
<6>[   13.148942] ipq8064-mdio 37000000.mdio eth1: nss_gmac_ipc_offload_init: enable Rx checksum
<6>[   13.155377] ipq8064-mdio 37000000.mdio eth1: nss_gmac_config_flow_control:
<6>[   13.163780] ipq8064-mdio 37000000.mdio eth1: nss_gmac_rx_pause_disable: disable Rx flow control
<6>[   13.170653] ipq8064-mdio 37000000.mdio eth1: nss_gmac_tx_pause_disable: disable Tx flow control
<6>[   13.179623] ipq8064-mdio 37000000.mdio eth1: nss_gmac_linkup: data plane open command successfully issued
<6>[   13.189168] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
<6>[   13.198153] IPv6: ADDRCONF(NETDEV_CHANGE): eth1.1: link becomes ready
<12>[   18.419977] Starting lvm2 during preinit
<14>[   19.688719] mount_root: loading kmods from internal overlay
<14>[   19.728154] kmodloader: loading kernel modules from //etc/modules-boot.d/*
<14>[   19.735028] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
<5>[   19.875992] UBIFS (ubi0:1): Mounting in unauthenticated mode
<5>[   19.877176] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 219
<5>[   20.014816] UBIFS (ubi0:1): recovery needed
<5>[   20.434249] UBIFS (ubi0:1): recovery completed
<5>[   20.434384] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
<5>[   20.437672] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
<5>[   20.445499] UBIFS (ubi0:1): FS size: 82280448 bytes (78 MiB, 648 LEBs), max 659 LEBs, journal size 4063232 bytes (3 MiB, 32 LEBs)
<5>[   20.455469] UBIFS (ubi0:1): reserved for root: 3886308 bytes (3795 KiB)
<5>[   20.467091] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID 4E261E68-E706-41E9-8AC8-7459F0A0CED9, small LPT model
<14>[   20.475656] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[   20.499789] block: extroot: not configured
<5>[   20.503324] UBIFS (ubi0:1): un-mount UBI device 0
<5>[   20.503391] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" stops
<5>[   20.511729] UBIFS (ubi0:1): Mounting in unauthenticated mode
<5>[   20.516000] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 220
<5>[   20.816211] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
<5>[   20.816253] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
<5>[   20.823033] UBIFS (ubi0:1): FS size: 82280448 bytes (78 MiB, 648 LEBs), max 659 LEBs, journal size 4063232 bytes (3 MiB, 32 LEBs)
<5>[   20.833014] UBIFS (ubi0:1): reserved for root: 3886308 bytes (3795 KiB)
<5>[   20.844647] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID 4E261E68-E706-41E9-8AC8-7459F0A0CED9, small LPT model
<14>[   20.902901] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[   20.909321] block: extroot: not configured
<14>[   20.912384] mount_root: switching to ubifs overlay
<12>[   20.934250] urandom-seed: Seeding with /etc/urandom.seed
<14>[   21.157821] procd: - early -
<14>[   21.158073] procd: - watchdog -
<14>[   21.159986] procd: Watchdog has previously reset the system
<14>[   21.833802] procd: - watchdog -
<14>[   21.833948] procd: Watchdog has previously reset the system
<14>[   21.839102] procd: - ubus -
<14>[   22.010705] procd: - init -
<14>[   22.832734] urngd: v1.0.2 started.
<14>[   22.988231] kmodloader: loading kernel modules from /etc/modules.d/*
<6>[   23.001964] NET: Registered PF_ALG protocol family
<5>[   23.027706] Key type dns_resolver registered
<6>[   23.050093] RPC: Registered named UNIX socket transport module.
<6>[   23.050133] RPC: Registered udp transport module.
<6>[   23.054874] RPC: Registered tcp transport module.
<6>[   23.059770] RPC: Registered tcp NFSv4.1 backchannel transport module.
<6>[   23.090293] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
<6>[   23.094996] gre: GRE over IPv4 demultiplexor driver
<6>[   23.096273] ip_gre: GRE over IPv4 tunneling driver
<5>[   23.129239] NFS: Registering the id_resolver key type
<5>[   23.129312] Key type id_resolver registered
<5>[   23.133375] Key type id_legacy registered
<6>[   23.140010] cryptodev: driver 1.12 loaded.
<6>[   23.146661] PPP generic driver version 2.4.2
<6>[   23.147896] NET: Registered PF_PPPOX protocol family
<6>[   23.161457] GACT probability on
<6>[   23.162569] Mirror/redirect action on
<6>[   23.170179] u32 classifier
<6>[   23.170210]     input device check on
<6>[   23.171812]     Actions configured
<6>[   23.189454] fuse: init (API version 7.34)
<6>[   23.205143] Loading modules backported from Linux version v6.1-rc8-0-g76dcd734eca2
<6>[   23.205202] Backport generated by backports.git v5.15.81-1-41-g02e352527db5
<6>[   23.223489] PPTP driver version 0.8.5
<6>[   23.240859] xt_time: kernel timezone is -0000
<6>[   23.380778] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
<6>[   23.380819] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
<7>[   23.408712] ath10k_pci 0000:01:00.0: assign IRQ: got 47
<6>[   23.409395] ath10k_pci 0000:01:00.0: enabling device (0140 -> 0142)
<7>[   23.409485] ath10k_pci 0000:01:00.0: enabling bus mastering
<7>[   23.409898] qcom-pcie 1b500000.pci: msi#1 address_hi 0x0 address_lo 0x4362b498
<6>[   23.410053] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
<6>[   28.214265] ath10k_pci 0000:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
<6>[   28.214317] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
<6>[   28.225624] ath10k_pci 0000:01:00.0: firmware ver 10.4-3.9.0.2-00157 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps,peer-fixed-rate,iram-recovery crc32 6cdc6ff9
<6>[   30.504941] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 0:1 crc32 826c8c57
<6>[   34.273653] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-nvmem max-sta 512 raw 0 hwcrypto 1
<7>[   34.363944] ath: EEPROM regdomain: 0x0
<7>[   34.363965] ath: EEPROM indicates default country code should be used
<7>[   34.363977] ath: doing EEPROM country->regdmn map search
<7>[   34.363993] ath: country maps to regdmn code: 0x3a
<7>[   34.364006] ath: Country alpha2 being used: US
<7>[   34.364018] ath: Regpair used: 0x3a
<3>[   34.364130] backports-6.1-rc8/net/mac80211/main.c:1325: ieee80211_register_hw: setting default CoDeL params for phy
<6>[   34.366518] wlan0: Created a NSS virtual interface
<7>[   34.376097] ath10k_pci 0001:01:00.0: assign IRQ: got 49
<6>[   34.377278] ath10k_pci 0001:01:00.0: enabling device (0140 -> 0142)
<7>[   34.378405] ath10k_pci 0001:01:00.0: enabling bus mastering
<7>[   34.378984] qcom-pcie 1b700000.pci: msi#1 address_hi 0x0 address_lo 0x43597498
<6>[   34.379162] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
<6>[   35.658305] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
<6>[   35.658365] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
<6>[   35.669605] ath10k_pci 0001:01:00.0: firmware ver 10.4-3.9.0.2-00157 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps,peer-fixed-rate,iram-recovery crc32 6cdc6ff9
<6>[   37.946210] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:2 crc32 826c8c57
<6>[   41.724069] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-nvmem max-sta 512 raw 0 hwcrypto 1
<7>[   41.813247] ath: EEPROM regdomain: 0x0
<7>[   41.813269] ath: EEPROM indicates default country code should be used
<7>[   41.813282] ath: doing EEPROM country->regdmn map search
<7>[   41.813297] ath: country maps to regdmn code: 0x3a
<7>[   41.813311] ath: Country alpha2 being used: US
<7>[   41.813324] ath: Regpair used: 0x3a
<3>[   41.813411] backports-6.1-rc8/net/mac80211/main.c:1325: ieee80211_register_hw: setting default CoDeL params for phy
<3>[   41.815588] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   41.823173] wlan1: Created a NSS virtual interface
<14>[   41.835409] kmodloader: done loading kernel modules from /etc/modules.d/*
<6>[   46.383364] ECM init ( QSDK 12.1.r5_CS1 2022-11-25 1c401a1 )
<6>[   46.383448] ECM database jhash random seed: 0x1619c734
<6>[   46.392645] ECM init complete
<6>[   49.902852] ipq8064-mdio 37000000.mdio eth0: nss_gmac_start_up: Force link up
<6>[   49.902899] ipq8064-mdio 37000000.mdio eth0: 1000 Mbps Full Duplex
<6>[   49.909126] ipq8064-mdio 37000000.mdio eth0: nss_gmac_dev_set_speed:NSS_ETH_CLK_DIV0(0xc) - 0x100
<6>[   49.915064] ipq8064-mdio 37000000.mdio eth0: nss_gmac_reset: eth0 resetting...
<6>[   51.047126] ipq8064-mdio 37000000.mdio eth0: GMAC reset completed in 113 jiffies; dma_bus_mode - 0x20100
<6>[   51.047363] ipq8064-mdio 37000000.mdio eth0: nss_gmac_change_mtu: Enabling Normal Frame MTU (Requested MTU [1500])
<6>[   51.055720] ipq8064-mdio 37000000.mdio eth0: Set addr a0:04:60:09:e4:14
<6>[   51.065938] ipq8064-mdio 37000000.mdio eth0: nss_gmac_ipc_offload_init: enable Rx checksum
<6>[   51.072361] ipq8064-mdio 37000000.mdio eth0: nss_gmac_config_flow_control:
<6>[   51.080769] ipq8064-mdio 37000000.mdio eth0: nss_gmac_rx_pause_disable: disable Rx flow control
<6>[   51.087641] ipq8064-mdio 37000000.mdio eth0: nss_gmac_tx_pause_disable: disable Tx flow control
<6>[   51.096610] ipq8064-mdio 37000000.mdio eth0: nss_gmac_linkup: data plane open command successfully issued
<6>[   51.113083] ipq8064-mdio 37000000.mdio eth1: nss_gmac_start_up: Force link up
<6>[   51.114548] ipq8064-mdio 37000000.mdio eth1: 1000 Mbps Full Duplex
<6>[   51.121760] ipq8064-mdio 37000000.mdio eth1: nss_gmac_dev_set_speed:NSS_ETH_CLK_DIV0(0xc) - 0x100
<6>[   51.127813] ipq8064-mdio 37000000.mdio eth1: nss_gmac_dev_set_speed: qsgmii_base(0xde980000) + PCS_MODE_CTL(0x68): 0x80008082
<6>[   51.136753] ipq8064-mdio 37000000.mdio eth1: Set addr a0:04:60:09:e4:13
<6>[   51.148031] ipq8064-mdio 37000000.mdio eth1: nss_gmac_ipc_offload_init: enable Rx checksum
<6>[   51.154395] ipq8064-mdio 37000000.mdio eth1: nss_gmac_config_flow_control:
<6>[   51.162802] ipq8064-mdio 37000000.mdio eth1: nss_gmac_rx_pause_disable: disable Rx flow control
<6>[   51.169663] ipq8064-mdio 37000000.mdio eth1: nss_gmac_tx_pause_disable: disable Tx flow control
<6>[   51.184251] ipq8064-mdio 37000000.mdio eth1: nss_gmac_linkup: data plane open command successfully issued
<6>[   51.196640] br-lan: port 1(eth1.1) entered blocking state
<6>[   51.196687] br-lan: port 1(eth1.1) entered disabled state
<6>[   51.202203] device eth1.1 entered promiscuous mode
<6>[   51.207440] device eth1 entered promiscuous mode
<6>[   51.222914] br-lan: port 1(eth1.1) entered blocking state
<6>[   51.222962] br-lan: port 1(eth1.1) entered forwarding state
<6>[   51.241200] br-vlan: port 1(eth1.3) entered blocking state
<6>[   51.241248] br-vlan: port 1(eth1.3) entered disabled state
<6>[   51.246435] device eth1.3 entered promiscuous mode
<6>[   51.262647] br-vlan: port 1(eth1.3) entered blocking state
<6>[   51.262697] br-vlan: port 1(eth1.3) entered forwarding state
<6>[   51.826917] wlan0: Destroyed NSS virtual interface
<6>[   51.956828] wlan1: Destroyed NSS virtual interface
<3>[   52.135444] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   52.135735] Created a NSS virtual interface for dev [nssifb]
<6>[   52.141315] NSS IFB data callback registered
<6>[   52.147015] NSS IFB transmit callback registered
<6>[   52.151782] NSS IFB module loaded.
<7>[   52.348604] ath: EEPROM regdomain: 0x824f
<7>[   52.348645] ath: EEPROM indicates we should expect a country code
<7>[   52.351605] ath: doing EEPROM country->regdmn map search
<7>[   52.357809] ath: country maps to regdmn code: 0x10
<7>[   52.363063] ath: Country alpha2 being used: PA
<7>[   52.367756] ath: Regpair used: 0x10
<7>[   52.372086] ath: regdomain 0x824f dynamically updated by user
<7>[   52.375501] ath: EEPROM regdomain: 0x824f
<7>[   52.381472] ath: EEPROM indicates we should expect a country code
<7>[   52.385374] ath: doing EEPROM country->regdmn map search
<7>[   52.391534] ath: country maps to regdmn code: 0x10
<7>[   52.396904] ath: Country alpha2 being used: PA
<7>[   52.401428] ath: Regpair used: 0x10
<7>[   52.405854] ath: regdomain 0x824f dynamically updated by user
<4>[   52.471809] __nss_qdisc_init[2175]:parent (65536) and TC_H_ROOT (-1))
<4>[   52.471835] __nss_qdisc_init[2176]:root->ops->owner (00000000) and THIS_MODULE (bf63f600))
<4>[   52.471886] __nss_qdisc_init[2177]:NSS qdisc c59b1c00 (type 1) used along with non-nss qdiscs, or the interface is currently down
<6>[   52.487534] 6ad4ab96: Found net device [eth0]
<6>[   52.498276] 6ad4ab96: Net device [eth0] has NSS intf_num [1]
<6>[   52.502578] Nexthop successfully set for [eth0] to [nssifb]
<3>[   54.336161] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   54.336435] phy1-ap0: Created a NSS virtual interface
<4>[   60.823099] ath10k_pci 0001:01:00.0: Unknown eventid: 36933
<3>[   60.826632] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   60.827775] phy0-ap0: Created a NSS virtual interface
<6>[   60.841083] br-lan: port 2(phy1-ap0) entered blocking state
<6>[   60.841128] br-lan: port 2(phy1-ap0) entered disabled state
<6>[   60.845757] device phy1-ap0 entered promiscuous mode
<6>[   60.851237] br-lan: port 2(phy1-ap0) entered blocking state
<6>[   60.856293] br-lan: port 2(phy1-ap0) entered forwarding state
<6>[   60.876178] br-lan: port 2(phy1-ap0) entered disabled state
<6>[   61.406414] br-lan: port 2(phy1-ap0) entered blocking state
<6>[   61.406467] br-lan: port 2(phy1-ap0) entered forwarding state
<6>[   61.626012] 6ad4ab96: Reset nexthop successful for net device [eth0].
<6>[   61.634010] NSS IFB transmit callback unregistered
<6>[   61.636672] NSS virtual interface destroyed for dev [nssifb]
<4>[   68.677653] ath10k_pci 0000:01:00.0: Unknown eventid: 36933
<6>[   68.680237] br-lan: port 3(phy0-ap0) entered blocking state
<6>[   68.682048] br-lan: port 3(phy0-ap0) entered disabled state
<6>[   68.687923] device phy0-ap0 entered promiscuous mode
<6>[   68.693265] br-lan: port 3(phy0-ap0) entered blocking state
<6>[   68.698441] br-lan: port 3(phy0-ap0) entered forwarding state
<6>[   68.705639] br-lan: port 3(phy0-ap0) entered disabled state
<3>[   68.818583] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   68.818906] Created a NSS virtual interface for dev [nssifb]
<6>[   68.824398] NSS IFB data callback registered
<6>[   68.830246] NSS IFB transmit callback registered
<4>[   68.970055] __nss_qdisc_init[2175]:parent (65536) and TC_H_ROOT (-1))
<4>[   68.970074] __nss_qdisc_init[2176]:root->ops->owner (00000000) and THIS_MODULE (bf63f600))
<4>[   68.970113] __nss_qdisc_init[2177]:NSS qdisc c96fe400 (type 1) used along with non-nss qdiscs, or the interface is currently down
<6>[   68.981607] 6ad4ab96: Found net device [eth0]
<6>[   68.995736] 6ad4ab96: Net device [eth0] has NSS intf_num [1]
<6>[   69.000140] Nexthop successfully set for [eth0] to [nssifb]
<6>[   69.074210] br-lan: port 3(phy0-ap0) entered blocking state
<6>[   69.074265] br-lan: port 3(phy0-ap0) entered forwarding state
<1>[ 2283.723608] NSS core 0 signal COREDUMP COMPLETE 4000
<1>[ 2283.723667] bf281400: Starting NSS-FW logbuffer dump for core 0
<1>[ 2283.727672] bf281400: Warn: trap[813]: Trap on CHIP ID 00050000
<1>[ 2283.733389] bf281400: Warn: trap[620]: Trapped: TRAP_TD(00000004) DCAPT(3C000080)
<1>[ 2283.739206] bf281400: Warn: trap[645]: Trapped: Thread: 2, reason: 00000020, PC: 4002FBF4, previous PC: 4002FBF0
<1>[ 2283.746847] bf281400: Warn: trap[594]: A0_3: 4F71A294 402321C0 3F019940 4F71A296
<1>[ 2283.757086] bf281400: Warn: trap[594]: A4_7: 4F71A296 40053D04 3F019940 3F00AEF0
<1>[ 2283.764474] bf281400: Warn: trap[599]: D0_3: 00000026 00000001 00000009 4F71A284
<1>[ 2283.771843] bf281400: Warn: trap[599]: D4_7: 00060000 00000026 00000024 000005DC
<1>[ 2283.779222] bf281400: Warn: trap[599]: D8_11: 00000024 C0A80104 4CF9E534 00000000
<1>[ 2283.786600] bf281400: Warn: trap[599]: D12_15: 00000000 00000000 00D84001 00004002
<1>[ 2283.793980] bf281400: Warn: trap[649]: Thread_2 has non-recoverable trap
<3>[ 2283.801683] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[ 2283.804457] NSS core 1 signal COREDUMP COMPLETE 4000
<3>[ 2283.808392] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[ 2283.815063] bf285b00: Starting NSS-FW logbuffer dump for core 1
<3>[ 2283.819935] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<0>[ 2283.826679] Kernel panic - not syncing: NSS FW coredump: bringing system down
<3>[ 2283.832268] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<2>[ 2283.839267] CPU0: stopping
<4>[ 2283.839277] CPU: 0 PID: 12 Comm: ksoftirqd/0 Not tainted 5.15.98 #0
<4>[ 2283.839287] Hardware name: Generic DT based system
<4>[ 2283.839297] [<c030e7e8>] (unwind_backtrace) from [<c030a194>] (show_stack+0x14/0x20)
<4>[ 2283.839320] [<c030a194>] (show_stack) from [<c064cee0>] (dump_stack_lvl+0x40/0x4c)
<4>[ 2283.839339] [<c064cee0>] (dump_stack_lvl) from [<c030cfbc>] (do_handle_IPI+0x144/0x184)
<4>[ 2283.839354] [<c030cfbc>] (do_handle_IPI) from [<c030d018>] (ipi_handler+0x1c/0x2c)
<4>[ 2283.839364] [<c030d018>] (ipi_handler) from [<c037d3cc>] (handle_percpu_devid_irq+0x84/0x16c)
<4>[ 2283.839379] [<c037d3cc>] (handle_percpu_devid_irq) from [<c0376c5c>] (handle_domain_irq+0x68/0x94)
<4>[ 2283.839397] [<c0376c5c>] (handle_domain_irq) from [<c0666dec>] (gic_handle_irq+0x8c/0xb4)
<4>[ 2283.839414] [<c0666dec>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[ 2283.839429] Exception stack(0xc146bb00 to 0xc146bb48)
<4>[ 2283.839438] bb00: 60000093 1cd49000 00000000 c0c442fc 0000003c 00000000 60000013 00000000
<4>[ 2283.839445] bb20: bf8365b0 c83dad00 00000000 c146bd10 00000000 c146bb50 c0374678 c037467c
<4>[ 2283.839450] bb40: 60000013 ffffffff
<4>[ 2283.839455] [<c0300b7c>] (__irq_svc) from [<c037467c>] (vprintk_emit+0x1d0/0x1f8)
<4>[ 2283.839471] [<c037467c>] (vprintk_emit) from [<c03746c8>] (vprintk_default+0x24/0x2c)
<4>[ 2283.839482] [<c03746c8>] (vprintk_default) from [<c03710b8>] (_printk+0x1c/0x2c)
<4>[ 2283.839496] [<c03710b8>] (_printk) from [<bf7e4054>] (ieee80211_set_ringparam+0x330/0xd50 [mac80211])
<4>[ 2283.839748] [<bf7e4054>] (ieee80211_set_ringparam [mac80211]) from [<bf7e5b3c>] (ieee80211_sta_ps_transition+0x9a0/0x17fc [mac80211])
<4>[ 2283.839990] [<bf7e5b3c>] (ieee80211_sta_ps_transition [mac80211]) from [<bf7e785c>] (ieee80211_is_our_addr+0xe1c/0x2ad4 [mac80211])
<4>[ 2283.840227] [<bf7e785c>] (ieee80211_is_our_addr [mac80211]) from [<bf7e9d40>] (ieee80211_mark_rx_ba_filtered_frames+0x82c/0x1520 [mac80211])
<4>[ 2283.840465] [<bf7e9d40>] (ieee80211_mark_rx_ba_filtered_frames [mac80211]) from [<bf7eb880>] (ieee80211_rx_list+0xe4c/0xe94 [mac80211])
<4>[ 2283.840702] [<bf7eb880>] (ieee80211_rx_list [mac80211]) from [<bf7eb8f0>] (ieee80211_rx_napi+0x28/0x90 [mac80211])
<4>[ 2283.840938] [<bf7eb8f0>] (ieee80211_rx_napi [mac80211]) from [<bf99c000>] (ath10k_htt_rx_pktlog_completion_handler+0xfa0/0x1428 [ath10k_core])
<4>[ 2283.841189] [<bf99c000>] (ath10k_htt_rx_pktlog_completion_handler [ath10k_core]) from [<bf9a0b8c>] (ath10k_htt_txrx_compl_task+0xa28/0x1240 [ath10k_core])
<4>[ 2283.841333] [<bf9a0b8c>] (ath10k_htt_txrx_compl_task [ath10k_core]) from [<bf9f0a54>] (ath10k_pci_napi_poll+0x5c/0x134 [ath10k_pci])
<4>[ 2283.841444] [<bf9f0a54>] (ath10k_pci_napi_poll [ath10k_pci]) from [<c085df2c>] (__napi_poll+0x5c/0x1d0)
<4>[ 2283.841472] [<c085df2c>] (__napi_poll) from [<c085e298>] (net_rx_action+0xe4/0x268)
<4>[ 2283.841485] [<c085e298>] (net_rx_action) from [<c0301304>] (__do_softirq+0x11c/0x2e4)
<4>[ 2283.841500] [<c0301304>] (__do_softirq) from [<c0325d98>] (run_ksoftirqd+0x38/0x4c)
<4>[ 2283.841516] [<c0325d98>] (run_ksoftirqd) from [<c034824c>] (smpboot_thread_fn+0xe8/0x1ac)
<4>[ 2283.841535] [<c034824c>] (smpboot_thread_fn) from [<c0343b50>] (kthread+0x158/0x174)
<4>[ 2283.841554] [<c0343b50>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<4>[ 2283.841565] Exception stack(0xc146bfb0 to 0xc146bff8)
<4>[ 2283.841573] bfa0:                                     00000000 00000000 00000000 00000000
<4>[ 2283.841579] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[ 2283.841585] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000