Netgear R7800 exploration (IPQ8065, QCA9984)

It has been quite stable. I have not seen a crash since we last discussed.

1 Like

I've had several crashes using NSS builds since cache scaling was disabled. I've put the data here and here2.
As I have several R7800s I can say that two other R7800s using the same NSS master build as mentioned above are pretty stable with more than 20-30 days uptime.
At the same time I left another R7800 running on older NSS master build (with kernel 5.10.100) and most probably without cache scaling disabled. It ran for more than 93 days without any crashes.

That is actually likely with the old CPU freq and old cache scaling driver, and that old cache scaling enabled. The problems materialised in the move to a new scaling driver along 5.15.

(But this is offtopic in my non-NSS build's discussion thread... Might be better to continue in the R7800 exploration... I will move this bit there.)

EDIT: moved.

1 Like

I've looked at the /sys/fs/pstore of all the R7800 routers under my control and found this.
First R7800 with currently over 17 days uptime. I've got the following.

<1>[563194.845040] 8<--- cut here ---
<1>[563194.845078] Unable to handle kernel paging request at virtual address ca1ee28e
<1>[563194.846999] pgd = b24e1e43
<1>[563194.854285] [ca1ee28e] *pgd=4c01141e(bad)
<0>[563194.857069] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[563194.861234] Modules linked in: ecm nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_compat nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables usblp sch_cake ppp_async nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pptp pptp qca_nss_pppoe
<4>[563194.861915]  pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev nfsv4 nfsv3 nfs msdos ip_gre gre ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel tun lockd sunrpc grace hfsplus hfs cdrom dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic twofish_common tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha1_generic seqiv ecb cmac authencesn authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd qca_nss_drv qca_nss_gmac ramoops reed_solomon pstore gpio_button_hotplug vfat fat f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys
<4>[563194.932121]  trusted tpm oid_registry asn1_encoder asn1_decoder
<4>[563195.040924] CPU: 1 PID: 7146 Comm: kworker/1:1 Not tainted 5.15.88 #0
<4>[563195.046912] Hardware name: Generic DT based system
<4>[563195.053248] Workqueue: events dbs_work_handler
<4>[563195.058019] PC is at 0xca1ee28e
<4>[563195.062533] LR is at krait_mux_set_parent+0xd4/0x11c
<4>[563195.066007] pc : [<ca1ee28e>]    lr : [<c06b5a1c>]    psr: 600000b3
<4>[563195.070956] sp : c61b9d68  ip : 00000000  fp : c1f1d600
<4>[563195.077378] r10: c1c17318  r9 : 00000000  r8 : c61b9db4
<4>[563195.082673] r7 : 20000013  r6 : 00000101  r5 : 00000001  r4 : c15ba35c
<4>[563195.087971] r3 : ca1ee28f  r2 : c0d9b300  r1 : 20000093  r0 : 000346dc
<4>[563195.094308] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA Thumb  Segment none
<4>[563195.100908] Control: 10c5787d  Table: 4c35406a  DAC: 00000051
<1>[563195.108544] Register r0 information: non-paged memory
<1>[563195.114270] Register r1 information: non-paged memory
<1>[563195.119392] Register r2 information: non-slab/vmalloc memory
<1>[563195.124514] Register r3 information: non-slab/vmalloc memory
<1>[563195.130330] Register r4 information: slab kmalloc-128 start c15ba300 pointer offset 92 size 128
<1>[563195.136063] Register r5 information: non-paged memory
<1>[563195.144913] Register r6 information: non-paged memory
<1>[563195.149862] Register r7 information: non-paged memory
<1>[563195.154982] Register r8 information: non-slab/vmalloc memory
<1>[563195.160105] Register r9 information: NULL pointer
<1>[563195.165918] Register r10 information: slab kmalloc-256 start c1c17300 pointer offset 24 size 256
<1>[563195.170614] Register r11 information: slab kmalloc-128 start c1f1d600 pointer offset 0 size 128
<1>[563195.179553] Register r12 information: NULL pointer
<0>[563195.188402] Process kworker/1:1 (pid: 7146, stack limit = 0x203ad98d)
<0>[563195.193006] Stack: (0xc61b9d68 to 0xc61ba000)
<0>[563195.199605] 9d60:                   c15ba368 c0d8099c ffffffff 00000002 c61b9db4 c06b70f4
<0>[563195.204038] 9d80: 00000000 c0d8099c ffffffff c0342cbc c1c17300 c0d8099c c1665d80 00000002
<0>[563195.212284] 9da0: 23c34600 2faf0800 00000000 c06a4484 c158ba00 c1c18100 23c34600 2faf0800
<0>[563195.220530] 9dc0: c158ba00 2faf0800 c1665d80 00000000 c15af0c0 c06a8a9c c1665c68 c15af0c0
<0>[563195.228776] 9de0: c06b5f1c c140b180 23c34600 2faf0800 00000000 c06a8ad4 c1665d80 2faf0800
<0>[563195.237024] 9e00: c15af0c0 c1f1d780 dd999010 2faf0800 00000000 c06a8d0c 000f387c 2faf0800
<0>[563195.245271] 9e20: 00000000 ffffffff 2faf0800 c0d8366c c1f1d840 c1f1e5c0 2faf0800 00000000
<0>[563195.253516] 9e40: c1f1d780 dd999010 2faf0800 c06a8ec0 c1d11400 c1f1d800 00000000 c1f1d780
<0>[563195.261763] 9e60: dd999010 c07de400 c1665d80 c1f1d780 c0d05038 2faf0800 00000000 c1d11400
<0>[563195.270008] 9e80: dd999010 00000006 2faf0800 c1f1d800 00000000 000c3500 00000000 c07de810
<0>[563195.278256] 9ea0: 00000000 c0342cbc c1d11e00 2faf0800 c1d11e00 c0dd1038 00000000 00000002
<0>[563195.286503] 9ec0: 000927c0 c07e30e8 c1d11e00 000927c0 000c3500 00000024 d2445a00 c1d11e00
<0>[563195.294749] 9ee0: c1f1db80 c1f1db00 c1f1db80 c1f1e740 c1f1db00 dd99f305 c613c8c0 c07e6474
<0>[563195.302996] 9f00: c1f1dbb8 00000000 c1f1db84 c0d90974 00000000 00000040 dd99f305 c07e71fc
<0>[563195.311244] 9f20: c1f1dbb8 c613c880 dd99c040 dd99f300 00000000 c0339484 c61b8000 dd99c040
<0>[563195.319489] 9f40: c0d03d00 c613c880 dd99c040 c613c898 c0d03d00 dd99c058 c61b8000 dd99c200
<0>[563195.327735] 9f60: 00000000 c033a0dc d20b3ed8 c6da5b40 c6c14080 c033a07c c613c880 c61b8000
<0>[563195.335983] 9f80: d20b3ed8 c6c140a0 00000000 c0341374 c6da5b40 c034121c 00000000 00000000
<0>[563195.344227] 9fa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[563195.352474] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.360722] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[563195.368969] [<c06b5a1c>] (krait_mux_set_parent) from [<c06b70f4>] (krait_notifier_cb+0x58/0xb8)
<0>[563195.377212] [<c06b70f4>] (krait_notifier_cb) from [<c0342cbc>] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[563195.386156] [<c0342cbc>] (srcu_notifier_call_chain) from [<c06a4484>] (__clk_notify+0x70/0x94)
<0>[563195.395269] [<c06a4484>] (__clk_notify) from [<c06a8a9c>] (clk_change_rate+0xfc/0x2b8)
<0>[563195.403688] [<c06a8a9c>] (clk_change_rate) from [<c06a8ad4>] (clk_change_rate+0x134/0x2b8)
<0>[563195.411673] [<c06a8ad4>] (clk_change_rate) from [<c06a8d0c>] (clk_core_set_rate_nolock+0xb4/0x22c)
<0>[563195.420007] [<c06a8d0c>] (clk_core_set_rate_nolock) from [<c06a8ec0>] (clk_set_rate+0x3c/0x170)
<0>[563195.429036] [<c06a8ec0>] (clk_set_rate) from [<c07de400>] (_set_opp+0x1fc/0x524)
<0>[563195.438061] [<c07de400>] (_set_opp) from [<c07de810>] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[563195.445440] [<c07de810>] (dev_pm_opp_set_rate) from [<c07e30e8>] (__cpufreq_driver_target+0xe0/0x240)
<0>[563195.453254] [<c07e30e8>] (__cpufreq_driver_target) from [<c07e6474>] (od_dbs_update+0xcc/0x1a0)
<0>[563195.462629] [<c07e6474>] (od_dbs_update) from [<c07e71fc>] (dbs_work_handler+0x38/0x74)
<0>[563195.471570] [<c07e71fc>] (dbs_work_handler) from [<c0339484>] (process_one_work+0x234/0x4b0)
<0>[563195.479643] [<c0339484>] (process_one_work) from [<c033a0dc>] (worker_thread+0x60/0x5c4)
<0>[563195.488063] [<c033a0dc>] (worker_thread) from [<c0341374>] (kthread+0x158/0x174)
<0>[563195.496223] [<c0341374>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<0>[563195.503684] Exception stack(0xc61b9fb0 to 0xc61b9ff8)
<0>[563195.510802] 9fa0:                                     00000000 00000000 00000000 00000000
<0>[563195.516016] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.524263] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[563195.532512] Code: f8fb 3c3b 1324 2170 (8d8b) e7c1 
<4>[563195.539363] ---[ end trace 313bacd35760440f ]---
<0>[563195.563539] Kernel panic - not syncing: Fatal exception
<2>[563195.563579] CPU0: stopping
<4>[563195.567916] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D           5.15.88 #0
<4>[563195.570438] Hardware name: Generic DT based system
<4>[563195.578079] [<c030e560>] (unwind_backtrace) from [<c030a17c>] (show_stack+0x14/0x20)
<4>[563195.582768] [<c030a17c>] (show_stack) from [<c0649ca0>] (dump_stack_lvl+0x40/0x4c)
<4>[563195.590753] [<c0649ca0>] (dump_stack_lvl) from [<c030cfbc>] (do_handle_IPI+0x144/0x184)
<4>[563195.598217] [<c030cfbc>] (do_handle_IPI) from [<c030d018>] (ipi_handler+0x1c/0x2c)
<4>[563195.606550] [<c030d018>] (ipi_handler) from [<c037a3dc>] (handle_percpu_devid_irq+0x84/0x16c)
<4>[563195.613932] [<c037a3dc>] (handle_percpu_devid_irq) from [<c0373c6c>] (handle_domain_irq+0x68/0x94)
<4>[563195.622612] [<c0373c6c>] (handle_domain_irq) from [<c0663bac>] (gic_handle_irq+0x8c/0xb4)
<4>[563195.631552] [<c0663bac>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[563195.639883] Exception stack(0xc0d01f10 to 0xc0d01f58)
<4>[563195.647434] 1f00:                                     20f9acf8 00000000 00000001 c03147e0
<4>[563195.652560] 1f20: 00000000 c0d04f24 c0d00000 00000000 00000000 ffffe000 c0d04ec8 c0d04f58
<4>[563195.660804] 1f40: c0dc0e80 c0d01f60 c030761c c0307620 60000013 ffffffff
<4>[563195.669048] [<c0300b7c>] (__irq_svc) from [<c0307620>] (arch_cpu_idle+0x44/0x50)
<4>[563195.675908] [<c0307620>] (arch_cpu_idle) from [<c03519cc>] (do_idle+0x258/0x2b8)
<4>[563195.683286] [<c03519cc>] (do_idle) from [<c0351d50>] (cpu_startup_entry+0x1c/0x20)
<4>[563195.690751] [<c0351d50>] (cpu_startup_entry) from [<c0c0119c>] (start_kernel+0x6b8/0x6c8)
<1>[563194.845040] 8<--- cut here ---
<1>[563194.845078] Unable to handle kernel paging request at virtual address ca1ee28e
<1>[563194.846999] pgd = b24e1e43
<1>[563194.854285] [ca1ee28e] *pgd=4c01141e(bad)
<0>[563194.857069] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[563194.861234] Modules linked in: ecm nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_compat nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables usblp sch_cake ppp_async nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pptp pptp qca_nss_pppoe
<4>[563194.861915]  pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev nfsv4 nfsv3 nfs msdos ip_gre gre ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel tun lockd sunrpc grace hfsplus hfs cdrom dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic twofish_common tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha1_generic seqiv ecb cmac authencesn authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd qca_nss_drv qca_nss_gmac ramoops reed_solomon pstore gpio_button_hotplug vfat fat f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys
<4>[563194.932121]  trusted tpm oid_registry asn1_encoder asn1_decoder
<4>[563195.040924] CPU: 1 PID: 7146 Comm: kworker/1:1 Not tainted 5.15.88 #0
<4>[563195.046912] Hardware name: Generic DT based system
<4>[563195.053248] Workqueue: events dbs_work_handler
<4>[563195.058019] PC is at 0xca1ee28e
<4>[563195.062533] LR is at krait_mux_set_parent+0xd4/0x11c
<4>[563195.066007] pc : [<ca1ee28e>]    lr : [<c06b5a1c>]    psr: 600000b3
<4>[563195.070956] sp : c61b9d68  ip : 00000000  fp : c1f1d600
<4>[563195.077378] r10: c1c17318  r9 : 00000000  r8 : c61b9db4
<4>[563195.082673] r7 : 20000013  r6 : 00000101  r5 : 00000001  r4 : c15ba35c
<4>[563195.087971] r3 : ca1ee28f  r2 : c0d9b300  r1 : 20000093  r0 : 000346dc
<4>[563195.094308] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA Thumb  Segment none
<4>[563195.100908] Control: 10c5787d  Table: 4c35406a  DAC: 00000051
<1>[563195.108544] Register r0 information: non-paged memory
<1>[563195.114270] Register r1 information: non-paged memory
<1>[563195.119392] Register r2 information: non-slab/vmalloc memory
<1>[563195.124514] Register r3 information: non-slab/vmalloc memory
<1>[563195.130330] Register r4 information: slab kmalloc-128 start c15ba300 pointer offset 92 size 128
<1>[563195.136063] Register r5 information: non-paged memory
<1>[563195.144913] Register r6 information: non-paged memory
<1>[563195.149862] Register r7 information: non-paged memory
<1>[563195.154982] Register r8 information: non-slab/vmalloc memory
<1>[563195.160105] Register r9 information: NULL pointer
<1>[563195.165918] Register r10 information: slab kmalloc-256 start c1c17300 pointer offset 24 size 256
<1>[563195.170614] Register r11 information: slab kmalloc-128 start c1f1d600 pointer offset 0 size 128
<1>[563195.179553] Register r12 information: NULL pointer
<0>[563195.188402] Process kworker/1:1 (pid: 7146, stack limit = 0x203ad98d)
<0>[563195.193006] Stack: (0xc61b9d68 to 0xc61ba000)
<0>[563195.199605] 9d60:                   c15ba368 c0d8099c ffffffff 00000002 c61b9db4 c06b70f4
<0>[563195.204038] 9d80: 00000000 c0d8099c ffffffff c0342cbc c1c17300 c0d8099c c1665d80 00000002
<0>[563195.212284] 9da0: 23c34600 2faf0800 00000000 c06a4484 c158ba00 c1c18100 23c34600 2faf0800
<0>[563195.220530] 9dc0: c158ba00 2faf0800 c1665d80 00000000 c15af0c0 c06a8a9c c1665c68 c15af0c0
<0>[563195.228776] 9de0: c06b5f1c c140b180 23c34600 2faf0800 00000000 c06a8ad4 c1665d80 2faf0800
<0>[563195.237024] 9e00: c15af0c0 c1f1d780 dd999010 2faf0800 00000000 c06a8d0c 000f387c 2faf0800
<0>[563195.245271] 9e20: 00000000 ffffffff 2faf0800 c0d8366c c1f1d840 c1f1e5c0 2faf0800 00000000
<0>[563195.253516] 9e40: c1f1d780 dd999010 2faf0800 c06a8ec0 c1d11400 c1f1d800 00000000 c1f1d780
<0>[563195.261763] 9e60: dd999010 c07de400 c1665d80 c1f1d780 c0d05038 2faf0800 00000000 c1d11400
<0>[563195.270008] 9e80: dd999010 00000006 2faf0800 c1f1d800 00000000 000c3500 00000000 c07de810
<0>[563195.278256] 9ea0: 00000000 c0342cbc c1d11e00 2faf0800 c1d11e00 c0dd1038 00000000 00000002
<0>[563195.286503] 9ec0: 000927c0 c07e30e8 c1d11e00 000927c0 000c3500 00000024 d2445a00 c1d11e00
<0>[563195.294749] 9ee0: c1f1db80 c1f1db00 c1f1db80 c1f1e740 c1f1db00 dd99f305 c613c8c0 c07e6474
<0>[563195.302996] 9f00: c1f1dbb8 00000000 c1f1db84 c0d90974 00000000 00000040 dd99f305 c07e71fc
<0>[563195.311244] 9f20: c1f1dbb8 c613c880 dd99c040 dd99f300 00000000 c0339484 c61b8000 dd99c040
<0>[563195.319489] 9f40: c0d03d00 c613c880 dd99c040 c613c898 c0d03d00 dd99c058 c61b8000 dd99c200
<0>[563195.327735] 9f60: 00000000 c033a0dc d20b3ed8 c6da5b40 c6c14080 c033a07c c613c880 c61b8000
<0>[563195.335983] 9f80: d20b3ed8 c6c140a0 00000000 c0341374 c6da5b40 c034121c 00000000 00000000
<0>[563195.344227] 9fa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[563195.352474] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.360722] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[563195.368969] [<c06b5a1c>] (krait_mux_set_parent) from [<c06b70f4>] (krait_notifier_cb+0x58/0xb8)
<0>[563195.377212] [<c06b70f4>] (krait_notifier_cb) from [<c0342cbc>] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[563195.386156] [<c0342cbc>] (srcu_notifier_call_chain) from [<c06a4484>] (__clk_notify+0x70/0x94)
<0>[563195.395269] [<c06a4484>] (__clk_notify) from [<c06a8a9c>] (clk_change_rate+0xfc/0x2b8)
<0>[563195.403688] [<c06a8a9c>] (clk_change_rate) from [<c06a8ad4>] (clk_change_rate+0x134/0x2b8)
<0>[563195.411673] [<c06a8ad4>] (clk_change_rate) from [<c06a8d0c>] (clk_core_set_rate_nolock+0xb4/0x22c)
<0>[563195.420007] [<c06a8d0c>] (clk_core_set_rate_nolock) from [<c06a8ec0>] (clk_set_rate+0x3c/0x170)
<0>[563195.429036] [<c06a8ec0>] (clk_set_rate) from [<c07de400>] (_set_opp+0x1fc/0x524)
<0>[563195.438061] [<c07de400>] (_set_opp) from [<c07de810>] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[563195.445440] [<c07de810>] (dev_pm_opp_set_rate) from [<c07e30e8>] (__cpufreq_driver_target+0xe0/0x240)
<0>[563195.453254] [<c07e30e8>] (__cpufreq_driver_target) from [<c07e6474>] (od_dbs_update+0xcc/0x1a0)
<0>[563195.462629] [<c07e6474>] (od_dbs_update) from [<c07e71fc>] (dbs_work_handler+0x38/0x74)
<0>[563195.471570] [<c07e71fc>] (dbs_work_handler) from [<c0339484>] (process_one_work+0x234/0x4b0)
<0>[563195.479643] [<c0339484>] (process_one_work) from [<c033a0dc>] (worker_thread+0x60/0x5c4)
<0>[563195.488063] [<c033a0dc>] (worker_thread) from [<c0341374>] (kthread+0x158/0x174)
<0>[563195.496223] [<c0341374>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<0>[563195.503684] Exception stack(0xc61b9fb0 to 0xc61b9ff8)
<0>[563195.510802] 9fa0:                                     00000000 00000000 00000000 00000000
<0>[563195.516016] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[563195.524263] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[563195.532512] Code: f8fb 3c3b 1324 2170 (8d8b) e7c1 
<4>[563195.539363] ---[ end trace 313bacd35760440f ]---

Second R7800 with currently over 25 days uptime. I've got nothing in pstore
Third R7800 was updated and restarted only 3 days ago and there was nothing in pstore.
I'll monitor all R7800s I have and when I see a restart I'll check the ptsore and put the data here.
All R7800s use the NSS master build default ondemand optimized settings.

echo 600000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
echo 600000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq
echo 25 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor

As this topic was widely discussed here on the forum. Isn't there a software tool which can be used to test for this cache issue or to test for other hardware faults? Something like MemTestx86 software.

@hnyman
I follow this issue for quite some time now and I can say that there was a point before the kernel 5.10.100 that this crash issue was really symptomatic for the builds at that time and the crashes were really frequent every few days. These builds haven't had pstore enabled yet and it was impossible to catch the real reason for a restart. Later the pstore was enabled for the master builds.
At one point (I cannot say the exact build) the crashes simply stopped (or at least they were very rare).
That's why I left one R7800 on this older build just to make sure this crash doesn't repeat again.
Then the master switched to kernel 5.15 (obviously using newer cache scaling driver which was disabled by @Ansuel later) and crashes still happened from time to time (at least on my R7800s). I cannot surely say if they were rarer or more frequent with cache scaling driver disabled.
I cannot be more exact about the crashes when using performance mode because I don't use it. I didn't switch to performance mode deliberately in order to be able to test/catch if the cache scaling crash would happen.
If the crashes continue frequently I'll set performance mode on one R7800 just to be completely sure that this cache scaling crash won't happen once again.

I saw other people reported crashes lately but I hope they read this topic and can say more by themselves.

@Ansuel , @quarky , @hnyman
I'm currently testing a NSS master build compiled by me based on Master acwififdude's 15 commits + @rickkz0r 's 13 commits with sqm nss script master/Kernel5.15. After approx. 30 hours I had a crash probably connected to the NSS firmware. Maybe it's not the cache scaling issue.

Panic#1 Part1
<6>[    7.897397] (unnamed net_device) (uninitialized): nss_gmac_clk_ctl_dev_init: ctx->clk_ctl_base(0xde880000) + CLK_HALT_NSSFAB0_NSSFAB1_STATEA(0x3c20): 0x5b00
<6>[    7.910775] (unnamed net_device) (uninitialized): nss_gmac_clk_ctl_dev_init: ctx->clk_ctl_base(0xde880000) + GMAC_COREn_CLK_CTL(2)(0x3cf4): 0x50
<6>[    7.924834] (unnamed net_device) (uninitialized): nss_gmac_dev_init: nss_base(0xde960000) + NSS_GMACn_CTL(2)(0x38): 0x80c0c
<6>[    7.937759] (unnamed net_device) (uninitialized): nss_gmac_dev_init: nss_base(0xde960000) + NSS_ETH_CLK_DIV0(0xc): 0x100
<6>[    7.948600] (unnamed net_device) (uninitialized): nss_gmac_qsgmii_dev_init: QSGMII_PHY_SGMII_1_CTL(0x13c) - 0xc09c408f
<6>[    7.959721] (unnamed net_device) (uninitialized): nss_gmac_qsgmii_dev_init: NSS_QSGMII_CLK_CTL(0x2c) - 0x0
<6>[    7.970217] (unnamed net_device) (uninitialized): SGMII Specific Init for GMAC2 Done!
<6>[    7.979889] (unnamed net_device) (uninitialized): ioremap OK. Size 0x4000. reg_base 0x37400000. mac_base 0x7bd0fd7a.
<6>[    7.987817] (unnamed net_device) (uninitialized): mdio bus '37000000.mdio-mii' OK.
<6>[    8.005838] ipq8064-mdio 37000000.mdio (unnamed net_device) (uninitialized): nss_gmac_ipc_offload_init: enable Rx checksum
<6>[    8.006392] ipq8064-mdio 37000000.mdio eth1: Initialized NSS GMAC2 interface eth1: (base = 0x37400000, irq = 40, PhyId = 0, PollLink = 0)
<1>[    8.102562] nss_driver - fw of size 544712  bytes copied to load addr: 40000000, nss_id : 0
<1>[    8.103254] nss_driver - Turbo Support 1
<1>[    8.109805] Supported Frequencies - 
<1>[    8.109811] 800Mhz 
<1>[    8.113891] 800Mhz 
<1>[    8.117513] 800Mhz 
<1>[    8.119271] 
<1>[    8.123696] bf281400: meminfo init succeed
<3>[    8.125228] (unnamed net_device) (dummy): netif_napi_add() called with weight 64
<1>[    8.150868] node size 1 # items 2
<1>[    8.150897] memory: 0 0 (avl 496586752) items 2 active_cores 2
<1>[    8.153181] addr/size storage words 1 1 # words 2 in DTS, ddr size 1000000
<1>[    8.158980] bf281400: nss core 0 booted successfully
<1>[    8.186258] nss_driver - fw of size 218860  bytes copied to load addr: 40800000, nss_id : 1
<1>[    8.187487] bf285b00: meminfo init succeed
<3>[    8.193611] debugfs: Directory 'dynamic_if' with parent 'stats' already present!
<3>[    8.197754] debugfs: File 'n2h' in directory 'strings' already present!
<3>[    8.205184] debugfs: File 'drv' in directory 'strings' already present!
<1>[    8.214335] node size 1 # items 2
<6>[    8.214445] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
<1>[    8.218116] memory: 0 0 (avl 496721920) items 2 active_cores 2
<1>[    8.218179] addr/size storage words 1 1 # words 2 in DTS, ddr size 1000000
<1>[    8.218206] bf285b00: nss core 1 booted successfully
<5>[    8.245916] SCSI subsystem initialized
<7>[    8.251348] libata version 3.00 loaded.
<4>[    8.254173] ahci 29000000.sata: supply ahci not found, using dummy regulator
<4>[    8.254361] ahci 29000000.sata: supply phy not found, using dummy regulator
<4>[    8.260538] ahci 29000000.sata: supply target not found, using dummy regulator
<6>[    8.267918] ahci 29000000.sata: forcing port_map 0x0 -> 0x1
<6>[    8.274297] ahci 29000000.sata: AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl platform mode
<6>[    8.279821] ahci 29000000.sata: flags: ncq sntf pm led clo only pmp pio slum part ccc apst 
<6>[    8.289203] scsi host0: ahci
<6>[    8.296879] ata1: SATA max UDMA/133 mmio [mem 0x29000000-0x2900017f] port 0x100 irq 32
<6>[    8.628366] ata1: SATA link down (SStatus 0 SControl 300)
<6>[    8.633340] fsl-ehci: Freescale EHCI Host controller driver
<6>[    8.634933] ehci-platform: EHCI generic platform driver
<6>[    8.649025] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
<6>[    8.650575] ohci-platform: OHCI generic platform driver
<6>[    8.795032] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
<6>[    8.795113] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
<6>[    8.799747] xhci-hcd xhci-hcd.0.auto: hcc params 0x0228f065 hci version 0x100 quirks 0x0000000002010010
<6>[    8.807157] xhci-hcd xhci-hcd.0.auto: irq 55, io mem 0x10000000
<6>[    8.816721] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
<6>[    8.822268] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
<6>[    8.827892] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
<6>[    8.836733] hub 1-0:1.0: USB hub found
<6>[    8.842139] hub 1-0:1.0: 1 port detected
<6>[    8.846765] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[    8.850302] hub 2-0:1.0: USB hub found
<6>[    8.857987] hub 2-0:1.0: 1 port detected
<6>[    8.861961] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
<6>[    8.865538] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 3
<6>[    8.871027] xhci-hcd xhci-hcd.1.auto: hcc params 0x0228f065 hci version 0x100 quirks 0x0000000002010010
<6>[    8.878435] xhci-hcd xhci-hcd.1.auto: irq 56, io mem 0x11000000
<6>[    8.887988] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
<6>[    8.893616] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 4
<6>[    8.899240] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
<6>[    8.907585] hub 3-0:1.0: USB hub found
<6>[    8.913384] hub 3-0:1.0: 1 port detected
<6>[    8.917533] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[    8.921638] hub 4-0:1.0: USB hub found
<6>[    8.929682] hub 4-0:1.0: 1 port detected
<6>[    8.937574] usbcore: registered new interface driver usb-storage
<6>[    8.938844] usbcore: registered new interface driver uas
<14>[    8.943087] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
<14>[    8.956666] init: - preinit -
<6>[   12.032172] ipq8064-mdio 37000000.mdio eth1: nss_gmac_start_up: Force link up
<6>[   12.032229] ipq8064-mdio 37000000.mdio eth1: 1000 Mbps Full Duplex
<6>[   12.038398] ipq8064-mdio 37000000.mdio eth1: nss_gmac_dev_set_speed:NSS_ETH_CLK_DIV0(0xc) - 0x100
<6>[   12.044386] ipq8064-mdio 37000000.mdio eth1: nss_gmac_dev_set_speed: qsgmii_base(0xde980000) + PCS_MODE_CTL(0x68): 0x80008082
<6>[   12.053432] ipq8064-mdio 37000000.mdio eth1: nss_gmac_reset: eth1 resetting...
<6>[   13.119105] ipq8064-mdio 37000000.mdio eth1: GMAC reset completed in 106 jiffies; dma_bus_mode - 0x20100
<6>[   13.119520] ipq8064-mdio 37000000.mdio eth1: nss_gmac_change_mtu: Enabling Normal Frame MTU (Requested MTU [1500])
<6>[   13.127790] ipq8064-mdio 37000000.mdio eth1: nss_gmac_change_mtu: Enabling Jumbo Frame MTU for eth1 (Requested MTU [1500])
<6>[   13.137917] ipq8064-mdio 37000000.mdio eth1: Set addr a0:04:60:09:e4:13
<6>[   13.148942] ipq8064-mdio 37000000.mdio eth1: nss_gmac_ipc_offload_init: enable Rx checksum
<6>[   13.155377] ipq8064-mdio 37000000.mdio eth1: nss_gmac_config_flow_control:
<6>[   13.163780] ipq8064-mdio 37000000.mdio eth1: nss_gmac_rx_pause_disable: disable Rx flow control
<6>[   13.170653] ipq8064-mdio 37000000.mdio eth1: nss_gmac_tx_pause_disable: disable Tx flow control
<6>[   13.179623] ipq8064-mdio 37000000.mdio eth1: nss_gmac_linkup: data plane open command successfully issued
<6>[   13.189168] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
<6>[   13.198153] IPv6: ADDRCONF(NETDEV_CHANGE): eth1.1: link becomes ready
<12>[   18.419977] Starting lvm2 during preinit
<14>[   19.688719] mount_root: loading kmods from internal overlay
<14>[   19.728154] kmodloader: loading kernel modules from //etc/modules-boot.d/*
<14>[   19.735028] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
<5>[   19.875992] UBIFS (ubi0:1): Mounting in unauthenticated mode
<5>[   19.877176] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 219
<5>[   20.014816] UBIFS (ubi0:1): recovery needed
<5>[   20.434249] UBIFS (ubi0:1): recovery completed
<5>[   20.434384] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
<5>[   20.437672] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
<5>[   20.445499] UBIFS (ubi0:1): FS size: 82280448 bytes (78 MiB, 648 LEBs), max 659 LEBs, journal size 4063232 bytes (3 MiB, 32 LEBs)
<5>[   20.455469] UBIFS (ubi0:1): reserved for root: 3886308 bytes (3795 KiB)
<5>[   20.467091] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID 4E261E68-E706-41E9-8AC8-7459F0A0CED9, small LPT model
<14>[   20.475656] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[   20.499789] block: extroot: not configured
<5>[   20.503324] UBIFS (ubi0:1): un-mount UBI device 0
<5>[   20.503391] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" stops
<5>[   20.511729] UBIFS (ubi0:1): Mounting in unauthenticated mode
<5>[   20.516000] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 220
<5>[   20.816211] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
<5>[   20.816253] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
<5>[   20.823033] UBIFS (ubi0:1): FS size: 82280448 bytes (78 MiB, 648 LEBs), max 659 LEBs, journal size 4063232 bytes (3 MiB, 32 LEBs)
<5>[   20.833014] UBIFS (ubi0:1): reserved for root: 3886308 bytes (3795 KiB)
<5>[   20.844647] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID 4E261E68-E706-41E9-8AC8-7459F0A0CED9, small LPT model
<14>[   20.902901] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[   20.909321] block: extroot: not configured
<14>[   20.912384] mount_root: switching to ubifs overlay
<12>[   20.934250] urandom-seed: Seeding with /etc/urandom.seed
<14>[   21.157821] procd: - early -
<14>[   21.158073] procd: - watchdog -
<14>[   21.159986] procd: Watchdog has previously reset the system
<14>[   21.833802] procd: - watchdog -
<14>[   21.833948] procd: Watchdog has previously reset the system
<14>[   21.839102] procd: - ubus -
<14>[   22.010705] procd: - init -
<14>[   22.832734] urngd: v1.0.2 started.
<14>[   22.988231] kmodloader: loading kernel modules from /etc/modules.d/*
<6>[   23.001964] NET: Registered PF_ALG protocol family
<5>[   23.027706] Key type dns_resolver registered
<6>[   23.050093] RPC: Registered named UNIX socket transport module.
<6>[   23.050133] RPC: Registered udp transport module.
<6>[   23.054874] RPC: Registered tcp transport module.
<6>[   23.059770] RPC: Registered tcp NFSv4.1 backchannel transport module.
<6>[   23.090293] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
<6>[   23.094996] gre: GRE over IPv4 demultiplexor driver
<6>[   23.096273] ip_gre: GRE over IPv4 tunneling driver
<5>[   23.129239] NFS: Registering the id_resolver key type
<5>[   23.129312] Key type id_resolver registered
<5>[   23.133375] Key type id_legacy registered
<6>[   23.140010] cryptodev: driver 1.12 loaded.
<6>[   23.146661] PPP generic driver version 2.4.2
<6>[   23.147896] NET: Registered PF_PPPOX protocol family
<6>[   23.161457] GACT probability on
<6>[   23.162569] Mirror/redirect action on
<6>[   23.170179] u32 classifier
<6>[   23.170210]     input device check on
<6>[   23.171812]     Actions configured
<6>[   23.189454] fuse: init (API version 7.34)
<6>[   23.205143] Loading modules backported from Linux version v6.1-rc8-0-g76dcd734eca2
<6>[   23.205202] Backport generated by backports.git v5.15.81-1-41-g02e352527db5
<6>[   23.223489] PPTP driver version 0.8.5
<6>[   23.240859] xt_time: kernel timezone is -0000
<6>[   23.380778] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
<6>[   23.380819] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
<7>[   23.408712] ath10k_pci 0000:01:00.0: assign IRQ: got 47
<6>[   23.409395] ath10k_pci 0000:01:00.0: enabling device (0140 -> 0142)
<7>[   23.409485] ath10k_pci 0000:01:00.0: enabling bus mastering
<7>[   23.409898] qcom-pcie 1b500000.pci: msi#1 address_hi 0x0 address_lo 0x4362b498
<6>[   23.410053] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
<6>[   28.214265] ath10k_pci 0000:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
<6>[   28.214317] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
<6>[   28.225624] ath10k_pci 0000:01:00.0: firmware ver 10.4-3.9.0.2-00157 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps,peer-fixed-rate,iram-recovery crc32 6cdc6ff9
<6>[   30.504941] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 0:1 crc32 826c8c57
<6>[   34.273653] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-nvmem max-sta 512 raw 0 hwcrypto 1
<7>[   34.363944] ath: EEPROM regdomain: 0x0
<7>[   34.363965] ath: EEPROM indicates default country code should be used
<7>[   34.363977] ath: doing EEPROM country->regdmn map search
<7>[   34.363993] ath: country maps to regdmn code: 0x3a
<7>[   34.364006] ath: Country alpha2 being used: US
<7>[   34.364018] ath: Regpair used: 0x3a
<3>[   34.364130] backports-6.1-rc8/net/mac80211/main.c:1325: ieee80211_register_hw: setting default CoDeL params for phy
<6>[   34.366518] wlan0: Created a NSS virtual interface
<7>[   34.376097] ath10k_pci 0001:01:00.0: assign IRQ: got 49
<6>[   34.377278] ath10k_pci 0001:01:00.0: enabling device (0140 -> 0142)
<7>[   34.378405] ath10k_pci 0001:01:00.0: enabling bus mastering
<7>[   34.378984] qcom-pcie 1b700000.pci: msi#1 address_hi 0x0 address_lo 0x43597498
<6>[   34.379162] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
<6>[   35.658305] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
<6>[   35.658365] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
<6>[   35.669605] ath10k_pci 0001:01:00.0: firmware ver 10.4-3.9.0.2-00157 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps,peer-fixed-rate,iram-recovery crc32 6cdc6ff9
<6>[   37.946210] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:2 crc32 826c8c57
<6>[   41.724069] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-nvmem max-sta 512 raw 0 hwcrypto 1
<7>[   41.813247] ath: EEPROM regdomain: 0x0
<7>[   41.813269] ath: EEPROM indicates default country code should be used
<7>[   41.813282] ath: doing EEPROM country->regdmn map search
<7>[   41.813297] ath: country maps to regdmn code: 0x3a
<7>[   41.813311] ath: Country alpha2 being used: US
<7>[   41.813324] ath: Regpair used: 0x3a
<3>[   41.813411] backports-6.1-rc8/net/mac80211/main.c:1325: ieee80211_register_hw: setting default CoDeL params for phy
<3>[   41.815588] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   41.823173] wlan1: Created a NSS virtual interface
<14>[   41.835409] kmodloader: done loading kernel modules from /etc/modules.d/*
<6>[   46.383364] ECM init ( QSDK 12.1.r5_CS1 2022-11-25 1c401a1 )
<6>[   46.383448] ECM database jhash random seed: 0x1619c734
<6>[   46.392645] ECM init complete
<6>[   49.902852] ipq8064-mdio 37000000.mdio eth0: nss_gmac_start_up: Force link up
<6>[   49.902899] ipq8064-mdio 37000000.mdio eth0: 1000 Mbps Full Duplex
<6>[   49.909126] ipq8064-mdio 37000000.mdio eth0: nss_gmac_dev_set_speed:NSS_ETH_CLK_DIV0(0xc) - 0x100
<6>[   49.915064] ipq8064-mdio 37000000.mdio eth0: nss_gmac_reset: eth0 resetting...
<6>[   51.047126] ipq8064-mdio 37000000.mdio eth0: GMAC reset completed in 113 jiffies; dma_bus_mode - 0x20100
<6>[   51.047363] ipq8064-mdio 37000000.mdio eth0: nss_gmac_change_mtu: Enabling Normal Frame MTU (Requested MTU [1500])
<6>[   51.055720] ipq8064-mdio 37000000.mdio eth0: Set addr a0:04:60:09:e4:14
<6>[   51.065938] ipq8064-mdio 37000000.mdio eth0: nss_gmac_ipc_offload_init: enable Rx checksum
<6>[   51.072361] ipq8064-mdio 37000000.mdio eth0: nss_gmac_config_flow_control:
<6>[   51.080769] ipq8064-mdio 37000000.mdio eth0: nss_gmac_rx_pause_disable: disable Rx flow control
<6>[   51.087641] ipq8064-mdio 37000000.mdio eth0: nss_gmac_tx_pause_disable: disable Tx flow control
<6>[   51.096610] ipq8064-mdio 37000000.mdio eth0: nss_gmac_linkup: data plane open command successfully issued
<6>[   51.113083] ipq8064-mdio 37000000.mdio eth1: nss_gmac_start_up: Force link up
<6>[   51.114548] ipq8064-mdio 37000000.mdio eth1: 1000 Mbps Full Duplex
<6>[   51.121760] ipq8064-mdio 37000000.mdio eth1: nss_gmac_dev_set_speed:NSS_ETH_CLK_DIV0(0xc) - 0x100
<6>[   51.127813] ipq8064-mdio 37000000.mdio eth1: nss_gmac_dev_set_speed: qsgmii_base(0xde980000) + PCS_MODE_CTL(0x68): 0x80008082
<6>[   51.136753] ipq8064-mdio 37000000.mdio eth1: Set addr a0:04:60:09:e4:13
<6>[   51.148031] ipq8064-mdio 37000000.mdio eth1: nss_gmac_ipc_offload_init: enable Rx checksum
<6>[   51.154395] ipq8064-mdio 37000000.mdio eth1: nss_gmac_config_flow_control:
<6>[   51.162802] ipq8064-mdio 37000000.mdio eth1: nss_gmac_rx_pause_disable: disable Rx flow control
<6>[   51.169663] ipq8064-mdio 37000000.mdio eth1: nss_gmac_tx_pause_disable: disable Tx flow control
<6>[   51.184251] ipq8064-mdio 37000000.mdio eth1: nss_gmac_linkup: data plane open command successfully issued
<6>[   51.196640] br-lan: port 1(eth1.1) entered blocking state
<6>[   51.196687] br-lan: port 1(eth1.1) entered disabled state
<6>[   51.202203] device eth1.1 entered promiscuous mode
<6>[   51.207440] device eth1 entered promiscuous mode
<6>[   51.222914] br-lan: port 1(eth1.1) entered blocking state
<6>[   51.222962] br-lan: port 1(eth1.1) entered forwarding state
<6>[   51.241200] br-vlan: port 1(eth1.3) entered blocking state
<6>[   51.241248] br-vlan: port 1(eth1.3) entered disabled state
<6>[   51.246435] device eth1.3 entered promiscuous mode
<6>[   51.262647] br-vlan: port 1(eth1.3) entered blocking state
<6>[   51.262697] br-vlan: port 1(eth1.3) entered forwarding state
<6>[   51.826917] wlan0: Destroyed NSS virtual interface
<6>[   51.956828] wlan1: Destroyed NSS virtual interface
<3>[   52.135444] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   52.135735] Created a NSS virtual interface for dev [nssifb]
<6>[   52.141315] NSS IFB data callback registered
<6>[   52.147015] NSS IFB transmit callback registered
<6>[   52.151782] NSS IFB module loaded.
<7>[   52.348604] ath: EEPROM regdomain: 0x824f
<7>[   52.348645] ath: EEPROM indicates we should expect a country code
<7>[   52.351605] ath: doing EEPROM country->regdmn map search
<7>[   52.357809] ath: country maps to regdmn code: 0x10
<7>[   52.363063] ath: Country alpha2 being used: PA
<7>[   52.367756] ath: Regpair used: 0x10
<7>[   52.372086] ath: regdomain 0x824f dynamically updated by user
<7>[   52.375501] ath: EEPROM regdomain: 0x824f
<7>[   52.381472] ath: EEPROM indicates we should expect a country code
<7>[   52.385374] ath: doing EEPROM country->regdmn map search
<7>[   52.391534] ath: country maps to regdmn code: 0x10
<7>[   52.396904] ath: Country alpha2 being used: PA
<7>[   52.401428] ath: Regpair used: 0x10
<7>[   52.405854] ath: regdomain 0x824f dynamically updated by user
<4>[   52.471809] __nss_qdisc_init[2175]:parent (65536) and TC_H_ROOT (-1))
<4>[   52.471835] __nss_qdisc_init[2176]:root->ops->owner (00000000) and THIS_MODULE (bf63f600))
<4>[   52.471886] __nss_qdisc_init[2177]:NSS qdisc c59b1c00 (type 1) used along with non-nss qdiscs, or the interface is currently down
<6>[   52.487534] 6ad4ab96: Found net device [eth0]
<6>[   52.498276] 6ad4ab96: Net device [eth0] has NSS intf_num [1]
<6>[   52.502578] Nexthop successfully set for [eth0] to [nssifb]
<3>[   54.336161] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   54.336435] phy1-ap0: Created a NSS virtual interface
<4>[   60.823099] ath10k_pci 0001:01:00.0: Unknown eventid: 36933
<3>[   60.826632] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   60.827775] phy0-ap0: Created a NSS virtual interface
<6>[   60.841083] br-lan: port 2(phy1-ap0) entered blocking state
<6>[   60.841128] br-lan: port 2(phy1-ap0) entered disabled state
<6>[   60.845757] device phy1-ap0 entered promiscuous mode
<6>[   60.851237] br-lan: port 2(phy1-ap0) entered blocking state
<6>[   60.856293] br-lan: port 2(phy1-ap0) entered forwarding state
<6>[   60.876178] br-lan: port 2(phy1-ap0) entered disabled state
<6>[   61.406414] br-lan: port 2(phy1-ap0) entered blocking state
<6>[   61.406467] br-lan: port 2(phy1-ap0) entered forwarding state
<6>[   61.626012] 6ad4ab96: Reset nexthop successful for net device [eth0].
<6>[   61.634010] NSS IFB transmit callback unregistered
<6>[   61.636672] NSS virtual interface destroyed for dev [nssifb]
<4>[   68.677653] ath10k_pci 0000:01:00.0: Unknown eventid: 36933
<6>[   68.680237] br-lan: port 3(phy0-ap0) entered blocking state
<6>[   68.682048] br-lan: port 3(phy0-ap0) entered disabled state
<6>[   68.687923] device phy0-ap0 entered promiscuous mode
<6>[   68.693265] br-lan: port 3(phy0-ap0) entered blocking state
<6>[   68.698441] br-lan: port 3(phy0-ap0) entered forwarding state
<6>[   68.705639] br-lan: port 3(phy0-ap0) entered disabled state
<3>[   68.818583] debugfs: File 'virt_if' in directory 'stats' already present!
<6>[   68.818906] Created a NSS virtual interface for dev [nssifb]
<6>[   68.824398] NSS IFB data callback registered
<6>[   68.830246] NSS IFB transmit callback registered
<4>[   68.970055] __nss_qdisc_init[2175]:parent (65536) and TC_H_ROOT (-1))
<4>[   68.970074] __nss_qdisc_init[2176]:root->ops->owner (00000000) and THIS_MODULE (bf63f600))
<4>[   68.970113] __nss_qdisc_init[2177]:NSS qdisc c96fe400 (type 1) used along with non-nss qdiscs, or the interface is currently down
<6>[   68.981607] 6ad4ab96: Found net device [eth0]
<6>[   68.995736] 6ad4ab96: Net device [eth0] has NSS intf_num [1]
<6>[   69.000140] Nexthop successfully set for [eth0] to [nssifb]
<6>[   69.074210] br-lan: port 3(phy0-ap0) entered blocking state
<6>[   69.074265] br-lan: port 3(phy0-ap0) entered forwarding state
<1>[ 2283.723608] NSS core 0 signal COREDUMP COMPLETE 4000
<1>[ 2283.723667] bf281400: Starting NSS-FW logbuffer dump for core 0
<1>[ 2283.727672] bf281400: Warn: trap[813]: Trap on CHIP ID 00050000
<1>[ 2283.733389] bf281400: Warn: trap[620]: Trapped: TRAP_TD(00000004) DCAPT(3C000080)
<1>[ 2283.739206] bf281400: Warn: trap[645]: Trapped: Thread: 2, reason: 00000020, PC: 4002FBF4, previous PC: 4002FBF0
<1>[ 2283.746847] bf281400: Warn: trap[594]: A0_3: 4F71A294 402321C0 3F019940 4F71A296
<1>[ 2283.757086] bf281400: Warn: trap[594]: A4_7: 4F71A296 40053D04 3F019940 3F00AEF0
<1>[ 2283.764474] bf281400: Warn: trap[599]: D0_3: 00000026 00000001 00000009 4F71A284
<1>[ 2283.771843] bf281400: Warn: trap[599]: D4_7: 00060000 00000026 00000024 000005DC
<1>[ 2283.779222] bf281400: Warn: trap[599]: D8_11: 00000024 C0A80104 4CF9E534 00000000
<1>[ 2283.786600] bf281400: Warn: trap[599]: D12_15: 00000000 00000000 00D84001 00004002
<1>[ 2283.793980] bf281400: Warn: trap[649]: Thread_2 has non-recoverable trap
<3>[ 2283.801683] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[ 2283.804457] NSS core 1 signal COREDUMP COMPLETE 4000
<3>[ 2283.808392] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[ 2283.815063] bf285b00: Starting NSS-FW logbuffer dump for core 1
<3>[ 2283.819935] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<0>[ 2283.826679] Kernel panic - not syncing: NSS FW coredump: bringing system down
<3>[ 2283.832268] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<2>[ 2283.839267] CPU0: stopping
<4>[ 2283.839277] CPU: 0 PID: 12 Comm: ksoftirqd/0 Not tainted 5.15.98 #0
<4>[ 2283.839287] Hardware name: Generic DT based system
<4>[ 2283.839297] [<c030e7e8>] (unwind_backtrace) from [<c030a194>] (show_stack+0x14/0x20)
<4>[ 2283.839320] [<c030a194>] (show_stack) from [<c064cee0>] (dump_stack_lvl+0x40/0x4c)
<4>[ 2283.839339] [<c064cee0>] (dump_stack_lvl) from [<c030cfbc>] (do_handle_IPI+0x144/0x184)
<4>[ 2283.839354] [<c030cfbc>] (do_handle_IPI) from [<c030d018>] (ipi_handler+0x1c/0x2c)
<4>[ 2283.839364] [<c030d018>] (ipi_handler) from [<c037d3cc>] (handle_percpu_devid_irq+0x84/0x16c)
<4>[ 2283.839379] [<c037d3cc>] (handle_percpu_devid_irq) from [<c0376c5c>] (handle_domain_irq+0x68/0x94)
<4>[ 2283.839397] [<c0376c5c>] (handle_domain_irq) from [<c0666dec>] (gic_handle_irq+0x8c/0xb4)
<4>[ 2283.839414] [<c0666dec>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[ 2283.839429] Exception stack(0xc146bb00 to 0xc146bb48)
<4>[ 2283.839438] bb00: 60000093 1cd49000 00000000 c0c442fc 0000003c 00000000 60000013 00000000
<4>[ 2283.839445] bb20: bf8365b0 c83dad00 00000000 c146bd10 00000000 c146bb50 c0374678 c037467c
<4>[ 2283.839450] bb40: 60000013 ffffffff
<4>[ 2283.839455] [<c0300b7c>] (__irq_svc) from [<c037467c>] (vprintk_emit+0x1d0/0x1f8)
<4>[ 2283.839471] [<c037467c>] (vprintk_emit) from [<c03746c8>] (vprintk_default+0x24/0x2c)
<4>[ 2283.839482] [<c03746c8>] (vprintk_default) from [<c03710b8>] (_printk+0x1c/0x2c)
<4>[ 2283.839496] [<c03710b8>] (_printk) from [<bf7e4054>] (ieee80211_set_ringparam+0x330/0xd50 [mac80211])
<4>[ 2283.839748] [<bf7e4054>] (ieee80211_set_ringparam [mac80211]) from [<bf7e5b3c>] (ieee80211_sta_ps_transition+0x9a0/0x17fc [mac80211])
<4>[ 2283.839990] [<bf7e5b3c>] (ieee80211_sta_ps_transition [mac80211]) from [<bf7e785c>] (ieee80211_is_our_addr+0xe1c/0x2ad4 [mac80211])
<4>[ 2283.840227] [<bf7e785c>] (ieee80211_is_our_addr [mac80211]) from [<bf7e9d40>] (ieee80211_mark_rx_ba_filtered_frames+0x82c/0x1520 [mac80211])
<4>[ 2283.840465] [<bf7e9d40>] (ieee80211_mark_rx_ba_filtered_frames [mac80211]) from [<bf7eb880>] (ieee80211_rx_list+0xe4c/0xe94 [mac80211])
<4>[ 2283.840702] [<bf7eb880>] (ieee80211_rx_list [mac80211]) from [<bf7eb8f0>] (ieee80211_rx_napi+0x28/0x90 [mac80211])
<4>[ 2283.840938] [<bf7eb8f0>] (ieee80211_rx_napi [mac80211]) from [<bf99c000>] (ath10k_htt_rx_pktlog_completion_handler+0xfa0/0x1428 [ath10k_core])
<4>[ 2283.841189] [<bf99c000>] (ath10k_htt_rx_pktlog_completion_handler [ath10k_core]) from [<bf9a0b8c>] (ath10k_htt_txrx_compl_task+0xa28/0x1240 [ath10k_core])
<4>[ 2283.841333] [<bf9a0b8c>] (ath10k_htt_txrx_compl_task [ath10k_core]) from [<bf9f0a54>] (ath10k_pci_napi_poll+0x5c/0x134 [ath10k_pci])
<4>[ 2283.841444] [<bf9f0a54>] (ath10k_pci_napi_poll [ath10k_pci]) from [<c085df2c>] (__napi_poll+0x5c/0x1d0)
<4>[ 2283.841472] [<c085df2c>] (__napi_poll) from [<c085e298>] (net_rx_action+0xe4/0x268)
<4>[ 2283.841485] [<c085e298>] (net_rx_action) from [<c0301304>] (__do_softirq+0x11c/0x2e4)
<4>[ 2283.841500] [<c0301304>] (__do_softirq) from [<c0325d98>] (run_ksoftirqd+0x38/0x4c)
<4>[ 2283.841516] [<c0325d98>] (run_ksoftirqd) from [<c034824c>] (smpboot_thread_fn+0xe8/0x1ac)
<4>[ 2283.841535] [<c034824c>] (smpboot_thread_fn) from [<c0343b50>] (kthread+0x158/0x174)
<4>[ 2283.841554] [<c0343b50>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<4>[ 2283.841565] Exception stack(0xc146bfb0 to 0xc146bff8)
<4>[ 2283.841573] bfa0:                                     00000000 00000000 00000000 00000000
<4>[ 2283.841579] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[ 2283.841585] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000

Interesting, I was just about to say I don't see these crashes on mine and when I checked the uptime to brag about how rock solid things are it appears that one of my R7800s crashed today with a similar dump :smiley: :

<1>[1025155.785915] NSS core 0 signal COREDUMP COMPLETE 4000
<1>[1025155.785957] bf283480: Starting NSS-FW logbuffer dump for core 0
<1>[1025155.790037] bf283480: Warn: trap[813]: Trap on CHIP ID 00050000
<3>[1025155.791659] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[1025155.796120] bf283480: Warn: trap[620]: Trapped: TRAP_TD(00000004) DCAPT(3C000080)
<1>[1025155.796130] bf283480: Warn: trap[645]: Trapped: Thread: 2, reason: 00000020, PC: 4002FBF4, previous PC: 4002FBF0
<3>[1025155.804330] wlan1-iot: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[1025155.808975] bf283480: Warn: trap[594]: A0_3: 506A6ED4 402321C0 3F00F760 506A6ED6
<1>[1025155.808985] bf283480: Warn: trap[594]: A4_7: 506A6ED6 40053D04 3F00F760 3F00AEF0
5112 DPT=853 WINDOW=65535 RES=0x00 SYN URGP=0
<3>[1025155.804330] wlan1-iot: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[1025155.808975] bf283480: Warn: trap[594]: A0_3: 506A6ED4 402321C0 3F00F760 506A6ED6
<1>[1025155.808985] bf283480: Warn: trap[594]: A4_7: 506A6ED6 40053D04 3F00F760 3F00AEF0
<3>[1025155.816687] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[1025155.826992] bf283480: Warn: trap[599]: D0_3: 00000026 00000006 00000009 506A6EC4
<3>[1025155.833877] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[1025155.841487] bf283480: Warn: trap[599]: D4_7: 00060000 00000026 00000002 000005DC
<1>[1025155.841496] bf283480: Warn: trap[599]: D8_11: FFFFFFFF C0A8B369 4CFD2DC0 00000000
<3>[1025155.849265] wlan1-iot: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[1025155.855888] bf283480: Warn: trap[599]: D12_15: 00000000 00000000 00D84001 00000702
<1>[1025155.855896] bf283480: Warn: trap[649]: Thread_2 has non-recoverable trap
<1>[1025155.859386] NSS core 1 signal COREDUMP COMPLETE 4000
<3>[1025155.863486] wlan1-iot: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<1>[1025155.870324] bf287b80: Starting NSS-FW logbuffer dump for core 1
<3>[1025155.878205] phy0-ap0: NSS TX failed with error: NSS_TX_FAILURE_NOT_READY
<0>[1025155.885412] Kernel panic - not syncing: NSS FW coredump: bringing system down
<2>[1025155.885425] CPU0: stopping
<4>[1025155.885434] CPU: 0 PID: 12 Comm: ksoftirqd/0 Not tainted 5.15.88 #0
<4>[1025155.885443] Hardware name: Generic DT based system
<4>[1025155.885452] [<c030e560>] (unwind_backtrace) from [<c030a17c>] (show_stack+0x14/0x20)
<4>[1025155.885478] [<c030a17c>] (show_stack) from [<c0649e60>] (dump_stack_lvl+0x40/0x4c)
<4>[1025155.885496] [<c0649e60>] (dump_stack_lvl) from [<c030cfbc>] (do_handle_IPI+0x144/0x184)
<4>[1025155.885509] [<c030cfbc>] (do_handle_IPI) from [<c030d018>] (ipi_handler+0x1c/0x2c)
<4>[1025155.885522] [<c030d018>] (ipi_handler) from [<c037a3dc>] (handle_percpu_devid_irq+0x84/0x16c)
<4>[1025155.885547] [<c037a3dc>] (handle_percpu_devid_irq) from [<c0373c6c>] (handle_domain_irq+0x68/0x94)
<4>[1025155.885564] [<c0373c6c>] (handle_domain_irq) from [<c0663d6c>] (gic_handle_irq+0x8c/0xb4)
<4>[1025155.885580] [<c0663d6c>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[1025155.885594] Exception stack(0xc146b9f0 to 0xc146ba38)
<4>[1025155.885602] b9e0:                                     c0d9f324 0000000a 00000046 00000020
<4>[1025155.885611] ba00: c0d71f50 00000046 c0d9f324 00000011 00000000 00000400 00000000 000003ed
<4>[1025155.885617] ba20: c0ac7c4c c146ba40 c036ea94 c06480a0 20000013 ffffffff
<4>[1025155.885624] [<c0300b7c>] (__irq_svc) from [<c06480a0>] (memchr+0x0/0x20)

I don't see anything interesting happening in syslog leading up to that time.
I run the performance governor and the NSS clock fixed at 800mhz and it had been up for about 10 days until now.

2 Likes

Yup, the NSS firmware crashed. Unfortunately there’s nothing much we can do about it. Btw this is QSDK11.4 firmware?

1 Like

Yes it is. Basically ACWifiDude's kernel-5.15-qsdk11 branch.
What might be interesting is that it was exactly at the same time we left the house. So that would result in two iPhones leaving and some home automation stuff going off. I'll keep an eye on it.

2 Likes

I saw this in the log.

[0.688265] krait-cc soc:clock-controller: CPU0 @ 800000 KHz
[0.688292] krait-cc soc:clock-controller: CPU1 @ Undefined rate. Forcing new rate.
[0.688431] krait-cc soc:clock-controller: CPU1 @ 384000 KHz
[0.688451] krait-cc soc:clock-controller: L2 @ Undefined rate. Forcing new rate.
[0.688643] krait-cc soc:clock-controller: L2 @ 384000 KHz

Is all of this OK. CPU0 is at 800 but CPU1 and L2 are at 384 MHz.

1 Like

@Ansuel or any other who may know the answer, why
cat /proc/sys/dev/nss/clock/current_freq
doesn't output anything?

Check your syslog, I don't know why but it sends output there :slight_smile:

1 Like

It's really whimsy.

I've tried on 22.03 build and this always gives 800000000 as output. Shouldn't it be lower sometimes supposedly the krait-cc clock switching is not disabled on 22.03 builds.

Guys can you test this patch ?

GitHub - Ansuel/openwrt at ipq806x-test-crash2

The test is a stability test... Uptime should be equal to current images...

4 Likes

So far uptime of 3 days 18 hours with your patch :wink:

I would say that it looks good enough for wider crash testing.

1 Like

Sorry for the irrelevant question but how do I apply that patch?

You can download it from GitHub to your source repo, apply it and then compile your firmware.

wget https://github.com/Ansuel/openwrt/commit/849ea62e4340f331b95384187f3ea03f55d0559d.patch
patch -p 1 -i 849ea62e4340f331b95384187f3ea03f55d0559d.patch

You can download any commit or PR commits from GitHub by simply adding ".patch" to end of the normal URL

2 Likes

Oh my, with this patch I am getting 923 Mbps on speedtest.net (On Wi-Fi 5Ghz AC @ 160Mhz). Previously, speed tests are maxing around 460-490Mbps with the NSS patch.

2 Likes

That said, the router crashed in about 13 hours:

<1>[48816.188447] pgd = 70549b80
<1>[48816.195644] [5e806028] *pgd=00000000
<0>[48816.198339] Internal error: Oops: 5 [#1] SMP ARM
<4>[48816.202074] Modules linked in: ecm pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath rndis_host pptp pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_meta_bridge nft_masq nft_log nft_limit nft_hash nft_fwd_netdev nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_dup_netdev nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack_bridge nf_conntrack mac80211 ip6_tables ip_tables cp210x ch341 cfg80211 cdc_ncm cdc_ether x_tables wireguard usbserial usbnet ts_kmp ts_fsm ts_bm tcp_bbr slhc nfnetlink_queue nfnetlink_log nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_dup_netdev nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c libchacha20poly1305 crc_itu_t crc_ccitt compat fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit
<4>[48816.202776]  act_mirred act_gact qca_nss_tunipip6 qca_nss_tun6rd qca_nss_qdisc qca_nss_crypto ledtrig_usbport cryptodev nfsv4 nfsv3 nfs msdos bonding ip_gre gre ifb dummy nat46 l2tp_ip6 l2tp_ip l2tp_eth sit l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 ipip ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel rpcsec_gss_krb5 auth_rpcgss tun xfrm_user xfrm_ipcomp af_key xfrm_algo ntfs lockd sunrpc grace cifs cifs_md4 cifs_arc4 dns_resolver nls_utf8 nls_iso8859_1 nls_cp437 wp512 twofish_generic twofish_common tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis seqiv md4 echainiv des_generic arc4 usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd qca_nss_drv qca_nss_gmac
<4>[48816.272481]  ramoops reed_solomon pstore gpio_button_hotplug vfat fat f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic encrypted_keys trusted tpm oid_registry asn1_encoder
<4>[48816.380565] CPU: 1 PID: 31023 Comm: kworker/1:2 Not tainted 5.15.108 #0
<4>[48816.399903] Hardware name: Generic DT based system
<4>[48816.406412] Workqueue: events dbs_work_handler
<4>[48816.411272] PC is at msm_read_timer_count+0xc/0x18
<4>[48816.415698] LR is at sched_clock+0x3c/0xb8
<4>[48816.420471] pc : [<c08606ac>]    lr : [<c03bd038>]    psr: 60000193
<4>[48816.424554] sp : cbcff4b8  ip : cbcfe000  fp : 00000000
<4>[48816.430716] r10: c1ca9418  r9 : cbcfe000  r8 : c0e78548
<4>[48816.435924] r7 : 00000120  r6 : c0e78540  r5 : 00000000  r4 : c0e78548
<4>[48816.441136] r3 : 5e806024  r2 : 00000000  r1 : ffffffff  r0 : c0e9bc08
<4>[48816.447732] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[48816.454246] Control: 10c5787d  Table: 47fe006a  DAC: 00000051
<1>[48816.461446] Register r0 information: non-slab/vmalloc memory
<1>[48816.467262] Register r1 information: non-paged memory
<1>[48816.472990] Register r2 information: NULL pointer
<1>[48816.477937] Register r3 information: non-paged memory
<1>[48816.482626] Register r4 information: non-slab/vmalloc memory
<1>[48816.487662] Register r5 information: NULL pointer
<1>[48816.493389] Register r6 information: non-slab/vmalloc memory
<1>[48816.497991] Register r7 information: non-paged memory
<1>[48816.503722] Register r8 information: non-slab/vmalloc memory
<1>[48816.508667] Register r9 information: non-slab/vmalloc memory
<1>[48816.514397] Register r10 information: slab kmalloc-256 start c1ca9400 pointer offset 24 size 256
<1>[48816.520044] Register r11 information: NULL pointer
<1>[48816.528805] Register r12 information: non-slab/vmalloc memory
<0>[48816.533408] Process kworker/1:2 (pid: 31023, stack limit = 0x292ed251)
<0>[48816.539226] Stack: (0xcbcff4b8 to 0xcbd00000)
<0>[48816.545649] f4a0:                                                       00000000 ffffffff
<0>[48816.550086] f4c0: 00000000 00000000 00000000 c0b2d510 cbcfe000 c1ca9418 00000000 c038916c
<0>[48816.558245] f4e0: 00000000 00000000 00000000 00000000 c0b2d510 0000792f 00000000 00000000
<0>[48816.566403] f500: 00000000 00000000 00000000 ffffffff 00000000 00000000 00000000 00000000
<0>[48816.574563] f520: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 a16b35ab
<0>[48816.582723] f540: 00000000 ffffffff 00000000 00000000 00000000 c0b2d510 cbcfe000 c1ca9418
<0>[48816.590883] f560: 00000000 c0389654 cbcff5bc 00000000 00000000 5e806028 cbcff620 00000005
<0>[48816.599042] f580: 00000000 c030f984 cbcfe000 c1ca9418 00000000 c0389780 cbcff5bc cbcfe000
<0>[48816.607203] f5a0: 5e806028 c038616c c068cd88 cbcff5bc cbcff620 c030f550 c0b2d510 5e806028
<0>[48816.615362] f5c0: 00000005 c0ec88ec 00000000 c030f850 00000000 c0e099e0 00000005 5e806028
<0>[48816.623522] f5e0: cbcff620 c030f984 cbcfe000 c1ca9418 00000000 c030fa8c 00000000 00000000
<0>[48816.631682] f600: 00000000 00000000 c08606ac 60000193 ffffffff cbcff654 c0e78548 c0300ae8
<0>[48816.639843] f620: c0e9bc08 ffffffff 00000000 5e806024 c0e78548 00000000 c0e78540 00000120
<0>[48816.648002] f640: c0e78548 cbcfe000 c1ca9418 00000000 cbcfe000 cbcff670 c03bd038 c08606ac
<0>[48816.656162] f660: 60000193 ffffffff 00000051 00000000 00000000 ffffffff 00000000 00000000
<0>[48816.664321] f680: 00000000 c0b2d510 cbcfe000 c1ca9418 00000000 c038916c 00000000 00000000
<0>[48816.672479] f6a0: 00000000 00000000 c0b2d510 0000792f 00000000 00000000 00000000 00000000
<0>[48816.680640] f6c0: 00000000 ffffffff 00000000 00000000 00000000 00000000 00000001 cd673100
<0>[48816.688800] f6e0: cc621da8 bfa87b9c 00000000 00000000 00000000 a16b35ab cd673100 ffffffff
<0>[48816.696960] f700: 00000000 00000000 00000000 c0b2d510 cbcfe000 c1ca9418 00000000 c0389654
<0>[48816.705119] f720: cbcff774 bfa8ee8c 00000004 5e806028 cbcff7d8 00000005 00000000 c030f984
<0>[48816.713279] f740: cbcfe000 c1ca9418 00000000 c0389780 cbcff774 cbcfe000 5e806028 c038616c
<0>[48816.721440] f760: c068cd88 cbcff774 cbcff7d8 c030f550 c0b2d510 5e806028 00000004 c0ec88ec
<0>[48816.729599] f780: 00000000 c030f850 00000000 c0e099e0 00000005 5e806028 cbcff7d8 c030f984
<0>[48816.737759] f7a0: cbcfe000 c1ca9418 00000000 c030fa8c 00000000 c5aa1cfc c5aa0000 00000034
<0>[48816.745918] f7c0: c08606ac 60000193 ffffffff cbcff80c c0e78548 c0300ae8 c0e9bc08 ffffffff
<0>[48816.754078] f7e0: 00000000 5e806024 c0e78548 00000000 c0e78540 00000120 c0e78548 cbcfe000
<0>[48816.762237] f800: c1ca9418 00000000 cbcfe000 cbcff828 c03bd038 c08606ac 60000193 ffffffff
<0>[48816.770397] f820: 00000051 c5a917e0 00000054 ffffffff 00000000 00000000 00000000 c0b2d510
<0>[48816.778557] f840: cbcfe000 c1ca9418 00000000 c038916c 00000001 00000000 00000000 00000000
<0>[48816.786716] f860: c0b2d510 0000792f c16f1880 00000800 00000000 47940000 df663000 ffffffff
<0>[48816.794877] f880: 00000000 0000ffff 00000000 c5a92ad4 c5a92ae4 dea23000 00000000 00000000
<0>[48816.803036] f8a0: c46450a4 bf889400 dea23010 a16b35ab c5a904c0 ffffffff 00000000 00000000
<0>[48816.811197] f8c0: 00000000 c0b2d510 cbcfe000 c1ca9418 00000000 c0389654 cbcff92c c60710f8
<0>[48816.819355] f8e0: c63cf600 5e806028 cbcff990 00000005 00000000 c030f984 cbcfe000 c1ca9418
<0>[48816.827515] f900: 00000000 c0389780 cbcff92c cbcfe000 5e806028 c038616c c068cd88 cbcff92c
<0>[48816.835675] f920: cbcff990 c030f550 c0b2d510 5e806028 00000003 c0ec88ec 00000000 c030f850
<0>[48816.843834] f940: c5a92aec c0e099e0 00000005 5e806028 cbcff990 c030f984 cbcfe000 c1ca9418
<0>[48816.851995] f960: 00000000 c030fa8c c5aa0000 00000034 000007d4 bfa5127c c08606ac 60000193
<0>[48816.860155] f980: ffffffff cbcff9c4 c0e78548 c0300ae8 c0e9bc08 ffffffff 00000000 5e806024
<0>[48816.868315] f9a0: c0e78548 00000000 c0e78540 00000120 c0e78548 cbcfe000 c1ca9418 00000000
<0>[48816.876473] f9c0: cbcfe000 cbcff9e0 c03bd038 c08606ac 60000193 ffffffff 00000051 00000001
<0>[48816.884634] f9e0: 00000001 ffffffff 00000000 00000000 00000000 c0b2d510 cbcfe000 c1ca9418
<0>[48816.892792] fa00: 00000000 c038916c cbcffb30 00000002 c0e05038 00000000 c0b2d510 0000792f
<0>[48816.900953] fa20: c14221c0 c1422280 00000179 00000800 00000000 ffffffff 00000000 00000156
<0>[48816.909112] fa40: 00000400 00000166 00000156 00000001 00000001 00000000 00000001 00000000
<0>[48816.917278] fa60: 00000000 a16b35ab 00000000 ffffffff 00000000 00000000 00000000 c0b2d510
<0>[48816.925435] fa80: cbcfe000 c1ca9418 00000000 c0389654 cbcffae4 00000000 cbcffaac 5e806028
<0>[48816.933593] faa0: cbcffb48 00000005 00000000 c030f984 cbcfe000 c1ca9418 00000000 c0389780
<0>[48816.941752] fac0: cbcffae4 cbcfe000 5e806028 c038616c c068cd88 cbcffae4 cbcffb48 c030f550
<0>[48816.949912] fae0: c0b2d510 5e806028 00000002 c0ec88ec 00000000 c030f850 fa83b2da c0e099e0
<0>[48816.958070] fb00: 00000005 5e806028 cbcffb48 c030f984 cbcfe000 c1ca9418 00000000 c030fa8c
<0>[48816.966231] fb20: cf6201e0 00000001 00000000 c565c930 c08606ac 60000193 ffffffff cbcffb7c
<0>[48816.974389] fb40: c0e78548 c0300ae8 c0e9bc08 ffffffff 00000000 5e806024 c0e78548 00000000
<0>[48816.982550] fb60: c0e78540 00000120 c0e78548 cbcfe000 c1ca9418 00000000 cbcfe000 cbcffb98
<0>[48816.990711] fb80: c03bd038 c08606ac 60000193 ffffffff 00000051 c0bf91e8 dd99c480 ffffffff
<0>[48816.998869] fba0: 00000000 00000000 00000000 c0b2d510 cbcfe000 c1ca9418 00000000 c038916c
<0>[48817.007030] fbc0: c565c800 c0d48480 00000000 00000000 c0b2d510 0000792f cbcffc24 1cc54000
<0>[48817.015189] fbe0: dd999440 c03aa168 cbcffc24 ffffffff 00000000 cbcfe000 00000002 c0b9c97c
<0>[48817.023349] fc00: c0b9c0cc c03a9640 cbcffc1c 40000013 004a0972 004a0972 c0e03d00 a16b35ab
<0>[48817.031508] fc20: c0b9c0cc ffffffff 00000000 00000000 00000000 c0b2d510 cbcfe000 c1ca9418
<0>[48817.039668] fc40: c151bf80 c0389654 cbcffc9c c0a6f480 00000004 5e806028 cbcffd00 00000005
<0>[48817.047828] fc60: 00000000 c030f984 cbcfe000 c1ca9418 c151bf80 c0389780 cbcffc9c cbcfe000
<0>[48817.055988] fc80: 5e806028 c038616c c068cd88 cbcffc9c cbcffd00 c030f550 c0b2d510 5e806028
<0>[48817.064147] fca0: 00000001 c0ec88ec 00000000 c030f850 c0b94a88 c0e099e0 00000005 5e806028
<0>[48817.072307] fcc0: cbcffd00 c030f984 cbcfe000 c1ca9418 c151bf80 c030fa8c c161bc40 c070e92c
<0>[48817.080468] fce0: c161be00 00000000 c08606ac a0000093 ffffffff cbcffd34 cbcffdb4 c0300ae8
<0>[48817.088627] fd00: c0e9bc08 20000093 c0ea4300 5e806024 c0ed829c 00000006 00000101 20000013
<0>[48817.096787] fd20: cbcffdb4 00000000 c1ca9418 c151bf80 00000000 cbcffd50 c08606e0 c08606ac
<0>[48817.104946] fd40: a0000093 ffffffff 00000051 00000001 c0ed829c c0689ff8 c0ed85e4 00000001
<0>[48817.113105] fd60: 00000101 c06f8bd8 c0ed85f0 c0e89de4 ffffffff 00000002 cbcffdb4 c06fa470
<0>[48817.121266] fd80: 00000000 c0e89de4 ffffffff c035a4cc c1ca9400 c0e89de4 c1667e40 00000002
<0>[48817.129426] fda0: 66d16940 23c34600 000927c0 c06e7520 c158be00 c15b8a00 66d16940 23c34600
<0>[48817.137586] fdc0: c158be00 23c34600 c1667e40 00000000 c15af0c0 c06ebb38 c1667d28 c15af0c0
<0>[48817.145744] fde0: c06f90b8 c140b180 66d16940 23c34600 000927c0 c06ebb70 c1667e40 23c34600
<0>[48817.153905] fe00: c15af0c0 c151d100 dd999010 23c34600 000927c0 c06ebda8 23c34600 23c34600
<0>[48817.162065] fe20: 00000000 ffffffff 23c34600 c0e8cab4 00061a80 c15d5080 23c34600 00000001
<0>[48817.170224] fe40: c151d100 dd999010 23c34600 c06ebf5c c173e600 c15b5280 00000001 c151d100
<0>[48817.178384] fe60: dd999010 c0822290 c1667e40 c151d100 00000000 23c34600 00000000 c173e600
<0>[48817.186544] fe80: dd999010 00000006 23c34600 c15b5280 00000000 000927c0 00000000 c0822638
<0>[48817.194704] fea0: c0e99b8c c035a4cc c173f000 23c34600 c173f000 c0eda3c8 00000000 00000001
<0>[48817.202863] fec0: 001a5248 c0826f3c c173f000 001a5248 000927c0 00000024 c565c800 c173f000
<0>[48817.211023] fee0: c151d300 c151d280 c151d300 c1508e00 c151d280 dd99f305 ca653140 c082a2d8
<0>[48817.219182] ff00: c151d338 00000000 c151d304 c0e99dd4 00000000 00000040 dd99f305 c082b068
<0>[48817.227342] ff20: c151d338 ca653100 dd99c080 dd99f300 00000000 c0350c18 cbcfe000 dd99c080
<0>[48817.235501] ff40: c0e03d00 ca653100 dd99c080 ca653118 c0e03d00 dd99c098 cbcfe000 dd99c240
<0>[48817.243661] ff60: 00000000 c0351870 c748bed8 ca105840 c63f4ec0 c0351810 ca653100 cbcfe000
<0>[48817.251821] ff80: c748bed8 c63f4ee0 00000000 c0358c0c ca105840 c0358ab4 00000000 00000000
<0>[48817.259980] ffa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[48817.268140] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[48817.276300] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[48817.284458] [<c08606ac>] (msm_read_timer_count) from [<c03bd038>] (sched_clock+0x3c/0xb8)
<0>[48817.292615] [<c03bd038>] (sched_clock) from [<c038916c>] (vprintk_store+0x84/0x47c)
<0>[48817.300775] [<c038916c>] (vprintk_store) from [<c0389654>] (vprintk_emit+0xf0/0x1f8)
<0>[48817.308239] [<c0389654>] (vprintk_emit) from [<c0389780>] (vprintk_default+0x24/0x2c)
<0>[48817.316226] [<c0389780>] (vprintk_default) from [<c038616c>] (_printk+0x1c/0x2c)
<0>[48817.323952] [<c038616c>] (_printk) from [<c030f550>] (__do_kernel_fault.part.0+0x2c/0x8c)
<0>[48817.331417] [<c030f550>] (__do_kernel_fault.part.0) from [<c030f850>] (do_page_fault+0x2a0/0x2bc)
<0>[48817.339490] [<c030f850>] (do_page_fault) from [<c030fa8c>] (do_DataAbort+0x48/0xd8)
<0>[48817.348344] [<c030fa8c>] (do_DataAbort) from [<c0300ae8>] (__dabt_svc+0x48/0x80)
<0>[48817.355807] Exception stack(0xcbcff620 to 0xcbcff668)
<0>[48817.363449] f620: c0e9bc08 ffffffff 00000000 5e806024 c0e78548 00000000 c0e78540 00000120
<0>[48817.368401] f640: c0e78548 cbcfe000 c1ca9418 00000000 cbcfe000 cbcff670 c03bd038 c08606ac
<0>[48817.376556] f660: 60000193 ffffffff
<0>[48817.384709] [<c0300ae8>] (__dabt_svc) from [<c08606ac>] (msm_read_timer_count+0xc/0x18)
<0>[48817.388014] [<c08606ac>] (msm_read_timer_count) from [<c03bd038>] (sched_clock+0x3c/0xb8)
<0>[48817.396000] [<c03bd038>] (sched_clock) from [<c038916c>] (vprintk_store+0x84/0x47c)
<0>[48817.404332] [<c038916c>] (vprintk_store) from [<c0389654>] (vprintk_emit+0xf0/0x1f8)
<0>[48817.411799] [<c0389654>] (vprintk_emit) from [<c0389780>] (vprintk_default+0x24/0x2c)
<0>[48817.419787] [<c0389780>] (vprintk_default) from [<c038616c>] (_printk+0x1c/0x2c)
<0>[48817.427509] [<c038616c>] (_printk) from [<c030f550>] (__do_kernel_fault.part.0+0x2c/0x8c)
<0>[48817.434976] [<c030f550>] (__do_kernel_fault.part.0) from [<c030f850>] (do_page_fault+0x2a0/0x2bc)
<0>[48817.443049] [<c030f850>] (do_page_fault) from [<c030fa8c>] (do_DataAbort+0x48/0xd8)
<0>[48817.451901] [<c030fa8c>] (do_DataAbort) from [<c0300ae8>] (__dabt_svc+0x48/0x80)
<0>[48817.459366] Exception stack(0xcbcff7d8 to 0xcbcff820)
<0>[48817.467004] f7c0:                                                       c0e9bc08 ffffffff
<0>[48817.471959] f7e0: 00000000 5e806024 c0e78548 00000000 c0e78540 00000120 c0e78548 cbcfe000
<0>[48817.480118] f800: c1ca9418 00000000 cbcfe000 cbcff828 c03bd038 c08606ac 60000193 ffffffff
<0>[48817.488276] [<c0300ae8>] (__dabt_svc) from [<c08606ac>] (msm_read_timer_count+0xc/0x18)
<0>[48817.496434] [<c08606ac>] (msm_read_timer_count) from [<c03bd038>] (sched_clock+0x3c/0xb8)
<0>[48817.504247] [<c03bd038>] (sched_clock) from [<c038916c>] (vprintk_store+0x84/0x47c)
<0>[48817.512580] [<c038916c>] (vprintk_store) from [<c0389654>] (vprintk_emit+0xf0/0x1f8)
<0>[48817.520045] [<c0389654>] (vprintk_emit) from [<c0389780>] (vprintk_default+0x24/0x2c)
<0>[48817.528031] [<c0389780>] (vprintk_default) from [<c038616c>] (_printk+0x1c/0x2c)
<0>[48817.535758] [<c038616c>] (_printk) from [<c030f550>] (__do_kernel_fault.part.0+0x2c/0x8c)
<0>[48817.543223] [<c030f550>] (__do_kernel_fault.part.0) from [<c030f850>] (do_page_fault+0x2a0/0x2bc)
<0>[48817.551296] [<c030f850>] (do_page_fault) from [<c030fa8c>] (do_DataAbort+0x48/0xd8)
<0>[48817.560149] [<c030fa8c>] (do_DataAbort) from [<c0300ae8>] (__dabt_svc+0x48/0x80)
<0>[48817.567612] Exception stack(0xcbcff990 to 0xcbcff9d8)
<0>[48817.575251] f980:                                     c0e9bc08 ffffffff 00000000 5e806024
<0>[48817.580207] f9a0: c0e78548 00000000 c0e78540 00000120 c0e78548 cbcfe000 c1ca9418 00000000
<0>[48817.588365] f9c0: cbcfe000 cbcff9e0 c03bd038 c08606ac 60000193 ffffffff
<0>[48817.596520] [<c0300ae8>] (__dabt_svc) from [<c08606ac>] (msm_read_timer_count+0xc/0x18)
<0>[48817.602945] [<c08606ac>] (msm_read_timer_count) from [<c03bd038>] (sched_clock+0x3c/0xb8)
<0>[48817.610932] [<c03bd038>] (sched_clock) from [<c038916c>] (vprintk_store+0x84/0x47c)
<0>[48817.619263] [<c038916c>] (vprintk_store) from [<c0389654>] (vprintk_emit+0xf0/0x1f8)
<0>[48817.626728] [<c0389654>] (vprintk_emit) from [<c0389780>] (vprintk_default+0x24/0x2c)
<0>[48817.634716] [<c0389780>] (vprintk_default) from [<c038616c>] (_printk+0x1c/0x2c)
<0>[48817.642442] [<c038616c>] (_printk) from [<c030f550>] (__do_kernel_fault.part.0+0x2c/0x8c)
<0>[48817.649908] [<c030f550>] (__do_kernel_fault.part.0) from [<c030f850>] (do_page_fault+0x2a0/0x2bc)
<0>[48817.657981] [<c030f850>] (do_page_fault) from [<c030fa8c>] (do_DataAbort+0x48/0xd8)
<0>[48817.666833] [<c030fa8c>] (do_DataAbort) from [<c0300ae8>] (__dabt_svc+0x48/0x80)
<0>[48817.674297] Exception stack(0xcbcffb48 to 0xcbcffb90)
<0>[48817.681937] fb40:                   c0e9bc08 ffffffff 00000000 5e806024 c0e78548 00000000
<0>[48817.686891] fb60: c0e78540 00000120 c0e78548 cbcfe000 c1ca9418 00000000 cbcfe000 cbcffb98
<0>[48817.695047] fb80: c03bd038 c08606ac 60000193 ffffffff
<0>[48817.703200] [<c0300ae8>] (__dabt_svc) from [<c08606ac>] (msm_read_timer_count+0xc/0x18)
<0>[48817.708240] [<c08606ac>] (msm_read_timer_count) from [<c03bd038>] (sched_clock+0x3c/0xb8)
<0>[48817.716054] [<c03bd038>] (sched_clock) from [<c038916c>] (vprintk_store+0x84/0x47c)
<0>[48817.724385] [<c038916c>] (vprintk_store) from [<c0389654>] (vprintk_emit+0xf0/0x1f8)
<0>[48817.731851] [<c0389654>] (vprintk_emit) from [<c0389780>] (vprintk_default+0x24/0x2c)
<0>[48817.739836] [<c0389780>] (vprintk_default) from [<c038616c>] (_printk+0x1c/0x2c)
<0>[48817.747562] [<c038616c>] (_printk) from [<c030f550>] (__do_kernel_fault.part.0+0x2c/0x8c)
<0>[48817.755029] [<c030f550>] (__do_kernel_fault.part.0) from [<c030f850>] (do_page_fault+0x2a0/0x2bc)
<0>[48817.763102] [<c030f850>] (do_page_fault) from [<c030fa8c>] (do_DataAbort+0x48/0xd8)
<0>[48817.771954] [<c030fa8c>] (do_DataAbort) from [<c0300ae8>] (__dabt_svc+0x48/0x80)
<0>[48817.779417] Exception stack(0xcbcffd00 to 0xcbcffd48)
<0>[48817.787060] fd00: c0e9bc08 20000093 c0ea4300 5e806024 c0ed829c 00000006 00000101 20000013
<0>[48817.792011] fd20: cbcffdb4 00000000 c1ca9418 c151bf80 00000000 cbcffd50 c08606e0 c08606ac
<0>[48817.800168] fd40: a0000093 ffffffff
<0>[48817.808319] [<c0300ae8>] (__dabt_svc) from [<c08606ac>] (msm_read_timer_count+0xc/0x18)
<0>[48817.811625] [<c08606ac>] (msm_read_timer_count) from [<c08606e0>] (msm_read_current_timer+0x18/0x24)
<0>[48817.819613] [<c08606e0>] (msm_read_current_timer) from [<c0689ff8>] (__timer_delay+0x2c/0x78)
<0>[48817.828988] [<c0689ff8>] (__timer_delay) from [<c06f8bd8>] (krait_mux_set_parent+0xd4/0x11c)
<0>[48817.837407] [<c06f8bd8>] (krait_mux_set_parent) from [<c06fa470>] (krait_notifier_cb+0x58/0xb8)
<0>[48817.845916] [<c06fa470>] (krait_notifier_cb) from [<c035a4cc>] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[48817.854336] [<c035a4cc>] (srcu_notifier_call_chain) from [<c06e7520>] (__clk_notify+0x70/0x94)
<0>[48817.863364] [<c06e7520>] (__clk_notify) from [<c06ebb38>] (clk_change_rate+0xfc/0x2b8)
<0>[48817.872042] [<c06ebb38>] (clk_change_rate) from [<c06ebb70>] (clk_change_rate+0x134/0x2b8)
<0>[48817.879942] [<c06ebb70>] (clk_change_rate) from [<c06ebda8>] (clk_core_set_rate_nolock+0xb4/0x22c)
<0>[48817.888188] [<c06ebda8>] (clk_core_set_rate_nolock) from [<c06ebf5c>] (clk_set_rate+0x3c/0x170)
<0>[48817.897129] [<c06ebf5c>] (clk_set_rate) from [<c0822290>] (_set_opp+0x264/0x524)
<0>[48817.905723] [<c0822290>] (_set_opp) from [<c0822638>] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[48817.913361] [<c0822638>] (dev_pm_opp_set_rate) from [<c0826f3c>] (__cpufreq_driver_target+0xe0/0x240)
<0>[48817.921088] [<c0826f3c>] (__cpufreq_driver_target) from [<c082a2d8>] (od_dbs_update+0xcc/0x1a0)
<0>[48817.930376] [<c082a2d8>] (od_dbs_update) from [<c082b068>] (dbs_work_handler+0x38/0x74)
<0>[48817.938883] [<c082b068>] (dbs_work_handler) from [<c0350c18>] (process_one_work+0x234/0x4b0)
<0>[48817.946870] [<c0350c18>] (process_one_work) from [<c0351870>] (worker_thread+0x60/0x5c4)
<0>[48817.955548] [<c0351870>] (worker_thread) from [<c0358c0c>] (kthread+0x158/0x174)
<0>[48817.963620] [<c0358c0c>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<0>[48817.970997] Exception stack(0xcbcfffb0 to 0xcbcffff8)
<0>[48817.978028] ffa0:                                     00000000 00000000 00000000 00000000
<0>[48817.983157] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[48817.991315] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[48817.999476] Code: e12fff1e e30a343c e34c30ed e5933000 (e5930004)
<4>[48818.005895] ---[ end trace 0a1224c70e67bbe1 ]---
<0>[48818.038776] Kernel panic - not syncing: Fatal exception
<2>[48818.038824] CPU0: stopping
<4>[48818.042809] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D           5.15.108 #0
<4>[48818.045590] Hardware name: Generic DT based system
<4>[48818.053143] [<c030e7e8>] (unwind_backtrace) from [<c030a194>] (show_stack+0x14/0x20)
<4>[48818.057832] [<c030a194>] (show_stack) from [<c068caa0>] (dump_stack_lvl+0x40/0x4c)
<4>[48818.065731] [<c068caa0>] (dump_stack_lvl) from [<c030cfbc>] (do_handle_IPI+0x144/0x184)
<4>[48818.073109] [<c030cfbc>] (do_handle_IPI) from [<c030d018>] (ipi_handler+0x1c/0x2c)
<4>[48818.081008] [<c030d018>] (ipi_handler) from [<c0392484>] (handle_percpu_devid_irq+0x84/0x16c)
<4>[48818.088650] [<c0392484>] (handle_percpu_devid_irq) from [<c038bd14>] (handle_domain_irq+0x68/0x94)
<4>[48818.097243] [<c038bd14>] (handle_domain_irq) from [<c06a69cc>] (gic_handle_irq+0x8c/0xb4)
<4>[48818.106096] [<c06a69cc>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[48818.114339] Exception stack(0xc0e01e28 to 0xc0e01e70)
<4>[48818.121806] 1e20:                   ffffe000 c0d48040 00000000 1cc48000 00000008 c0e03d00
<4>[48818.126846] 1e40: c0e00000 04200002 0000000a c0d48040 c0d48040 c0e01e78 00000002 c0e01e78
<4>[48818.135004] 1e60: c033b028 c0301298 60000113 ffffffff
<4>[48818.143158] [<c0300b7c>] (__irq_svc) from [<c0301298>] (__do_softirq+0xb0/0x2e4)
<4>[48818.148195] [<c0301298>] (__do_softirq) from [<c033b028>] (irq_exit+0xc8/0x114)
<4>[48818.155660] [<c033b028>] (irq_exit) from [<c038bd18>] (handle_domain_irq+0x6c/0x94)
<4>[48818.162691] [<c038bd18>] (handle_domain_irq) from [<c06a69cc>] (gic_handle_irq+0x8c/0xb4)
<4>[48818.170331] [<c06a69cc>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[48818.178663] Exception stack(0xc0e01f10 to 0xc0e01f58)
<4>[48818.186128] 1f00:                                     0fd4dfc6 00000000 00000001 c0314a80
<4>[48818.191170] 1f20: 00000000 c0e04f28 c0e00000 00000000 00000000 ffffe000 c0e04ec8 c0e04f5c
<4>[48818.199327] 1f40: c0ec9ec0 c0e01f60 c0307628 c030762c 60000013 ffffffff
<4>[48818.207481] [<c0300b7c>] (__irq_svc) from [<c030762c>] (arch_cpu_idle+0x44/0x50)
<4>[48818.213906] [<c030762c>] (arch_cpu_idle) from [<c0369a18>] (do_idle+0x25c/0x2bc)
<4>[48818.221545] [<c0369a18>] (do_idle) from [<c0369d9c>] (cpu_startup_entry+0x1c/0x20)
<4>[48818.228924] [<c0369d9c>] (cpu_startup_entry) from [<c0d0119c>] (start_kernel+0x6b8/0x6c8)

@swg0101 out of curiosity can you post the initial bootlog? I need to check the cpuid and i need the first 10 lines of dmesg from an early uptime