Ipq806x NSS build (Netgear R7800 / TP-Link C2600 / Linksys EA8500)

vochong · 2022-07-09T17:47:39.286Z

These logs seem to depend largely upon specific wifi clients. Some clients never trigger them, but others may do very frequently and occasionally at such high rates that rate throttling logs are also generated for them.

echelon · 2022-07-09T18:22:28.653Z

Thanks for reassuring, I don't want to bash ACwifidude's wonderful works but currently I am running with other fork (don't want to mentioned it in here), my experiences with ACwifidude previous build was 'unstable' for 2.4 for large amount of clients like I have mentioned above.

Yes I am already aware of 2.4 minuses, what's I've meant by throughput was in the baseline of 2.4 limitation.

Currently building myself ACwifidude's master fork, hopefully it will be fine.

Reznale · 2022-07-09T20:28:13.743Z

Any chance that sqm packages could be bundled in with the Master builds? I just installed the 2 day old master build but can't install SQM with opkg because the kernel version is incompatible

ACwifidude · 2022-07-10T11:34:24.873Z

I removed SQM because it was dependent on firewall 3. I was following a thread several months ago where they were trying to move SQM from fw3 => fw4. Looks like at least one of the .qos files was still written for fw3. I’m going to run fw4 for a little bit and see how it works.

Master and 22.03 (both have fw4) were updated yesterday. 21.02 was updated yesterday and has fw3 and SQM still in it.

There are so many packages out there and everyone uses a different combo. I recommend cloning my repo and building your flavor (especially if you want to use @KONG nss.qos). The instructions on how to build are in post #2 if you have a Linux computer handy.

slh · 2022-07-10T11:56:21.016Z

sqm works with fw4 (and without fw3), but it does pull in an iptables provider (iptables-nft, make sure not to let it pull in iptables-legacy; you may have to disable the later explicitly).

D43m0n · 2022-07-12T08:21:06.586Z

I've created a new build based on the 22.03 release and rebased to the 22.03 tree on July 7. This build incorporates the AQL WiFi fixes from a few weeks ago. So far WiFi responsiveness has improved with multiple clients and AQL enabled. I have seen a spontaneous crash just recently. I've got two ramoops files that I'll share here:

<1>[   53.513375] 8<--- cut here ---                                                                                  
<6>[   53.513835] Nexthop successfully set for [eth0] to [nssifb]                                      
<1>[   53.516736] Unable to handle kernel NULL pointer dereference at virtual address 00000138                                                                                                                                              
<1>[   53.522252] pgd = 66bda50b                                                                                                                                                                                                            
<1>[   53.530560] [00000138] *pgd=476ee835, *pte=00000000, *ppte=00000000                                     
<0>[   53.533030] Internal error: Oops: 17 [#1] SMP ARM                                                               
<4>[   53.539188] Modules linked in: nss_ifb ecm ath10k_pci ath10k_core ath wireguard nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet mac80211 libchacha20poly1305 ipt_REJECT curve25519_neon cfg80211 xt_time xt_tcpu
dp xt_tcpmss xt_statistic xt_state xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_cgroup xt_addrtype xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_L
OG xt_HL xt_DSCP xt_CT xt_CLASSIFY sch_cake ppp_async poly1305_arm nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib
_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_flow_table nf_conntrack_netlink libcurve25519_generic iptable_nat iptable_mangle iptable_filter ipt_ECN ip_tab
les crc_ccitt compat chacha_neon fuse sch_tbf sch_ingress sch_htb                      
<4>[   53.539570]  sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pppoe pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev xt_set ip_set_lis
t_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port 
ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsv3 nfs nfs_ssc msdos b
onding ifb ip6_udp_tunnel udp_tunnel sit qca_nss_drv qca_nss_gmac oid_registry tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo vfat fat lockd sunrpc grace hfsplus hfs cdrom cifs dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_
1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic                                                                
<4>[   53.609422]  twofish_common tgr192 tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg s
ha512_generic sha1_generic seqiv md5 md4 kpp echainiv ecb des_generic libdes cmac authenc arc4 uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_
platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug xfs libcrc32c f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc3
2c_generic crc32_generic cbc encrypted_keys trusted tpm                                                               
<4>[   53.762019] CPU: 0 PID: 2924 Comm: modprobe Not tainted 5.10.127 #0                    
<4>[   53.784246] Hardware name: Generic DT based system                                                              
<4>[   53.790324] PC is at eth_type_trans+0x20/0x20c                                                                  
<4>[   53.795177] LR is at nss_ifb_data_cb+0x20/0x54 [nss_ifb]                                 
<4>[   53.799604] pc : [<c087720c>]    lr : [<bfaf52e8>]    psr: 80000113
<4>[   53.805072] sp : c5ba9c80  ip : bf69f0a8  fp : 00000002          
<4>[   53.811061] r10: 00000034  r9 : c5577800  r8 : bf695040
<4>[   53.816271] r7 : 00000000  r6 : bf695350  r5 : 00000000  r4 : c597e600
<4>[   53.821480] r3 : 00000001  r2 : bf695690  r1 : 00000000  r0 : c597e600
<4>[   53.828078] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[   53.834592] Control: 10c5787d  Table: 47bac06a  DAC: 00000051
<0>[   53.841794] Process modprobe (pid: 2924, stack limit = 0x0dbb961a)
<0>[   53.847529] Stack: (0xc5ba9c80 to 0xc5baa000)
<0>[   53.853606] 9c80: c32c56c0 00000034 c597e600 bf695690 bf695350 00000000 bf695040 bfaf52e8
<0>[   53.858037] 9ca0: c597e600 bfaf52c8 bf695350 bf64db18 00000000 bfaf528c bf695690 c32c5040
<0>[   53.866197] 9cc0: bf695058 0000007f 0000001c 00000002 00000001 000000e0 c1598a00 bf695680
<0>[   53.874356] 9ce0: 00000002 00000001 c1598a04 00000001 00000002 bf695690 00000040 bf695040
<0>[   53.882515] 9d00: bf6a2000 00000000 bf677c20 bf695680 00000002 bf64ee20 00000000 00000040
<0>[   53.890674] 9d20: c1406600 00000002 c1406600 00000001 bf695690 00000040 c5ba9d77 c5ba9d78
<0>[   53.898836] 9d40: c0d03d00 1cd49000 c5ba9d80 c082d0a0 bf695690 dd991700 0000012c c0c48700
<0>[   53.906996] 9d60: c5ba9d78 c082d3bc 0000630f ffff9fb8 00000000 0046cb70 c5ba9d78 c5ba9d78
<0>[   53.915155] 9d80: c5ba9d80 c5ba9d80 c583cb40 00000000 00000003 c0d0308c c0d03080 40000003
<0>[   53.923315] 9da0: c5ba8000 00000100 c5ba9db0 c0301378 04060000 c583ca80 c5ba9dcc c0d03080
<0>[   53.931473] 9dc0: c0c41220 0000000a c0c411ac c0c47940 c0c47940 ffff9fb7 c0d03d00 00400000
<0>[   53.939634] 9de0: c5ba9e28 ffffe000 00000000 00000000 00000001 c1410000 de80200c c5ba9e28
<0>[   53.947794] 9e00: 00000071 c032311c c0c46cb8 c0371848 c0d05334 c5ba9e48 c0d7e664 de802000
<0>[   53.955953] 9e20: c0c46cc4 c064c100 c063dfd0 40000013 ffffffff c5ba9e7c c594fcc0 c5ba8000
<0>[   53.964114] 9e40: c53f2f60 c0300b8c c53f2f70 c5b99c04 c0447808 b5b7a000 c53f2f60 c5b99c00
<0>[   53.972272] 9e60: c19e6340 c5df0918 c594fcc0 c5b99c00 c53f2f60 00000071 00000001 c5ba9e98
<0>[   53.980432] 9e80: c04480b4 c063dfd0 40000013 ffffffff 00000051 bf000000 c5df0910 c5df0918
<0>[   53.988592] 9ea0: b6f11000 00000000 c594fcc0 c044b8f4 c5df0910 00000000 c583ca80 00000000
<0>[   53.996751] 9ec0: 00000000 00004000 00000004 c583ca80 00000000 00000000 00000000 00004000
<0>[   54.004911] 9ee0: c03122d8 00004000 00000002 00000071 c583ca80 b6f11000 00000001 00000004
<0>[   54.013071] 9f00: c5b99c00 c044bf18 00000000 c5ba9f58 000017ff 00000071 00000001 c5b99c44
<0>[   54.021231] 9f20: c5ba9f58 c583ca80 00000000 00003fa4 00000001 000000c0 00000000 c042e22c
<0>[   54.029391] 9f40: 00000002 00000000 c5ba9f54 c5ba9f58 62c587c0 00000000 c5ba9f58 c5ba9f58
<0>[   54.037550] 9f60: 00000002 00000002 00000001 00003fa4 00000000 00000000 c583ca80 c04495dc
<0>[   54.045710] 9f80: 00000002 00000000 00000000 00000000 00000000 00000002 000000c0 c03002c4
<0>[   54.053869] 9fa0: c5ba8000 c0300060 00000000 00000000 00000000 00003fa4 00000001 00000002
<0>[   54.062030] 9fc0: 00000000 00000000 00000002 000000c0 00000000 00003fa4 00000001 00000000
<0>[   54.070189] 9fe0: 00022f6c be878ab0 00012548 b6f7e6bc 40000010 00000000 00000000 00000000
<0>[   54.078354] [<c087720c>] (eth_type_trans) from [<bfaf52e8>] (nss_ifb_data_cb+0x20/0x54 [nss_ifb])
<0>[   54.086688] [<bfaf52e8>] (nss_ifb_data_cb [nss_ifb]) from [<bf64db18>] (nss_core_send_buffer+0x1b18/0x1fd8 [qca_nss_drv])
<0>[   54.095478] [<bf64db18>] (nss_core_send_buffer [qca_nss_drv]) from [<bf64ee20>] (nss_core_handle_napi+0x194/0x238 [qca_nss_drv])
<0>[   54.106361] [<bf64ee20>] (nss_core_handle_napi [qca_nss_drv]) from [<c082d0a0>] (__napi_poll+0x34/0x150)
<0>[   54.117928] [<c082d0a0>] (__napi_poll) from [<c082d3bc>] (net_rx_action+0xdc/0x270)
<0>[   54.127389] [<c082d3bc>] (net_rx_action) from [<c0301378>] (__do_softirq+0x110/0x2b8)
<0>[   54.134769] [<c0301378>] (__do_softirq) from [<c032311c>] (irq_exit+0xb8/0x118)
<0>[   54.142753] [<c032311c>] (irq_exit) from [<c0371848>] (__handle_domain_irq+0x94/0xf4)
<0>[   54.149873] [<c0371848>] (__handle_domain_irq) from [<c064c100>] (gic_handle_irq+0x90/0xb8)
<0>[   54.157858] [<c064c100>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
<0>[   54.166012] Exception stack(0xc5ba9e48 to 0xc5ba9e90) 
<0>[   54.173656] 9e40:                   c53f2f70 c5b99c04 c0447808 b5b7a000 c53f2f60 c5b99c00
<0>[   54.178696] 9e60: c19e6340 c5df0918 c594fcc0 c5b99c00 c53f2f60 00000071 00000001 c5ba9e98
<0>[   54.186851] 9e80: c04480b4 c063dfd0 40000013 ffffffff 
<0>[   54.195010] [<c0300b8c>] (__irq_svc) from [<c063dfd0>] (__rb_insert_augmented+0x0/0x1c4)
<0>[   54.200047] [<c063dfd0>] (__rb_insert_augmented) from [<c044b8f4>] (mmap_region+0x418/0x694)
<0>[   54.208206] [<c044b8f4>] (mmap_region) from [<c044bf18>] (do_mmap+0x3a8/0x46c)
<0>[   54.216628] [<c044bf18>] (do_mmap) from [<c042e22c>] (vm_mmap_pgoff+0x8c/0xd4)
<0>[   54.223655] [<c042e22c>] (vm_mmap_pgoff) from [<c04495dc>] (sys_mmap_pgoff+0x98/0xd8)
<0>[   54.230859] [<c04495dc>] (sys_mmap_pgoff) from [<c0300060>] (ret_fast_syscall+0x0/0x54)
<0>[   54.238756] Exception stack(0xc5ba9fa8 to 0xc5ba9ff0) 
<0>[   54.246570] 9fa0:                   00000000 00000000 00000000 00003fa4 00000001 00000002
<0>[   54.251787] 9fc0: 00000000 00000000 00000002 000000c0 00000000 00003fa4 00000001 00000000
<0>[   54.259941] 9fe0: 00022f6c be878ab0 00012548 b6f7e6bc 
<0>[   54.268100] Code: e58d8010 e58de014 e24dd008 e5801008 (e5913138) 
<4>[   54.273287] ---[ end trace aca6b4fc0e4b4ae1 ]---

And the other one:

<1>[   53.513375] 8<--- cut here ---                                                                                  
<6>[   53.513835] Nexthop successfully set for [eth0] to [nssifb]                                      
<1>[   53.516736] Unable to handle kernel NULL pointer dereference at virtual address 00000138                                                                                                                                              
<1>[   53.522252] pgd = 66bda50b                                                                                                                                                                                                            
<1>[   53.530560] [00000138] *pgd=476ee835, *pte=00000000, *ppte=00000000                                     
<0>[   53.533030] Internal error: Oops: 17 [#1] SMP ARM
<4>[   53.539188] Modules linked in: nss_ifb ecm ath10k_pci ath10k_core ath wireguard nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet mac80211 libchacha20poly1305 ipt_REJECT curve25519_neon cfg80211 xt_time xt_tcpu
dp xt_tcpmss xt_statistic xt_state xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_cgroup xt_addrtype xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_L
OG xt_HL xt_DSCP xt_CT xt_CLASSIFY sch_cake ppp_async poly1305_arm nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib
_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_flow_table nf_conntrack_netlink libcurve25519_generic iptable_nat iptable_mangle iptable_filter ipt_ECN ip_tab
les crc_ccitt compat chacha_neon fuse sch_tbf sch_ingress sch_htb
<4>[   53.539570]  sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pppoe pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev xt_set ip_set_lis
t_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port 
ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsv3 nfs nfs_ssc msdos b
onding ifb ip6_udp_tunnel udp_tunnel sit qca_nss_drv qca_nss_gmac oid_registry tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo vfat fat lockd sunrpc grace hfsplus hfs cdrom cifs dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_
1 nls_cp850 nls_cp437 nls_cp1250 wp512 twofish_generic
<4>[   53.609422]  twofish_common tgr192 tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg s
ha512_generic sha1_generic seqiv md5 md4 kpp echainiv ecb des_generic libdes cmac authenc arc4 uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_
platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug xfs libcrc32c f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc3
2c_generic crc32_generic cbc encrypted_keys trusted tpm
<4>[   53.762019] CPU: 0 PID: 2924 Comm: modprobe Not tainted 5.10.127 #0
<4>[   53.784246] Hardware name: Generic DT based system
<4>[   53.790324] PC is at eth_type_trans+0x20/0x20c
<4>[   53.795177] LR is at nss_ifb_data_cb+0x20/0x54 [nss_ifb]
<4>[   53.799604] pc : [<c087720c>]    lr : [<bfaf52e8>]    psr: 80000113
<4>[   53.805072] sp : c5ba9c80  ip : bf69f0a8  fp : 00000002
<4>[   53.811061] r10: 00000034  r9 : c5577800  r8 : bf695040
<4>[   53.816271] r7 : 00000000  r6 : bf695350  r5 : 00000000  r4 : c597e600
<4>[   53.821480] r3 : 00000001  r2 : bf695690  r1 : 00000000  r0 : c597e600
<4>[   53.828078] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[   53.834592] Control: 10c5787d  Table: 47bac06a  DAC: 00000051
<0>[   53.841794] Process modprobe (pid: 2924, stack limit = 0x0dbb961a)
<0>[   53.847529] Stack: (0xc5ba9c80 to 0xc5baa000)
<0>[   53.853606] 9c80: c32c56c0 00000034 c597e600 bf695690 bf695350 00000000 bf695040 bfaf52e8
<0>[   53.858037] 9ca0: c597e600 bfaf52c8 bf695350 bf64db18 00000000 bfaf528c bf695690 c32c5040
<0>[   53.866197] 9cc0: bf695058 0000007f 0000001c 00000002 00000001 000000e0 c1598a00 bf695680
<0>[   53.874356] 9ce0: 00000002 00000001 c1598a04 00000001 00000002 bf695690 00000040 bf695040
<0>[   53.882515] 9d00: bf6a2000 00000000 bf677c20 bf695680 00000002 bf64ee20 00000000 00000040
<0>[   53.890674] 9d20: c1406600 00000002 c1406600 00000001 bf695690 00000040 c5ba9d77 c5ba9d78
<0>[   53.898836] 9d40: c0d03d00 1cd49000 c5ba9d80 c082d0a0 bf695690 dd991700 0000012c c0c48700
<0>[   53.906996] 9d60: c5ba9d78 c082d3bc 0000630f ffff9fb8 00000000 0046cb70 c5ba9d78 c5ba9d78
<0>[   53.915155] 9d80: c5ba9d80 c5ba9d80 c583cb40 00000000 00000003 c0d0308c c0d03080 40000003
<0>[   53.923315] 9da0: c5ba8000 00000100 c5ba9db0 c0301378 04060000 c583ca80 c5ba9dcc c0d03080
<0>[   53.931473] 9dc0: c0c41220 0000000a c0c411ac c0c47940 c0c47940 ffff9fb7 c0d03d00 00400000
<0>[   53.939634] 9de0: c5ba9e28 ffffe000 00000000 00000000 00000001 c1410000 de80200c c5ba9e28
<0>[   53.947794] 9e00: 00000071 c032311c c0c46cb8 c0371848 c0d05334 c5ba9e48 c0d7e664 de802000
<0>[   53.955953] 9e20: c0c46cc4 c064c100 c063dfd0 40000013 ffffffff c5ba9e7c c594fcc0 c5ba8000
<0>[   53.964114] 9e40: c53f2f60 c0300b8c c53f2f70 c5b99c04 c0447808 b5b7a000 c53f2f60 c5b99c00
<0>[   53.972272] 9e60: c19e6340 c5df0918 c594fcc0 c5b99c00 c53f2f60 00000071 00000001 c5ba9e98
<0>[   53.980432] 9e80: c04480b4 c063dfd0 40000013 ffffffff 00000051 bf000000 c5df0910 c5df0918
<0>[   53.988592] 9ea0: b6f11000 00000000 c594fcc0 c044b8f4 c5df0910 00000000 c583ca80 00000000
<0>[   53.996751] 9ec0: 00000000 00004000 00000004 c583ca80 00000000 00000000 00000000 00004000
<0>[   54.004911] 9ee0: c03122d8 00004000 00000002 00000071 c583ca80 b6f11000 00000001 00000004
<0>[   54.013071] 9f00: c5b99c00 c044bf18 00000000 c5ba9f58 000017ff 00000071 00000001 c5b99c44
<0>[   54.021231] 9f20: c5ba9f58 c583ca80 00000000 00003fa4 00000001 000000c0 00000000 c042e22c
<0>[   54.029391] 9f40: 00000002 00000000 c5ba9f54 c5ba9f58 62c587c0 00000000 c5ba9f58 c5ba9f58
<0>[   54.037550] 9f60: 00000002 00000002 00000001 00003fa4 00000000 00000000 c583ca80 c04495dc                                                                                                                                             
<0>[   54.045710] 9f80: 00000002 00000000 00000000 00000000 00000000 00000002 000000c0 c03002c4
<0>[   54.053869] 9fa0: c5ba8000 c0300060 00000000 00000000 00000000 00003fa4 00000001 00000002
<0>[   54.062030] 9fc0: 00000000 00000000 00000002 000000c0 00000000 00003fa4 00000001 00000000
<0>[   54.070189] 9fe0: 00022f6c be878ab0 00012548 b6f7e6bc 40000010 00000000 00000000 00000000
<0>[   54.078354] [<c087720c>] (eth_type_trans) from [<bfaf52e8>] (nss_ifb_data_cb+0x20/0x54 [nss_ifb])
<0>[   54.086688] [<bfaf52e8>] (nss_ifb_data_cb [nss_ifb]) from [<bf64db18>] (nss_core_send_buffer+0x1b18/0x1fd8 [qca_nss_drv])
<0>[   54.095478] [<bf64db18>] (nss_core_send_buffer [qca_nss_drv]) from [<bf64ee20>] (nss_core_handle_napi+0x194/0x238 [qca_nss_drv])
<0>[   54.106361] [<bf64ee20>] (nss_core_handle_napi [qca_nss_drv]) from [<c082d0a0>] (__napi_poll+0x34/0x150)
<0>[   54.117928] [<c082d0a0>] (__napi_poll) from [<c082d3bc>] (net_rx_action+0xdc/0x270)
<0>[   54.127389] [<c082d3bc>] (net_rx_action) from [<c0301378>] (__do_softirq+0x110/0x2b8)
<0>[   54.134769] [<c0301378>] (__do_softirq) from [<c032311c>] (irq_exit+0xb8/0x118)
<0>[   54.142753] [<c032311c>] (irq_exit) from [<c0371848>] (__handle_domain_irq+0x94/0xf4)
<0>[   54.149873] [<c0371848>] (__handle_domain_irq) from [<c064c100>] (gic_handle_irq+0x90/0xb8)
<0>[   54.157858] [<c064c100>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
<0>[   54.166012] Exception stack(0xc5ba9e48 to 0xc5ba9e90) 
<0>[   54.173656] 9e40:                   c53f2f70 c5b99c04 c0447808 b5b7a000 c53f2f60 c5b99c00
<0>[   54.178696] 9e60: c19e6340 c5df0918 c594fcc0 c5b99c00 c53f2f60 00000071 00000001 c5ba9e98
<0>[   54.186851] 9e80: c04480b4 c063dfd0 40000013 ffffffff 
<0>[   54.195010] [<c0300b8c>] (__irq_svc) from [<c063dfd0>] (__rb_insert_augmented+0x0/0x1c4)
<0>[   54.200047] [<c063dfd0>] (__rb_insert_augmented) from [<c044b8f4>] (mmap_region+0x418/0x694)
<0>[   54.208206] [<c044b8f4>] (mmap_region) from [<c044bf18>] (do_mmap+0x3a8/0x46c)
<0>[   54.216628] [<c044bf18>] (do_mmap) from [<c042e22c>] (vm_mmap_pgoff+0x8c/0xd4)
<0>[   54.223655] [<c042e22c>] (vm_mmap_pgoff) from [<c04495dc>] (sys_mmap_pgoff+0x98/0xd8)
<0>[   54.230859] [<c04495dc>] (sys_mmap_pgoff) from [<c0300060>] (ret_fast_syscall+0x0/0x54)
<0>[   54.238756] Exception stack(0xc5ba9fa8 to 0xc5ba9ff0) 
<0>[   54.246570] 9fa0:                   00000000 00000000 00000000 00003fa4 00000001 00000002
<0>[   54.251787] 9fc0: 00000000 00000000 00000002 000000c0 00000000 00003fa4 00000001 00000000
<0>[   54.259941] 9fe0: 00022f6c be878ab0 00012548 b6f7e6bc 
<0>[   54.268100] Code: e58d8010 e58de014 e24dd008 e5801008 (e5913138) 
<4>[   54.273287] ---[ end trace aca6b4fc0e4b4ae1 ]---

In this second ramoops file, I found some other relevant part as well after the --- cut here --- and --- end trace --- part:

<0>[   54.286251] Kernel panic - not syncing: Fatal exception in interrupt  
<2>[   54.286284] CPU1: stopping                                                                                      
<4>[   54.291663] CPU: 1 PID: 85 Comm: kworker/1:8 Tainted: G      D           5.10.127 #0
<4>[   54.294184] Hardware name: Generic DT based system                                                              
<4>[   54.302088] Workqueue: events dbs_work_handler                                                                  
<4>[   54.306692] [<c030e4ac>] (unwind_backtrace) from [<c030a22c>] (show_stack+0x14/0x20)                            
<4>[   54.311119] [<c030a22c>] (show_stack) from [<c0632d68>] (dump_stack+0x94/0xa8)           
<4>[   54.319015] [<c0632d68>] (dump_stack) from [<c030d1d0>] (do_handle_IPI+0x140/0x184)      
<4>[   54.326045] [<c030d1d0>] (do_handle_IPI) from [<c030d230>] (ipi_handler+0x1c/0x2c)       
<4>[   54.333598] [<c030d230>] (ipi_handler) from [<c0371844>] (__handle_domain_irq+0x90/0xf4) 
<4>[   54.341237] [<c0371844>] (__handle_domain_irq) from [<c064c100>] (gic_handle_irq+0x90/0xb8)
<4>[   54.349485] [<c064c100>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)        
<4>[   54.357552] Exception stack(0xc1eebce0 to 0xc1eebd28)                                    
<4>[   54.365198] bce0: dd99f980 00000000 dd99f998 00002140 dd99f980 c0d03d00 c1cc8a18 c1eea000
<4>[   54.370236] bd00: 00000002 c0ae8db0 c0ae8cf0 c1eebd44 00000001 c1eebd30 c0339e40 c0339ebc
<4>[   54.378387] bd20: 60000013 ffffffff                                                                             
<4>[   54.386550] [<c0300b8c>] (__irq_svc) from [<c0339ebc>] (wq_worker_sleeping+0xc0/0xcc)    
<4>[   54.389852] [<c0339ebc>] (wq_worker_sleeping) from [<c09d2f9c>] (schedule+0x48/0x110)    
<4>[   54.397838] [<c09d2f9c>] (schedule) from [<c09d6b8c>] (schedule_timeout+0x74/0xd8)       
<4>[   54.405645] [<c09d6b8c>] (schedule_timeout) from [<c09d3dc4>] (wait_for_completion_timeout+0x9c/0x11c)
<4>[   54.413122] [<c09d3dc4>] (wait_for_completion_timeout) from [<c06f1044>] (qcom_rpm_write+0x154/0x1dc)
<4>[   54.422415] [<c06f1044>] (qcom_rpm_write) from [<c0699a44>] (clk_rpm_set_rate+0x118/0x124)
<4>[   54.431696] [<c0699a44>] (clk_rpm_set_rate) from [<c068cfac>] (clk_change_rate+0x258/0x2b8)
<4>[   54.439850] [<c068cfac>] (clk_change_rate) from [<c068d0c0>] (clk_core_set_rate_nolock+0xb4/0x1f8)
<4>[   54.448096] [<c068d0c0>] (clk_core_set_rate_nolock) from [<c068d240>] (clk_set_rate+0x3c/0x170)
<4>[   54.457123] [<c068d240>] (clk_set_rate) from [<c07bc15c>] (set_target+0xf0/0x1e4)        
<4>[   54.465721] [<c07bc15c>] (set_target) from [<c07b5f8c>] (__cpufreq_driver_target+0x1a0/0x5b4)
<4>[   54.473358] [<c07b5f8c>] (__cpufreq_driver_target) from [<c07b9524>] (od_dbs_update+0xcc/0x1a0)
<4>[   54.481862] [<c07b9524>] (od_dbs_update) from [<c07ba20c>] (dbs_work_handler+0x38/0x74)
<4>[   54.490368] [<c07ba20c>] (dbs_work_handler) from [<c03388d0>] (process_one_work+0x1fc/0x470)
<4>[   54.498356] [<c03388d0>] (process_one_work) from [<c0338bb8>] (worker_thread+0x74/0x5d4)
<4>[   54.507039] [<c0338bb8>] (worker_thread) from [<c033ea9c>] (kthread+0x15c/0x160)
<4>[   54.515107] [<c033ea9c>] (kthread) from [<c0300148>] (ret_from_fork+0x14/0x2c)
<4>[   54.522481] Exception stack(0xc1eebfb0 to 0xc1eebff8) 
<4>[   54.529514] bfa0:                                     00000000 00000000 00000000 00000000
<4>[   54.534645] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   54.542803] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000

I don't really know how to diagnose this or what additional info is relevant. This particular build has version r19385+159-f765f2f114 and includes some extra packages such as igmpproxy, dawn, prometheus-node-exporter. I've also got a USB-stick plugged in that I use to run AdGuardHome from and store logging from AdGuardHome.

From what I can tell it's a kernel issue but I don't know how to diagnose, interpret. Anyone with some guidance?

pattagghiu · 2022-07-12T11:53:08.980Z

D43m0n:

This build incorporates the AQL WiFi fixes from a few weeks ago.

hello, as far as you know are these fixes also in 21.03? i've been encountering for a while some "laggyness" in Alexas' wifi connection..

D43m0n · 2022-07-12T13:10:09.710Z

From the looks of it, these fixes are not (yet?) backported to 21.03.

pattagghiu · 2022-07-12T13:30:27.670Z

that's unfair
thanks

Mpilon · 2022-07-12T16:41:48.027Z

In general, it looks like a null pointer (0x0) dereference - attempting to read from address 0x0.

There's probably 2 issues here -

some code in the kernel is attempting a memory access from a variable it hasn't checked to be non-null.

B) best guess is that something in the kernel is appempting to alloc memory, failing, and going on to try to use that memory without checking for that alloc failure.

Sorry if I've oversimplified, I'll take a look at your ramoops dumps to focus a little more.

vochong · 2022-07-12T17:41:20.575Z

I saw the same crash @Unable to handle kernel NULL pointer dereference at virtual address 00000138 running my own rebased master snapshot yesterday. Things are getting really flaky these days.

Mpilon · 2022-07-12T17:43:19.368Z

OK ... in THIS post,

D43m0n has 3 crash dumps.

This reply covers dump #1.
line numbers don't count automatic editor line wraps - I widened emacs to look at the dump.

The highlights:

line 3: <1>[   53.516736] Unable to handle kernel NULL pointer dereference
at virtual address 00000138
 -- some dereference 0x138 bytes (312 bytes decimal) off of null.

line 6: Internal error: Oops: 17 [#1] SMP ARM
 --

line 21: <4>[   53.762019] CPU: 0 PID: 2924 Comm: modprobe Not tainted
5.10.127 #0                    
 -- process ID 2924: modprobe running on CPU 0.

line 23, 24:
 <4>[   53.790324] PC is at eth_type_trans+0x20/0x20c                                                                  
 <4>[   53.795177] LR is at nss_ifb_data_cb+0x20/0x54 [nss_ifb]
 -- in function eth_type_trans, called from nss_ifb_data_cb,
 
    the byte offsets are where in each func the thing happened, first
    (PC is at ... ): the instruction causing the exception,

    second (LR is at ...) LR is link register - holds the return
    address for the function call to where func eth_type_trans must return.

    *** if you disassemble (man objdump) the module (or the kernel itself?)
    containing nss_ifb_data_cb,
    look just before byte offset 0x20/0x54 in that disassembly - the call
    to eth_type_trans should be there. it's probably being passed a bad pointer.

    ... or for that matter, look in the src where eth_type_trans is called from
    nss_ifb_data_cb.

lines 62 - 71:
 -- the entire call tree at crash.  looks like it's crashing on the way out
 of some ISR, maybe net_rx_action ... I haven't looked at how the ARM kernel
 handles any given interrupt.


lines 67 - 61:
<0>[   54.127389] [<c082d3bc>] (net_rx_action) from [<c0301378>] (__do_softirq+0x110/0x2b8)
<0>[   54.134769] [<c0301378>] (__do_softirq) from [<c032311c>] (irq_exit+0xb8/0x118)
<0>[   54.142753] [<c032311c>] (irq_exit) from [<c0371848>] (__handle_domain_irq+0x94/0xf4)
<0>[   54.149873] [<c0371848>] (__handle_domain_irq) from [<c064c100>] (gic_handle_irq+0x90/0xb8)
<0>[   54.157858] [<c064c100>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
 -- specifically: the raw hardware interrupt being serviced is gic (?)

lines 76 - 81:
<0>[   54.195010] [<c0300b8c>] (__irq_svc) from [<c063dfd0>] (__rb_insert_augmented+0x0/0x1c4)
<0>[   54.200047] [<c063dfd0>] (__rb_insert_augmented) from [<c044b8f4>] (mmap_region+0x418/0x694)
<0>[   54.208206] [<c044b8f4>] (mmap_region) from [<c044bf18>] (do_mmap+0x3a8/0x46c)
<0>[   54.216628] [<c044bf18>] (do_mmap) from [<c042e22c>] (vm_mmap_pgoff+0x8c/0xd4)
<0>[   54.223655] [<c042e22c>] (vm_mmap_pgoff) from [<c04495dc>] (sys_mmap_pgoff+0x98/0xd8)
<0>[   54.230859] [<c04495dc>] (sys_mmap_pgoff) from [<c0300060>] (ret_fast_syscall+0x0/0x54)

 -- the thing that was interrupted -- probably an innocent bystander unless
 the memory mapping being done in the kernel (do_mmap) was somehow interacting
 with data manipulated by the ISR.

--------------------

That's what I have for this one.  You also may want to look at the errata for
this *specific* CPU involved.  Compiler usually handles this stuff but if the
crash looks similar to something mention in the errata that's a direction too -
probably the -m cpu specifier option to the compiler. the less likely thing,
unless CFLAGS for the build have been changed.

for the adventurous with wide editors I've attached my work file. no, I haven't - my reply window doesn't have that option ?!

M.

vochong · 2022-07-12T17:50:26.308Z

I specifically asked Felix to backport his ATF WIFI commits to 21.02.x as well so the next 21.02.4 release would no longer have the WIFI problem. Unfortunately he never replied so I guess you will have to move on to 22.03.x.

Mpilon · 2022-07-12T17:56:34.373Z

Re D43m0n's 2nd crash dump HERE ...

it looks the same as the first. I don't know if modprobe is the cause or the victim ... I wouldn't think it should be used often enough to get clobbered randomly by net_rx_action --
if it is invoked that often then that's a system design issue (over-using modprobe), -- if not then the module being probed probably isn't handling its insert without stepping on something or maybe disabling interrupts until its startup code is done.

EDIT: looking at man modprobe there are some things to keep in mind there, too.

D43m0n · 2022-07-12T18:13:07.532Z

Thank you for looking into it. I rebased on 22.03-rc4 just before -rc5 was labeled. I’m guessing minor kernel/module updates will get pulled in when we’re closing in on the new stable 22.03 stable release. I ran a similar build with from April this year on 22.03-rc2 or 3 I think. It was pretty stable but that didn’t have the AQL/WiFi fix from @nbd yet. This build now does, WiFi seems improved a lot on responsiveness and even handles bufferbloat better based on numerous results from waveforms tool.

I’ll keep an eye on any new ramoops files that appear and share results if they appear.

Mpilon · 2022-07-12T18:26:19.356Z

you're welcome -- you may want to find that null pointer dereference in nss_ifb_data_cb+0x20/0x54 [nss_ifb]) /

nss_ifb_data_cb [nss_ifb] from
nss_core_send_buffer

This may be a symptom of the kernel unable to alloc memory - something else has exhausted memory - and nss_ifb_data_cb is just the victim.

In any case, it should be checking for that memory alloc failure.

vochong · 2022-07-12T19:47:29.507Z

@ACwifidude,

Please remove the following firewall setting because we no longer need the qca-nss-ecm iptables script:

config include 'qcanssecm'
        option type 'script'
        option path '/etc/firewall.d/qca-nss-ecm'
        option family 'any'  
        option reload '1'

Leaving it in the firewall config will cause the firewall startup script to throw errors as shown below:

/etc/init.d/firewall restart
Section qcanssecm option 'family' is not supported by fw4
Section qcanssecm option 'reload' is not supported by fw4
Section qcanssecm specifies unreachable path '/etc/firewall.d/qca-nss-ecm', ignoring section

pattagghiu · 2022-07-13T07:07:48.510Z

i'd have to go back to master, but i'm not ready for firewall4 since the new dnsmasq-full is still not ready.. bad luck

vochong · 2022-07-13T19:13:50.627Z

@ACwifidude,

FYI, Felix has just committed another multicast-triggered latency bug fix for WIFI in both master and 22.03 branches. Unfortunately, I don't think he will make such commit for 21.02 so people having WIFI problem with 21.02 can only stick with 21.02.1 or move on to 22.03 RC5 or later.

https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=ec9f82fa18c7c8deb4875152d7907855d186f4c6

xeonpj · 2022-07-13T19:43:50.080Z

Can you go from 22.02 to 23.01 keeping the configuration?