Pending Airtime Underflow

I have an Armor Z2 (running a pretty recent snapshot) that seems to crash every week or so, something I have never been able to understand. Anyway I keep looking for any debugging info I can, and I have recently had this message twice ...

[719448.567574] ------------[ cut here ]------------
[719448.567807] WARNING: CPU: 0 PID: 0 at backports-5.8-rc2-1/net/mac80211/sta_info.c:1929 ieee80211_sta_update_pending_airtime+0x1f8/0x1fc [mac80211]
[719448.571318] STA f0:c3:71:10:fa:85 AC 2 txq pending airtime underflow: 4294967096, 200
[719448.571324] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT rndis_host pppox ppp_generic oti6858 nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mac80211 ipt_REJECT huawei_cdc_ncm ebtable_nat ebtable_filter ebtable_broute ch341 cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial usbnet slhc sch_cake nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat cdc_wdm br_netfilter fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route
[719448.584781]  cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsd nfs msdos ifb rpcsec_gss_krb5 auth_rpcgss oid_registry tun snd_rawmidi snd_seq_device snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd_hwdep snd_compress snd soundcore vfat fat ntfs lockd sunrpc grace hfsplus hfs autofs4 dns_resolver dm_mirror dm_region_hash dm_log dm_crypt nls_utf8 nls_iso8859_1 nls_cp437 sha1_generic md5 ecb des_generic libdes cts cbc arc4 dm_mod dax uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd
[719448.657941]  phy_qcom_dwc3 ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug xfs libcrc32c ext4 mbcache jbd2 crc32c_generic
[719448.766880] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.52 #0
[719448.785175] Hardware name: Generic DT based system
[719448.791454] [<c030f954>] (unwind_backtrace) from [<c030b96c>] (show_stack+0x14/0x20)
[719448.796044] [<c030b96c>] (show_stack) from [<c08bbe00>] (dump_stack+0x94/0xa8)
[719448.804038] [<c08bbe00>] (dump_stack) from [<c031e7b4>] (__warn+0xb4/0xd0)
[719448.811152] [<c031e7b4>] (__warn) from [<c031e850>] (warn_slowpath_fmt+0x80/0x90)
[719448.818280] [<c031e850>] (warn_slowpath_fmt) from [<bf851b4c>] (ieee80211_sta_update_pending_airtime+0x1f8/0x1fc [mac80211])
[719448.826098] [<bf851b4c>] (ieee80211_sta_update_pending_airtime [mac80211]) from [<bf84c174>] (ieee80211_report_low_ack+0x250/0x514 [mac80211])
[719448.837365] [<bf84c174>] (ieee80211_report_low_ack [mac80211]) from [<bf84c44c>] (ieee80211_free_txskb+0x14/0x2c [mac80211])
[719448.850021] [<bf84c44c>] (ieee80211_free_txskb [mac80211]) from [<bf9978fc>] (ath10k_txrx_tx_unref+0x608/0x738 [ath10k_core])
[719448.861469] [<bf9978fc>] (ath10k_txrx_tx_unref [ath10k_core]) from [<bf9918a4>] (ath10k_htt_t2h_msg_handler+0xe74/0x1290 [ath10k_core])
[719448.872618] [<bf9918a4>] (ath10k_htt_t2h_msg_handler [ath10k_core]) from [<bf9ec768>] (ath10k_pci_htt_rx_cb+0x178/0x230 [ath10k_pci])
[719448.885099] [<bf9ec768>] (ath10k_pci_htt_rx_cb [ath10k_pci]) from [<bf9b131c>] (ath10k_ce_per_engine_service+0x9c/0x10c [ath10k_core])
[719448.896985] [<bf9b131c>] (ath10k_ce_per_engine_service [ath10k_core]) from [<bf9b140c>] (ath10k_ce_per_engine_service_any+0x80/0xd8 [ath10k_core])
[719448.908988] [<bf9b140c>] (ath10k_ce_per_engine_service_any [ath10k_core]) from [<bf9ee104>] (ath10k_pci_napi_poll+0x54/0x15c [ath10k_pci])
[719448.922109] [<bf9ee104>] (ath10k_pci_napi_poll [ath10k_pci]) from [<c0775084>] (net_rx_action+0x118/0x374)
[719448.934593] [<c0775084>] (net_rx_action) from [<c0302298>] (__do_softirq+0x130/0x2d4)
[719448.944307] [<c0302298>] (__do_softirq) from [<c0322c10>] (irq_exit+0xbc/0xe0)
[719448.952291] [<c0322c10>] (irq_exit) from [<c036e238>] (__handle_domain_irq+0x6c/0xd0)
[719448.959507] [<c036e238>] (__handle_domain_irq) from [<c05c78e0>] (gic_handle_irq+0x5c/0xb8)
[719448.967487] [<c05c78e0>] (gic_handle_irq) from [<c0301a8c>] (__irq_svc+0x6c/0x90)
[719448.976064] Exception stack(0xc0c01ee0 to 0xc0c01f28)
[719448.983478] 1ee0: 00000000 00028e55 1d054000 ddb93a40 dcc32000 00000000 ddb92e30 00028e55
[719448.988603] 1f00: 00028e55 00000000 ac4ba180 ac43a7a0 00000015 c0c01f30 c0717e48 c0717e4c
[719448.996821] 1f20: 90000013 ffffffff
[719449.005071] [<c0301a8c>] (__irq_svc) from [<c0717e4c>] (cpuidle_enter_state+0x94/0x498)
[719449.008803] [<c0717e4c>] (cpuidle_enter_state) from [<c0718294>] (cpuidle_enter+0x30/0x4c)
[719449.016877] [<c0718294>] (cpuidle_enter) from [<c034a804>] (do_idle+0x1d8/0x240)
[719449.024946] [<c034a804>] (do_idle) from [<c034ab14>] (cpu_startup_entry+0x1c/0x20)
[719449.032585] [<c034ab14>] (cpu_startup_entry) from [<c0b00e5c>] (start_kernel+0x4dc/0x4ec)
[719449.040156] ---[ end trace 1e152b8707d44ab9 ]--- 

and

[95637.387820] ------------[ cut here ]------------
[95637.387972] WARNING: CPU: 0 PID: 0 at backports-5.8-rc2-1/net/mac80211/sta_info.c:1929 ieee80211_sta_update_pending_airtime+0x1f8/0x1fc [mac80211]
[95637.391604] STA f0:c3:71:10:fa:85 AC 3 txq pending airtime underflow: 4294966324, 972
[95637.391608] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT rndis_host pppox ppp_generic oti6858 nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mac80211 ipt_REJECT huawei_cdc_ncm ebtable_nat ebtable_filter ebtable_broute ch341 cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial usbnet slhc sch_cake nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat cdc_wdm br_netfilter fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route
[95637.404723]  cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsd nfs msdos ifb rpcsec_gss_krb5 auth_rpcgss oid_registry tun snd_rawmidi snd_seq_device snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd_hwdep snd_compress snd soundcore vfat fat ntfs lockd sunrpc grace hfsplus hfs autofs4 dns_resolver dm_mirror dm_region_hash dm_log dm_crypt nls_utf8 nls_iso8859_1 nls_cp437 sha1_generic md5 ecb des_generic libdes cts cbc arc4 dm_mod dax uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd
[95637.477678]  phy_qcom_dwc3 ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug xfs libcrc32c ext4 mbcache jbd2 crc32c_generic
[95637.586753] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.52 #0
[95637.604955] Hardware name: Generic DT based system
[95637.610790] [<c030f954>] (unwind_backtrace) from [<c030b96c>] (show_stack+0x14/0x20)
[95637.615644] [<c030b96c>] (show_stack) from [<c08bbe00>] (dump_stack+0x94/0xa8)
[95637.623550] [<c08bbe00>] (dump_stack) from [<c031e7b4>] (__warn+0xb4/0xd0)
[95637.630576] [<c031e7b4>] (__warn) from [<c031e850>] (warn_slowpath_fmt+0x80/0x90)
[95637.637543] [<c031e850>] (warn_slowpath_fmt) from [<bf829b4c>] (ieee80211_sta_update_pending_airtime+0x1f8/0x1fc [mac80211])
[95637.645176] [<bf829b4c>] (ieee80211_sta_update_pending_airtime [mac80211]) from [<bf824174>] (ieee80211_report_low_ack+0x250/0x514 [mac80211])
[95637.656394] [<bf824174>] (ieee80211_report_low_ack [mac80211]) from [<bf82444c>] (ieee80211_free_txskb+0x14/0x2c [mac80211])
[95637.668985] [<bf82444c>] (ieee80211_free_txskb [mac80211]) from [<bf9738fc>] (ath10k_txrx_tx_unref+0x608/0x738 [ath10k_core])
[95637.680340] [<bf9738fc>] (ath10k_txrx_tx_unref [ath10k_core]) from [<bf96d8a4>] (ath10k_htt_t2h_msg_handler+0xe74/0x1290 [ath10k_core])
[95637.691476] [<bf96d8a4>] (ath10k_htt_t2h_msg_handler [ath10k_core]) from [<bf9c8768>] (ath10k_pci_htt_rx_cb+0x178/0x230 [ath10k_pci])
[95637.703486] [<bf9c8768>] (ath10k_pci_htt_rx_cb [ath10k_pci]) from [<bf98d31c>] (ath10k_ce_per_engine_service+0x9c/0x10c [ath10k_core])
[95637.715633] [<bf98d31c>] (ath10k_ce_per_engine_service [ath10k_core]) from [<bf98d40c>] (ath10k_ce_per_engine_service_any+0x80/0xd8 [ath10k_core])
[95637.727585] [<bf98d40c>] (ath10k_ce_per_engine_service_any [ath10k_core]) from [<bf9ca104>] (ath10k_pci_napi_poll+0x54/0x15c [ath10k_pci])
[95637.740660] [<bf9ca104>] (ath10k_pci_napi_poll [ath10k_pci]) from [<c0775084>] (net_rx_action+0x118/0x374)
[95637.753064] [<c0775084>] (net_rx_action) from [<c0302298>] (__do_softirq+0x130/0x2d4)
[95637.762693] [<c0302298>] (__do_softirq) from [<c0322c10>] (irq_exit+0xbc/0xe0)
[95637.770590] [<c0322c10>] (irq_exit) from [<c036e238>] (__handle_domain_irq+0x6c/0xd0)
[95637.777716] [<c036e238>] (__handle_domain_irq) from [<c05c78e0>] (gic_handle_irq+0x5c/0xb8)
[95637.785609] [<c05c78e0>] (gic_handle_irq) from [<c0301a8c>] (__irq_svc+0x6c/0x90)
[95637.793761] Exception stack(0xc0c01ee0 to 0xc0c01f28)
[95637.801416] 1ee0: 00000000 000056fb 1d054000 ddb93a40 dcc1e000 00000000 ddb92e30 000056fb
[95637.806454] 1f00: 000056fb 00000000 50866e40 50863a60 00000015 c0c01f30 c0717e48 c0717e4c
[95637.814600] 1f20: 00000013 ffffffff
[95637.822758] [<c0301a8c>] (__irq_svc) from [<c0717e4c>] (cpuidle_enter_state+0x94/0x498)
[95637.826060] [<c0717e4c>] (cpuidle_enter_state) from [<c0718294>] (cpuidle_enter+0x30/0x4c)
[95637.834046] [<c0718294>] (cpuidle_enter) from [<c034a804>] (do_idle+0x1d8/0x240)
[95637.842378] [<c034a804>] (do_idle) from [<c034ab14>] (cpu_startup_entry+0x1c/0x20)
[95637.849931] [<c034ab14>] (cpu_startup_entry) from [<c0b00e5c>] (start_kernel+0x4dc/0x4ec)
[95637.857404] ---[ end trace 5847a187770c0975 ]--- 

Not sure this is helpful, but thought I would share it ...

How recent is your snapshot? Try compiling a new image since this patch seems to address your issue.

Well spotted and thanks for that.

I am updating my snapshot now. I really don't understand why the router crashes as often as it does, but as this error repeated I thought I would post it.

The other couple of examples I have are rather less reliable.

I have an example of "swap_info_get: Bad swap file entry" (but I don't use any swap) and "WARNING: CPU: 1 PID: 20989 at drivers/clk/clk.c:190 clk_enable_unlock+0xd8/0x100" (PID is for a kworker process).

The swap message in particular suggests to me that there must be some sort of overflow, or that a block of memory is being re-used after it is freed.