I can echo sentiments above that with my testing on the RT3200 the latency on 2.4 is much worse than on 5. Any idea why? Will this get improved with the recent developments? And will these ultimately get out into 22.03?
Actually I just realised all my work in getting low latency despite LTE is undone for WiFi clients given the latency variation they see.
After patches in version 1 and version 2 I have bugs and not usable device- according to post. After newest commit I have more errors than older (i.e. many in 24h vs one-two in 24 hours).
I will be testing @nbd patch, but this is only surely a temporary prosthesis 
.
Please just test and report back if your willing and able. Too many questions like these at this point in development have a tendency to drive the "devs" quiet and away from the forums.
If you look back 1-2 years ago in this thread, I (and others) have posted here and elsewhere in the forums about "poor" wifi quality/latency issues that seem to have started about the time AQL/ATF was implemented. I now know that not all the issues I observe result from AQL/ATF; but the recent events are the first time any real attention has been put to investigating if AQL/ATF might be a contributing factor.
After tests @nbd patch... no work for ea6350v3- many restarts, after add patch to mac80211 and ath10-ct...
Files of copy before add patches of airtime fairness have been restored and so far there is silence in the log and router worked normal.
My R7800 rebooted twice after I applied the patch. Managed to get this from the router.
<1>[ 2106.043670] 8<--- cut here ---
<1>[ 2106.043701] Unable to handle kernel NULL pointer dereference at virtual address 00000001
<1>[ 2106.045646] pgd = 6e5d9f1c
<1>[ 2106.053956] [00000001] *pgd=00000000
<0>[ 2106.056386] Internal error: Oops: 5 [#1] SMP ARM
<4>[ 2106.060112] Modules linked in: ecm iptable_nat ath10k_pci ath10k_core ath xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD wireguard nf_nat nf_flow_table_hw nf_flow_table nf_conntrack mac80211 libchacha20poly1305 libblake2s ipt_REJECT ebtable_nat ebtable_filter ebtable_broute curve25519_neon cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_quota xt_pkttype xt_physdev xt_owner xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_addrtype xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY ppp_async poly1305_arm nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 macvlan libcurve25519_generic libblake2s_generic l2tp_ppp iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat chacha_neon sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact
<4>[ 2106.060313] qca_nss_tunipip6 qca_nss_tun6rd qca_nss_ipsecmgr qca_nss_cfi_cryptoapi qca_nss_qdisc qca_nss_crypto qca_nss_vlan qca_nss_pppoe pppoe pppox ppp_generic slhc qca_nss_gre qca_nss_bridge_mgr ledtrig_usbport xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 qca_mcs bonding ip6_gre ip_gre gre sit qca_nss_drv l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 ipip ip6_tunnel qca_nss_gmac tunnel6 tunnel4 ip_tunnel tun qca_ssdk xfrm_user xfrm_ipcomp af_key xfrm_algo shortcut_fe_drv shortcut_fe_ipv6 shortcut_fe sha1_generic md5 echainiv des_generic libdes cbc authenc
<4>[ 2106.129562] usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 crc32c_generic
<4>[ 2106.241792] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.194 #0
<4>[ 2106.264020] Hardware name: Generic DT based system
<4>[ 2106.270383] PC is at ieee80211_next_txq+0x88/0x368 [mac80211]
<4>[ 2106.274929] LR is at ieee80211_next_txq+0x170/0x368 [mac80211]
<4>[ 2106.280689] pc : [<bf943bd8>] lr : [<bf943cc0>] psr: 20000113
<4>[ 2106.286417] sp : c0c01cb8 ip : 00000000 fp : 00000002
<4>[ 2106.292577] r10: 00000000 r9 : d98f0df4 r8 : d98f0c40
<4>[ 2106.297786] r7 : d98f0d00 r6 : 00000001 r5 : d936f2b4 r4 : 00000001
<4>[ 2106.302997] r3 : 00000001 r2 : d98f0df4 r1 : d936f2b4 r0 : 00000000
<4>[ 2106.309598] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
<4>[ 2106.316107] Control: 10c5787d Table: 5adc806a DAC: 00000051
<0>[ 2106.323310] Process swapper/0 (pid: 0, stack limit = 0x94cfd9bc)
<0>[ 2106.329038] Stack: (0xc0c01cb8 to 0xc0c02000)
<0>[ 2106.335119] 1ca0: 00000006 00000030
<0>[ 2106.339390] 1cc0: 00000004 d98f0df0 597a9523 000001ea 00000ab4 d98f0c40 00000000 00000002
<0>[ 2106.347548] 1ce0: 00000000 00000002 00000ab4 d98f2020 c0c01e18 bfa30e00 d98f2a7c d98f63e0
<0>[ 2106.355708] 1d00: 00000040 d98f23e0 00000040 1ce4e000 d98f2020 bfa4285c d98f2020 dd9c1000
<0>[ 2106.363867] 1d20: 00000440 00000046 d98f2940 00000000 0004b400 bfa5d834 c0c01d60 d98f2020
<0>[ 2106.372027] 1d40: c0c01d60 c09c98d8 00000000 00000001 d98f6748 d98f2a9c d98f2a60 00000010
<0>[ 2106.380186] 1d60: c0c01d60 c0c01d60 00000000 00000040 00001d94 00000000 c0c01d78 c0c01d78
<0>[ 2106.388346] 1d80: 00000000 d98f6740 0004b400 bfa93348 c0c01e18 bfa5de3c 00000006 00000000
<0>[ 2106.396505] 1da0: d98f2020 00000001 00000040 d98f2020 d98f63e0 00000040 d98f23e0 00000040
<0>[ 2106.404665] 1dc0: 1ce4e000 c0c03d00 c0c01e18 bfa90b28 00000001 d98f63e0 00000040 c0c01e0f
<0>[ 2106.412825] 1de0: c0c01e10 c0793210 d98f63e0 c0b42400 0000012c dd990400 c0c01e10 c0793564
<0>[ 2106.420985] 1e00: c0c01e14 0002c17e 00000001 0066009c c0c01e10 c0c01e10 c0c01e18 c0c01e18
<0>[ 2106.429144] 1e20: dd654e58 00000000 00000003 c0c0308c c0c03080 40000003 ffffe000 00000100
<0>[ 2106.437304] 1e40: c0c03080 c0302318 dca55c68 c0370e80 dca55c00 c0b3b210 c0b41640 0000000a
<0>[ 2106.445464] 1e60: c0b3b19c 0002c17d c0c03d00 c09caffc 00200002 c09cafc4 c0c05250 ffffe000
<0>[ 2106.453625] 1e80: 00000000 00000001 00000000 dd410000 de803000 59f87b00 59f6cbc0 c03232d0
<0>[ 2106.461784] 1ea0: c0b41614 c0370438 c0c05250 c0c19100 de80200c de802000 c0c01ee0 c05e27a8
<0>[ 2106.469944] 1ec0: c0734be4 20000013 ffffffff c0c01f14 000001ea c0c00000 59f87b00 c0301b0c
<0>[ 2106.478103] 1ee0: 00000000 000001ea 1ce4e000 dd98fa80 dd7f7000 00000000 dd98ee30 000001ea
<0>[ 2106.486262] 1f00: 000001ea 00000000 59f87b00 59f6cbc0 00000015 c0c01f30 c0734be0 c0734be4
<0>[ 2106.494423] 1f20: 20000013 ffffffff 00000051 00000000 0006d560 00000000 dd7f7000 dd98f10c
<0>[ 2106.502582] 1f40: dd98ee30 77359400 c0c01f8f dd98ee30 dd7f7000 00000000 00000001 dd98ee30
<0>[ 2106.510742] 1f60: c0b40e28 c0c0cd60 dd7f7000 c073502c ffffe000 c0c04e68 c0c04ea8 c034b5d4
<0>[ 2106.518903] 1f80: 60000013 00000000 00000051 01000000 10c5387d 000000cf c0c04e40 00000000
<0>[ 2106.527061] 1fa0: c0c3202c 00000001 c0b2ea58 10c5387d 00000000 c034b8e4 c0c32044 c0b00fc8
<0>[ 2106.535221] 1fc0: ffffffff ffffffff 00000000 c0b006cc c0b2ea58 3cc023d3 00000000 c0b00470
<0>[ 2106.543380] 1fe0: 00000051 10c0387d 0000136c 42ef6a28 512f04d0 00000000 00000000 00000000
<4>[ 2106.551733] [<bf943bd8>] (ieee80211_next_txq [mac80211]) from [<bfa30e00>] (ath10k_mac_tx_push_pending+0xa0/0xfc [ath10k_core])
<4>[ 2106.559796] [<bfa30e00>] (ath10k_mac_tx_push_pending [ath10k_core]) from [<bfa4285c>] (ath10k_htt_txrx_compl_task+0x804/0x10a0 [ath10k_core])
<4>[ 2106.571022] [<bfa4285c>] (ath10k_htt_txrx_compl_task [ath10k_core]) from [<bfa90b28>] (ath10k_pci_napi_poll+0x5c/0x114 [ath10k_pci])
<4>[ 2106.583876] [<bfa90b28>] (ath10k_pci_napi_poll [ath10k_pci]) from [<c0793210>] (__napi_poll+0x34/0x168)
<4>[ 2106.595808] [<c0793210>] (__napi_poll) from [<c0793564>] (net_rx_action+0xd8/0x21c)
<4>[ 2106.604918] [<c0793564>] (net_rx_action) from [<c0302318>] (__do_softirq+0x130/0x2d4)
<4>[ 2106.612555] [<c0302318>] (__do_softirq) from [<c03232d0>] (irq_exit+0xbc/0xe0)
<4>[ 2106.620541] [<c03232d0>] (irq_exit) from [<c0370438>] (__handle_domain_irq+0x6c/0xd0)
<4>[ 2106.627669] [<c0370438>] (__handle_domain_irq) from [<c05e27a8>] (gic_handle_irq+0x5c/0xb8)
<4>[ 2106.635560] [<c05e27a8>] (gic_handle_irq) from [<c0301b0c>] (__irq_svc+0x6c/0x90)
<4>[ 2106.643711] Exception stack(0xc0c01ee0 to 0xc0c01f28)
<4>[ 2106.651363] 1ee0: 00000000 000001ea 1ce4e000 dd98fa80 dd7f7000 00000000 dd98ee30 000001ea
<4>[ 2106.656402] 1f00: 000001ea 00000000 59f87b00 59f6cbc0 00000015 c0c01f30 c0734be0 c0734be4
<4>[ 2106.664549] 1f20: 20000013 ffffffff
<4>[ 2106.672712] [<c0301b0c>] (__irq_svc) from [<c0734be4>] (cpuidle_enter_state+0x94/0x498)
<4>[ 2106.676011] [<c0734be4>] (cpuidle_enter_state) from [<c073502c>] (cpuidle_enter+0x30/0x4c)
<4>[ 2106.683998] [<c073502c>] (cpuidle_enter) from [<c034b5d4>] (do_idle+0x1d8/0x240)
<4>[ 2106.692327] [<c034b5d4>] (do_idle) from [<c034b8e4>] (cpu_startup_entry+0x1c/0x20)
<4>[ 2106.699882] [<c034b8e4>] (cpu_startup_entry) from [<c0b00fc8>] (start_kernel+0x4dc/0x4ec)
<0>[ 2106.707266] Code: e3540000 0a000078 e1a03004 e1a02009 (e4931004)
<4>[ 2106.715603] ---[ end trace 2c6f6a606a0a2ee8 ]---
Hi all,
Openwrt 22.03
IPQ8064, QCA9984, QCA9980
256MB RAM
4x4 for both 2.4GHz and 5GHz radios
ath10k-ct 5.15
Patch1: https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=96012227e578a0d8dcfa86823db97345e98e2c8f
Patch 2: https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=958785508cc802de75698607670f9a236883d24d
Patch 3: ndb's recent patch: https://pastebin.com/LqdrpxCJ
I have a fairly robust test setup and it's reproducible. Running 20 clients with iperf3 and the AP is the DUT. I built a bare minimum version: No NSS, no routing/firewall, no dhcp (dnsmasq), no VLANs. Have it setup as a dumb AP.
Test 1:
Test 2:
Test 3:
Test 4:
Test 5:
Test 1 with Patch 1 and 2 Crash report
[ 139.747915] netifd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[ 139.747951] CPU: 0 PID: 679 Comm: netifd Not tainted 5.10.120 #0
[ 139.756454] Hardware name: Generic DT based system
[ 139.762457] [<c030e54c>] (unwind_backtrace) from [<c030a2d0>] (show_stack+0x14/0x20)
[ 139.767057] [<c030a2d0>] (show_stack) from [<c061b42c>] (dump_stack+0x94/0xa8)
[ 139.774956] [<c061b42c>] (dump_stack) from [<c041438c>] (dump_header+0x58/0x1a8)
[ 139.781983] [<c041438c>] (dump_header) from [<c0414b9c>] (oom_kill_process+0x1ec/0x1f0)
[ 139.789534] [<c0414b9c>] (oom_kill_process) from [<c0415584>] (out_of_memory+0x194/0x358)
[ 139.797268] [<c0415584>] (out_of_memory) from [<c0458dbc>] (__alloc_pages_nodemask+0x9ac/0xed4)
[ 139.805598] [<c0458dbc>] (__alloc_pages_nodemask) from [<c0411414>] (pagecache_get_page+0x17c/0x38c)
[ 139.814102] [<c0411414>] (pagecache_get_page) from [<c0412ea0>] (filemap_fault+0x6fc/0x8a0)
[ 139.823481] [<c0412ea0>] (filemap_fault) from [<c043cdac>] (__do_fault+0x44/0x140)
[ 139.831550] [<c043cdac>] (__do_fault) from [<c0441c34>] (handle_mm_fault+0xad4/0x1378)
[ 139.839191] [<c0441c34>] (handle_mm_fault) from [<c030f740>] (do_page_fault+0x11c/0x290)
[ 139.847089] [<c030f740>] (do_page_fault) from [<c030fba8>] (do_PrefetchAbort+0x48/0xac)
[ 139.855334] [<c030fba8>] (do_PrefetchAbort) from [<c0301188>] (ret_from_exception+0x0/0x18)
[ 139.863056] Exception stack(0xc26a5fb0 to 0xc26a5ff8)
[ 139.871390] 5fa0: 00000000 b6f35268 0000000a 00000351
[ 139.876608] 5fc0: 00000004 b6f35268 0000000a 00000351 00000000 00000000 0000201d 00000001
[ 139.884764] 5fe0: beae1d40 beae1d68 b6f8a350 b6f5c7c0 60000010 ffffffff
[ 139.893154] Mem-Info:
[ 139.899437] active_anon:38 inactive_anon:1334 isolated_anon:0
[ 139.899437] active_file:1 inactive_file:3 isolated_file:0
[ 139.899437] unevictable:0 dirty:0 writeback:0
[ 139.899437] slab_reclaimable:1020 slab_unreclaimable:3640
[ 139.899437] mapped:4 shmem:15 pagetables:97 bounce:0
[ 139.899437] free:3911 free_pcp:1 free_cma:0
[ 139.910011] Node 0 active_anon:152kB inactive_anon:5336kB active_file:4kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:16kB dirty:0kB writeback:0kB shmem:60kB writeback_tmp:0kB kernel_stack:760kB all_unreclaimable? yes
[ 139.935206] Normal free:41440kB min:20480kB low:24576kB high:28672kB reserved_highatomic:0KB active_anon:152kB inactive_anon:5336kB active_file:76kB inactive_file:156kB unevictable:0kB writepending:0kB present:229376kB managed:216724kB mlocked:0kB pagetables:388kB bounce:0kB free_pcp:108kB local_pcp:4kB free_cma:0kB
[ 139.965408] lowmem_reserve[]: 0 0 0
[ 139.984622] Normal: 345*4kB (UME) 254*8kB (UME) 141*16kB (UME) 768*32kB (UME) 126*64kB (UME) 11*128kB (UM) 5*256kB (M) 2*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 42020kB
[ 139.988556] 125 total pagecache pages
[ 140.002662] 57344 pages RAM
[ 140.006164] 0 pages HighMem/MovableOnly
[ 140.008845] 3163 pages reserved
[ 140.012586] Tasks state (memory values in pages):
[ 140.015713] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 140.020722] [ 182] 81 182 269 22 10240 0 0 ubusd
[ 140.029199] [ 183] 0 183 291 11 10240 0 0 ash
[ 140.037368] [ 218] 0 218 216 13 8192 0 0 urngd
[ 140.045231] [ 505] 514 505 387 15 10240 0 0 logd
[ 140.053568] [ 506] 0 506 306 23 8192 0 0 logread
[ 140.061712] [ 615] 0 615 602 12 8192 0 0 hostapd
[ 140.069798] [ 637] 101 637 1163 127 14336 0 0 hostapd
[ 140.078079] [ 679] 0 679 409 44 10240 0 0 netifd
[ 140.090760] [ 792] 0 792 907 82 12288 0 -1000 sshd
[ 140.102889] [ 959] 0 959 913 316 12288 0 0 lua
[ 140.102963] [ 1079] 0 1079 289 6 10240 0 0 udhcpc
[ 140.113832] [ 1153] 0 1153 602 24 12288 0 0 ntpd
[ 140.121581] [ 1182] 123 1182 289 6 8192 0 0 ntpd
[ 140.127288] [ 1210] 0 1210 1529 320 14336 0 0 lua
[ 140.143122] [ 1983] 0 1983 989 163 12288 0 0 sshd
[ 140.151464] [ 1985] 0 1985 291 12 10240 0 0 ash
[ 140.159804] [ 1994] 0 1994 292 14 8192 0 0 top
[ 140.167458] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=lua,pid=1210,uid=0
[ 140.175573] Out of memory: Killed process 1210 (lua) total-vm:6116kB, anon-rss:1280kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:14kB oom_score_adj:0
Test 1 with Patch 3 Crash report
00:19 root:~# [ 176.196324] ath10k_pci 0000:01:00.0: Invalid VHT mcs 15 peer stats
[ 189.766760] kworker/1:7: page allocation failure: order:0, mode:0x800(GFP_NOWAIT), nodemask=(null),cpuset=/,mems_allowed=0
[ 189.766816] CPU: 1 PID: 94 Comm: kworker/1:7 Not tainted 5.10.120 #0
[ 189.776710] Hardware name: Generic DT based system
[ 189.783234] Workqueue: ubiblock0_1 ubiblock_do_work
[ 189.787835] [<c030e54c>] (unwind_backtrace) from [<c030a2d0>] (show_stack+0x14/0x20)
[ 189.792608] [<c030a2d0>] (show_stack) from [<c061b42c>] (dump_stack+0x94/0xa8)
[ 189.800602] [<c061b42c>] (dump_stack) from [<c045838c>] (warn_alloc+0xd4/0x158)
[ 189.807624] [<c045838c>] (warn_alloc) from [<c0459228>] (__alloc_pages_nodemask+0xe18/0xed4)
[ 189.814835] [<c0459228>] (__alloc_pages_nodemask) from [<c045e710>] (allocate_slab+0x1f0/0x264)
[ 189.823512] [<c045e710>] (allocate_slab) from [<c04607cc>] (___slab_alloc.constprop.0+0x340/0x4e0)
[ 189.831929] [<c04607cc>] (___slab_alloc.constprop.0) from [<c04611b4>] (kmem_cache_alloc+0x1f0/0x2f0)
[ 189.840962] [<c04611b4>] (kmem_cache_alloc) from [<c0687584>] (adm_prep_slave_sg+0x184/0x68c)
[ 189.850248] [<c0687584>] (adm_prep_slave_sg) from [<c0704d0c>] (prep_adm_dma_desc+0x128/0x220)
[ 189.858752] [<c0704d0c>] (prep_adm_dma_desc) from [<c07061f0>] (read_reg_dma+0x6c/0xa0)
[ 189.867258] [<c07061f0>] (read_reg_dma) from [<c0706304>] (config_nand_cw_read+0x68/0xb0)
[ 189.875158] [<c0706304>] (config_nand_cw_read) from [<c07071d0>] (read_page_ecc+0xf8/0x580)
[ 189.883495] [<c07071d0>] (read_page_ecc) from [<c06f6f64>] (nand_read_oob+0x1d4/0x790)
[ 189.891655] [<c06f6f64>] (nand_read_oob) from [<c06df968>] (mtd_read_oob+0x94/0x168)
[ 189.899638] [<c06df968>] (mtd_read_oob) from [<c06dfa7c>] (mtd_read+0x40/0x5c)
[ 189.907544] [<c06dfa7c>] (mtd_read) from [<c0717d00>] (ubi_io_read+0xd4/0x368)
[ 189.914570] [<c0717d00>] (ubi_io_read) from [<c0715a30>] (ubi_eba_read_leb+0xa8/0x3e0)
[ 189.921775] [<c0715a30>] (ubi_eba_read_leb) from [<c0715dec>] (ubi_eba_read_leb_sg+0x84/0x194)
[ 189.929672] [<c0715dec>] (ubi_eba_read_leb_sg) from [<c0714444>] (ubi_leb_read_sg+0x94/0xd4)
[ 189.938271] [<c0714444>] (ubi_leb_read_sg) from [<c0720000>] (ubiblock_do_work+0x94/0x134)
[ 189.946866] [<c0720000>] (ubiblock_do_work) from [<c0338c00>] (process_one_work+0x1fc/0x474)
[ 189.954936] [<c0338c00>] (process_one_work) from [<c0338eec>] (worker_thread+0x74/0x5d4)
[ 189.963529] [<c0338eec>] (worker_thread) from [<c033edd4>] (kthread+0x15c/0x160)
[ 189.971597] [<c033edd4>] (kthread) from [<c0300148>] (ret_from_fork+0x14/0x2c)
[ 189.978969] Exception stack(0xc1f69fb0 to 0xc1f69ff8)
[ 189.986009] 9fa0: 00000000 00000000 00000000 00000000
[ 189.991136] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 189.999295] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 190.007443] Mem-Info:
[ 190.013864] active_anon:35 inactive_anon:1267 isolated_anon:0
[ 190.013864] active_file:0 inactive_file:97 isolated_file:0
[ 190.013864] unevictable:0 dirty:0 writeback:0
[ 190.013864] slab_reclaimable:907 slab_unreclaimable:3599
[ 190.013864] mapped:17 shmem:15 pagetables:95 bounce:0
[ 190.013864] free:4583 free_pcp:52 free_cma:0
[ 190.024446] Node 0 active_anon:140kB inactive_anon:5068kB active_file:0kB inactive_file:388kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:68kB dirty:0kB writeback:0kB shmem:60kB writeback_tmp:0kB kernel_stack:720kB all_unreclaimable? yes
[ 190.047020] Normal free:47520kB min:24576kB low:28672kB high:32768kB reserved_highatomic:0KB active_anon:140kB inactive_anon:5068kB active_file:200kB inactive_file:4kB unevictable:0kB writepending:0kB present:229376kB managed:216724kB mlocked:0kB pagetables:380kB bounce:0kB free_pcp:264kB local_pcp:64kB free_cma:0kB
[ 190.075048] lowmem_reserve[]: 0 0 0
[ 190.097278] Normal: 330*4kB (UME) 276*8kB (UME) 153*16kB (UME) 852*32kB (UME) 186*64kB (UME) 12*128kB (UM) 6*256kB (UME) 2*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 49240kB
[ 190.100593] 120 total pagecache pages
[ 190.115856] 57344 pages RAM
[ 190.119585] 0 pages HighMem/MovableOnly
[ 190.122191] 3163 pages reserved
[ 190.126803] 8<--- cut here ---
[ 190.129211] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[ 190.132271] pgd = 43db503e
[ 190.140587] [00000004] *pgd=00000000
[ 190.143031] Internal error: Oops: 17 [#1] SMP ARM
[ 190.146759] Modules linked in: nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath wireguard nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 libchacha20poly1305 curve25519_neon cfg80211 poly1305_arm nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcurve25519_generic libcrc32c crc_ccitt compat chacha_neon ledtrig_usbport ledtrig_oneshot ip6_udp_tunnel udp_tunnel seqiv cmac leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd ledtrig_transient phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug crc32c_generic
[ 190.214722] CPU: 1 PID: 94 Comm: kworker/1:7 Not tainted 5.10.120 #0
[ 190.236952] Hardware name: Generic DT based system
[ 190.243478] Workqueue: ubiblock0_1 ubiblock_do_work
[ 190.248065] PC is at submit_descs+0x64/0x1a4
[ 190.252841] LR is at vchan_tx_submit+0x74/0x88
[ 190.257349] pc : [<c0705a7c>] lr : [<c06862d4>] psr: 20000113
[ 190.261604] sp : c1f69c98 ip : c4477038 fp : 000001f4
[ 190.267766] r10: ffffffff r9 : c1dc3800 r8 : c12c0f40
[ 190.272974] r7 : 00000000 r6 : c12c0f40 r5 : c12c0f84 r4 : c2e65ec0
[ 190.278187] r3 : 0000a51c r2 : c442cf38 r1 : 20000113 r0 : fffffff4
[ 190.284784] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 190.291295] Control: 10c5787d Table: 44b7006a DAC: 00000051
[ 190.298502] Process kworker/1:7 (pid: 94, stack limit = 0x750446db)
[ 190.304228] Stack: (0xc1f69c98 to 0xc1f6a000)
[ 190.310309] 9c80: c1349040 00000000
[ 190.314833] 9ca0: 0000001b 00000004 c12c0f40 c0707248 00000000 20000000 00000000 00000000
[ 190.322997] 9cc0: 00000000 c1dc3000 00000000 0000328a 0000328a c1dc3000 c12c0f40 00000000
[ 190.331156] 9ce0: c1f69dc8 c1349040 c0707718 00000800 00000000 00000000 00000000 c1f69dc8
[ 190.339316] 9d00: c134911c c06f6f64 c10201c0 c1f69d78 00000000 00000000 c1dc3000 00000800
[ 190.347474] 9d20: 0000328a 00000000 00000000 c1dc3000 c1dc3000 00001000 00000000 0000328a
[ 190.355635] 9d40: 00000000 00000000 00000000 00000040 00000000 c0dda304 00000002 00000000
[ 190.363796] 9d60: 00000000 00000000 c1f69e68 c1349040 c136fc00 c1f69dc8 00000000 01945000
[ 190.371955] 9d80: 00000000 00000002 00000000 c06df968 c1f69dc8 00000000 00000000 00000000
[ 190.380115] 9da0: 00000000 00005000 000000ca 00001000 00000000 c0abedcc 01945000 c06dfa7c
[ 190.388274] 9dc0: c1f69dc8 00000c40 00000000 00001000 00000000 00000000 00000000 00000000
[ 190.396433] 9de0: c1dc3000 00000000 0000000a 00000004 c1908000 c0717d00 00001000 c1f69e1c
[ 190.404593] 9e00: c1dc3000 c1908000 c1b14400 c1dc3000 c0a48aac c0abed98 00000001 c0715070
[ 190.412753] 9e20: 0000000a 00000000 c1908000 c1b14400 0000000a c1dc3000 000000ca 00000001
[ 190.420914] 9e40: 0000000a c0715a30 00001000 cdde0d40 c1ea9d40 0000002c 2cc4bfa0 c1b39418
[ 190.429073] 9e60: c1b393f8 00001000 0001b000 00000003 c1908000 c1b14400 0000000a c0715dec
[ 190.437233] 9e80: 00004000 00001000 00000000 00000000 00000400 c1f68000 00000000 00000000
[ 190.445392] 9ea0: 00002c00 c1b14400 c1b393f8 0001c400 c1b39340 c1908000 00000001 c0714444
[ 190.453552] 9ec0: 00005000 0001c400 00000000 cdde0900 ff7f8e00 0001f000 00002c00 0001c800
[ 190.461713] 9ee0: 0000000b c1b393f8 c1b39340 c1b28000 00000000 c0720000 0001c400 00000000
[ 190.469870] 9f00: c0c46d00 c1b395d0 c1f69f44 c1b393e8 c1ebe900 cdde0900 ff7f8e00 00000000
[ 190.478031] 9f20: 00000040 00000000 c1f68000 c0338c00 00000008 cdde0918 c1ebe900 c1ebe914
[ 190.486191] 9f40: cdde0900 00000008 cdde0918 c0d03d00 cdde0ac0 c0338eec c0daeb48 c0d0c02c
[ 190.494351] 9f60: c1ebe900 c1ebf240 c1ebfc00 00000000 c1f68000 c0338e78 c1ebe900 c1eb1ec4
[ 190.502510] 9f80: c1ebf264 c033edd4 00000000 c1ebfc00 c033ec78 00000000 00000000 00000000
[ 190.510670] 9fa0: 00000000 00000000 00000000 c0300148 00000000 00000000 00000000 00000000
[ 190.518830] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 190.526989] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 190.535146] [<c0705a7c>] (submit_descs) from [<c0707248>] (read_page_ecc+0x170/0x580)
[ 190.543304] [<c0707248>] (read_page_ecc) from [<c06f6f64>] (nand_read_oob+0x1d4/0x790)
[ 190.551116] [<c06f6f64>] (nand_read_oob) from [<c06df968>] (mtd_read_oob+0x94/0x168)
[ 190.558926] [<c06df968>] (mtd_read_oob) from [<c06dfa7c>] (mtd_read+0x40/0x5c)
[ 190.566830] [<c06dfa7c>] (mtd_read) from [<c0717d00>] (ubi_io_read+0xd4/0x368)
[ 190.573857] [<c0717d00>] (ubi_io_read) from [<c0715a30>] (ubi_eba_read_leb+0xa8/0x3e0)
[ 190.581063] [<c0715a30>] (ubi_eba_read_leb) from [<c0715dec>] (ubi_eba_read_leb_sg+0x84/0x194)
[ 190.588961] [<c0715dec>] (ubi_eba_read_leb_sg) from [<c0714444>] (ubi_leb_read_sg+0x94/0xd4)
[ 190.597556] [<c0714444>] (ubi_leb_read_sg) from [<c0720000>] (ubiblock_do_work+0x94/0x134)
[ 190.606155] [<c0720000>] (ubiblock_do_work) from [<c0338c00>] (process_one_work+0x1fc/0x474)
[ 190.614222] [<c0338c00>] (process_one_work) from [<c0338eec>] (worker_thread+0x74/0x5d4)
[ 190.622820] [<c0338eec>] (worker_thread) from [<c033edd4>] (kthread+0x15c/0x160)
[ 190.630887] [<c033edd4>] (kthread) from [<c0300148>] (ret_from_fork+0x14/0x2c)
[ 190.638258] Exception stack(0xc1f69fb0 to 0xc1f69ff8)
[ 190.645295] 9fa0: 00000000 00000000 00000000 00000000
[ 190.650424] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 190.658582] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 190.666743] Code: e1550004 03a00000 0a000005 e594001c (e5903010)
[ 190.673255] ---[ end trace b7271a14c5af6e76 ]---
[ 190.679476] Kernel panic - not syncing: Fatal exception
[ 190.684096] CPU0: stopping
[ 190.689037] CPU: 0 PID: 10 Comm: ksoftirqd/0 Tainted: G D 5.10.120 #0
[ 190.691815] Hardware name: Generic DT based system
[ 190.699725] [<c030e54c>] (unwind_backtrace) from [<c030a2d0>] (show_stack+0x14/0x20)
[ 190.704323] [<c030a2d0>] (show_stack) from [<c061b42c>] (dump_stack+0x94/0xa8)
[ 190.712220] [<c061b42c>] (dump_stack) from [<c030d270>] (do_handle_IPI+0x140/0x184)
[ 190.719249] [<c030d270>] (do_handle_IPI) from [<c030d2d0>] (ipi_handler+0x1c/0x2c)
[ 190.726804] [<c030d2d0>] (ipi_handler) from [<c0371c20>] (__handle_domain_irq+0x90/0xf4)
[ 190.734446] [<c0371c20>] (__handle_domain_irq) from [<c06346d4>] (gic_handle_irq+0x90/0xb8)
[ 190.742689] [<c06346d4>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[ 190.750757] Exception stack(0xc1053a80 to 0xc1053ac8)
[ 190.758404] 3a80: bf238820 c3c79038 c423c844 0321f004 c4b41ab2 c40b8900 c52a6098 c1053af8
[ 190.763442] 3aa0: 0b188c6c c423c76c c52a6000 c2370cc0 80000000 c1053ad0 bf238820 bf239100
[ 190.771593] 3ac0: 60000113 ffffffff
[ 190.779848] [<c0300b8c>] (__irq_svc) from [<bf239100>] (ieee80211_tx_dequeue+0x9e8/0x11b4 [mac80211])
[ 190.783171] [<bf239100>] (ieee80211_tx_dequeue [mac80211]) from [<bf3f2334>] (ath10k_mac_tx_push_txq+0xa0/0x31c [ath10k_core])
[ 190.792496] [<bf3f2334>] (ath10k_mac_tx_push_txq [ath10k_core]) from [<bf3f2788>] (ath10k_mac_op_wake_tx_queue+0xcc/0xd8 [ath10k_core])
[ 190.803785] [<bf3f2788>] (ath10k_mac_op_wake_tx_queue [ath10k_core]) from [<bf23ba8c>] (ieee80211_schedule_txq+0x644/0x91c [mac80211])
[ 190.815868] [<bf23ba8c>] (ieee80211_schedule_txq [mac80211]) from [<bf23cf48>] (__ieee80211_subif_start_xmit+0x834/0xc20 [mac80211])
[ 190.828021] [<bf23cf48>] (__ieee80211_subif_start_xmit [mac80211]) from [<bf23d37c>] (ieee80211_subif_start_xmit+0x48/0x310 [mac80211])
[ 190.840053] [<bf23d37c>] (ieee80211_subif_start_xmit [mac80211]) from [<c080f7c4>] (dev_hard_start_xmit+0xf8/0x1a0)
[ 190.851896] [<c080f7c4>] (dev_hard_start_xmit) from [<c080fee0>] (__dev_queue_xmit+0x594/0x8f8)
[ 190.862317] [<c080fee0>] (__dev_queue_xmit) from [<c0961414>] (br_dev_queue_push_xmit+0x68/0x158)
[ 190.870991] [<c0961414>] (br_dev_queue_push_xmit) from [<c096381c>] (br_handle_frame_finish+0x24c/0x468)
[ 190.880017] [<c096381c>] (br_handle_frame_finish) from [<c0963e04>] (br_handle_frame+0x3cc/0x438)
[ 190.889567] [<c0963e04>] (br_handle_frame) from [<c08105e0>] (__netif_receive_skb_core.constprop.0+0x250/0xbfc)
[ 190.898336] [<c08105e0>] (__netif_receive_skb_core.constprop.0) from [<c0812b2c>] (__netif_receive_skb_one_core+0x28/0x58)
[ 190.908231] [<c0812b2c>] (__netif_receive_skb_one_core) from [<c0812de0>] (process_backlog+0x100/0x1e8)
[ 190.919338] [<c0812de0>] (process_backlog) from [<c0812498>] (__napi_poll+0x34/0x150)
[ 190.928627] [<c0812498>] (__napi_poll) from [<c08127b4>] (net_rx_action+0xdc/0x270)
[ 190.936612] [<c08127b4>] (net_rx_action) from [<c0301378>] (__do_softirq+0x110/0x2b8)
[ 190.944078] [<c0301378>] (__do_softirq) from [<c0323044>] (run_ksoftirqd+0x38/0x4c)
[ 190.952065] [<c0323044>] (run_ksoftirqd) from [<c0344634>] (smpboot_thread_fn+0xe0/0x1b0)
[ 190.959532] [<c0344634>] (smpboot_thread_fn) from [<c033edd4>] (kthread+0x15c/0x160)
[ 190.967862] [<c033edd4>] (kthread) from [<c0300148>] (ret_from_fork+0x14/0x2c)
[ 190.975668] Exception stack(0xc1053fb0 to 0xc1053ff8)
[ 190.982705] 3fa0: 00000000 00000000 00000000 00000000
[ 190.987833] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 190.995991] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 191.004150] Rebooting in 3 seconds..
Thanks for testing. I will revert the patches for now until I have a proper solution
You got it. I updated my post with more information
Hello uniqe13,
Can you point to me where I can get the copy of files before the patches and airtime fairness? I want to revert and test it out. Thanks!
Hi sjpacket,
I believe openwrt 21.02.1 is the version using the round-robin scheduler. Starting from 21.02.2 and any later version, the switch to virtual time-based airtime scheduler for ATF started causing all kinds of issues.
Ahh...got it. Thank you!
From e.g. stable version (first version 21.02 or 19.07) OpenWrt... But I have removed patches of 332 and 333 (330 and 331 left from patches v2) from mac80211 and compile. Now testing, and now without bugs is, but speed in 10 meters is probably worse than before any changes of airtime.
There are prior reports of latency issues that seem to be client specific (apple devices, also quarky reported running a fair amount of apple devices but is confident these issues are present on other OS/devices as well). Are your clients all one OS or a mix? I'm not looking for a complete list, just an overview.
My devices are a mix:
7 apple laptops (mix of m1 and intel with latest macOS) - mix of 2x2 and 3x3
3 iOS devices (ipad and iphone) - 2x2
5 Pi with latest raspbian - 1x1
5 linux machines (mix of usb-to-wifi dongles, and on board wifi ac) - mix of 3x3 and 4x4
sorry, edited the above count. Just counted all devices again.
nice test setup and testing btw. It might be interesting to try testing (without nbd's recent patches 1-3) without any apple devices connected just to see if there is a difference - but I'll leave that up to you.
I can test that first thing tomorrow 
Building that branch now.
For Apple devices mayble setting up DTIM perioid in settings of OpenWrt to 3 or 6 (from standard 2) for more stable connect (and mayble more stable pings propably). I not have Apple devices, I Android (mobile and on TV) here and PC testings.
This suggests to use DTIM >=3 for apple devices.
This suggests DTIM should not be set to >2:.
Yes, I this suggested....
Where I read somewhere that 6 gives good results (in layman's terms, a multiple of '3'; yes, I'm a farmer and gave up computer science many years ago, but I still enjoy tinkering for satisfaction, a minor aside 
).
Still no errors, but the speed is inconclusive (fast.com and speedtest.pl) show 2x higher than speedtest.net - that's how it's been for me lately and I'm leaning more towards the former sites. When I come back from work in the evening I will switch on the next of the patches for testing (unless they throw errors in log earlier).
Edit: 5-6 hours ago all patches (330, 331, 332, 333) have deleted in master tree- 1 and 2
In terms of bisecting this, I know when AQL worked - see the bug's start. I am also confused as to AQL+ATF, was it 22.02.1 -> 22.02.2 where things went to heck or 19.x ->