Ath10k-ct keeps crashing

I am having 2x2 wave 2 device which contains qca9888 chipset for 5 GHz. It is keep crashing. I have used CT driver for it. I have also used ath10k (non CT ) driver but it had issue where after connecting 8-10 clients on device, no new client was able to connect. My kernel is 4.9.152. when there are not wlan on 5GHz device works perfectly but after switching to 5GHz device acts unevenly.Sometimes device reaches to the state where all functions stop working no serial console,no wlan,no arp , no reboot,nothing. it is observed while running 5GHz radio. dmesg is attached here. Ref board is AP-152.

[ 3493.618596] ath10k_pci 0000:00:00.0: firmware crashed! (uuid 22ac6352-5e22-49a7-be3a-d65f0fd5be67)
[ 3493.627905] ath10k_pci 0000:00:00.0: qca9888 hw2.0 target 0x01000000 chip_id 0x00000000 sub 0000:0000
[ 3493.637448] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[ 3493.656849] ath10k_pci 0000:00:00.0: firmware ver 10.4-ct-9888-fW-011-cf79c7f api 5 features peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,cust-stats-CT crc32 0f624b45
[ 3493.681660] ath10k_pci 0000:00:00.0: board_file api 2 bmi_id 0:16 crc32 d65c17b1
[ 3493.689319] ath10k_pci 0000:00:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1
[ 3493.701772] ath10k_pci 0000:00:00.0: firmware register dump:
[ 3493.707631] ath10k_pci 0000:00:00.0: [00]: 0x0000000C 0x000015B3 0x009C10BA 0x00975B31
[ 3493.715817] ath10k_pci 0000:00:00.0: [04]: 0x009C10BA 0x00060F30 0x00000019 0x0043D210
[ 3493.723989] ath10k_pci 0000:00:00.0: [08]: 0x0043D520 0x0043E1F8 0x00000CC0 0x00424984
[ 3493.732168] ath10k_pci 0000:00:00.0: [12]: 0x00000009 0x00000000 0x009C10D6 0x009C10DA
[ 3493.740347] ath10k_pci 0000:00:00.0: [16]: 0x00973820 0x00973A9E 0x009606CA 0x00000000
[ 3493.748531] ath10k_pci 0000:00:00.0: [20]: 0x409C10BA 0x00405DBC 0x00000002 0x00000040
[ 3493.756711] ath10k_pci 0000:00:00.0: [24]: 0x8099731A 0x00405E1C 0x00000000 0xC09C10BA
[ 3493.764882] ath10k_pci 0000:00:00.0: [28]: 0x80998356 0x00405E3C 0x0043E1F8 0x0043D298
[ 3493.773063] ath10k_pci 0000:00:00.0: [32]: 0x80997D92 0x00405E5C 0x0043D314 0x0043E1F8
[ 3493.781243] ath10k_pci 0000:00:00.0: [36]: 0x8099A095 0x00405EBC 0x0043D314 0x0043E1F8
[ 3493.789420] ath10k_pci 0000:00:00.0: [40]: 0x8099530D 0x00405EDC 0x004274E0 0x0043BC1C
[ 3493.797603] ath10k_pci 0000:00:00.0: [44]: 0x8099306E 0x00405F0C 0x00000001 0x004274E0
[ 3493.805785] ath10k_pci 0000:00:00.0: [48]: 0x8098EC88 0x00405F4C 0x00425E50 0x00000000
[ 3493.813957] ath10k_pci 0000:00:00.0: [52]: 0x80963BFA 0x004060DC 0x00425E50 0x0098EC80
[ 3493.822136] ath10k_pci 0000:00:00.0: [56]: 0x80960EC8 0x004060FC 0x0000001F 0x00400000
[ 3493.830311] ath10k_pci 0000:00:00.0: Copy Engine register dump:
[ 3493.836442] ath10k_pci 0000:00:00.0: [00]: 0x0004a000  14  14   3   3
[ 3493.843099] ath10k_pci 0000:00:00.0: [01]: 0x0004a400  24  24 350 351
[ 3493.849764] ath10k_pci 0000:00:00.0: [02]: 0x0004a800  47  47  46  47
[ 3493.856430] ath10k_pci 0000:00:00.0: [03]: 0x0004ac00   1   1   3   1
[ 3493.863089] ath10k_pci 0000:00:00.0: [04]: 0x0004b000 792 792  50  10
[ 3493.869754] ath10k_pci 0000:00:00.0: [05]: 0x0004b400  14  14 141 142
[ 3493.876420] ath10k_pci 0000:00:00.0: [06]: 0x0004b800  18  18  18  18
[ 3493.883078] ath10k_pci 0000:00:00.0: [07]: 0x0004bc00   1   1   1   1
[ 3493.889743] ath10k_pci 0000:00:00.0: [08]: 0x0004c000   0   0 127   0
[ 3493.896410] ath10k_pci 0000:00:00.0: [09]: 0x0004c400   1   1   1   1
[ 3493.903073] ath10k_pci 0000:00:00.0: [10]: 0x0004c800   0   0   0   0
[ 3493.909734] ath10k_pci 0000:00:00.0: [11]: 0x0004cc00   0   0   0   0
[ 3493.918406] ath10k_pci 0000:00:00.0: debug log header, dbuf: 0x41b558  dropped: 0
[ 3493.927156] ath10k_pci 0000:00:00.0: [0] next: 0x41b540 buf: 0x417940 sz: 1500 len: 28 count: 1 free: 0
[ 3493.937872] ath10k_pci 0000:00:00.0: ath10k_pci ATH10K_DBG_BUFFER:
[ 3493.944253] ath10k: [0000]: 4EE43500 0100FC17 BA109C00 B3150000 B3150000 AC5C4000 69451091
[ 3493.952791] ath10k_pci 0000:00:00.0: ATH10K_END
[ 3493.958483] ath10k_pci 0000:00:00.0: [1] next: 0x41b558 buf: 0x417350 sz: 1500 len: 0 count: 0 free: 0
[ 3493.995419] ath10k_pci 0000:00:00.0: removing peer, cleanup-all, deleting: peer 83973800 vdev: 0 addr: f8:16:54:2b:da:2c 
[ 3494.006782] ath10k_pci 0000:00:00.0: removing peer, cleanup-all, deleting: peer 87291000 vdev: 0 addr: 58:61:63:00:10:52 
[ 3494.158495] ieee80211 phy0: Hardware restart was requested
[ 3496.784678] ath10k_pci 0000:00:00.0: firmware crashed! (uuid e1f1b42a-71e2-4e63-853d-0d0bc80cc2cf)
[ 3496.793998] ath10k_pci 0000:00:00.0: qca9888 hw2.0 target 0x01000000 chip_id 0x00000000 sub 0000:0000
[ 3496.803542] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[ 3496.822948] ath10k_pci 0000:00:00.0: firmware ver 10.4-ct-9888-fW-011-cf79c7f api 5 features peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,cust-stats-CT crc32 0f624b45
[ 3496.847762] ath10k_pci 0000:00:00.0: board_file api 2 bmi_id 0:16 crc32 d65c17b1
[ 3496.855427] ath10k_pci 0000:00:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1
[ 3496.867867] ath10k_pci 0000:00:00.0: firmware register dump:
[ 3496.873715] ath10k_pci 0000:00:00.0: [00]: 0x0000000C 0x00000000 0x00000000 0x00000000
[ 3496.881898] ath10k_pci 0000:00:00.0: [04]: 0x00000000 0x00060024 0x00000000 0x00000000
[ 3496.890081] ath10k_pci 0000:00:00.0: [08]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.898264] ath10k_pci 0000:00:00.0: [12]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.906445] ath10k_pci 0000:00:00.0: [16]: 0x0096F487 0x000CAD7B 0x00000000 0x00000000
[ 3496.914618] ath10k_pci 0000:00:00.0: [20]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.922797] ath10k_pci 0000:00:00.0: [24]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.930978] ath10k_pci 0000:00:00.0: [28]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.939154] ath10k_pci 0000:00:00.0: [32]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.947339] ath10k_pci 0000:00:00.0: [36]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.955522] ath10k_pci 0000:00:00.0: [40]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.963692] ath10k_pci 0000:00:00.0: [44]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.971870] ath10k_pci 0000:00:00.0: [48]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.980048] ath10k_pci 0000:00:00.0: [52]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.988231] ath10k_pci 0000:00:00.0: [56]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3496.996410] ath10k_pci 0000:00:00.0: Copy Engine register dump:
[ 3497.002530] ath10k_pci 0000:00:00.0: [00]: 0x0004a000  14  14   3   3
[ 3497.009195] ath10k_pci 0000:00:00.0: [01]: 0x0004a400   3   3   9  10
[ 3497.015860] ath10k_pci 0000:00:00.0: [02]: 0x0004a800   1   1 127   1
[ 3497.022518] ath10k_pci 0000:00:00.0: [03]: 0x0004ac00   0   0   2   0
[ 3497.029184] ath10k_pci 0000:00:00.0: [04]: 0x0004b000  44  44  41  22
[ 3497.035850] ath10k_pci 0000:00:00.0: [05]: 0x0004b400   0   0   0   0
[ 3497.042519] ath10k_pci 0000:00:00.0: [06]: 0x0004b800   0   0   0   0
[ 3497.049185] ath10k_pci 0000:00:00.0: [07]: 0x0004bc00   1   1   1   1
[ 3497.055848] ath10k_pci 0000:00:00.0: [08]: 0x0004c000   0   0 127   0
[ 3497.062508] ath10k_pci 0000:00:00.0: [09]: 0x0004c400   0   0   0   0
[ 3497.069172] ath10k_pci 0000:00:00.0: [10]: 0x0004c800   0   0   0   0
[ 3497.075839] ath10k_pci 0000:00:00.0: [11]: 0x0004cc00   0   0   0   0
[ 3497.084504] ath10k_pci 0000:00:00.0: debug log header, dbuf: 0x41b540  dropped: 0
[ 3497.093254] ath10k_pci 0000:00:00.0: [0] next: 0x41b558 buf: 0x417350 sz: 1500 len: 0 count: 0 free: 0
[ 3497.103878] ath10k_pci 0000:00:00.0: [1] next: 0x41b540 buf: 0x417940 sz: 1500 len: 0 count: 0 free: 0
[ 3501.855003] ath10k_pci 0000:00:00.0: wmi service ready event not received
[ 3501.948683] ath10k_pci 0000:00:00.0: Could not init core: -145
[ 3501.954721] ------------[ cut here ]------------
[ 3501.959661] WARNING: CPU: 0 PID: 110 at backports-2017-11-01/net/mac80211/util.c:1867 ieee80211_reconfig+0x88/0xa20 [mac80211]
[ 3501.971427] Hardware became unavailable during restart.
[ 3501.976819] Modules linked in: mtd_rw ath9k ath9k_common pppoe ppp_async l2tp_ppp batman_adv ath9k_hw ath10k_pci ath10k_core ath pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_physdev xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_coova xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache libcrc32c iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_snat ebt_redirect
[ 3502.050294]  ebt_pkttype ebt_nflog ebt_mark_m ebt_mark ebt_log ebt_limit ebt_ip ebt_dnat ebt_arpreply ebt_arp ebt_among ebt_802_3 crc_ccitt compat br_netfilter sch_cake nf_conntrack act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_tbf sch_htb sch_hfsc sch_ingress ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables ifb l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel tun ledtrig_transient ehci_platform
[ 3502.122252]  ehci_hcd gpio_button_hotplug usbcore nls_base usb_common crc16 crc32c_generic crypto_hash
[ 3502.130593] CPU: 0 PID: 110 Comm: kworker/0:1 Not tainted 4.9.152 #0
[ 3502.137235] Workqueue: events_freezable ieee80211_alloc_hw_nm [mac80211]
[ 3502.144160] Stack : 87ce6998 00000008 80460000 800a966c 87ca1cac 80451267 80406180 0000006e
[ 3502.152876]         804060ec 87d49d04 80450000 800d781c 80460000 800a966c 8040baac 80450000
[ 3502.161576]         00000003 87d49d04 80450000 800950d4 80460000 87d49d3c 00000000 00000008
[ 3502.170277]         00000002 00000000 8700064c 81102700 87c36000 6576656e 74735f66 7265657a
[ 3502.178977]         61626c65 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 3502.187676]         ...
[ 3502.190213] Call Trace:
[ 3502.192740] [<8006b220>] show_stack+0x70/0x8c
[ 3502.197268] [<80080264>] __warn+0x110/0x118
[ 3502.201593] [<8008029c>] warn_slowpath_fmt+0x30/0x3c
[ 3502.206808] [<870320d4>] ieee80211_reconfig+0x88/0xa20 [mac80211]
[ 3502.213160] [<870006ec>] ieee80211_alloc_hw_nm+0x68c/0x6d0 [mac80211]
[ 3502.219894] [<80093130>] process_one_work+0x1e8/0x318
[ 3502.225137] [<80093f3c>] worker_thread+0x2f4/0x424
[ 3502.230086] [<800986e0>] kthread+0xdc/0xe4
[ 3502.234331] [<80065fd8>] ret_from_kernel_thread+0x14/0x1c
[ 3502.239915] ---[ end trace 0668bd83755cb52c ]---
[ 3502.244745] ------------[ cut here ]------------
[ 3502.249633] WARNING: CPU: 0 PID: 110 at backports-2017-11-01/net/mac80211/driver-ops.h:17 drv_remove_interface+0x48/0x7c [mac80211]
[ 3502.261850] mon0:  Failed check-sdata-in-driver check, flags: 0x0
[ 3502.268144] Modules linked in: mtd_rw ath9k ath9k_common pppoe ppp_async l2tp_ppp batman_adv ath9k_hw ath10k_pci ath10k_core ath pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_physdev xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_coova xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache libcrc32c iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_snat ebt_redirect
[ 3502.341620]  ebt_pkttype ebt_nflog ebt_mark_m ebt_mark ebt_log ebt_limit ebt_ip ebt_dnat ebt_arpreply ebt_arp ebt_among ebt_802_3 crc_ccitt compat br_netfilter sch_cake nf_conntrack act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_tbf sch_htb sch_hfsc sch_ingress ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables ifb l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel tun ledtrig_transient ehci_platform
[ 3502.413587]  ehci_hcd gpio_button_hotplug usbcore nls_base usb_common crc16 crc32c_generic crypto_hash
[ 3502.421928] CPU: 0 PID: 110 Comm: kworker/0:1 Tainted: G        W       4.9.152 #0
[ 3502.429840] Workqueue: events_freezable ieee80211_alloc_hw_nm [mac80211]
[ 3502.436783] Stack : 00000001 0000000c 85b82bd0 800a966c 87ca1cac 80451267 80406180 0000006e
[ 3502.445483]         804060ec 87d49bdc 80450000 800d781c 85b82bd0 800a966c 8040baac 80450000
[ 3502.454174]         00000003 87d49bdc 80450000 800950d4 85b82bd0 87d49c14 00000000 00000008
[ 3502.462874]         804634e0 00000000 8700064c 81102700 87c36000 6576656e 74735f66 7265657a
[ 3502.471574]         61626c65 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 3502.480275]         ...
[ 3502.482812] Call Trace:
[ 3502.485347] [<8006b220>] show_stack+0x70/0x8c
[ 3502.489869] [<80080264>] __warn+0x110/0x118
[ 3502.494196] [<8008029c>] warn_slowpath_fmt+0x30/0x3c
[ 3502.499403] [<87003090>] drv_remove_interface+0x48/0x7c [mac80211]
[ 3502.505870] [<8701590c>] ieee80211_del_virtual_monitor+0x84c/0x8d8 [mac80211]
[ 3502.513265] ---[ end trace 0668bd83755cb52d ]---
[ 3502.522571] ath10k_pci 0000:00:00.0: cannot restart a device that hasn't been started

@tirth21 Did you find a solution to this issue? thanks!

@tirth21

Judging by the kernel version, this looks a bit older firmware.

Could you retest using the latest master?

Thanks,

Koen