[Solved] Mt7915e Down - HW problem - can not stop rx aggregation

[dsouza]

I have four Redmi AX6S (mt7622) access points, all of them are running OpenWrt in "dumb configuration" (firewall/dnsmasq/dhcpd/... disabled). Main router is a NanoPi R4S also with OpenWrt. All AX6S are wired (no mesh in use), and they share the same SSID and passwords for roaming.

All devices are running OpenWrt 23.05.2.

One of the AX6S recently started having issues. After ~24 hours working, wireless suddenly stops working. The error is:

[122255.188788] Timeout waiting for MCU reset state 10
[122255.193733] WARNING: CPU: 0 PID: 3646 at mt7915_mcu_rf_regval+0x1f8/0x4e0 [mt7915e]
[122255.201512] Modules linked in: mt7915e mt7615e mt7615_common mt76_connac_lib mt76 mac80211 cfg80211 hwmon compat seqiv leds_gpio gpio_button_hotplug

And the call trace is:

[122255.326805] Call trace:
[122255.329329] mt7915_mcu_rf_regval+0x1f8/0x4e0 [mt7915e]
[122255.334637] mt7915_mac_reset_work+0xd1c/0xd70 [mt7915e]
[122255.340031] process_one_work+0x210/0x3b0
[122255.344123] worker_thread+0x170/0x4d0
[122255.347951] kthread+0x11c/0x130
[122255.351262] ret_from_fork+0x10/0x20

After this issue happen it keeps repeating every 20 seconds.

The only thing different on this device is that recently I've added a Windows Laptop with an Intel AX211 wifi adapter.

I will start investigating this issue, but just reporting here just in case someone already faced a similar issue.

[hecatae]

You may want to try snapshot, in the main mt76 wireless driver debugging there's several fixes that were added in January that you may not see on your 23.05.2 November release

Mt76 wireless driver debugging

Thanks for the tip. There are very recent firmware files available here also: https://git01.mediatek.com/plugins/gitiles/openwrt/feeds/mtk-openwrt-feeds/+/af91065e8254da2e6ff2a5619ba4e7eca20f60f4/autobuild_mac80211_release/package/kernel/mt76/src/firmware I did briefly test these ones on my mt7986 device but got wireless lockups after a few days. Maybe the ones for the mt7915 are stable but don't have a device to test on.

[dsouza]

I think I've found the curlpit. Sometime ago I enabled WED on this exact same device that is having issues:

# /etc/modules.com
options mt7915e wed_enable=Y

This is the only configuration that was different from the other AX6S which are not having issues.

I just removed this config and I will observe to confirm the problem is solved.

EDIT: confirmed that disabling WED solved the problem. So enabling WED with 23.05.2 on AX6S is causing the device to crash after ~20hours of operation.

[dsouza]

BTW:

You may want to try snapshot

Unfortunately the snapshots for the AX6S have been disabled, so using snapshot builds on this device is currently not an option. See here:

https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=dadad6bb731ff1025125c619da7ccc196ad6ea01