Just out of curiosity I extracted mac80211.ko from the ipk file and copied it to the /lib/modules/5.4.143/ folder, overwriting the existing mac80211.ko on my WRT3200ACM with Openwrt v21.02.0. After a restart Wifi was stable and any cutouts had gone.
I want to thank all involved in resolving this issue for their dedication and invested time and efforts. Thank you so much!!
Due to my lack of technical knowledge I did not understand all of the technical details discussed here. So, please, allow me one more question:
Retrospectively, was that issue caused by faulty Linksys firmware and this could be overcome by adapting the Linux kernel module? Or was the issue caused only by a regression flaw in the kernel module?
It was caused by a regression in upstream kernel module (mac80211), which regressions only affected some wifi drivers. nbd specified mwlwifi and ath9k in the commit message...
Latency seems to improved significantly on my WRT3200ACM. Previously, I would always get something like Unloaded 70 ms and around 500-700 ms Loaded latency on Fast speedtest.
After the fix, I just got Unloaded 14 ms and 78 ms Loaded latency. I didn't believe it at first until I ran it a few times.
Same settings as always, the only difference is the applied fix.
The most interesting fact is that the regression affected a rather specific group of devices, most notably Apple devices, while others were unaffected. I wonder how that can be explained.
I was actually hoping that somebody could explain that as well. I've been very curious as to why Apple devices trigger this issue more than any other devices.
With my mt7615 router, I've noticed iPhone 6S+ traffic getting stuck. Flipping wifi on/off fixes it. I haven't been following the thread closely, but is this the issue?
The problem is triggered by tearing down and re-establishing a-mpdu sessions. This depends on traffic patterns and client behavior. The first agg session is likely to work, since it will have a low seq number and the driver's seqno will be mostly in sync with mac80211's seqno. Subsequent sessions will trigger this more severely.
I had to trigger the wireless cutouts as a part of bisecting OpenWrt commits with @adworacz and then bisecting mac80211 commits with @cotequeiroz.
After triggering hundreds of wireless cutouts, near the end of it all, I had developed such a good technique and repetition that I was consistently able to trigger a wireless cutout within 45 seconds to 90 seconds.
I am still in some sort of shock-like surprise about this patch, but I have not been able to trigger a single wireless cutout since this patch came out. And I have been using my iPhone XR (trigger device) non-stop since then.
Congratulations everyone. I think many of us questioned whether or not we would ever get to the root cause of this issue. Seriously.
The perseverance of @adworacz, @cotequeiroz, @arinc9, and the community here in general all made this happen. Many great tips and suggestions along the way from @slh and @hnyman as well.
And special thanks to @nbd for patching master and 21.02 branches and upstreaming the patch promptly.
This truly is awesome, so much work you guys put into this! I was already researching replacement devices, but I really like these particular devices and now have 4 of them. So this already in the nightly snapshot build of 21.02 for these devices may be in 21.02.2 when it comes out ?? or can I install 21.02.1 and update via the opkg method you mention in an earlier comment??
Yes.
If you want it right now into 21.02, you could install the semi-hidden 21.02 snapshot firmware (current 21.02 HEAD. Later than 21.02.1 but not yet officially 21.02.2)