@nbd thanks a lot for taking this to the linux-stable@. I didn't look into, though hope you did it right and we will get your patch upstreamed, thus credible openwrt changes. I have couple of 3200acm I'd like to run on a recent openwrt release. Sorry I couldn't test patches, though you guys made an incredible work to narrow down the issue. Looking forward to see fixed and working upcoming release. <3 Thanks!
An outstanding piece of troubleshooting.
Just wanted to say as a WRT3200ACM user I have been following the progress of this thread and dedication of the troubleshooting and fault finding undertaken to find the issue has been amazing.
Thanks to everyone who tested and helped find the offending commit, certainly looks promising for using 21.02 on the WRT3200ACM in the next release in a more stable fashion for wireless when it gets backported.
Amazing work.
2 Likes
Thanks to each and every one of you for pouring your time into troubleshooting and finally fixing the issue upstream. I added information on the main post with @nbd's replies.
Cheers!
2 Likes
Wow, that is really good news!
Just out of curiosity I extracted mac80211.ko from the ipk file and copied it to the /lib/modules/5.4.143/ folder, overwriting the existing mac80211.ko on my WRT3200ACM with Openwrt v21.02.0. After a restart Wifi was stable and any cutouts had gone.
I want to thank all involved in resolving this issue for their dedication and invested time and efforts. Thank you so much!!
Due to my lack of technical knowledge I did not understand all of the technical details discussed here. So, please, allow me one more question:
Retrospectively, was that issue caused by faulty Linksys firmware and this could be overcome by adapting the Linux kernel module? Or was the issue caused only by a regression flaw in the kernel module?
Thanks again to all
Alexander
1 Like
It was caused by a regression in upstream kernel module (mac80211), which regressions only affected some wifi drivers. nbd specified mwlwifi and ath9k in the commit message...
The patch fixes this upstream commit:
5 Likes
Latency seems to improved significantly on my WRT3200ACM. Previously, I would always get something like Unloaded 70 ms and around 500-700 ms Loaded latency on Fast speedtest.
After the fix, I just got Unloaded 14 ms and 78 ms Loaded latency. I didn't believe it at first until I ran it a few times.
Same settings as always, the only difference is the applied fix.
1 Like
The most interesting fact is that the regression affected a rather specific group of devices, most notably Apple devices, while others were unaffected. I wonder how that can be explained.
3 Likes
I was actually hoping that somebody could explain that as well. I've been very curious as to why Apple devices trigger this issue more than any other devices.
And within the family, never saw the issue with an IPad Air 2, but an IPad Air 4 suffers.
does that mean the patch will be pulled on next git pull ??
Yes, it will. For both master and 21.02
1 Like
Guess i am running another build tonight then. Thanks.
With my mt7615 router, I've noticed iPhone 6S+ traffic getting stuck. Flipping wifi on/off fixes it. I haven't been following the thread closely, but is this the issue?
1 Like
Could be anything including iOS version. Broadcom's drivers for iOS are totally closed.
Yes, it is.
Hrm that implies I'm having the issue as well. I'll spin up a build soon to see if master fixes that issue.
The problem is triggered by tearing down and re-establishing a-mpdu sessions. This depends on traffic patterns and client behavior. The first agg session is likely to work, since it will have a low seq number and the driver's seqno will be mostly in sync with mac80211's seqno. Subsequent sessions will trigger this more severely.
5 Likes
Although I am not a good candidate (does not use Apple products) I updated mac80211 and cfg8011 on WRT32X and WRT1900ACS, both are working correctly.
1 Like
This patch is a success for the wireless cutouts.
I had to trigger the wireless cutouts as a part of bisecting OpenWrt commits with @adworacz and then bisecting mac80211 commits with @cotequeiroz.
After triggering hundreds of wireless cutouts, near the end of it all, I had developed such a good technique and repetition that I was consistently able to trigger a wireless cutout within 45 seconds to 90 seconds.
I am still in some sort of shock-like surprise about this patch, but I have not been able to trigger a single wireless cutout since this patch came out. And I have been using my iPhone XR (trigger device) non-stop since then.
Congratulations everyone. I think many of us questioned whether or not we would ever get to the root cause of this issue. Seriously.
The perseverance of @adworacz, @cotequeiroz, @arinc9, and the community here in general all made this happen. Many great tips and suggestions along the way from @slh and @hnyman as well.
And special thanks to @nbd for patching master and 21.02 branches and upstreaming the patch promptly.
Cheers!
14 Likes