WRT1900ACS Wi-Fi stops working randomly

Hi,

for a long time I was running OpenWrt 19.07.4 on this router. Then, a while ago, I updated to 23.05.0 to get some new features. Since then I have noticed random Wi-Fi dropouts, where no device is able to connect to Wi-Fi, although it does show up when scanning. A reboot fixes the problem but it comes back after a few days. A week ago I updated to 23.05.5 since it has a slightly updated mwlwifi driver (10.4.10.0). It looked promising, but today it stopped working again. Is there anything I can do to debug this or should I just give up on this router and Wi-Fi.

The dmesg output when Wi-Fi doesn't work is like this (starts at line 4):

...
[  351.617243] mv88e6085 f1072004.mdio-mii:00 lan4: Link is Up - 1Gbps/Full - flow control rx/tx
[  351.625909] br-lan: port 4(lan4) entered blocking state
[  351.631165] br-lan: port 4(lan4) entered forwarding state
[129981.246531] ieee80211 phy0: cmd 0x9125=BAStream timed out
[129981.252698] ieee80211 phy0: return code: 0x1125
[129981.258419] ieee80211 phy0: timeout: 0x1125
[129982.158461] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[130002.204031] ieee80211 phy0: cmd 0x9125=BAStream timed out
[130002.209562] ieee80211 phy0: return code: 0x1125
[130002.214201] ieee80211 phy0: timeout: 0x1125
[130224.874275] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[130244.987938] ieee80211 phy0: cmd 0x9122=UpdateEncryption timed out
[130244.994167] ieee80211 phy0: return code: 0x1122
[130244.998805] ieee80211 phy0: timeout: 0x1122
[130245.003095] phy0-ap0: failed to remove key (0, 94:45:60:3a:8b:25) from hardware (-5)
[130245.084024] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[130265.213594] ieee80211 phy0: cmd 0x9111=SetNewStation timed out
[130265.219553] ieee80211 phy0: return code: 0x1111
[130265.224205] ieee80211 phy0: timeout: 0x1111
[130268.263595] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[130288.319209] ieee80211 phy0: cmd 0x9122=UpdateEncryption timed out
[130288.325437] ieee80211 phy0: return code: 0x1122
[130288.330076] ieee80211 phy0: timeout: 0x1122
[130288.334375] phy0-ap0: failed to remove key (0, 5c:87:9c:0a:36:b5) from hardware (-5)
[130288.413213] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[130308.528053] ieee80211 phy0: cmd 0x9122=UpdateEncryption timed out
[130308.534281] ieee80211 phy0: return code: 0x1122
[130308.538918] ieee80211 phy0: timeout: 0x1122
[130308.543213] phy0-ap0: failed to remove key (0, 04:34:f6:13:38:7b) from hardware (-5)
[130308.622937] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[130328.728528] ieee80211 phy0: cmd 0x9111=SetNewStation timed out
[130328.734495] ieee80211 phy0: return code: 0x1111
[130328.739135] ieee80211 phy0: timeout: 0x1111
[130328.782674] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[130348.872178] ieee80211 phy0: cmd 0x9111=SetNewStation timed out
[130348.878134] ieee80211 phy0: return code: 0x1111
[130348.882782] ieee80211 phy0: timeout: 0x1111
[130348.888047] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[130368.943845] ieee80211 phy0: cmd 0x801c=80211RadioControl timed out
[130368.950153] ieee80211 phy0: return code: 0x001c
[130368.954805] ieee80211 phy0: timeout: 0x001c

Hi @sasodoma,

Hoping it's the firmware, we can test.

As you're running 23.05.3 :
Please connect to your OpenWrt device using ssh and download the firmware and replace it in openwrt

cd /lib/firmware/mwlwifi/
wget https://github.com/jbsky/mwlwifi/raw/5e4ffc4c2178b95b186eca6143a26a7e46308ca2/bin/firmware/88W8864.bin

Reboot and see if you feel better.

If not, and if you've got a bit of time, we'll try to find out where the regression is ;).

Best regard,

Jbs

1 Like

Thank you for the reply. Glad to get some help with this. I have replaced the firmware and rebooted. I will let you know if the problem comes back, though last time it took about a week.

Best regard

1 Like

There is a new 88W8864.bin BLOB out in the wilds?

2 Likes

As I'm still not sure about the firmware, since it's the closed source, the first thing to test is the previous version.

Best,

Jbs

Right, but since the BLOB is closed source, my question is from where and how did this come into existence.

It was born from the latest 2018 firmware from linksys,.
https://downloads.linksys.com/downloads/releasenotes/WRT1900ACv2_Customer_Release_Notes_2.0.8.187766.txt
It was extracted from it.

1 Like

is this something good for the 3200 too?

Different radio in that device.

@jbsky , IIUC, you are thinking to test issue with your updated driver code on previous firmware. 23.x does not have all the latest driver code, but not sure if OP is trying WPA3 here.

i know but take a look, souce code of 1900 and 32xx

@anomeome,
Firmware is something we can't control. Since there have been changes with 23.05.3, I'd rather ask you to go back over this file to see if it's any better.

It took 4 days for sasodoma to reproduce the problem.

Why WPA3? Here, the conf files have not yet been requested.

Best,

Jbs

@jbsky just an update.

For the record I haven’t run wifi on my 1900ACS for about the last 2 years.

8 days ago I lost my unifi ap out of the blue (2 days beyond warranty that they declined to RMA :grimacing:) but I digress. In a pinch I brought out my spare 1900ACS, put current stable on it and turned it into a wireless AP to get me through. Pretty quickly I started losing 5Ghz occasionally that required a reboot to restore.

This thread was still fresh in my mind so I replaced your firmware from here.

After 3 days, wifi 5Ghz remained stable and I disco’d all my wired devices and have been running everything on the 1900-AP 5Ghz since without a hiccup.

I don’t really fully understand the conversation @anomeome and you had, or the context of why WPA3 played into it, but on a whim I tried to enable WPA3 tonight that just wouldn’t take on either band.

I did though test 5Ghz and WPA3 from your work a few months ago over ~3-4 days that seemed reasonably stable on a 6.1 main/master build.

Don't know if this helps here, but willing to help if I can.

1 Like

Guess I can also give an update from my side, I've had a power outage last week, but right now the router is at 6 days uptime and I haven't noticed any dropouts, there is also no errors in dmesg. I'm thinking of letting it run maybe another week, and then switching back to the newer firmware, to see how long that one takes to fail.

1 Like

@sasodoma
Thank you for your feedback. For my part, I reproduced the timeout. The router being vertically against the wall, the heat made it buggy.

@RuralRoots
There's no connection with WPA3. WPA3 was unblocked by telling the firmware to let through the managed packets sent by the mac80211 layer. The encryption methods remain the same.

I don't understand, WPA3 didn't work for you? The only thing that doesn't make it work is mixed mode.

For what it's worth, at first I did some switching between new and old firmware, and with new firmware the error showed up quite often. Now recently I've been running old firmware and everything has been stable until today. At around 34 days of uptime the Wi-Fi stopped working and I got the same error:

[1879393.558189] br-lan: port 4(lan4) entered blocking state
[1879393.563617] br-lan: port 4(lan4) entered forwarding state
[2955822.554329] ieee80211 phy0: cmd 0x9125=BAStream timed out
[2955822.559938] ieee80211 phy0: return code: 0x1125
[2955822.564675] ieee80211 phy0: timeout: 0x1125
[2955831.960157] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[2955851.969798] ieee80211 phy0: cmd 0x9122=UpdateEncryption timed out
[2955851.976132] ieee80211 phy0: return code: 0x1122
[2955851.980876] ieee80211 phy0: timeout: 0x1122
[2955851.985261] phy0-ap0: failed to remove key (0, 5c:87:9c:0a:36:b5) from hardware (-5)
[2955851.993403] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[2955871.999432] ieee80211 phy0: cmd 0x9111=SetNewStation timed out
[2955872.005487] ieee80211 phy0: return code: 0x1111
[2955872.010230] ieee80211 phy0: timeout: 0x1111
[2956077.595720] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[2956097.611370] ieee80211 phy0: cmd 0x9122=UpdateEncryption timed out
[2956097.617684] ieee80211 phy0: return code: 0x1122
[2956097.622410] ieee80211 phy0: timeout: 0x1122
[2956097.626801] phy0-ap0: failed to remove key (0, 28:39:26:e2:68:b9) from hardware (-5)
[2956097.685440] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[2956117.704989] ieee80211 phy0: cmd 0x9111=SetNewStation timed out
[2956117.711034] ieee80211 phy0: return code: 0x1111
[2956117.715769] ieee80211 phy0: timeout: 0x1111
[2956117.775014] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[2956137.782636] ieee80211 phy0: cmd 0x9122=UpdateEncryption timed out
[2956137.788948] ieee80211 phy0: return code: 0x1122
[2956137.793672] ieee80211 phy0: timeout: 0x1122
[2956137.798055] phy0-ap0: failed to remove key (0, da:bb:55:72:56:6e) from hardware (-5)
[2956137.844672] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[2956157.862278] ieee80211 phy0: cmd 0x9122=UpdateEncryption timed out
[2956157.868702] ieee80211 phy0: return code: 0x1122
[2956157.873471] ieee80211 phy0: timeout: 0x1122
[2956157.877995] phy0-ap0: failed to remove key (0, 94:45:60:3a:8b:25) from hardware (-5)
[2956157.944618] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[2956177.971910] ieee80211 phy0: cmd 0x9125=BAStream timed out
[2956177.977532] ieee80211 phy0: return code: 0x1125
[2956177.982258] ieee80211 phy0: timeout: 0x1125
[2956177.986896] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000
[2956197.989545] ieee80211 phy0: cmd 0x9111=SetNewStation timed out
[2956197.995599] ieee80211 phy0: return code: 0x1111
[2956198.000325] ieee80211 phy0: timeout: 0x1111
[2956198.043626] ieee80211 phy0: MACREG_REG_INT_CODE: 0x0000