Rekeying Issue - “driver can’t safely do that.”

I am seeing the error below on an Archer C6 v3.2 running a snapshot build from last week (I have one device as AP+Router plus other two as AP - all same HW version):

I've already found an existing discussion about this here.

[32455.456810] Rekeying PTK for STA xx:xx:xx:xx:82:11 but driver can't safely do that.
[33990.272681] Rekeying PTK for STA xx:xx:xx:xx:82:11 but driver can't safely do that.
[78458.702378] Rekeying PTK for STA xx:xx:xx:xx:f1:c6 but driver can't safely do that.
[78917.761234] Rekeying PTK for STA xx:xx:xx:xx:f1:c6 but driver can't safely do that.
[81114.968419] Rekeying PTK for STA xx:xx:xx:xx:f1:c6 but driver can't safely do that.
[82795.467225] Rekeying PTK for STA xx:xx:xx:xx:82:11 but driver can't safely do that.
[163715.545652] Rekeying PTK for STA xx:xx:xx:xx:f1:c6 but driver can't safely do that.

However, per comment in the former thread this would be related to using WPA-EAP (WPA Enterprise), which is not my case (I am using WPA-PSK+Force CCPM-AES).

However I have 802.11r enabled and correctly configured but I so far I have not able to have seamless roaming working across the 3 access points.

So I am wondering if these errors could be somehow related to 802.11r or if this is a potential bug with m76 wifi drivers.

Any comment would be welcome.

2 Likes

These warnings are not tied to WPA-EAP. They are triggered when a PSK is being replaced for a STA and it's unclear if the driver is handling all the races around that.
Which at least theoretically can only happen when the PSK is rekeyed during an association.
And at least hostapd has PTK-rekeying only enabled by default when you use WPA-EAP. But with the correct configuration settings you can enable PTK-rekeying with WPA-PSK, too. (And keep in mind that either end can decide to rekey the connection.)

Now why the key is replaced for you here can't be determined by the log you shared. You probably need debug logs from the AP and the client to figure out what the SW is trying to do and why this looks like a rekey to mac80211 on the router. It may well be a bug somewhere....

Or at least I do not see why roaming should have any need to update a running PSK. (Deleting a key and installing a new one can't trigger the warning, only replacing a active key with another one...)

1 Like

Thanks. Unfortunately I don't have the tools required to troubleshoot the clients.

Interestingly enough these errors only appears with Apple devices (there are two iPads and two iPhones at home, and only their MAC address appear in these error messages).

Anyway, I am still suspecting it is somehow related to a device roaming across different OpenWRT access points. See errors below, they are from DMESG from two different OpenWRT access points, probably happened when the devices were roaming (however this is only an hypothesis and I have not tested enough to confirm). Or it's an iOS bug...

Anyway, I rebooted all devices last night and so far only two of these errors appeared. For now I will ignore them, but I have a feeling that in fact there is a bug somewhere (either in iOS or OpenWRT).

One possible test I will do sometime is to disable 802.11r in all devices and see if these errors disappear.

Additionally apple might be trying to do some "smart roaming" which might be somehow conflicting with OpenWRT implementation - namely "PMKID caching" used by Apple, see here and here).

See the following comments from Cisco:

(...) This is possible because, every time a client is fully EAP-authenticated, the client and Authentication Server derive an MSK, which is used in order to derive the PMK. This is used as the seed for the WPA2 4-Way handshake in order to derive the final unicast encryption key (PTK) that is used for the session (until the client roams to another AP or the session expires); hence, this method prevents the EAP authentication phase when roaming because it reutilizes the original PMK cached by the client and the AP. The client only has to go through the WPA2 4-Way handshake in order to derive new encryption keys.
(...)
This method is optional and is not supported by all WPA2 devices, because the purpose of the 802.11i amendment does not concern fast-secure roaming, and the IEEE was already working on another amendment to standardize fast-secure roaming for WLANs (802.11r, which is covered later in this document).(...)

OpenWRT Access point 1 (192.168.1.1):

[   23.129576] br-lan: port 6(wlan1) entered blocking state
[   23.134921] br-lan: port 6(wlan1) entered forwarding state
[  794.739624] Rekeying PTK for STA xx:xx:xx:xx:82:11 but driver can't safely do that.
[  799.229220] Rekeying PTK for STA xx:xx:xx:xx:3b:f3 but driver can't safely do that.
EOF

OpenWRT Access point 3 (192.168.1.3):

[   19.436263] br-lan: port 6(wlan1) entered forwarding state
[  842.181201] Rekeying PTK for STA xx:xx:xx:xx:82:11 but driver can't safely do that.
[  843.943349] Rekeying PTK for STA xx:xx:xx:xx:3b:f3 but driver can't safely do that.
EOF

EDITED: below are some additional errors from logread which might or might not be related to these issues:

Thu Sep 16 23:09:09 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Thu Sep 16 23:09:09 2021 daemon.err hostapd: nl80211: NL80211_ATTR_STA_VLAN (addr=xx:xx:xx:xx:82:11 ifname=wlan1 vlan_id=0) failed: -2 (No such file or directory)
Thu Sep 16 23:09:09 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Thu Sep 16 23:09:09 2021 daemon.err hostapd: nl80211: NL80211_ATTR_STA_VLAN (addr=xx:xx:xx:xx:3b:f3 ifname=wlan1 vlan_id=0) failed: -2 (No such file or directory)
Fri Sep 17 06:33:47 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Sep 17 06:56:02 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Sep 17 06:56:02 2021 daemon.err hostapd: nl80211: NL80211_ATTR_STA_VLAN (addr=xx:xx:xx:xx:3b:f3 ifname=wlan1 vlan_id=0) failed: -2 (No such file or directory)
Fri Sep 17 07:06:52 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Sep 17 07:06:52 2021 daemon.err hostapd: nl80211: NL80211_ATTR_STA_VLAN (addr=xx:xx:xx:xx:82:11 ifname=wlan1 vlan_id=0) failed: -2 (No such file or directory)
Fri Sep 17 08:10:42 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed

Same error with two dir-882 (mt76 with fast roaming) and two iPad with iPadOS 15.1.
Were you able to understand the cause?

I haven’t figured out a solution so far. Currently I have 802.11r disabled since it was not working as expected and it was causing problems. Disabling 802.11r made these errors disappear, so they were indeed related to 802.11r.

However mt76 drivers have been updated on the master builds recently, so when I have some time I will test 802.11r again.

I also want to test more with non-Apple devices, since I suspect this issue might be specific with iOS devices. But it is only a hypothesis and I have not confirmed this.

1 Like

Yes I confirm, I have many android and Windows devices connected but the problem is only with iOS and iPadOS.

Have you also tried to change this?

Generate PMK locally
When using a PSK, the PMK can be automatically generated. When enabled, the R0/R1 key options below are not applied. Disable this to use the R0 and R1 key options.

Yes, I have always enabled this option (ft_psk_generate_local). Below are the configs I use to enable 802.11r just for reference.

BTW, I just updated all my devices in my home network (four Archer C6 v3.2 and one RE305 v3) to a recent snapshot build I did yesterday (Oct 31st r17903-a2fcd3900c). I also re-enabled 802.11r on all my access points, but I'm away from home now and did this remotely. Tomorrow I will be back home and I should be able to see if there is any improvement on the recent builds regarding iOS fast roaming.

uci set wireless.default_radio0.ieee80211r='1'
uci set wireless.default_radio0.mobility_domain='6465'
uci set wireless.default_radio0.ft_over_ds='1'
uci set wireless.default_radio0.ft_psk_generate_local='1'
uci set wireless.default_radio1.ieee80211r='1'
uci set wireless.default_radio1.mobility_domain='6465'
uci set wireless.default_radio1.ft_over_ds='1'
uci set wireless.default_radio1.ft_psk_generate_local='1'
uci commit wireless
wifi up
1 Like

Great, I try to deactivate ft_psk_generate_local and use instead static R0 and R1.

Just back home and tested again with the new build.

Unfortunately the issue still remains. I tested with an iPhone 11 (MAC 9e:05:xx:xx:xx:11) and a Windows 10 laptop (Intel WiFi 6E AX210).

In both devices packets are lost when roaming from one AP to another, however the error is appearing in OpenWRT logs only for iPhone (despite Windows 10 is losing packets while roaming).

I am giving up on 802.11r for now and I will disable it on all APs since it makes roaming worse... :frowning:

root@ap1-router:~# logread -f
Tue Nov  2 17:42:55 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Tue Nov  2 17:42:55 2021 daemon.err hostapd: nl80211: NL80211_ATTR_STA_VLAN (addr=9e:05:xx:xx:xx:11 ifname=wlan1 vlan_id=0) failed: -2 (No such file or directory)
Tue Nov  2 17:42:55 2021 daemon.info hostapd: wlan1: STA 9e:05:xx:xx:xx:11 IEEE 802.11: associated (aid 1)
Tue Nov  2 17:42:55 2021 daemon.notice hostapd: wlan1: AP-STA-CONNECTED 9e:05:xx:xx:xx:11
root@ap1-router:~# dmesg | grep Rek
[63054.152535] Rekeying PTK for STA 9e:05:xx:xx:xx:11 but driver can't safely do that.
[63795.619941] Rekeying PTK for STA 9e:05:xx:xx:xx:11 but driver can't safely do that.

Windows 10 while roaming and 802.11r enabled:

C:\>ping 192.168.1.1 -t

Pinging 192.168.1.1 with 32 bytes of data:
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=4ms TTL=64
Reply from 192.168.1.1: bytes=32 time=45ms TTL=64
Reply from 192.168.1.1: bytes=32 time=10ms TTL=64
Reply from 192.168.1.1: bytes=32 time=14ms TTL=64
Reply from 192.168.1.1: bytes=32 time=7ms TTL=64
Reply from 192.168.1.1: bytes=32 time=7ms TTL=64
Reply from 192.168.1.1: bytes=32 time=8ms TTL=64
Reply from 192.168.1.1: bytes=32 time=8ms TTL=64
Reply from 192.168.1.1: bytes=32 time=6ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=5ms TTL=64
Reply from 192.168.1.1: bytes=32 time=9ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=3ms TTL=64
Reply from 192.168.1.1: bytes=32 time=4ms TTL=64
Reply from 192.168.1.1: bytes=32 time=10ms TTL=64
Reply from 192.168.1.1: bytes=32 time=5ms TTL=64
Reply from 192.168.1.1: bytes=32 time=9ms TTL=64
Reply from 192.168.1.1: bytes=32 time=6ms TTL=64
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.1.1: bytes=32 time=8ms TTL=64
Reply from 192.168.1.1: bytes=32 time=6ms TTL=64
Reply from 192.168.1.1: bytes=32 time=5ms TTL=64

iPhone 11 while roaming and 802.11r enabled:

1 Like

Same thing for me too!
Now I try with the R0 and R1 and let you know...

I've got no pertinent info to contribute, but, doesn't Apple devices MAC randomize by default on connection to an AP? Could it be that it's scrambling the MAC on the IOS device and trying to connect "Roaming" to the second AP and cause issues? Again, not speaking from any point of knowledge other than remembering about IOS playing monkey with the MACs

OK, i have tried also this configuration and many others but the result is always:
Rekeying PTK for STA 8a:54:64:e3:xx:xx but driver can't safely do that.
With packet loss when switching from one ap to another.

Maybe @nbd can help us... :crossed_fingers:

@Grommish any advice is always accepted, MAC address is the same and it is not changing, I think it changes only with networks of different names.

2 Likes

This could help us:
https://bugs.openwrt.org/index.php?do=details&task_id=3159&string=802.11r&type[0]=&sev[0]=&pri[0]=&due[0]=&reported[0]=&cat[0]=&status[0]=open&percent[0]=&opened=&dev=&closed=&duedatefrom=&duedateto=&changedfrom=&changedto=&openedfrom=&openedto=&closedfrom=&closedto=

4 Likes

Good finding, and all seems to make sense.

I will test the recommended workaround settings this weekend and I will post here the results of my tests.

1 Like

For tests I suggest you enable level 1 login :slightly_smiling_face:

1 Like

Today I tried my ipad from a friend with a mesh system AVM 7530 + AVM 1200 (connected with cable), while switching between router and repeater wifi network it lost a lot of packets.
OpenWrt with fast roaming ( FT over the air) is much more stable ...

I changed ft_over_ds='0' (FT Over the Air) and set reassociation_deadline='20000'. My configuration is as follows:

uci set wireless.default_radio0.ieee80211r='1'
uci set wireless.default_radio0.mobility_domain='6465'
uci set wireless.default_radio0.ft_over_ds='0'
uci set wireless.default_radio0.ft_psk_generate_local='1'
uci set wireless.default_radio0.reassociation_deadline='20000'
uci set wireless.default_radio1.ieee80211r='1'
uci set wireless.default_radio1.mobility_domain='6465'
uci set wireless.default_radio1.ft_over_ds='0'
uci set wireless.default_radio1.ft_psk_generate_local='1'
uci set wireless.default_radio1.reassociation_deadline='20000'
uci commit wireless
wifi up

I've done just some quick tests with an iPad and a Windows 10 laptop. I don't know if it's placebo, but while I am still getting lost packets during roaming it seems to have improved.

If I have time tomorrow I will do some additional testing tomorrow.

The good news is that in this quick test I haven't seen the error "driver can’t safely do that" so far. However on the other hand the error hostapd: nl80211: kernel reports: key addition failed is still being logged in all devices involved in the roaming tests and I suspect that the FT roaming is not really working:

root@ap1-router:~# logread | grep nl80211
Fri Nov  5 12:41:11 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 12:41:11 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 12:45:12 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 13:05:53 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 13:06:27 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 13:22:42 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 13:57:28 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 13:57:28 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 14:14:51 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 14:44:41 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 16:28:24 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 16:48:25 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:10:24 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:41:15 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:42:14 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:43:35 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:44:44 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:46:03 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:59:59 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:26:43 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:39:31 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:40:15 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:41:32 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:42:48 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:43:42 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 20:06:36 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 20:08:23 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 20:35:43 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed

(...)
root@ap3:~# logread | grep nl80211
Fri Nov  5 13:22:48 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:39:44 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:41:42 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 20:03:38 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
(...)
root@ap4:~# logread | grep nl80211
Fri Nov  5 12:41:42 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 13:06:30 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 14:15:20 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 14:47:55 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 16:29:37 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:10:40 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:45:43 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 18:47:56 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:41:00 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 19:43:06 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 20:07:17 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 20:33:45 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed
Fri Nov  5 20:36:04 2021 daemon.err hostapd: nl80211: kernel reports: key addition failed

OK, I did some additional tests with iOS and Windows 10. Definitively changing ft_over_ds='0' and reassociation_deadline='20000' did improve roaming for the Windows 10 laptop as recommended in the bug report you referred previously.

While I am still getting some lost packets while moving from one AP to another, roaming with Windows 10 is now better/faster than without 802.11r. So at this moment I will keep 802.11r enabled.

However with iOS roaming is not much better with 802.11r. While at some moments I was able to roam with no packet loss on the iPad, it was just on one or two occasions. I believe this is related to the error "hostapd: nl80211: kernel reports: key addition failed" errors which seems specific to iOS. As mentioned in the bug report, this may be the result from iOS keeping open connections to multiple access points and this is causing FT roaming to not work properly with iOS devices when moving back to an AP previously authenticated.

Well, since I'm the OP, I will mark this post as a solution to help anyone with this issue in the future to quickly find this information. I'm also adding below the reference to bug report #3159 that did shed some light on this issue. Once again thank you @craz for finding and sharing this bug report! :+1:

https://bugs.openwrt.org/index.php?do=details&task_id=3159

5 Likes

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.