Only certain Apple devices (note the glaring omission of MacBooks pre-Apple Silicon) support 802.11r per https://support.apple.com/en-us/HT202628.
Further, Apple has pretty interesting roaming thresholds based on the band: https://support.apple.com/en-us/HT206207
And even more reading for roaming candidate selection: https://support.apple.com/en-us/HT203068
Nice links. My wife has an iPhone 11 with iOS 16 and iPad Pro 12 and in our network with 3x RT3200s using WPA2-PSK and 802.11r she experiences things like when she unlocks her phone the WiFi symbol shows connected but she actually doesn't get internet connectivity until she disables and re-enables WiFi. I don't know how to fix this but would love to try out any suggestions.
For example, might trying WPA3-SAE with roaming help?
You could try, but it will likely also not work fully due to the bug I explained in the other thread: 802.11r Fast Transition how to understand that FT works? - #178 by huaracheguarache
Getting 802.11r to work properly is apparently too challenging for Apple's engineers...
FWIW, I have ~20 Apple devices (Intel MacBooks, M1 MacBook, iPhones, iPads, Apple Watches, etc) that happily migrate between my 3x RT3200s without any loss of connectivity requiring manual intervention. I chose the word "migrate" specifically to avoid labeling it "roaming" for fear that confounds the syntax with mixed 802.11r support on the client level.
My 3x RT3200s (dumb APs) are ethernet connected back to a core switch where I have multiple VLANs supplied to the APs for multiple SSIDs.
I'm not saying my configuration is perfect, but I can say without a doubt I am not having the issue you described. Here is a snippet of my wireless config. I omitted all of the SSIDs I have configured because they effectively duplicate what I show here.
config wifi-device 'radio0'
option type 'mac80211'
option path 'platform/18000000.wmac'
option band '2g'
option htmode 'HT20'
option country 'US'
option cell_density '0'
option log_level '1'
option channel '11'
option txpower '6'
config wifi-device 'radio1'
option type 'mac80211'
option path '1a143000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0'
option band '5g'
option country 'US'
option cell_density '0'
option he_bss_color '30'
option htmode 'HE80'
option log_level '1'
option channel '157'
option he_su_beamformer '1'
option he_su_beamformee '1'
option he_mu_beamformer '1'
option he_mu_beamformee '1'
option txpower '19'
config wifi-iface 'default_radio0'
option device 'radio0'
option network 'lan'
option mode 'ap'
option key '<redacted>'
option dtim_period '3'
option ssid '<redacted>'
option ieee80211r '1'
option ft_over_ds '0'
option ft_psk_generate_local '1'
option reassociation_deadline '20000'
option ieee80211k '1'
option time_advertisement '2'
option time_zone 'GMT0'
option wnm_sleep_mode '1'
option bss_transition '1'
option rrm_neighbor_report '1'
option rrm_beacon_report '1'
option encryption 'psk2+ccmp'
option ieee80211w '1'
option max_inactivity '15'
option mbo '1'
config wifi-iface 'default_radio1'
option device 'radio1'
option network 'lan'
option mode 'ap'
option key '<redacted>'
option dtim_period '3'
option ssid '<redacted>'
option ieee80211r '1'
option ft_over_ds '0'
option ft_psk_generate_local '1'
option reassociation_deadline '20000'
option ieee80211k '1'
option time_advertisement '2'
option time_zone 'GMT0'
option wnm_sleep_mode '1'
option bss_transition '1'
option rrm_neighbor_report '1'
option rrm_beacon_report '1'
option encryption 'psk2+ccmp'
option ieee80211w '1'
option max_inactivity '15'
option mbo '1'
...
Many thanks indeed @_FailSafe for posting your config. Here is mine:
root@OpenWrt:~/cake-autorate# cat /etc/config/wireless
config wifi-device 'radio0'
option type 'mac80211'
option path 'platform/18000000.wmac'
option channel '1'
option band '2g'
option htmode 'HT40'
option country 'GB'
option noscan '1'
option cell_density '2'
config wifi-iface 'default_radio0'
option device 'radio0'
option mode 'ap'
option wds '1'
option ssid 'XXX'
option encryption 'psk2'
option key 'XXX'
option ieee80211r '1'
option reassociation_deadline '20000'
option ft_over_ds '0'
option ft_psk_generate_local '1'
option network 'guest'
option dtim_period '3'
config wifi-device 'radio1'
option type 'mac80211'
option path '1a143000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0'
option channel '36'
option band '5g'
option htmode 'HE80'
option country 'GB'
option he_bss_color '8'
option he_su_beamformee '1'
option cell_density '2'
config wifi-iface 'default_radio1'
option device 'radio1'
option network 'lan'
option mode 'ap'
option wds '1'
option ssid 'XXX'
option encryption 'psk2'
option key 'XXX'
option ieee80211r '1'
option reassociation_deadline '20000'
option ft_over_ds '0'
option ft_psk_generate_local '1'
option dtim_period '3'
config wifi-iface 'wifinet2'
option device 'radio0'
option mode 'ap'
option ssid 'XXX'
option encryption 'psk2'
option key 'XXX'
option ieee80211r '1'
option reassociation_deadline '20000'
option ft_over_ds '0'
option ft_psk_generate_local '1'
option network 'lan'
option dtim_period '3'
As you can see my config is super simple. Any idea what setting I should try from your config as you have a lot of extra settings set? It may be significant that my RT3200's are connected to each other via WDS.
I am trying to think what out of your settings might be the most significant. For example what about 'ieee80211w'? Or ' 'wnm_sleep_mode'?
Not that any of these are certain fixes, but a few things that stand out to me are:
cell_density - You have '2' and I have '0'he_bss_color values or are all set to '8'? (My he_bss_color is unique per AP)Interesting! One thing I've noticed is that not all Apple devices are affected by that bug. An iPad 5th gen I have roams without issues, and an iPhone 8 that used to be in the household roamed just fine too. The bug got introduced at a later point it seems.
Just out of curiosity, none of your system logs contain long roam attempts like mine here?
Sun Feb 26 17:02:04 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:02:04 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:02:04 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:02:04 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 MLME: MLME-AUTHENTICATE.indication(66:d8:12:d5:a0:02, FT)
Sun Feb 26 17:02:19 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:02:19 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:02:19 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:02:19 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 MLME: MLME-AUTHENTICATE.indication(66:d8:12:d5:a0:02, FT)
Sun Feb 26 17:02:33 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:02:33 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:02:33 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:02:33 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 MLME: MLME-AUTHENTICATE.indication(0e:84:ea:0b:b3:37, FT)
Sun Feb 26 17:02:45 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:02:45 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:02:45 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:02:45 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 MLME: MLME-AUTHENTICATE.indication(0e:84:ea:0b:b3:37, FT)
Sun Feb 26 17:02:56 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:02:56 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:02:56 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:02:56 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 MLME: MLME-AUTHENTICATE.indication(66:d8:12:d5:a0:02, FT)
Sun Feb 26 17:02:59 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:02:59 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:02:59 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:02:59 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 MLME: MLME-AUTHENTICATE.indication(0e:84:ea:0b:b3:37, FT)
Sun Feb 26 17:03:55 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:03:55 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:03:55 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:03:55 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 MLME: MLME-AUTHENTICATE.indication(0e:84:ea:0b:b3:37, FT)
Sun Feb 26 17:04:28 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:04:28 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:04:28 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:04:28 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 MLME: MLME-AUTHENTICATE.indication(0e:84:ea:0b:b3:37, FT)
Sun Feb 26 17:04:30 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:04:30 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:04:30 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:04:30 2023 daemon.debug hostapd: phy0-ap0: STA 0e:84:ea:0b:b3:37 MLME: MLME-AUTHENTICATE.indication(0e:84:ea:0b:b3:37, FT)
Sun Feb 26 17:04:46 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:04:46 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:04:46 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:04:46 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 MLME: MLME-AUTHENTICATE.indication(66:d8:12:d5:a0:02, FT)
Sun Feb 26 17:04:48 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:04:48 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:04:48 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:04:48 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 MLME: MLME-AUTHENTICATE.indication(66:d8:12:d5:a0:02, FT)
Sun Feb 26 17:04:51 2023 daemon.err hostapd: nl80211: kernel reports: key addition failed
Sun Feb 26 17:04:51 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: binding station to interface 'phy0-ap0'
Sun Feb 26 17:04:51 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 IEEE 802.11: authentication OK (FT)
Sun Feb 26 17:04:51 2023 daemon.debug hostapd: phy0-ap0: STA 66:d8:12:d5:a0:02 MLME: MLME-AUTHENTICATE.indication(66:d8:12:d5:a0:02, FT)
Thank you so much for your help here.
I understand that's a good idea because mine are fairly close together. This setting change from '0' didn't seem to affect this issue for me.
All '8'. Are they supposed to be different? I don't actually know what the significance of this feature is actually.
Seems like I should switch to 'HT20'. Presumably that's also slightly higher range as well owing to greater power concentration over narrower bandwidth.
This seems like it could be major. What do 802.11k and 802.11v do I wonder? Do they help roaming?
Wait a minute... You're using 802.11v as well? That's not really an apples to apples comparison with my setup then.
https://www.extremenetworks.com/extreme-networks-blog/what-is-bss-color-in-802-11ax/
Perhaps someone else can weigh in on whether BSS color only applies to differentiating between transmissions on identical base stations using the same frequency.
Yes, they're quite significant. Here is the Apple link that explains the purposes of them: https://support.apple.com/en-us/HT202628. There are certainly other sources outside of Apple that explain them as well.
Indeed--per the Usteer setup link I posted previously, I assume many readers (including myself initially) miss this important note:
Note: On previous versions of OpenWrt, the config option “bss_transition” used to be called “ieee80211v”. If migrating from an older version, please update this config-option too!
Usteer kicks devices to force transition rather than leaving it up to the client devices?
Seems from that link like BSS color should be different for all APs?
Hmm, ok. In that case 802.11v could be masking any issues in the 802.11r code by forcibly yeeting clients that stick to an AP.
Sort of, but out of the box it's less forcible than Dawn and is more of a "helper" for roaming. This probably explains it better than I just did: https://openwrt.org/docs/guide-user/network/wifi/usteer?s[]=usteer#setting_up_usteer_and_band-steering_in_openwrt
I have had really good success with Usteer (as opposed to Dawn). Here is how I have Usteer configured:
config usteer
# The network interface for inter-AP communication
option 'network' 'lan'
# Log messages to syslog (0/1)
option 'syslog' '1'
# Disable network communication (0/1)
option local_mode '0'
# Use IPv6 for remote exchange
option 'ipv6' '0'
# Minimum level of logged messages
# 0 = fatal
# 1 = info
# 2 = verbose
# 3 = some debug messages
# 4 = network packet information
# 5 = all debug messages
option 'debug_level' '2'
# Maximum number of neighbor reports set for a node
#option max_neighbor_reports 8
# Maximum amount of time (ms) a station may be blocked due to policy decisions
#option sta_block_timeout 30000
# Maximum amount of time (ms) a local unconnected station is tracked
#option local_sta_timeout 120000
# Maximum amount of time (ms) a measurement report is stored
#option measurement_report_timeout 120000
# Local station information update interval (ms)
#option local_sta_update 1000
# Maximum number of consecutive times a station may be blocked by policy
#option max_retry_band 5
# Maximum idle time of a station entry (ms) to be considered for policy decisions
#option seen_policy_timeout 30000
# Minimum number of stations delta between APs before load balancing policy is active
#option load_balancing_threshold 0
# Minimum number of stations delta between bands before band steering policy is active
#option band_steering_threshold 5
# Interval (ms) between sending state updates to other APs
#option remote_update_interval 1000
# Number of remote update intervals after which a remote-node is deleted
#option remote_node_timeout 10
# Allow rejecting assoc requests for steering purposes (0/1)
option assoc_steering 1
# Allow ignoring probe requests for steering purposes (0/1)
#option probe_steering 0
# Minimum signal-to-noise ratio or signal level (dBm) to allow connections
#option min_connect_snr 0
# Minimum signal-to-noise ratio or signal level (dBm) to remain connected
#option min_snr 0
# Timeout after which a station with snr < min_snr will be kicked
#option min_snr_kick_delay 5000
# Timeout (ms) for which a client will not be steered after rejecting a BSS-transition-request
#option steer_reject_timeout 60000
# Timeout (in ms) after which a association following a disassociation is not seen
# as a roam
#option roam_process_timeout 5000
# Minimum signal-to-noise ratio or signal level (dBm) before attempting to trigger
# client scans for roaming
#option roam_scan_snr 0
# Maximum number of client roaming scan trigger attempts
#option roam_scan_tries 3
# Retry scanning when roam_scan_tries is exceeded after this timeout (in ms)
# In case this option is set to 0, the client is kicked instead
#option roam_scan_timeout 0
# Minimum time (ms) between client roaming scan trigger attempts
#option roam_scan_interval 10000
# Minimum signal-to-noise ratio or signal level (dBm) before attempting to trigger
# forced client roaming
#option roam_trigger_snr 0
# Minimum time (ms) between client roaming trigger attempts
#option roam_trigger_interval 60000
# Timeout (ms) for client roam requests. usteer will kick the client after this times out.
#option roam_kick_delay 10000
# Minimum signal strength difference until AP steering policy is active
#option signal_diff_threshold 0
# Initial delay (ms) before responding to probe requests (to allow other APs to see packets as well)
#option initial_connect_delay 0
# Enable kicking client on excessive channel load (0/1)
#option load_kick_enabled 0
# Minimum channel load (%) before kicking clients
#option load_kick_threshold 75
# Minimum amount of time (ms) that channel load is above threshold before starting to kick clients
#option load_kick_delay 10000
# Minimum number of connected clients before kicking based on channel load
#option load_kick_min_clients 10
# Reason code on client kick based on channel load (default: WLAN_REASON_DISASSOC_AP_BUSY)
#option load_kick_reason_code 5
# Attempting to steer clients to a higher frequency-band every n ms.
# A value of 0 disabled band-steering.
#option band_steering_interval 120000
# Minimal SNR or absolute signal a device has to maintain over band_steering_interval to be
# steered to a higher frequency band
#option band_steering_min_snr -60
# Interval (ms) the device is sent a link-measurement request to help assess
# the bi-directional link quality. Setting the interval to 0 disables link-measurements.
#option link_measurement_interval 30000
# Script to run after bringing up a node
#option node_up_script ''
# Message types to include in log
# Available types:
# - probe_req_accept
# - probe_req_deny
# - auth_req_accept
# - auth_req_deny
# - assoc_req_accept
# - assoc_req_deny
# - load_kick_trigger
# - load_kick_reset
# - load_kick_min_clients
# - load_kick_no_client
# - load_kick_client
# - signal_kick
#list event_log_types ''
# List of SSIDs to enable steering on
list ssid_list '<redacted SSID 1>'
list ssid_list '<redacted SSID 2>'
list ssid_list '<redacted SSID 3>'
The most significant modification from OOB settings (outside of adding your SSIDs to the ssid_list is this:
# Allow rejecting assoc requests for steering purposes (0/1)
option assoc_steering 1
Here's another link to those interested in some ongoing Usteer conversation: Lets talk about usteer
I would caution that you start minimal with Usteer and only start tweaking once you have a good baseline established.
See my minimal config here: 802.11ax worse than 802.11ac with mt76 driver? - #292 by _FailSafe
It seems from reading like I should set bss_color to 1, 2 and 3 for my 3x AP's (5.4Ghz only, since 2.4 on RT3200 doesn't support AX). This ought to be helpful because they all operate on same frequency given my use of WDS. Does that seem correct to you?
I forgot about your APs being on the same channel due to WDS! My understanding is that you would be correct in setting the BSS color of each AP to a unique number. This seems even more beneficial in your case than mine given my APs are all on different channels.
I'm super hopeful that one of these changes might fix my problem. Many thanks indeed.
So for a WDS server/client pair is it better to set he_bss_color to the same or different values on each device?
I was avoiding dropping a Cisco link in here originally, but I feel like the inquiring minds need more details than I can confidently provide. 
Reference link: https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-7/config-guide/b_wl_17_7_cg/m_bss_coloring_ewlc.pdf
Specific excerpts from the link:
BSS Coloring is a new provision that allows devices operating in the same frequency space to quickly distinguish between packets from their own BSS and packets from an Overlapping BSS (OBSS), by simply looking at the BSS color value contained in the HE PHY header. In some scenarios, Spatial Reuse allows devices, to transmit at the same time as the OBSS packets they receive, instead of deferring transmissions because of legacy interference thresholds. Since every Wi-Fi 6 device understands the BSS color, it can be leveraged to increase power savings by dropping packets earlier, and to identify spatial reuse opportunities.
BSS Coloring is a method used to differentiate between the BSS of access points and their clients on the same RF channel. Wi-Fi 6 enables each AP radio to assign a value (from 1 to 63), known as BSS color, to be included in the PHY header of all HE transmissions from devices in its BSS. With devices of each BSS transmitting a locally-unique color, a device can quickly and easily distinguish transmissions coming from its BSS from those of a neighboring BSS.
With all that said, I'm not sure exactly how this relates specifically to WDS. I am not a WDS expert. Might be a question better addressed by experts like @nbd.