After adding some prints to the driver, it would seem the failure originates
in build_dir/target-mips_24kc_musl/linux-ath79_generic/backports-5.15.58-1/net/mac80211/cfg.c in ieee80211_add_key() :
499 if (!sta || !test_sta_flag(sta, WLAN_STA_ASSOC)) {
500 ieee80211_key_free_unused(key);
501 err = -ENOENT;
502 goto out_unlock;
503 }
The TODO in this code fragment seems to be clear: "accept the key if we have a station entry and add it to the device after the station." This was noted before here: 802.11r Fast Transition how to understand that FT works? - #62 by cotequeiroz
Now figuring out how to do that 
--- update ---
According to patch #1626e0fa :
During FT roaming, wpa_supplicant attempts to set the key before association. This used to be rejected, but as a side effect of my commit 66e67e41 ("mac80211: redesign auth/assoc") the key was accepted causing hardware crypto to not be used for it as the station isn't added to the driver yet. It would be possible to accept the key and then add it to the driver when the station has been added. However, this may run into issues with drivers using the state- based station adding if they accept the key only after association like it used to be. For now, revert to the behaviour from before the auth and assoc change.
Hence it would seem that worst-case may be to loose the hardware key decryption... I guess I'll just see what my router does with the ASSOC condition removed...
--- update 2 ---
Removing the WLAN_STA_ASSOC test indeed get's rid of the error message. Roaming seems to be a bit faster (roaming between AP 1 with this change, and an identical AP 2 without this change). Will keep an eye on this to see if there are negative events popping up in daily use.