rtl8xxxu is a very young driver, covering a large amount of different chipset versions and under quite active development. It might be that the current mac80211 snapshot is simply too old for rtl8xxxu (it probably doesn't really help that rtl8xxxu isn't used in LEDE's main target platforms, routers, so it probably isn't really in the main development focus of mac80211).