WMM Mode causing unstable WiFi connection

Hi,
I use a Xiaomi Router 3G running OpenWrt 22.03.2 as an AP and have some issues with the WiFi connection. As soon as WMM is enabled on the WiFi interfaces the connection becomes quite unstable, this is best reproducible by copying files from my NAS to my laptop. It starts out copying at full speed (~100mbps via 2.4G) but then drops to 0b/s after 10-30s, it then stays at 0b/s for another 10-30s and resume copying at full speed. At first i thought this was caused by windows or the WiFi adapter in my laptop but after testing i found the exact same thing also happens on other devices.
Additionally i have a Netgear WAC104 as a spare AP that i tested this with and the same behaviour can be observed so that leads me to believe it's an issue with OpenWRT or my configuration.
Now disabling WMM fixes this issue but limits the rate to 54mbps which is just not acceptable for me.
The AP is also connected to my main PC via LAN (so acts as a LAN switch too) and there are no speed issues, it's just WiFi.
I checked the System and Kernel Logs and nothing is logged when this issue occurs.
Does anyone know what this could be caused by?
Thanks in advance!