Let's start with the fake 1.2 Gbps number, and to be very clear it does not matter what the vendor says. Your connected device is likely the one limiting you. Let's say it's a common device 2x2 MIMO (MCS 9, doubt you can get MCS 11), you will be talking about 960 Mbps which 60% (if you are lucky and your air is not congested in the channel) will be your real bandwidth (one way, as WiFi is not full-duplex). So, there you go 576 Mbps.
Now, if your bandwidth with the ISP is 300 Mbps, you should configure your SQM to be between 85-95% of it, let say 90%. To see the real benefit of SQM you must use 270 Mbps as the bandwidth configured in SQM.
QQ, is your AX1800 running OpenWrt? If not, expect spikes of +20 ms at least under heavy load. For a real world example see the following:
- ISP -> RBPi4 (SQM) -> Dumb AP WiFi 5 (Nano HD running OpenWrt with AQL/AQM support)
- ISP bandwidth 1000/50 Mbps
- SQM bandwidth 800/42 Mbps
Test ran while downloading a file in another computer (computer connection 2x2 MIMO MCS 9 867 Mbps), FYI.
https://www.waveform.com/tools/bufferbloat?test-id=9f6095d9-0924-45c0-84ca-ed3ea72de14c
In summary, TL;DR, what you are seeing when you don't limit your bandwidth (SQM set to 300 Mbps) is the poor dump AP latency in action.