It is cleaner to use the built in ethernet port for WAN and the adapter for LAN.
It can also cause issues if you have inserted anything in that USB port or moved the adapter from another USB port.
I have all that equipment you have and never had issues using packet steering and SQM.
I know I got better SQM results with packet steering.
Can you detail how 480Mbps -10% overheads usb2 port managed to get to 500Mbps in any test?
I never tried it with USB2. That would have been one of the next steps if the problem had still occured.
One possible reason for drops on the UE300 could be power related. Make sure you have a power supply for your Pi4 that can source a full 5V @ 3A (15W). Less than that and you may end up with brown-out conditions when the CPU is taxed.
And, make sure you do not have other things connected to the USB ports since you need to make sure that your UE300 can always get enough power from that subsystem.