you can disable those packet offloads, and in fact I think SQM does so automatically. If you have an x86 I would use that, and choose a decent switch with VLANs and QoS: I use a ZyXEL gs1900-24e and a few TP-Link sg108e...
With my whole family at home, zooming, gaming, youtubing etc we can have easily 8 to 10 devices active at once, including 2 or 3 lines of VOIP calls and kids gaming on minecraft etc, as well as at least 3 machines using an NFS server for home directories. it all works well. Of course I'm lucky enough to get gigabit fiber from ATT, but the thread on extreme gaming stability shows that with appropriate settings in customized scripts, it's possible to get rock steady game latency even at 700kbps. Since you have well over 3000kbps each direction, you should be able to get very good latency, less than 15ms round trip time even with regular SQM, and probably better than that if you mark VOIP packets etc and use diffserv4
HOWEVER that will be just latency on exit of your router heading towards the modem. If the modem is borked and fluttering around with insufficient CPU or intentional packet shaping issues to discourage torrenting, etc then all bets are off.
EDIT:
When it comes to latency issues, there are two very very sensitive applications: Gaming and VOIP. I think SQM doesn't quite get to the state where gaming and VOIP are comfortable at speeds less than about 3000kbps but this is more or less "to be expected". However, above that speed, with SQM and diffserv4 I'd be surprised if at least VOIP wasn't very comfortable. Again, assuming your modem doesn't bork things.