I followed a Reddit user's suggestion and I believe it is now resolved.
I also posted this to reddit in case it looks familiar, I just thought there might be more of a response here.
I’ve run into a bit of a snag using SQM with Cake + layer_cake. I tried piece_of_cake as well, much to the same result.
Let me preface this by saying my household has a 12/1 Frontier ADSL2+ line that at any given moment is hammered by a number of people/devices. First impressions of SQM were good, buffer bloat scored an A on DSLReports and you could still browse the web when someone’s iCloud backup decides it has to start now. For fun here's a test with the stock ISP gateway when no one's using it.
The issue is with how this fancy QoS plays along with our T-Mobile Cell Spot. It uses the home DSL connection to tunnel to their network – it's like a Wi-Fi router but for 4G LTE, and allows for cell phones to work where there's otherwise no cell signal. It's connected to a LAN port like any other device.
Right now, with the ISP’s stock do-it-all gateway, when any device is using this CellSpot over 4G, it seems to somehow get absolute priority on the network. This is great for phone calls, which are important given that there is limited reception where I am, and why we have to use such a device in the first place. The problem is that when something decides to saturate the uplink over 4G data, which is also provided by the CellSpot and goes through the DSL line just the same, every other device on LAN will act as if it has lost its connection. I’ve seen it where nearly every single ping will timeout for minutes at a time, despite the DSL link still being operational. Any other device doing the same on LAN / Wi-Fi doesn’t cause quite the same issue, ping will skyrocket and sometimes packets will drop, but something eventually gets through. It seems the CellSpot uses UDP for tunneling, could that be related?
Now, with SQM enabled none of this happens, but since the CellSpot is not able to be so ‘pushy’ with SQM enabled, phone calls going through it now cut out when another device on LAN saturates the uplink. My simple test was uploading a video to YouTube while trying to make a phone call. It never cut out entirely but it sounded robotic enough that it was practically unusable, clearly being bandwidth starved. I could hear them fine (download), but they couldn't hear me (upload). All the while I ran a test on DSLReports with another device, which once again gave an A for bufferbloat (although quality dropped?), and I was able to do web browsing where before such an upload would make the connection quite unusable (except for somehow to the CellSpot).
I guess this giant wall of text is all to say – is there any way I can give priority to phone calls over the CellSpot while at the same time forcing it to ‘play nice’ with all the other devices on the network. From what I can tell it at least does its own internal QoS, since if someone were saturating the 4G data link, a call was still perfectly clear. Could I mix in classical QoS, perhaps guaranteeing half the uplink?
I am using a pretty generic router for all this, but it seems to run OpenWRT fine and CPU usage barely topped 20% under a fully loaded connection (it's using a dual-core MT7621AN MIPS SoC).
Thanks to anyone who make it this far through the post.