I have a question about configuring SQM (Layer Cake) for a T-Mobile 5G connection. Without SQM enabled, I consistently get in the upper 600 Mbps (680 Mbps is probably average) to sometimes 750 mbps on the downstream. On the upstream, I consistently get over 100 Mbps (with a maximum of 153 Mbps and an average of about 130 Mbps).
However, when I enable SQM and specify 770 Mbps for the download bandwidth and 170 Mbps for the upload bandwidth, I usually get the download averaging around 600 Mbps and the upload averaging about 80 Mbps.
In the beginning, I accidally kept the upload shaping bandwidth at 22.5 Mbps (which was earlier used for my Comcast cable internet), and the speedtest showed between 15 Mbps and 16 Mbps. So, it appears that the that SQM induces about a 30%-35% bandwidth throttling with the WAN connection beting T-Mobile 5G.
I can't find any information on the size of the overhead for 5G to enter it in the SQM configuration page or any other settings, such as the minimum packet size, etc. So, I left all of those blank by selecting Link Layer as none. But before that, I tried Link Layer as Ethernet and put the overhead size of 50 bytes. It didn't really change anything vis-Ă -vis the bandwidth being throttled (both downstream and upstream) by with SQM enabled by 30%-35%. My router is capable of shaping at up to 910 Mbps (with Comcast) without causing any packet drops (or errors on the Ookla Speedtest app in macOS).
If anyone has any insight, please share it here. Thank you.
If you are wondering about the loss of bandwidth, this is just SQM doing it's thing. It works like this on every single connection type, part of the price. If you want to try and experiment on recovering some of that bandwidth or improve latency, then you could consider trying out Cake with adaptive bandwidth.
And in regards to the link layer adaption, unfortunately you have to experiment as no easy answer is found in regards to LTE and 5g, depends on your ISP. I personally leave it as is on my LTE connection as I never saw any difference
Thank you for your comment. I’ve been using Layer Cake on the Comcast cable internet connection for 2.5 years now. The loss of throughout with Layer Cake enabled is about 5-7%.
With the 5G T-Mobile Internet connection, I’m getting about 30% of throughout loss when I enable SQM with Layer Cake. So, there is something fundamentally different with L2 overhead on 5G. I have to disable SQM completely to recover this “lost throughout” on the T-Mobile 5G connection.
Can you elaborate on Cake with adaptive bandwidth?
I would get a packet capture of such speedtests and then look with e.g. wireshark how "bursty" the data actually flows, cake accepts some bursts, but if these get too large cake will reign them in resulting in the affected flows slowing down, costing you some throughput. This is unavoidable as these burst cause sojourntime (aka latency spikes) in cake's queues and cake responds to these. You can configure cake to use a larger interval (it will also adjust the target parameter accordingly) but by doing so you generally will have to accept higher latency and jitter.
One thing to keep in mind is that traffic shapers by necessity need to be configured with gross rates, while speedtests measure payload rates which are smaller than the gross rates.
e.g. even without a configured overhead value (linux will default likely to 14 bytes overhead) you can expect the following difference between gross shaper and measured goodput:
IPv4: 100 * ((1500-20-20)/(1500+14)) = 96.4% -> 3.6% point reduction
IPv6: 100 * ((1500-40-20)/(1500+14)) = 95.1 -> 4.9% point reduction
and more if you e.g. use TCP timestamps or other TCP options...
Now this is much smaller than the reduction you see, but it will contribute.
That is odd, on egress your router is unlikely to get back-pressure from the 5G-modem and hence should not notice the bursty transmission to the base station...
The thing is we really do not know (just as we do not now the actual gross rate for the 5G link, not that this would help much as it is likely variable anyway).
While I am always for setting the correct overhead values, here the problem is that without a correct gross shaper rate getting the overhead correct is pretty irrelevant...
You might want to have a look at either lua-autorate (sqm-autorate) or bash-autprate (cake-autorate) discussed over in this (monster of a thread):
As shown above that is generally okay given the theoretical limits...
I bet it is less smooth and more bursty medium access, which results in delays. In addition there was an observation that mobile schedulers apparently want to see some standing queue on their side before allotting more capacity to a specific link, if cake keeps the queue in the modem too short you suffer from getting less transmit slot or so.
See the links above...
Well, precise overhead really becomes useful once the precise gross rate is known and set, if the gross rate is too large getting the overhead slightly wrong is not going to matter.
What hardware do you have? It's likely you're overloading the router CPU with SQM at 700 Mb.
Non-adaptive SQM should be set slightly less than the worst case speed of the ISP. This means of course that during good times the speed you can use is still limited.
I have an overclocked Raspberry Pi 4B. It can shape up to 920 Mbps without errors while pushing up to 886 Mbps through the firewall when the ISP it’s connected to is Comcast. So, it’s not the router that’s causing this issue with the T-Mobile 5G connection.
I am running a OC rpi4 too, and yes it should be powerful enough to handle up to those speeds. Have you tried experimenting with packet steering and irqbalance?
Oh, and thanks @moeller0 for a much more thorough explanation
Edit2: TBH, I do actually think I might be observing a similar harder hit on bandwidth on my LTE connection with SQM enabled, the trouble for me is pinning down such changes since my speeds are considerably lower and HIGHLY variable depending on traffic on my tower. If those speeds you are seeing are rather constant, you have a better opportunity to troubleshoot and observe changes when you experiment.
Over what timeframe? Really in your shoes I would try to get a packet capture and look how smoothly the download data comes in... cake will be start dropping by default if the queue stays above 5ms for >= 100ms, so looking at averaged rates over say a second will likely hide problematic bursts.
That seems too good to be true to me. Or at least too good to be true forever. I believe many 5G base stations operate with a wireless backhaul with circa 10-20Gbps. Surely only a matter of time before the per-user bandwidth allocation waxes and wanes as more and more users connect to the base station.
This is a rather lucky situation and gives you a unique opportunity to try out and troubleshoot what works, if of course your data plan allows it at those speeds.
I personally have to average over several weeks to get a sense of changes I make, since my tower congestion gives me speeds from anywhere between 40 to 140 Mbps, and with latency measured up to a second in worst conditions, and that is even though I am similarly close to my tower.
Edit: with those speeds, have you tried taking a look an the statistics of your cpu cores when you are doing heavy downloads?
There are users close to those line speeds who have reported better results when using tools and methods that spread out the load better over several cores when using rpi4
Yes, I did yesterday, using top. Only one core is engaged, and the utilization goes up to mid 60%. I need to try to spread the load across two CPUs. I have a nanoPi R6S on order, which should help with CPU over utilization.
However, the same setup with Comcast can pull 918 Mbps, so I don’t think I’m hitting the one-core capacity on the Raspberry Pi4B.
This is unlikely to happen. There is Comcast cable and AT&T fiber in the neighborhoods around that offer much better download throughputs. Only nerds would do T-Mobile, and there aren’t many nerds around. There is really no point going T-Mobile unless one is in IT and is interested in this as a project. The time frame so far is 10 days. Multiple speed tests (probably 100 tests over the last 10 days) prove very consistent download and upload throughput and the latency between 50 ms and 60 ms to the closest Ookla Speedtest server, which is 2 miles away. Of course, I can’t vouch for what’s going to happen in the future. I can always fall back to Comcast if the throughput on T-Mobile drops or the latency goes up.
The only real reason for me to switch over to T-Mobile from Comcast is the upstream throughout, which I consistently get above 100 Mbps on T-Mobile but only 20 Mbps on Comcast. That, and unlimited data without a 1.2 TB cap like what Comcast imposes.
From personal experience, I can say that it can be hard to troubleshoot on LTE lines, there is a lot of technologies in play here.
But there are a couple of things you can try and test out right now that should help with utilising your cores better, so you can see if that doesn't help recover some potential bandwidth.
Enable packet steering, run some test, turn it off, run some tests
After that, install irqbalance, (needs to be activated too in script, search forums) run some tests and so on
I've just repeated the speed test via the Ookla app on the M1 Pro MacBook Pro (wired to a switch). The device running OpenWRT is a Raspberry Pi 4B overclocked to 2 GHz and using the same physical interface as a WAN and LAN logical interfaces (separated by VLANs).
1. SQM disabled:
Packet steering enabled: maximum CPU utilization on Core 0 is 53%
Packet steering disabled: maximum CPU utilization on Core 0 is 64%.
Note: Enabling or disabling packet steering doesn't seem to affect the downstream / upstream throughput or the latency. The Ookla speed test results are pretty consistent with packet steering enabled and disabled
2. SQM enabled:
Packet steering enabled: maximum CPU utilization on Core 0 is 83%
Packet steering disabled: maximum CPU utilization on Core 0 is 74%.
Note 1: Enabling or disabling packet steering doesn't seem to affect the downstream / upstream throughput or the latency.
Note 2: Enabling SQM results in a throughput penalty of about 25-30% compared to when SQM is disabled.
Note 3: Running a speed test with SQM enabled results in lower CPU utilization when packet steering is disabled compared with when packet steering is enabled, which is counterintuitive.
Doing these kind of tests, as you can see, are informative.
Now they might not tell you how to recover that hit of 25% you are experiencing with SQM on on this particular line, but it tells you where to maximise other areas of the rpi4.
Going through tests like these are basically the only way to find out what works or not on your particular line.
There is another question you might also want to ask, if enabling SQM gives you enough benefit in your use case that you are willing to sacrifice the bandwidth?
Because it is not necessarily the case that it is possible to recover it, and it might take a lot of experiments to get some results.
I know that in my usecase with considerable less bandwidth than yours, SQM delivers some other benefits that I can use, but this is all personal preference
Any updates to this?
I am also using T-Mobile 5g Home Internet and experimenting with SQM.
Always looking to change OpenWrt config values to something better...
