And that is A-OK, as I said it is a local policy each network admin needs to take for their network.
My point that this is an individual decision that each network needs to take based on local condition and requirements? What is controversial or requiring a proof? If for you no download shaping wirks well, by all means make this decision and be happy. But claiming, as you did,
is not a generally useful statement... for one the ISP will also limit upload traffic and the ISP might do this limiting in a way that result in high working latency and low responsiveness of the link under load. But as I said, I am not doubting that for your network this is not the right policy, just that this is true for all networks.
type or paste code hereNow, I am the first to admit that whether to use sqm or not s a policy decision each network needs to make individually; BUT there is no magical access rate from which on sqm becomes futile... if your ISP delivers acceptable latency under full saturating loads, by all means do not use local AQM, but that is true independent of rate...
Say your ISP uses e.g. libreqos for your download traffic then in all likelihood you do not need to run a local sqm instance yourself for the download even on a 20 Mbps link, but there are enough links in the 1 Gbps class with atrocious latency under load, where SQM can really improve things...
And on a DOCSIS 3.1 link the mandatory PIE instance in the upload direction might also be good enough for an individual network's requirements.
Hence I always recommend to try bot egress and ingress shaping to figure out for each individual network which policy to select.
I think I can construct an example where SQM download makes sense.
Suppose your Internet download speed is faster than your local network. For example if your Internet is 1.1Gbps and your local network is 1Gbps ethernet. In that case, the input queue to your router get filled at 1.1G and can only be drained at 1Gbps. Further suppose that the input queue is 1MB in length. The delay could be 125KB/1MB ==> 1/8 second, worst case. Every so often the queue overflows, and the router throws it all away, as it should. Then the delay drops to 0 and gradually increases again. So the average delay gradually increases to 1/8 second over a 10 second period, drops to zero, and repeats. In this case, an SQM limit of say 990,000 bps makes sense. That would eliminate delay.
I've never had a network like that, so I didn't think of it until just now.
A queue will fill if for long enough ingress > egress, it is that simple, if the queue is well managed that does not need to result in poor performance, but typically ISP do not manage these queues well and so ingress shaping becomes an improvement, IFF the link is actually saturated often enough. ISPs often operate their backbone sich that the load never significantly exceeds 80% of capacity and then no queues will fill up unduly (except when smoothing out the occasional burst, exactly what queues are designed to mitigate). But for an access link that is hard, escpecially given that e.g. TCP is designed with the goal to saturate every link...
Again, I am not doubting your decision for your network, I am just trying to explain the principles and why I recommend for every network to try ot figure out the best fitting local policy.
I got all the numbers wrong. Typing faster than I was thinking. Should have been a delay of 1 MBps / 1 Gbps ==> 8 ms. The bufffer would flush every 1 MBps / .1Gbps ==> 80 ms. I need to think before I type.
Good point. rsolis76 has a 2.5 Gbps ingress connected to a 1 Gbps egress. The average input is 1.5 Gbps, but the cable modem will be bursing data to the router at a rate above 1.5 Gbps. So there will be a queue backup in the egress. Probably the delay will average less than 10 msec, but it is a delay that could be reduced with SQM. I suspect that this particular router will be able to handle it w/o CPU saturation, but the CPU be checked under test conditions. CPU saturation can cause all sorts of problems.
To understand all this best, I recommend:
An Engineering Approach to Computer Networking: Atm Networks, the Internet, and the Telephone Network 1997. Srinisav Keshav
It's old, but the math surrounding networking hasn't changed. It's a remarkable book. Still available on Amazon.
The biggest issue typically is the queue at the ISP's side of the link. The ISP's device typically has more capacity towards the internet than to the customer and hence is likely to act as bottleneck link. Now such ISP devices often have buffers that are over-sized but under-managed, and that under load fill up and cause undesired delay. The whole idea behind ingress shaping is to move the effective bottleneck from the ISP side to one's homerouter, but that is only approximate, as high enough inrushing traffic might still cause (hopefully transient) queueing at the ISP side.
Just dont complicate your life with 20 variables to adjust. One day some telco engineer will run their favourite FPS game and change cake-dt to codel-dt in everybodys docsis profile.
Unlikely, DOCSIS mandates PIE and low latency DOCSIS a variant of the IETF's DualQ, neither cake nor codel are mandated, let alone proper fq scheduling...
Yeah, this is a great question. I didn't follow the details. It seems like fq_codel + BQL would by default be enabled on the link between the router and the switch and this should do its job. I didn't go back and look in detail to see how the performance was during that operation. Can you provide some quotes of the specific test examples where this difference was visible?
I did some experiments on my 1 G fiber link connected through an OpenWRT router on a 2.5 G lan to a computer on the lan. The results show some improvement using SQM. Perhaps the sweet spot is 970M down & up. That reduces latency variation by 10 msec when the link is saturated. Might be important to a 200 Hz gamer downloading one game while playing another.
Hard to find another situation where it is detectable to a human.
I was doing some tests today, trying to simplify the setup.
Basically just using 900mb limit on the download and 46mb limit on the upload.
And I found something interesting while comparing different qdiscs.
The thing is that if I use cake/piece of cake with the parameters previously shared in previous posts (Based on the openwrt SQM guide)
I get ratings that stil have bufferbloat, I tried some variations like not using advanced parameters such as nat, mpu, etc, and still same results. Both sites, waveform and fast dot com consistently show an increased latency in the order of 15ms to 25ms while using cake and piece of cake.
However if I use FQ_DEL with Simplest_tbf with the same limits, then I get very good results.
This is more or less the way things are supposed to work. Maybe more like 10-15ms but it's basically expected that you'll have some increase during load.
However, what you're describing may be caused by excessive CPU usage in Cake. You may be able to mitigate this by using received packet steering, irqbalance, and some other tricks to spread CPU usage among multiple cores.
As I said, you are responsible for setting policy for your link, and I will not second guess your decision as long as you are happy/satisfied. If your ISP does not over buffer, great, and if that ISP also does appropriate sharing under load even better. From my limited experience I would say not all ISPs are that enlightened.
The biggest cost, last time I looked closer some years ago, was the actual traffic shaper, the scheduler and AQM even with bells and whistles are comparatively frugal in their CPU needs. Simple/simplest/simplest_TBF.qos all use somewhat cheaper traffic shapers (and HTB TBF as configured by SQM will compromise throughput more than latency whgen CPU starved, while cake tends to allow more latency under load).
I wonder whether you would be willing to try HTB+cake and/or TBF+cake as that might be even better and allow some of cake's more advanced features? If yes, let me know and I will modify the respectice qos.scripts.
@hnyman, since simplest_tbf is your work, would you be opposed to simply allowing cake as leaf qdisc? This, cake as leaf qdisc (hanging off a HTB tree) is what libreqos uses quite successfully? If not I will whip up a new script...
But you get 810/63... so either fast.com fumbles the upload test and reports bogus numbers (IMHO that is likely, these numbers are probably NOT taking at the remote end, but simply are what the browser reported and modern browsers are works of marvel, but not the best environments for high precision measurements).
Here are the expected throughput limits for your configuration:
Your download (810 Mbps) looks OK (but could already indicate that the CPU will not allow the 850Mbps, hard to say, as most transfers are not 100% efficient), but the upload (63 Mbps) is off...
Please do, also, in simplest_tbf.qos try changing:
sqm_prepare_script() {
do_modules
verify_qdisc "tbf" || return 1
case $QDISC in
cake*)
sqm_warn "Cake is not supported with this script; falling back to FQ-CoDel"
QDISC=fq_codel ;;
esac
}
to
sqm_prepare_script() {
do_modules
verify_qdisc "tbf" || return 1
#case $QDISC in
# cake*)
# sqm_warn "Cake is not supported with this script; falling back to FQ-CoDel"
# QDISC=fq_codel ;;
#esac
}
And set cake as qdisc, to see whether the issue is cake's traffic shaper (which would then imply CPU starvation for cake as shaper).
That is also not wrong, but the increase for cake as shaper in download direction strikes me as higher than easily explained by cake's known behaviour.
I think the interrupts are reasonably distributed already...but that is certainly something to look at as well.
