I am trying to configure SQM to eliminate buffer-bloat. I am using TP-Link Archer C6U router with latest currently available OpenWRT version.
My internet topology is like this:
Internet - OLT - Fiber - ISP ONT (Bridged, VLAN setting goes here) - OpenWRT Router - PC
My internet connection is FTTH, AFAIK its GPON. PPPoE is also used.
My question has probably been asked many times here and I apologize in advance for it.
What should I put in exactly in Link Layer Adaptation tab?
Link Layer :
Per Packet Overhead (bytes) :
I have read sqm and sqm-details page on OpenWRT docs. The pages say to put in "Fiber - choose Ethernet, and set overhead 44 (mpu 84)".
Is Ethernet frame overhead implicitly accounted in the above option? So I should only include 8Bytes of PPPoE overhead?
I am for RTFM, but I feel there is ambiguity in LuCI and docs, using ChatGPT has only furthered it. So here am I asking. I'd appreciate if anyone could explain their calculations too. I am no expert, far from it and have very shaky foundation in networking concepts. If my ignorance and dumbness annoys you, I humbly apologize.
Follow the instructions on the wiki; if you have any additional questions, you can ask me.
For GPON fiber/PPPOE connections, enter 44.
You don't need to add anything extra.
Turn on software stream acceleration, turn off packet tightening, and configure your download/upload shaping accordingly. That's how I use it; my router's hardware is a bit old, but it can handle the traffic this way.
You need to select this for PPPoe connection.
If this forum isn't tired of my questions, they won't say anything to you, you can relax.
First your routers CPU is fairly limited. like 200-300Mbps without offload 600-900 with sq offload.
(highly below 1000+1000 what gigabit wire is capable of)
First effort would be to measure latencies using https://www.waveform.com/tools/bufferbloat without offload and at 2 other offload levels. Reduced latency means CPU was overwhelmed just forwarding without QoS and choose least offload level that fills the wire.
So here is the issue, for traffic shaping to be effective it needs to properly account for the per-packet overhead at the bottleneck and the maximal gross rate at the same bottleneck. Sometimes it is relatively easy to deduce the relevant parameters of that bottleneck, but most often it is not, and so we need to use heuristics. Also note the relevant parameter can come from the properties of a specific link-layer but can also just come from an ISPs traffic shaper.
For typical ethernet things are relatively well documented so we know the frame overhead and typically we also know the fixed gross rate (e.g. 1000 Mbps for gigabit-ethernet).
For DOCSIS/cable the link-layer details are quite difficult (lot of things depend on how your ISP configured its system), BUT the docsis standards also mandate how end-user rates are shaped, including information about the relevant ethernet overhead (18 bytes, as the docsis accounting ignores a few overhead components relevant for real ethernet) so what we are missing is the ISPs configured gross rate, so we recommend to iteratively probe this.
For ADSL with ATM/AAL5 we can actually measure the overhead and the ATM/AAL5 cell-quantisation effects, so we are only looking for the gross rate (which might be as simple as figuring out the sync rate, but it might also vcome from a traffic shaper on the ISP side).
For FTTH things are tricky again, AON likely will behave like ethernet, so we have a decent idea about overhead, but gross rate might be limited either via the link speed or via an ISP side traffic shaper. For the PONs however (GPON, XGSPON) things get trickier, for one we can calculate the overhead on the wire per packet, but we have no idea about the ISPs traffic shaper configuration (which might be in the OLT or might be in the BNG or both). I have a theoretical idea how this could be measured, but that requires super precise and repeatable measurements that in reality this will likely never work (and since I have no FTTH link my self I am not actively pursuing that approach).
So given these uncertainties I recommend to try to err on the side of "a little too much" for the overhead and "a little too little" for the gross shaper rate (especially for ingress/download shaping, as that is approximate anyway and needs larger reserves than egress/uplink shaping).
As far as I can tell overhead 44 should work OK for FTTH as should mpu 84.. if your ISP uses PPPoE and you instantiate the shaper on pppoe-wan (where the kernel does not see the pppoe overhead) sure, add 8 bytes to the per packet overhead to be on the safe side.
What is the speed you have and what is the speed you want to shape it down to?
You need excess CPU headroom to shape traffic. Once CPU hits 100%, you are worse with shaping than without. Also, SQM seems to be inherently single-threaded. I am afraid that your 880Mhz MT7621 is unlikely to be able to shape traffic at higher speeds, unless we are talking <100Mbit.
Just so you understand, SQM will not give you higher speeds. It might give you better latency under load if you throttle down the speeds and your CPU is able to keep up.
100% correct, but sqm-scripts is really just a script top set-up kernel qdiscs... which essentially operate in single thread mode (per instance, one can still have the ingress shaper operate on one CPU and the egress shaper on another CPU). However mq-cake is on the way into the upstream kernel and that, at least on multi-queue capable ethernet NICs will allow to shape on multiple CPUs; but note this works by processing individual flows on a fixed CPU and hashing the flows to CPUs, so in a single flow test this will still be limited by what a single CPU can traffic shape...
My line speed is 100Mbps symmetric with 10% extra-provisioning if ISP person is to believed. So CPU capacity is non-issue.
Buffer-bloat grade stays A+ on Waveform and LibreQOS test (except for Bi-directional load test its A or B)
My concern was occasional packet loss and very high jitter which seems random at times but would subside turning off SQM instance. So I suspected any misconfiguration on my end.
PS. All offloads are off. On Router and NIC alike.
That is a dual core MIPS at 880 MHz, that will be hard pressed to shape 100/100 Mbps unless packet steering is enabled (for all CPUs) and only if the router does not do any other heavy lifting.
In such situations try logging into the router and run htop (configured to show detailed cpu times and also to not hide kernel threads) if one or both CPUs are running close to 90-100% (mostly SIRQ/softinterrupt context) then your SQM likely gets throttled.
Unfortunately cake's failure mode when running out of CPU is to let latency spikes come through (simple.qos with fq_codel should show a hard throughput ceiling but not really any latency spikes).
Just to be clear, do you mean putting 44 in the ‘Per Packet Overhead’ field? I remember reading underestimating is worse than overestimating. But it also said a gentle overestimation 44 > 8
As far as I can tell, I only have to acccount PPPoE overhead and nothing else. The VLAN ID is tagged on ONT after it leaves the router. So only 8 bytes.
I am assuming the drop-down menu ‘Link Layer : Ethernet with Overhead’ implicitly adds 38bytes of overhead into the calculation.
I am willing to trade bandwidth for stability (no latency spikes, dropped packets or jitter). My line speed is 100Mbps. I can shape it down to absolute minimum as i am the only user.
No, very much not so... unless your ISP is an incompetent doofus and also only accounts 8 bytes overhead in his traffic shaper.
You need to account for the overhead that will exist at the bottleneck even if you do not see that overhead on you side...
No, it is not doing this at all (this is why the GUI has the following hint for that drop down selection: "Which link layer technology to account for")... this allows you to switch between a "boring" ethernet link layer and funky ATM/AAL5 where each packet is transported inside an integer number of 53 byte ATM cells (with 48 bytes payload each) where the last cell is additionally padded with zeros if the actual payload (typically a partial ethernet frame) does not organically fill the last cell.
Try to select some text in a post, then a small pop up should pop up and offer "Quote" or "Copy Quote", the first will transform the selected text into a proper quote in the edit widget at the bottom of the page.
There are some tricks to save some CPU
a/ Easiest is to shape only upload direction
b/ a bit more complex - make "wan" port a member of "br-wan" bridge, bring all wan IP configuration to use br-wan, enable soft offload (between br-lan br-wan, other subnet bridges) then shape using sqm on the physical WAN port.
That tends to work better on asymmetrical links with really large download rates, but on a symmetric 100/100 I am not sure that is going to help all that much...
I wouldn't call my ISP most competent person but he does lend an ear to my problems. Being located in very remote region doesn't help either. I digress.
Don't we move up the bottleneck on our router by introducing shaper and SQM. (Isn't this akin to a frame limiter of a game where capping render framerate reduces input latency)
If not, how far down the line do I need to account overhead?
YEs, but to do that you need to make sure that the true bottleneck dies not (under some conditions) still be narrower than our artificial bottleneck, so we need to account for the overhead (and the rate) at the true bottleneck. We then need to set our shaper rate below the true bottleneck rate, and the shaper overhead to >= the true bottleneck overhead.
Great, you can then try to iteratively increase the shaper rates again...
I am on a VDSL2 link with 116.7/46.7 Sync, and I have may shaper set to 100/45 Mbps, the ingress shaping needs more bandwidth sacrifice to make accidental "back-spill" (when traffic rushes in so fast that it already saturates the true bottleneck before our shaper can signal flows to slow down) less likely. But by all means if 50/50 works well for your typical use cases try this a few days to confirm this really results in robust low latency performance-
44 > 8