SQM / Cake Link Layer Adaptation

I am not sure whether the details page will reach the intended audience :wink: as there is a less technical introduction page in the documentation. But sure most of what I can see on my phone seems an improvement, maybe with the exception of trying ATM as that will cost 9% bandwidth just for the 48/53 encapsulation, but I need to read it on a real screen first....
Thanks for taking the time to improve the documentation!

Edited the post to use blockquote, and added some enumerated list... should be easier to read now.

I'm going on the basis that our friend @deuteragenie was pointing to this documentation so someone like that will wind up here and I wanted them to get a holistic impression.

I figure based on reading the details that it's safe to overestimate the overhead on cable at 22. I think this is the only real discrepancy between what I put and what is on the simpler page. So I edited the simpler page to put 22 there, and to add a "when in doubt it's safe to overestimate, use 44" item

A couple of years back you helped me with figuring out the overhead for my VDSL connection and it ended up being 34, but the recommendation in IpenWRT is still 22. Why is the difference?

VDSL2 (IEEE 802.3-2012 61.3 VDSL2 with PPPoE and VLAN tag):
2 Byte PPP + 6 Byte PPPoE + 4 Byte VLAN + 1 Byte Start of Frame (S), 1 Byte End of Frame (Ck), 2 Byte TC-CRC (PTM-FCS), = 16 Byte

Residual Ethernetheader:
4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) + 2 (ethertype) = 18 byte

Total: 18+16 = 34 Bytes.

I would not recommend 22 for cable, as that is a value backed up by the standards, unless ds-lite is used, but I digress. The 22 for vdsl2 is much less likely than the other values.
I am fine with a "if in a hurry, set overhead to XX" preamble, but I would really like to keep the correct values, as far as these rather generic recommendations can be correct.
But let's ask Rich brown about his opinion, he wrote the sqm pages to begin with and is very good at keeping things simple, meaning he might agree with you :wink:

I'd like it to be accurate as well, but I think from a decision theory perspective, the cost of underestimating is higher than the cost of overestimating... so we should set the numbers in the "quick start" section to be the high end of what we think could happen for that technology (ie. assume there's a VLAN tag and a ds-lite and a whatever stacked up, and tell them to set that). Being too high by 2 or 4 or even 10 bytes has negligible downside for almost everyone who has 5Mbps or more. On the other hand, being too low by 10 bytes would be bad, because then when you're running a big download and you try to squeeze a VOIP call through... it'll break up because your actual bytes sent is larger than what the shaper thinks, so a buffer will form.

I think the see more details section below the list of basic advice on the details page should have absolutely accurate description of what the technology does down to the byte. If they're reading into that section it's now up to them to buy both pieces if they break it :wink: I didn't alter that section.

I'll see if I can summon @richb-hanover-priv

1 Like

The difference is due to 2 Byte PPP + 6 Byte PPPoE + 4 Byte VLAN which is not guaranteed to be used on all VDSL2-links, it just happens that your ISP uses that encapsulation. I have way too little information about typical VDSL2-encapsulations encountered in the wild to make a better guess.

Your friend here :slight_smile:

If it does not hurt too much, why not make 44 the default value that is used by Cake (or is it already the case ?)

And BTW, thanks to Cake my bufferbloat is significantly reduced and it not only shows on the DSL Report (A/A+), but also certainly feels on my daily use. Great work!

I think it would be a good idea to create a little tabular "components of overhead" table for the details section. If we do that what are some of the components we should have in it and what are the correct values? (I kind of know where to start looking this stuff up in wikipedia but it doesn't come to the tip of my tongue, perhaps we can come up with a quick list here)

  • VLAN
  • PPPoE
  • PPP
  • DS-Lite encap
  • Ethernet overhead
  • ATM padding
  • Carrier Pidgeon food...

Yes, and I am all for putting this in a preamble so everybody has a chance of digesting that :wink:

Unlikely on a DOCSIS link, but as far as I can tell not completely alien on other technologies.

That is rather hefty, as it will pull in an additional 40 byte IPv6 header, for a total of 58 bytes on DOCSIS, this is going to be fine if most traffic is IPv6, but in today's still majority? IPv4 world that is an unfortunate price to pay...

True :wink:

Yes, that's a big overhead, and it really should only be applied to ipv4 not to ipv6 packets. so let's leave that out for the moment or put it near the top of the more-details section.

Yepp, since I have no ds-lite link available for testing, I would rather stay quiet about the implications as I could only speculate....

I like your changes a lot (modulo the DOCSIS 22 byte recommendation :wink: ), now that I read them on a big screen. I also took the opportunity to clean up the DOCSIS section a abit, who acres about what DOCSIS really uses as overhead, it is completely opaque to the enduser, so let the cable ISPs and cablelabs worry about that level....

But remember, even if we put 100 bytes and there are really 0 bytes overhead, on normal bulk transfer using ~ 1500 byte MTU packets this has something like a 1/15 = 7% reduction in speed. A 1 hour download will now take 64 minutes....

On phone calls at 100kbit/s = 250 byte packets it's a 40% overhead so that you can only make 71% as many calls as you could otherwise. For a 10Mbps link that's 71 calls instead of 100... So if we're talking even a small office with say 20 people in it and no more than 10 calls at a time, it's still not a major downside, and particularly so if you said 100 bytes and actual amount was 38...

Truth is I don't think we should advocate 100 bytes of overhead for anyone, but on the basis of this kind of calculation, I'm inclined to over-estimate by 5 or 10 rather than under-estimate by 5-10, so if DOCSIS is really 18, and there are even 10% of DOCSIS users who have some kind of VLAN tag upstream to support a rollout of an ISP provided VOIP system or something, then 22 seems fine to me and if FTTP is really ethernet at 38, maybe we should say 42 or even 48, and if the VDSL is really 22 but PPPoE is pretty common, maybe we should say 34 like fantom-x has?

EDIT: and then encourage users to bump their speed estimates up closer to sync speeds

EDIT: 10 bytes too much overhead on 250 byte voip packets filling your whole LAN is still only 4% excess overhead... worst case if we're encouraging people to overestimate overhead by 4% it seems like a good tradeoff, particularly if now they stop underestimating throughput by setting say 85% of measured speed and instead set something like 95% of measured speed.

and then accurate details in the details section.

And yes, I like your DOCSIS simplification too it was getting pretty full of irrelevant parentheticals the end user had little use for.

This is a bit tricky, setting ATM/44@pppoe-wanon aVDSL link will have a similar effect as setting the shaper bandwidth 100-100*(1500-20-20)/(ceil((1500+44)/48)*53) = 16.5% lower than given by the number in the config file, unless a link is short on bandwidth, this will result in better bufferbloat performance, than Ethernet-with-overhead/34@pppoe-wan (100-100*((1500-8-20-20)/(1525)) = 4.78 %) but partly because this increases the difference between the true bottleneck and the artifical one created by the shaper. So one easily can get fooled to set ATM by a quick and dirty test, but once small packets come into play this will be nasty (the border case of the AAL5 method is to drag in a full 53 for a single byte, if your packet 49 bytes this will cost you 106 bytes on the wire for a whopping 100-100*((49)/(106)) = 53.7% of overhead (on top on the normal IP/TCP headers). Now ethernet frames typically are at least 64 bytes, but the issue still is terrible at 482+1 = 97 frame: 100-100((97)/(3*53)) = 38.99 % overhead. Lots of small packets and the achievable bandwidth takes a hit.

This is the reason not to default to ATM/44 :wink: (especially since ATM/AAL5 is in the extinction process and will get rarer and rarer).

What do you think is the better recommendation? Try VDSL next and then ATM? Remember we're assuming the person is completely clueless as to what they have. Hopefully most OpenWrt users will know the difference between say cable and VDSL or FTTP and ATM... so we're talking to the near totally uninitiated at this point in the docs.

Given what's out there, probably the most common connections are DOCSIS, and VDSL, with FTTP next and ADSL last... but there are big swaths of rural US where ADSL is all you'll get :frowning:

I am not opposed to this, but I really think this is optimizing at the wrong end, as I have never seen VLAN usage from DOCSIS ISPs, but ds-lite is quite common. I am not haggling about this, if you want to set it back to 22, I will leave it alone, since I agree with your position.

I have avoided going that route as I have no reliable information about the prevalence of the different encapsulation schemes for the different technologies.

Nah, I dislike this, as only correct overhead settings allow real bandwidth settings, effectively over-accounted overhead results in a weird packet-size dependent over-shaping, once the overhead is wrong overhead bandwidth are less independent than they should be.

It is a gradual thing, as seen by the ATM/44 example, but I agree within reason a larger overhead is a decent recommendation for people in a hurry, or folks that have better things to do :wink:

Yes, and while I claim ATM is dying, it is not doing so fast; I would hope that ADSL/ATM/AAL5 users would know about their predicament...

The cake man page actually recommends that. The bigger issue is to totally on the safe side one would need to also default to ATM/AAL5 encapsulation, but that costs anywhere from 10-50% Bandwidth (dependent on packet-size distribution), a rather steep cost to pay for people on non-ATM carriers IMHO.

Yes this is true, but here's my basic assumption: small packet flows are almost always going to be a small fraction of the total bandwidth. Few people are saturating VDSL or FTTP or DOCSIS with telephone calls, or LAN party games. How else would you get large flows with 200 or 400 byte packets?

For the bandwidth sensitive flows, which are almost entirely TCP web downloads and uploads, a 10 byte over-accounted packet adds less than 1% of error. On a 5Mbps "slow" VDSL uplink, if you fire up 4 telephone calls and a game all at once, you will use in the range of 4*100 + 600 = 1000 kbps out of your 5000 or 20% of your upload. Your excess overhead is 10 bytes on something like 250 byte packets, so you're talking about over-accounting by 4% on 20% of your upload, or "wasting" around 0.8% of your total bandwidth. Compare that to telling people to set say 90% of their bandwidth to leave some overhead... and 0.8% compared to 10% is much preferred.

Yes at the worst case, if you're filling 5Mbps upload with 250 byte packets somehow and you have 10 byte excess you'll be losing 4% unnecessarily, but still better than losing 10% unnecessarily because you set 90% of your upload rate :wink:

Mild Over-accounting has negligible impact on typical high bandwidth flows, and has the good property that it reserves a little bandwidth extra ~5% for latency sensitive flows. Severe errors in accounting are not a good idea so I'd be against upping everything by 40 bytes over reality.

I guess we agree, here within measure it is better to over-estimate the overhead, than to underestimate it. And unlike my staw-man argument you are not proposing to make ATM/AAL5 accounting the default :wink: