Hello there!
I am using a PPPoE over Fiber 100/50 connection with my provider's modem in bridge mode and my LEDE router doing the PPPoE call and I also get proper IPV6 (no Teredo or anything like that). My question is:
Since I'm using SQM-Scripts, what is the proper overhead and MTU setting that I should set?
I couldn't find anything really relevant on the internet.
Thanks!
So if you instantiate sqm on pppoe-wan (as recommended) you will need at least to specify an overhead of 8 for the PPPoE encapsulation (2 Byte PPP + 6 Byte PPPoE) that will be performed by the kernel. For everything else we need to guess. Maybe if you could post a link to your ISPs website with your relevant bandwidth plan that could help. Based on mostly thin air I would assume that you probably send full ethernet headers with frame check sequences over the link so I would probably add:
4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) + 2 (ethertype) = 18 byte
That way you would add a total of 26 bytes. Now your link might be using a VLAN tag, ten add another 4bytes. About the encapsulation on the fiber itself I have no clue (and it will most likely depend on the definition of fiber and the actual technology, say FTTC versus FTTH and active optical network versus passive optical network). Unfortunately I have no simple solution how to deduce the applicable overhead empirically (in theory measuring the link capacity with multiple different MTU/MSSs should allow to empirically find the fixed per packet overhead, but that is far from simple).
I hope other's with more fiber experience will chime in as well....
Best Regards
Thanks for that, that was a very detailed reply. The only certain thing I know is that my fiber to home is literal fiber with a fiber modem up until the modem itself, with the outside cable coming directly into the house, and I know that they are using VLAN. Unfortunately the modem itself is completely locked, I enabled bridge mode through what is essentially a hack with a direct URL.
Ah, thinking a bit more about it, I assume wanting to know the overhead on the fiber link probably is moot. I assume the fiber itself will transmit way more that 100/50, so there is a traffic shaper involved at your ISP, all you need/want to know is which overhead and bandwidth that shaper is set to. In my case my VDSL2 carrier shapes at the BRAS/BNG level, but has the shaper configured to have exactly as much overhead as the VDSL2-PTM, the challenge is to find the bandwidth setting of the ISPs shaper. (Then again it is not that hard, simply start with say 50% of the contracted bandwidth and then do a binary search of the bandwidth that still shows acceptable latency under load increase, and just do it for both directions).
That is relatively easy, in SQM scripts ignore the tcMTU setting completely, while called MTU this really just affects up to which packet size the size tables will be calculated, it has nothing to do with the real MTU. Unless you have set your ethernet interface connecting the router to the modem you will have the standard ethernet MTU of 1500 otherwise you would have needed to enable jumbo frames on that interface. Some ISPs (e.g. BT) use so called baby-jumbo frames of 1508 bytes, so that the 8 byte PPPoE overhead does still allow MTU 1500 for its payload, but I assume that this level of thoughtfulness is rare... Anyway to confirm the MTU follow the advise on e.g. http://muzso.hu/2009/05/17/how-to-determine-the-proper-mtu-size-with-icmp-pings...
Best Regards
I just did a full run of the ATM_overhead_detector script from the router itself (provided by another user in the forum as a link). I have the data and I'm in the processing phase.
As for MTU size, my guess is that it's really 1492. Since it's impossible to have the proper version of ping in LEDE, I did it from my windows machine. The only packets that don't fragment are 1464 in size. That means 1464+28 for headers=1492. I hope my guess is correct. That would also mean that the minimum overhead for SQM should be 28 bytes.
Are all these assumptions correct? I'll return to the topic once I pass the script through R.
EDIT: I just realized you're the person who wrote the script, LOL! 
The program gave me a 30-byte size packet, which is abit odd. I believe that 28 seems to be correct in the end.
Here is the report from it:
Found 145000 ping packets in D:\USER\Desktop\ping_sweep_ping_test_20170427_175804_NOSQM.txt
Elapsed time is 456.453 seconds.
Minimum size of ping payload used: 16 bytes.
warning: division by zero
warning: called from
ATM_overhead_detector at line 201 column 15
warning: legend: ignoring extra labels
warning: called from
legend at line 428 column 9
ATM_overhead_detector at line 320 column 1
Saved figure (1) to: D:\USER\Desktop\ping_sweep_ping_test_20170427_175804_NOSQM_data.png
lower bound estimate for one ATM cell RTT based of specified up and downlink is 0.014296 ms.
estimate for one ATM cell RTT based on linear fit of the ping sweep data is 0.014296 ms.
Starting brute-force search for optimal stair fit, might take a while...
Best staircase fit cumulative difference is: 0.50106
Best linear fit cumulative difference is: 0.29849
Quantized ATM carrier UNLIKELY (cummulative residual: stair fit 0.50106 linear fit 0.29849
remaining ATM cell length after ICMP header is 23 bytes.
ICMP RTT of a single ATM cell is 0.012152 ms.
Estimated overhead preceding the IP header: 30 bytes
Saved figure (2) to: D:\USER\Desktop\ping_sweep_ping_test_20170427_175804_NOSQM_results.png
According to http://ace-host.stuart.id.au/russell/files/tc/tc-atm/
30 bytes overhead indicate
a protocol stack this program does NOT know (yet)...
Add the following to both the egress root qdisc:
A) Assuming the router connects over ethernet to the DSL-modem:
stab mtu 2048 tsize 128 overhead 30 linklayer atm
Add the following to both the ingress root qdisc:
A) Assuming the router connects over ethernet to the DSL-modem:
stab mtu 2048 tsize 128 overhead 30 linklayer atm
Elapsed time is 492.058 seconds.
Done...
Hi PrMinisterGR,
I can guarantee, that ATM_overhead_detector will fail to successfully predict the overhead on your link correctly. I relies on the fact that ATM/aal5 cause a specific steplike increase in packet transfer time, that simply does not exist in that form on other technologies. Now it is possible that your ISP uses ATM on its fiber links but it is very very unlikely. So unlikely in fact, that I would not bother to run this on your link at all.
Regarding the 28 bytes of overhead you noticed above, these live inside the IP packet, it is 20 bytes for the IPv4 header itself and 8bytes for the ICMP header. These are irrelevant for sqm since sqm sees these headers simply as part of the payload, so in other words these are accounted for, it really is those headers on top or around the IP packets that need to be accounted for manually. And those are somewhat tricky to figure out. As I said in your case I expect at least 30 bytes overhead, but have no way to check this short of asking your ISP.
Your fragmentation test indicates that the fiber link has a MTU of 1500 (as your router adds the 8byte PPPoE headers, packets with MTU 1492 have size 1500 once they reach the modem.
And yes, clumsy fingers on a touch screen...
Best Regards
Thanks for the reply again 
I have ended up (for now) with these settings. They are the ones giving me constant A+ in everything when I do the bufferbloat/speedtest at dslreports, with 16/16 streams up/down and compression disabled.
Hi,
thanks for posting, typically it os shorter to post the output of "cat /etc/config/sqm" instead of using screen shots. Here is an example:
root@router:~# cat /etc/config/sqm
config queue
option debug_logging '0'
option verbosity '5'
option enabled '1'
option interface 'pppoe-wan'
option download '46246'
option upload '9545'
option linklayer 'ethernet'
option overhead '34'
option linklayer_advanced '1'
option tcMTU '2047'
option tcTSIZE '128'
option tcMPU '64'
option linklayer_adaptation_mechanism 'default'
option qdisc 'cake'
option script 'layer_cake.qos'
option qdisc_advanced '1'
option ingress_ecn 'ECN'
option egress_ecn 'NOECN'
option qdisc_really_really_advanced '1'
option iqdisc_opts 'nat dual-dsthost'
option eqdisc_opts 'nat dual-srchost'
option squash_dscp '0'
option squash_ingress '0'
That subtle UNLIKELY in there tells you that you have no ATM carrier. In that case the estimated overhead really is untrustworthy as the method depends on ATM cell quantization.
So I would recommend to set the overhead to 34, and select ethernet as the linklayer. You selected ATM which will result in two things:
a) The shaper will automatically account for ATMs expansion, where for every 48 payload size ATM uses an atm cell of 53 bytes (so you effectively shape 100-10048/53 = 9.4% below your set bandwidth values; that might still result in a decent shaper setting, but would it not be clearer to simply specify less bandwidth in the first place).
b) Since AAL5 requires that each user data frame is packaged into an integer number of ATM cells, where the last ATM cell is padded out to reach the 48 bytes payload size; specifying linklayer atm will tell sqm to account for that so that depending on the packet size sqm will add up to 47 additional bytes to account for the padding, this is especially dire for small packets as in the limit condition a data packet (including overhead) worth 49 bytes will require 2 atm cells, resulting in an ATM encapsulation cost of 100-10049/(53*2) = 53.77%, so more than half of your bandwidth will be lost to account for ATM's quirks. This is fine if you actually have an ATM carrier since the accounting reflects what is happening on the wire, but for non-ATM links this will be wrong.
I hope this helps.
Best Regards
Since there is literally no way to learn this from my provider, I followed your advice. Here's my sqm config. I get consistent A+ from dslreports, and it's the one that also enables me to have the largest possible upload. Thank you so much for all the replies here.
config queue 'eth1'
option interface 'pppoe-wan'
option debug_logging '0'
option verbosity '5'
option enabled '1'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option qdisc_advanced '1'
option squash_dscp '1'
option squash_ingress '1'
option ingress_ecn 'ECN'
option egress_ecn 'NOECN'
option qdisc_really_really_advanced '1'
option iqdisc_opts 'nat dual-dsthost'
option eqdisc_opts 'nat dual-srchost'
option linklayer 'ethernet'
option overhead '34'
option upload '53200'
option download '93600'
Reading your post I'm wondering if your ISP is doing the same as mine. Is your fibre terminating into a SFP module? You might be able to pull it out...
https://www.dslreports.com/forum/r31118482-Yes-you-CAN-bypass-the-HomeHub-3000
In my case there is no extra equipment. The fiber cable goes directly into the fiber port of the modem/router that they give. I set that one to bridge mode and that's it.
I guess I got lucky then that my ISP picked modem/router boxes that use separate SFP modules and that those can be pulled out.
Well, I'm getting a "real" 89Mbit down over 100MBps ethernet, and a "real" 50Mbit up, with zero bufferbloat and constant A+ in the dslreports test, as well as while running Speedtest.net with multiple pings on the back that stay stable.
I'm very very happy with my current setup 
can you pls show me how to do that?
A word of caution, the ping collector scripts create logfiles that are rather large (multiple MB to multiple dozens of MBs) and might easily overwhelm a router's memory, so make sure that your router has sufficient disk-space/memory in the directory you call the ping collector script from. If you have a flash/hard disk attached to your router you should be covered, but otherwise beware...
Also, please note that that script will only report trust-worthy overhead results for real ATM links, not docsis, fiber vdsl2/ptm links...
Best Regards
You need a "proper" computer to run that. I'm using an Intel Atom-based mini PC to do that. Unless your connection is really "weird", you might not even need to do it. What are you using exactly?
My connection is OK. My setup is rather typical. Windows PC and two tp-link routers (one in bridge mode). My hobby is online competitive gaming and I'm in search of the best low-ping and congestion free connection possible.
I'm also using a Netduma only for the Geofilter feature. Don't want to spam your thread though with my quest.
Well, the collection script itself, should work on the router itself (at least it did some time ago*); it is just as stated above that the logging is way to verbose and will drive most home router's into out of memory situations, so it is not recommended to run directly on the router unless one knows that there is enough space available.
Best Regards
So @moeller0, do you think the Wiki page of the SQM could be updated with this info about pppoe over Fiber?
My spanish ISP Movistar, uses the same approach, MTU 1492 and the router adds the 8byte. Also i have confirmed the ISP uses MTU 1500 all over their network, so my educated guess is that 34 bytes of overhead is the correct one.
What do you think?
BTW thank for the detailed explanation....
How? The only thing to add might be the fact that if using pppoe interfaces one needs to add 8 bytes but for the rest nobody knows well enough to give a correct recommendation that will fit in all cases. But 34 bytes is not wrong completely as it would cover the full ethernet overhead with double VLAN tags and pppoe, which might be a relative conservative estimate...
Your ISP might have some documentation about the frame sizes your on its network (if for example it needs to allow resellers to use its network, or if itself a reseller the company running the network might have some documentation about frame sizes for its resellers somewhere).
Best Regards
