Well, i think you are right, still we don't for sure if the 34 byte is a one-size-fits-all solution :-D. In my case it is, but there are other ISP providers that use DHCP instead of PPPoE for WAN connections ... so maybe it is better to bookmark this thread.
I don't know about this (documentation), but in a support forum many users complaint about why PPPoE instead of DHCP, and also asked if baby jumbo frames were allowed. The official answer was no, only 1500 MTU. That's the info i know for sure.
lesandie Le Sandie
August 18
So @moeller0, do you think the Wiki page of the SQM could be updated with this info about pppoe over Fiber?
My spanish ISP Movistar, uses the same approach, MTU 1492 and the router adds the 8byte. Also i have confirmed the ISP uses MTU 1500 all over their network, so my educated guess is that 34 bytes of overhead is the correct one.
Well, using PPPoE on a standard frame size ethernet connection will lead to a MTU of 1492 after PPPoE decapsulation (on the link from your modem to the ISP the MTU from ethernet's point of view is 1500, it is only that the 8 pppoe bytes do not carry anything you are interested in and are therefore lost to you, baby-jumbo frames, as you probably know, are a work around in that each of your packets will still carry 8 "useless bytes" between the modem and the ISP, only with MTU1508 packets after the 8 bytes are removed you end up with the standards ethernet MTU 1500 towards the internet, but I digress).
The 34 bytes really cover the case that your ISP uses 2 VLAN tags, if he uses no VLAN tags you would have only 26 bytes, about moviestar I do not know exactly... but I would be amazed if the true overhead would be larger so I believe this is a decent bet...
It uses VLAN tags for 3 different VLANs, internet, TV and Voip. The pppoe connection is a virtual bridge configured in the ONT and i do not need to configure vlan tags in the router. So my educated guess in that the overhead would be 34 + 8 = 42 bytes.
Okay, so I would guess more like:
outside MTU1500:
4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) + 2 (ethertype) = 18 byte
As far as I can tell GPON GEM encapsulation will drop the following ethernet overhead parts:
FAST-ETHERNET: 7 Byte Preamble + 1 Byte start of frame delimiter (SFD) + 12 Byte inter frame gap (IFG) = 20 Byte
But it will add 5 Bytes for the GEM header plus an (to me) unknown number of bytes for lower level encapsulations, add you single VLAN tag ad you should get:
18 + 5 + 4 = 27 Bytes on top of an ethernet interface and
18 + 5 + 4 + 8 = 35 on top of an pppoe interface
But the question I can not answer is at what level does the GPON shaper actually work. From looking at the figures in https://sites.google.com/site/amitsciscozone/home/gpon/gpon-fundamentals it looks like downstream transmissions carry additional GTC frame variable information called the GTC header (containing the variable upstream BW map for requested upstream sending). But I guess each downstream slot is 38880 bytes long and the GTC header is 30 + N8 bytes long with N being >= the number of connected ONTs, so depending on your ISP's split ration that might be 32 (assuming one T-CONT per ONT) 32 + 328 = 288Bytes, but that seems 100*288/38880 = 0.74% be a cost of less than one percent so that might be okay to ignore...
So best guess currently would be Overhead 35 bytes or 39 bytes if your ISP uses Q-in-Q double VLANs (which might be completely invisible to you). THe three different VLANs in use simply use a different VLAN-ID number in the same VLAN header (as any packet can only belong to one VLAN anyway).
But take this with a grain of salt, as y experience with GPON is almost non-existent....
The good thing is you will need to set the shaper a tad below your provisioned rates anyhow and then getting the overhead slightly wrong typically is not catastrophic anymore (setting the overhead to high will simply sacrifice more throughput for the same latency under load performance). I guess what I want to say, maybe just test wether you are happy with the performance and if yes, just call it a day...
I've just confirmed that the ISP (movistar) uses Q-in-Q. They have a nice knowledge base i can query :-D. As always your reply is very thorough, thank you very much for that, as my experience with the link layer is very weak.
With a 34 overhead the shaper is performing nicely. I'll crank up to 39 to see how it behaves.
Doing a MTU empirical test i get this numbers:
Empirical MTU test completed [Tried,Actual] local->remote=[1522,1472] remote->local=[1525,1525] that is 33 bytes of overhead on top of a pppoe (1492) interface. I'll investigate further and post some results in a couple of days.
To be honest, as I hinted above my GPON experience consist out of looking at a few articles/descriptions Google found, I have not even looked into the ITU documents, so let e chime in "my link layer experience" is also weak, and my advice could be wrong...
Ah, that really is a nice find, could you post a link to that information here (just to help quench my curiosity)?
What kind of test is that, and against which server? Could you maybe post a link to the test (I am constantly on the lookout for interesting network tests that help debug odd situations).
(Some overhead like the 5 bytes GEM overhead will only exist on the GPON fiber segment and will be invisible from anywhere else). Well, you should still be able to measure them somehow, but only indirectly. (For example if you would know your exact provisioned rates you could test different overheads with the shaper set to that 100% bandwidth and all overheads smaller than the true overheads should lead to an increase of the latency under load (bufferbloat); now given that you have a fast fiber link this increase in latency might be hard to notice)
But just as an anecdote I have a VDSL2 link and I had set the overhead to what I know to be correct (my ISP documents that indirectly) and still saw bad bufferbloat when measuring at 100*64/65 = 98.46% of sync rate (to account for PTM's 64/65 encoding). It turned out my ISP actually has a traffic shaper at the BRAS/BNG level and I was simply admitting to much data onto the link and was easuring the queue increases of that shaper. Empirically I found acceptable rates and later got confirmation from another user that the bandwidth shaper settings of my ISP are very close to my empirically deduced limit shaper setting. For completeness on egress I am quite happy to set my shaper to 100% of that ISP shaper rate, but for ingress I still reduce the rate further to make the ingress shaper snappier and less sensitive to multiple inrushing flows, but I digress. My point is, a bit of systematic measurements can get you a long way in getting decent and robust shaper settings.
Here! the bad thing is that it is in spanish. Page 22. I hope you can use google translate :-D. If not, just tell me and i can translate some parts to you. Telefonica is the main company owner of the Brand Movistar.
About the test, it is nice that you ask. I was using the typical ping test, but i discovered that the new version of OpenVPN 2.4 has a empirical test you can activate in the config file or via command line.
--mtu-test
To empirically measure MTU on connection startup, add the --mtu-test option to your configuration. OpenVPN will send ping packets of various sizes to the remote peer and measure the largest packets which were successfully received. The --mtu-test process normally takes about 3 minutes to complete.
I have a Openvpn server running in my R7800 at home (Movistar fiber). That is the remote server i usually connect to from other locations. Also i connect from home to another location with a OpenVPN server, also movistar fiber hosted, with a similar router. All routers run LEDE 17.01 snapshot.
The results of the mtu-test are published in the openvpn.log, everytime i connect.
Just tell me if you need something else!
This is interesting, i can test that, increasing little by little the overhead until reaching the sweet spot
Thanks for the information, Spanish is not my forte so I guess all I can do is look at the diagrams but from looking at those I believe your interpretation to be correct.
Ah, that openvpn test looks useful, now all I need is to set up openvpn
For testing the effect of the specified overhead on bufferbloat I would recomment to perform the latency probe measurements while concurrently satuarating both uplink and downlink. https://github.com/tohojo/flent/tree/master/flent is a great tool to achieve this (look at the rrul or rrul_cs8 tests).
After doing some tests i can say that i'm very happy with the results (using 39 bytes of overhead).
I used, as @moeller0 pointed out, the flent tool and also at the scripts provided by the bufferbloat project at:
betterspeedtest.sh for speed test and for a RRUL test the netperfrunner.sh
I found some SQM cake config at this post:
with some pretty interesting tweaks by @r43k3n , but mainly for Internet Cable providers, because the tweaks are related to the link layer advanced options like tcMTU, TCtsize etc. I believe those options are for MTU > 1500 so for our "pppoe-wan" case they seem to be not elegible (pppoe MTU 1492)
These are only evaluated when use use tc_stab as LLA-method, I will at least hook up the tcMPU for cake, but ATM you will need the advanced configuration strings to tell cake about the MPU... So you are correct in that these values are not suitable for cake (just the rationale s different ;))
Sorry to bring this thread again but i was wondering... i'm also on fiber with 20mb/1mb net and noticed an improve in my net after setting sqm to ethernet and 35 packet overhead.
My question is... Im using a DIR-835 behind a Huawei HG8245H which i cannot put to bridge mode (because of my ISP settings, but is using PPPoE) though i have the huawei modem in DMZ, i believe im dual natting or at least adding a hop to the network... should i increase the packet overhead or keep these settings???
Mmh, that is nice then, but does your ISP use (PPPoE and) a VLAN-tag? The point is that specifying more overhead than necessary is equivalent with a (packet-size dependent) reduction of the bandwidth and will give the shaper more leeway. Conceptually one would like to measure the real overhead and account for that and then only reduce the bandwidth until the performances/latency under load increase becomes acceptable. But 35 might actually be the correct number for your link, all I want to point out that overhead and bandwidth are not simply orthogonal to each other...
Yes, as far as I can tell you are running a dualNAT setup there. But for the overhead accounting you really only need to model/specify the overhead applicable at the bottleneck link independent on all the links before and after. (But note for many internet flows the bottleneck will be completely outside of your control, so in reality you will have trouble statically accounting for anything more remote than your direct access link, as those links are typically shared between users and over-subscribed and hence do not guarantee fixed bandwidths per user).
BTW, 20/1? That seems quite "cruel" from your ISP to not go at least 20/10... One of the advantages of fiber is that it it allows much higher upstream bandwidth than the typically way more asymmetric XDSL or DOCSIS links...
Yeah, you have a DualNAT setup . DMZing the D-Link only opens all ports but the double NAT is still there. You should try to find a solution to put the huawei gateway into bridge mode, so you can configure the D-link wan interface for a pppoe wan connection. Also adding up some more "cruel" stuff, your ISP should provide you with the login/pass for the huawei gateway ...
Actually as @moeller0 said in a previous post, 35 overhead should be OK and you can test 39 if your ISP uses QinQ. If you are not sure about that 4 byte overhead difference, as @moeller0 said too, don't worry you'll notice a little bandwidth decrease but nothing to be worried about.
thanks all for your answers, yeah, applying 39 decreases bandwidth a bit.. i have the login and password for my router but the bridge and ppoe setup is first time only then it seems the settings are applied from the isp side so they become unavailable to change.. im from dominican republic and 1mb upload has been an standard for decades xD, there is up to 100mb download but it is still expensive, i could change my net to 20mb/3mb or 40mb/5mb but i feel comfortable with 1mb..
i will leave the setting at 35 packets overhead as it seems to work very well, when we have the ability to measure whats really happening on the traffic flow then i will change it.
btw, by vlan tag you mean if the modem is used for VoIP and IPTV? if so, the modem do have those settings but my contract is only for internet so i dont think the other vlan tags are active.
Well, a number of ISP simply use VLAN(s) on the access link independent of the usage, so also for internet access. That should be documented somewhere potetntially even somewhere accessible to end users
I've beed looking for a gpon onu on ebay so i can replace the huawei for another i can put in bridge mode, but do you think ill notice any improvement?? i really want to only have the dlink as my main thou..
also, is there any cheap gpon modem that you would recommend???
Did you try hooking up a serial cable to your Huawei?? I had to do that trick to get the ātelecom adminā login instead of a āuserā login. Then you can probably get it into bridge mode. As for the PPPoE login; most huawei just āhideā the password with HTML code; easy the change and view.
Older firmware huawei modems let you save (backup) the config settings; those include the required passwords/logins.
but from looking at those I believe your interpretation to be correct.
. DMZing the D-Link only opens all ports but the double NAT is still there. You should try to find a solution to put the huawei gateway into bridge mode, so you can configure the D-link wan interface for a pppoe wan connection. Also adding up some more "cruel" stuff, your ISP should provide you with the login/pass for the huawei gateway ...