SQM (cake) overhead compensation for PPPoE link that is not *DSL

Installing and Using OpenWrt Network and Wireless Configuration
1

Hi.

I'm trying to configure SQM for my link and I'm really confused about all the overhead types.

Every time PPPoE is mentioned, it seems to be only discussed in connection with some type of *DSL link. However, my ISP had me set up a PPPoE connection on a bare Ethernet cable that is going into my premises. So as far as I'm aware, there is no DSL modem. The service is rated for 100 Mbps in both directions; speedtest.net without any sort of SQM maxes out the connection at ~91 Mbps.

The SQM instance uses the layer_cake.qos script and is configured on the pppoe interface (pppoe-wan). With this in mind:

  • If I try to use any sort of ATM-based overhead compensation (pppoe-vcmux, or any sort of overhead N atm with N between 20-40), the connection settles on ~80 Mbps in both directions (regardless of N), with no bufferbloat.
  • If I try to use any sort of PTM-based overhead compensation (like pppoe-ptm) or Ethernet with overhead between 0 and 80, the connection settles on ~90 Mbps in both directions with noticeable signs of bufferbloat on the ping test.

What sort of overhead compensation should I pick for my link? Or should I leave it as is at 80 Mbps with ATM compensation and ignore the wasted bandwidth?

2

@moeller0 is the expert on this. I am going to guess that 'overhead 38' is the correct setting and I'm likely wrong.

4

that should be 22 classic ethernet overhead + 8 pppoe session overhead (VLAN if required take +4)

5

ppo 34 worked fine when I had VDSL2 on vlan35.

6

No wonder, ATM/AAL5 adds around 9% additional overhead (100-100*48/53 = 9.43% actually that is the best case, due to AAL5 the per packet overhead can packet-size dependently increase up to an additional 47 bytes). The overhead value between 20 and 40 will have a considerably smaller effect than the ATM/AAL5 effects so are probably just "in the noise".

PTM, will really just assume a 64/65 encoding (albeit in a packet by packet manner) so results in >= 100-100*64/65 = 1.54% less goodput, so much less than ATM's ~9.4%.

My question is what gross shaper rate did you configure?

Nah, that is rather crude atm encapsulation is 'nasty'... and ptm encapsulation is better handled by simply reducing the gross shaper rate by 1.6%-age points....

The rule of thumb is that 38 bytes for full ethernet overhead, plus in your case 8 bytes for PPPoE should be enough, resulting in 46 bytes.

The way to test that is to make a throughput test with ~1500 byte MTU with overhead 46 bytes and iteratively decrease the gross shaper rate until bufferbloat is well controlled, and then repeat the experiment with clamping the MSS to a smaller value (like ~145 bytes if possible, reducing the packet size by a factor of ~10). If gross shaper rate and overhead are conservative, you should still see bufferbloat free measurements, if the overhead was set too small you will see bufferbloat. (CAVEAT: reducing the MSS will increase the packet rate by a factor of 10 and your router might not be able to actually deal with that, in that case reduce the MSS by a gentler factor, say start with a factor of 2-4)
See the OpenWrt sqm details wiki for some elaboration on the interdependence of overhead and gross shaper rate.

Hope that helps.

Almost, ethernet overhead is actually 38 bytes on the wire (what counts if you want to correctly shape for ethernet traffic), the points about adding 8 bytes for PPPoE anf 4 bytes for each VLAN tag are spot on.

2 Likes
7

Thank you so much for the comprehensive answer.

I should have really said this in the OP. The gross shaper rate was set to 100000 kbps, which is probably too much.

Alright. So, setting cake to overhead 46 mpu 84 noatm and iteratively reducing gross shaper rate by 1000 kbps at a time. It'll have to wait until the evening for the tiny packet stress test, but I'll certainly do that and get back with the results.

Just for completeness: what exactly is "well controlled", what should I look for?

8

I would guess so. Reverse calculation of your gross rate from your achievable goodput for TCP/IPv4 (likely what your test used) implies:
91 * ((1500-8+46) / (1500-8-20-20)) = 96.40 Mbps
so setting the shaper to 100 Mbps is likely to result in bufferbloat.

My rule of thumb is: run a few speedtest so you get a feeling which goodput you can reliably get, and then use that as gross shaper rate for sqm/cake. Yes, this will mean your shaper has some slack, but at that is generally a good idea to not set a traffic shaper to 100% of the true bottleneck rate...

That is the amount of bufferbloat you are willing to accept/tolerate; this is a local policy decision you need to take for your network (taking into account what is possible to achieve).

Personally, as a first quick and dirty method I tend to use any reliable speedtests (like speedtest.net, or https://www.waveform.com/tools/bufferbloat, fast.com) to load my link all the while running
gping 8.8.8.8 9.9.9.9 --buffer 60
to visualize the latency to two typically well-connected DNS-servers. The goal is to not see intolerable RTTs during the tests up- and download phases compared to the idle phases.

That is hardly a scientific test (for those I use flent, but for flent you need publicly available netperf servers, which are hard to come by).

Alternatively you could use Apple's networkQuality tool (if you use Apple Monterey) or the go-responsiveness version of networkQuality which you would need to build yourself.

time ./networkQuality --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config  --sattimeout 60 --extended-stats --logger-filename go_networkQuality_$(date +%Y%m%d_%H%M%S)

which result in output like:

08-24-2022 20:52:45 UTC Go Responsiveness to mensura.cdn-apple.com:443...
Creating a CSV data logger: go_networkQuality_20220824_225245-self-08-24-2022-20-52-45!
Creating a CSV data logger: go_networkQuality_20220824_225245-foreign-08-24-2022-20-52-45!
Creating a CSV data logger: go_networkQuality_20220824_225245-throughput-download08-24-2022-20-52-45!
Creating a CSV data logger: go_networkQuality_20220824_225245-throughput-upload08-24-2022-20-52-45!
Download:   7.369 Mbps (  0.921 MBps), using 20 parallel connections.
Upload:     2.748 Mbps (  0.344 MBps), using 16 parallel connections.
RPM:   249
Extended Statistics:
	Maximum Segment Size: 1440
	Total Bytes Retransmitted: 5632
	Retransmission Ratio: 9.47%
	Total Bytes Reordered: 2016000
	Average RTT: 768.9375


real	0m13.331s
user	0m0.464s
sys	0m0.442s

The reported rates should be around your true rates as seen in other speedtesrs, and the RPM value tells you how responsive your link is, RPM really is just a different way to show the latency under load, but since it is the reciprocal (RPM [1/min] = 60000/(RTT [ms]; 60000/RPM[1/min] = RTT [ms]) RPM gets larger the less latency under load you experience (lower latency under load typically is considered better).

2 Likes
9

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.