SQM Link Layer Adaptation question (VDSL2)

Hi All

I am trying to figure out the right value for the Link Layer Adaptation setting in SQM.

My connection is a Fiber to the cabinet (FTTC, in the UK) VDSL2 (80Mbps Down/20Mbps Up).

I have a VDSL2 modem in bridge mode hooked up to a Linksys WRT1900ACS running the latest LEDE stable build.

I am having issues understanding the right value for my type of connection. But here are some clues if someone can help!

My ISP (TalkTalk) doesn't require any authentication on my side (they don't use PPPoE or PPPoA). My modem settings are attached. TLDR of settings; turn on Dynamic IP, assign correct VLAN tag.

Any ideas? I currently have the value set at 26. The rationale behind it is the following:

4 Byte (FCS) + 6 (dMAC) + 6 (sMAC) + 2 (Ethertype) + 8 (double VLAN tag?)

The results are okay, but I just want to make sure I am not wasting any bandwidth by overestimating my overhead :slight_smile:

Thanks in advance!
2211

That sounds sane, but it really depends on were the bottleneck actually sits at the DSLAM/MSAN level or at the BRAS/BNG level.

Using 8 bytes successfully. I tried also 16,24,32 bytes.

This reminds me, thre unfortunately is a difference between selecting the right overhead and configuring it. So could you please switch to using simple.qos/fq_codel without any overhead accounting, run a speedtest and report the output of "tc -s qdisc ; tc -d qdisc" here in the thread, please?

The issue is that for some interfaces (mainly true ethernet interfaces) the kernel will automatically add 14 bytes of overhead and we need to take these into account (so if the kernel adds these 14 bytes and you want a total of 26 you need to request overhead 12, but please note that cake currently has even deeper issues).

Thanks everyone for the input. Much appreciated. I am running the tests below with ethernet hooked up directly to my laptop and all other devices are disconnected from the router.

At this point I can't get A+ on all three things (quality,bufferbloat and speed). Here is a few things I tried:

SQM disabled:

disabled

fq_codel/simple.qos/none:

None-Fq_qodel

cake/piece_of_cake.qos/none:

None-Cake

cake/piece_of_cake.qos/8:

8

cake/piece_of_cake.qos/12:

12

cake/piece_of_cake.qos/26:

26

While running on fq_codel/simple, output of tc -s qdisc: https://pastebin.com/V95F4U70
Also while running on fq_codel/simple ouput of tc -d qdisc: https://pastebin.com/TYyNQ5Su

The ingress/egress limit for all of these tests is common: 69000/17400

Thank you so much for the input guys!

Sorry for the long post above. Quick update here. After a quick search around while waiting for more feedback, I saw this command in the forum:

ethtool -K eth0 tso off gso off gro off

I ran this on all my interfaces, and now with cake/piece_of_cake.qos/none, I get A+ on all three metrics:

28985390

Does this make any sense? Why do you think this happened?

From your pastebin:

maxpacket 1514 (eth0)

this indicates that your kernel adds 14 bytes so only request "overhead 12".

maxpacket 7370 (ifb4eth0)

this tells us that on ingress you operate GRO which ties in with your last post.

Could you post the ouput of "tc -s qdisc" and "tc -d qdisc" again with your final settings as well as the output of "cat /etc/config/sqm", please?

tc -s qdisc: https://pastebin.com/W1Nsabh0
tc -d qdisc: https://pastebin.com/iSE1g4jv

config queue 'eth0'
option qdisc_advanced '0'
option interface 'eth0'
option debug_logging '0'
option verbosity '5'
option enabled '1'
option download '70800'
option upload '18210'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option linklayer 'ethernet'
option overhead '12'

And here is a test over wifi, not necessarily having everything else disconnected:

28991910

Have you enabled baby jumbo frames on the WAN connection by setting the MTU to 1508?

Why should he do that? WIthout PPPoE eating into the MTU I see no reason for that as the MTU to the wider inoernet will not exceed 1500 anyways.

Best Regards

So egress no reports:

max_len 1526

which seems sane, but ingress reports

max_len 2974

which indicates that GRO still seems to be active (which in theory cake should handle okay)

70.800 * (1500-20-20)/(1526) = 67.737 Mbps
18.210 * (1500-20-20)/(1526) = 17.422 Mbps
So ingress seems to be okay, but ingress seems a bit low.
Could you run a speedtest while also running "top -d 1" via ssh on the router? Then keep looking at both the idle and the sirq number in the second row IIRC; if idle gets close to zero and sirq quite high, your CPU might not be fast enough to shape at the intended bandwidth....

Thanks.

Is there any way to confirm GRO is off other than 'ethtool -k [interface]'? Running this on every intreface I have, I can definitely see that GRO is off. Any ideas?

I just did a Speedtest while watching the output of top, the maximum sirq got to (end of downstream test) was 20%.

Really appreciate the feedback and insightful input!

As @moeller0 suggested, this isn't needed since my ISP doesn't use PPPoE.

Not that I know of, and if you also disabled GSO and TSO there should simply be no way to create these large super/meta packets, but as cake unlike fq_codel knows how to deal with these packets it probably does not matter too much, I just noticed from the tc output.

Ah, great that probably also means that idle was never close to 0%.

Best Regards

Sorry, missed that he wasn't on PPPoE, there are a couple of providers in the UK offering FTTC who don't use it, but the majority do, so yes, baby jumbo frames not needed.

I actually missed a few virtual interfaces I made and my guest wifi interface. I disabled them now and rebooted. max_len on -d is still 2974!
FYI, it is not possible to disable TSO on any wlan interface. They all give this error:

ethtool -K wlan0 tso off
Cannot change tcp-segmentation-offload

However, GRO and GSO can be disabled just fine.

Thanks for the help!

tbh I was surprised. I prepared for my FTTC move (from Virgin cable) by getting a modem (Vigor 130) that supports jumbo frames. It turns out on TalkTalk it is not needed since they don't use PPPoE.

Ah, sorry for my brash tone, your post actually made me think and I failed to a) note how this was a great idea to check and b) to actually reword my first rough draft that should have never been sent un edited. So, again I did not intend the tone that my response implies.

Hey, no worries at all, it's always hard to convey tone in messages, and I didn't take it as being brash, and I should have read the original post more fully before chipping in :slight_smile:

I believe Sky don't use PPPoE either, instead relying on some DHCP value.

Xiaomi MiWifi Nano router LEDE17.01.4 Jan.30/18

tc -s qdisc ; tc -d qdisc

TekSavvy SR505n VDSL2 modem: Rate (Kbps): 16192 11328 down/up (kbps)
SQM: 15000/10800kbps/pppoe-wan, 92.6%/95.3% of modem rate, getting great results

  1. Default optimum setting for the last few weeks:
    SQM: 15000/10800kbps/pppoe-wan/cake/layer_cake/8bytes_overhead
    14.33 megabit/s down, 7.68 megabit/s up
    Overall A, BufferBloat A+, Quality A, Speed C
    13ms/17ms/17msec idle/dl/ul Bufferbloat

root@MiWifiNano:~# tc -s qdisc ; tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 46465872 bytes 56958 pkt (dropped 0, overlimits 0 requeues 4)
backlog 0b 0p requeues 4
maxpacket 1514 drop_overlimit 0 new_flow_count 8 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8004: dev eth0.2 root refcnt 2 bandwidth 10800Kbit diffserv3 triple-isolate rtt 100.0ms raw total_overhead 18 hard_header_len 18
Sent 22479209 bytes 25868 pkt (dropped 28, overlimits 24425 requeues 0)
backlog 0b 0p requeues 0
memory used: 22880b of 4Mb
capacity estimate: 10800Kbit
Bulk Best Effort Voice
thresh 675Kbit 10800Kbit 2700Kbit
target 27.0ms 5.0ms 6.7ms
interval 122.0ms 100.0ms 13.5ms
pk_delay 0us 82us 0us
av_delay 0us 23us 0us
sp_delay 0us 6us 0us
pkts 0 25896 0
bytes 0 22521825 0
way_inds 0 0 0
way_miss 0 351 0
way_cols 0 0 0
drops 0 28 0
marks 0 0 0
sp_flows 0 1 0
bk_flows 0 1 0
un_flows 0 0 0
max_len 0 1522 0

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 23795885 bytes 30014 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8005: dev ifb4eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
Sent 23868541 bytes 29522 pkt (dropped 492, overlimits 35272 requeues 0)
backlog 0b 0p requeues 0
memory used: 84672b of 4Mb
capacity estimate: 15Mbit
Tin 0
thresh 15Mbit
target 5.0ms
interval 100.0ms
pk_delay 612us
av_delay 155us
sp_delay 6us
pkts 30014
bytes 24456193
way_inds 0
way_miss 360
way_cols 0
drops 492
marks 0
sp_flows 1
bk_flows 1
un_flows 0
max_len 1522

qdisc cake 800d: dev pppoe-wan root refcnt 2 bandwidth 10800Kbit diffserv3 triple-isolate rtt 100.0ms raw total_overhead 26 hard_header_len 26
Sent 21889966 bytes 25244 pkt (dropped 148, overlimits 27862 requeues 0)
backlog 0b 0p requeues 0
memory used: 77312b of 4Mb
capacity estimate: 10800Kbit
Bulk Best Effort Voice
thresh 675Kbit 10800Kbit 2700Kbit
target 27.0ms 5.0ms 6.7ms
interval 122.0ms 100.0ms 13.5ms
pk_delay 0us 947us 91us
av_delay 0us 242us 2us
sp_delay 0us 14us 2us
pkts 0 25380 12
bytes 0 22105364 1301
way_inds 0 0 0
way_miss 0 273 12
way_cols 0 0 0
drops 0 148 0
marks 0 0 0
sp_flows 0 1 0
bk_flows 0 0 0
un_flows 0 0 0
max_len 0 3070 270

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
Sent 22879447 bytes 28893 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 800e: dev ifb4pppoe-wan root refcnt 2 bandwidth 15Mbit besteffort triple-isolate wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
Sent 23110591 bytes 28893 pkt (dropped 0, overlimits 11450 requeues 0)
backlog 0b 0p requeues 0
memory used: 8064b of 4Mb
capacity estimate: 15Mbit
Tin 0
thresh 15Mbit
target 5.0ms
interval 100.0ms
pk_delay 103us
av_delay 13us
sp_delay 4us
pkts 28893
bytes 23110591
way_inds 19
way_miss 288
way_cols 0
drops 0
marks 0
sp_flows 1
bk_flows 1
un_flows 0
max_len 1500

qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc cake 8004: dev eth0.2 root refcnt 2 bandwidth 10800Kbit diffserv3 triple-isolate rtt 100.0ms raw total_overhead 18 hard_header_len 18
linklayer ethernet overhead 8
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc cake 8005: dev ifb4eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
linklayer ethernet overhead 8
qdisc cake 800d: dev pppoe-wan root refcnt 2 bandwidth 10800Kbit diffserv3 triple-isolate rtt 100.0ms raw total_overhead 26 hard_header_len 26
linklayer ethernet overhead 8
qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
qdisc cake 800e: dev ifb4pppoe-wan root refcnt 2 bandwidth 15Mbit besteffort triple-isolate wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
linklayer ethernet overhead 8
root@MiWifiNano:~#

  1. SQM disabled:
    15.17 megabit/s down, 8.92 megabit/s up
    Overall B, BufferBloat B, Quality B, Speed B


    12/38/72msec idle/dl/ul Bufferbloat

root@MiWifiNano:~# tc -s qdisc ; tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 338667 bytes 1211 pkt (dropped 0, overlimits 0 requeues 3)
backlog 0b 0p requeues 3
maxpacket 542 drop_overlimit 0 new_flow_count 3 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev pppoe-wan root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5.0ms interval 100.0ms ecn
Sent 42971 bytes 290 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc noqueue 0: dev eth0.2 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc fq_codel 0: dev pppoe-wan root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5.0ms interval 100.0ms ecn
root@MiWifiNano:~#

  1. Simple/fq_codel request from moeller0:
    SQM: 15000/10800/pppoe/simple/fq_codel/0bytes
    14.51 megabit/s, 9.52 megabit/s
    Overall A, BufferBloat A, Quality A, Speed A
    12/21/16msec idle/dl/ul Bufferbloat

root@MiWifiNano:~# tc -s qdisc ; tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 46673912 bytes 61534 pkt (dropped 0, overlimits 0 requeues 3)
backlog 0b 0p requeues 3
maxpacket 542 drop_overlimit 0 new_flow_count 7 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev pppoe-wan root refcnt 2 r2q 10 default 12 direct_packets_stat 2 direct_qlen 3
Sent 23176317 bytes 30244 pkt (dropped 151, overlimits 42224 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 110: dev pppoe-wan parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
Sent 245730 bytes 2672 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 576 drop_overlimit 0 new_flow_count 2225 ecn_mark 0
new_flows_len 0 old_flows_len 9
qdisc fq_codel 120: dev pppoe-wan parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
Sent 22929816 bytes 27567 pkt (dropped 151, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 4476 drop_overlimit 0 new_flow_count 7407 ecn_mark 0
new_flows_len 1 old_flows_len 4
qdisc fq_codel 130: dev pppoe-wan parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
Sent 22461950 bytes 33975 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev ifb4pppoe-wan root refcnt 2 r2q 10 default 10 direct_packets_stat 0 direct_qlen 32
Sent 21971198 bytes 33540 pkt (dropped 435, overlimits 31516 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4pppoe-wan parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 21971198 bytes 33540 pkt (dropped 435, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 1492 drop_overlimit 0 new_flow_count 8466 ecn_mark 0
new_flows_len 1 old_flows_len 13
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc noqueue 0: dev eth0.2 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc htb 1: dev pppoe-wan root refcnt 2 r2q 10 default 12 direct_packets_stat 2 ver 3.17 direct_qlen 3
qdisc fq_codel 110: dev pppoe-wan parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 120: dev pppoe-wan parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 130: dev pppoe-wan parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
qdisc htb 1: dev ifb4pppoe-wan root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 direct_qlen 32
qdisc fq_codel 110: dev ifb4pppoe-wan parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
root@MiWifiNano:~#