SQM setting question - Link Layer Adaptation

if my router doesn't have nat I can use mpu 64 dual-dsthost y mpu 64 dual-srchost ?

Not sure what you mean here?

Probably, but you can always test wether it does the right thing:

A) On two computers run 1 speedtest each at the same time (use something like fast.com that can be configured for a relatively ;ong run time, so starting multiple instances get easier).
With or without the dual-xxxhost options you should see roughly the same rate for each speedtest

B) On the first computer start one speedtest like above, and on the second computer start two speedtests (in two different browsers probably).
Now you either get identical results for all three speedtests idicating that dual-xxxhost did not do its thing or roughly: X Mbps on the first Computer and X/2 on each of the speedtest on the second computer which would show dial-xxxhost working as intended.

Quick question though: Why post this on a thread about Link Layer Adaptation instead of starting a new focussed thread with an appropriate title?

to save time also what I am looking for is in this topic

When i use mpu 64 nat dual-dsthost the bandwidth is reduced because I think the problem here is NAT

In a public forum like this, it is always a good idea to look at "time-saving" from two perspectives:
A) What is the fastest way to get an answer
B) What way will be easiest/fastest to find by others that might stumble over the same issue and would be helped if they could find that post/thread easily.

Could you post speedtest results with and without that option, together with the matching output of tc -s qdisc?

Might well, be, but I still have not understood what the exact problem is to begin with, maybe you can help with describing your set-up and why you think that nat is problematic for you?

root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8048: dev eth0 root refcnt 6 bandwidth 23Mbit besteffort dual-srchost                                                                                                                                                              nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 18 mpu 64
 Sent 148268 bytes 927 pkt (dropped 0, overlimits 101 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 10368b of 4Mb
 capacity estimate: 23Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh         23Mbit
  target            5ms
  interval        100ms
  pk_delay        796us
  av_delay         46us
  sp_delay          2us
  backlog            0b
  pkts              927
  bytes          148268
  way_inds            0
  way_miss           89
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            5
  bk_flows            1
  un_flows            0
  max_len          6056
  quantum           701

qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
 Sent 293346 bytes 1646 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 ta                                                                                                                                                             rget 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 7784797232 bytes 6251436 pkt (dropped 0, overlimits 0 requeues 1)
 backlog 0b 0p requeues 1
  maxpacket 1514 drop_overlimit 0 new_flow_count 8 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 root refcnt 2 limit 10240p flows 1024 quantum 1514 ta                                                                                                                                                             rget 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 1388475424 bytes 1549819 pkt (dropped 1, overlimits 0 requeues 5)
 backlog 0b 0p requeues 5
  maxpacket 1514 drop_overlimit 0 new_flow_count 9 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8049: dev ifb4eth0 root refcnt 2 bandwidth 95Mbit besteffort dual-dst                                                                                                                                                             host nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 18 mpu 64
 Sent 316782 bytes 1646 pkt (dropped 0, overlimits 85 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 13760b of 4750000b
 capacity estimate: 95Mbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh         95Mbit
  target            5ms
  interval        100ms
  pk_delay         27us
  av_delay          4us
  sp_delay          3us
  backlog            0b
  pkts             1646
  bytes          316782
  way_inds            0
  way_miss           91
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len          2892
  quantum          1514

root@OpenWrt:~#

because as a router I have rpi4 and I think it does not have NAT

I thought that removing nat and using mpu 64 dual-dsthost it could be the same

That might be, I am not sure what the nat keyword does when conntrack is not active though. I presumed that it would just operate on the IP addresses it does see. So, can you please run the test I proposed above and report the outcome here (preferably run the test twice, once with and once without the nat keyword).

I've long used SQM cake on my WRT32X (500Mbit cable modem) and it works great, A+ bufferbloat/A+ quality on dslreports.com/speedtest while maxing upload/download at 97% bandwidth (3% reserved). I used to set the overhead to 18 as per the man page:
Source: https://man7.org/linux/man-pages/man8/tc-cake.8.html

But recently I don't even bother setting overhead, I get the same results not touching that config, I think it recieves the overhead from the kernel it says in the man page. It would be nice to know exactly what to enter to optimize this.

mpu 64 dual-dsthost
mpu 64 dual-srchost
1 PC: bajada = 56Mb subida = 12Mb
2 PC: bajada = 35Mb subida = 7.4Mb

no mpu 64 dual-srchost
no mpu 64 dual-srchost
1 PC: = 44Mb = 7.2Mb
2 PC: = 44Mb = 11Mb

1 PC = BRAVE: = 41Mb = 7.5Mb
Edge: = 26Mb = 5.6Mb

2 PC : = 25Mb = 5.2Mb

mpu 64 nat dual-dsthost
mpu 64 nat dual-srchost

1 PC : = 58Mb = 8.0 Mb
2 PC : = 37Mb = 6.1 Mb

Yes and no. So yes, I do believe you that you see similar results in speedtests with and without "overhead 18" , and no, the results are conceptually different :wink:

When you do not explicitly request an overhead either as overhead NN or with one of the symbolic overhead keywords like docsis the kernel will fill in its default of 14 bytes (these are essentially those 14 bytes of an ethernet header the core kernel needs to handle, like source and destination mac address, and these exists inside SKBs IIRC and hence the kernel will track these in the SKB's len field).

So you are missing the real overhead by 4 bytes (14 instead of 18) while at the same time giving the shaper a 3% margin to work with....

Here is the maximum goodput your link will carry (with 100% being what ever your ISP supplies, let's keep this in percent for easier math):
true goodput limit: 100 * ((1500-20-20)/(1500+18)) = 96.18%

Now with your shaper setting you at max admit
97 * ((1500-20-20)/(1500+14)) = 93.54%
of a possible 96.18% onto the link, while the arguably more correct value would be:
97 * ((1500-20-20)/(1500+18)) = 93.29%

Note that both these values are smaller than the true bottleneck, and hence you will not see any bufferbloat. Also note how this assumes maximally sized packets.

Let's repeat that exercise with small packets of 100 Bytes:

true goodput limit: 100 * ((100-20-20)/(100+18)) = 50.85%
your shaper setting
97 * ((100-20-20)/(100+14)) = 51.05%
correct shaper settings:
97 * ((100-20-20)/(100+18)) = 49.32%

Note how 51.05% is too much for the links 50.85% capacity, that is under (unlikely) conditions in which your link is saturated with 100bytes packets, your shaper will not keep bufferbloat under control. Note that you can make up for this by simply reducing the shaper rate a bit:
96 * ((100-20-20)/(100+14)) = 50.53%

Personally I would always opt to get the per-packet-overhead right, or better yet if hard to deduce precisely just mildly overestimate it.

1 Like

Strangely, the configuration I applied works fine last night but in the morning I had latency spikes

So this is one test each on two PCs? Rate is not shared as fair as I would expect it, wg=hich sppedtest did you use?

Could you please repeat this test but also supply the following information (always taken after tests)

A) mpu 64 dual-dsthost/mpu 64 dual-srchost: Computer1: speedtest, Computer2: speedtest, then tc -s qdisc (this should be your first test above?)

B)mpu 64 dual-dsthost/mpu 64 dual-srchost: Computer1: speedtest, Computer2: 2 speedtests, then tc -s qdisc (this is like your first test but with two browsers on computer2, but with the cake settings)

A) nat mpu 64 dual-dsthost/mpu 64 dual-srchost: Computer1: speedtest, Computer2: speedtest, then tc -s qdisc (this should be your first test above but with the nat keyword)

B)nat mpu 64 dual-dsthost/mpu 64 dual-srchost: Computer1: speedtest, Computer2: 2 speedtests, then tc -s qdisc (this is like your first test but with two browsers on computer2, but with the nat keyword)

Note how I would like to see the tc-s output after each test.

doing another test is already difficult because the PCs are already occupied by other people, ignoring that my router does not have NAT, what would be a better option?

Then let's wait until the machines are not used any more and the network quiet, these tests really are the best way I can see to answer your question, but there probably is no hurry, no?

Point taken and good info. I'll set my overhead back to 18 tonight and leave it. Not sure how to confirm my cable modem really uses that amount though. I haven't actually notice a place to just put the keyword "docsis" like the man page said too.

For DOCSIS these 18 Bytes overhead are mandated in the specifications (which you can find on the cablelabs website) from an earier version of the standard:

"C.2.2.7.2 Maximum Sustained Traffic Rate 632 This parameter is the rate parameter R of a token-bucket-based rate limit for packets. R is expressed in bits per second, and MUST take into account all MAC frame data PDU of the Service Flow from the byte following the MAC header HCS to the end of the CRC, including every PDU in the case of a Concatenated MAC Frame. This parameter is applied after Payload Header Suppression; it does not include the bytes suppressed for PHS. The number of bytes forwarded (in bytes) is limited during any time interval T by Max(T), as described in the expression: Max(T) = T * (R / 8) + B, (1) where the parameter B (in bytes) is the Maximum Traffic Burst Configuration Setting (refer to Annex C.2.2.7.3). NOTE: This parameter does not limit the instantaneous rate of the Service Flow. The specific algorithm for enforcing this parameter is not mandated here. Any implementation which satisfies the above equation is conformant. In particular, the granularity of enforcement and the minimum implemented value of this parameter are vendor specific. The CMTS SHOULD support a granularity of at most 100 kbps. The CM SHOULD support a granularity of at most 100 kbps. NOTE: If this parameter is omitted or set to zero, then there is no explicitly-enforced traffic rate maximum. This field specifies only a bound, not a guarantee that this rate is available."

So in essence DOCSIS users need to only account for 18 Bytes of ethernet overhead in both ingress and egress directions under non-congested conditions.

It is somewhat tricky to tests though as you would need an otherwise quiescent docsis segment...

1 Like

What about under congested conditions if I may ask?

Well, if your DOCSIS segment is congested you will not be getting your contracted speed no matter whether the overhead is set correctly or not, AND you will experience the full bufferbloat of your CMTS and/or modem (depending on the congested direction).... the only thing you could to to ameliorate the issue is try to new autorate script that @Lynx kicked-off and that is worked on by a great team over in the Cake /w Adaptive Bandwidth thread.

3 Likes