Difficulties setting up SQM on a PPPoE + DS-Lite network

As the title indicates, I don't understand how to correctly configure SQM on a PPPoE + DS-Lite network. The way I have now done it is to set up two SQM instances. One for the IPv4 DS-Lite, and the other one for the PPPoE-Wan interface for the "pure" IPv6 and tunneled IPv4 traffic. Each with their own per packet overhead.

My questions are now the following:

  1. Does my network require two SQM instances?

  2. Are the configured download/upload, qdisk options, and overhead correctly configured for both? If not, is this something my ISP would know more about?

  3. How do I test for bufferbloat on the pure IPv6 traffic side? Waveform and Dslreports don't seem to support IPv6, and I also don't know how to disable only the IPv4 traffic.


This is what my interface looks like in LuCI:

Untitled


And this is my SQM config:
config queue 'eth1'
        option enabled '1'
        option interface 'ds-wan_6_4'
        option download '400000'
        option upload '100000'
        option qdisc 'cake'
        option script 'piece_of_cake.qos'
        option linklayer 'ethernet'
        option debug_logging '0'
        option verbosity '5'
        option qdisc_advanced '1'
        option squash_dscp '1'
        option squash_ingress '1'
        option ingress_ecn 'ECN'
        option egress_ecn 'ECN'
        option qdisc_really_really_advanced '1'
        option iqdisc_opts 'mpu 96 nat dual-dsthost ingress'
        option eqdisc_opts 'mpu 96 nat dual-srchost'
        option overhead '74'

config queue
        option enabled '1'
        option interface 'pppoe-wan'
        option download '400000'
        option upload '100000'
        option qdisc 'cake'
        option script 'piece_of_cake.qos'
        option linklayer 'ethernet'
        option debug_logging '0'
        option verbosity '5'
        option qdisc_advanced '1'
        option squash_dscp '1'
        option squash_ingress '1'
        option ingress_ecn 'ECN'
        option egress_ecn 'ECN'
        option qdisc_really_really_advanced '1'
        option iqdisc_opts 'mpu 96 nat dual-dsthost ingress'
        option eqdisc_opts 'mpu 96 nat dual-srchost'
        option overhead '34'

Could you post the output of tc -s qdisc from just before and just after after running an IPv4 and an IPv6 speedtest please? The goal is to understand what the cake statistics report right now.

My guess is the best way forward would be to run a single SQM instance on pppoe-wan and just adjust the overhead setting to cover the wan_6_4 traffic as well.

It might be good to collect some data via tcp-dump to see the actual overhead...

Note for others: Cloudflare's speedtest supports IPv6, and I switched between them by disabling the unwanted IPv# protocol in the windows network adapter settings.

Hopefully this is right:

tc output: IPv4 before speedtest
root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 766085503 bytes 1422538 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 4461678 bytes 16027 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 66 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1363421 bytes 7177 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 56 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1078509 bytes 7327 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth4 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 2203891020 bytes 1799904 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1595 drop_overlimit 0 new_flow_count 52 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth5 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 696553 bytes 7277 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 99 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.10 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.66 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8019: dev pppoe-wan root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 96
 Sent 204024 bytes 737 pkt (dropped 0, overlimits 86 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 4352b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           72 /    1492
 min/max overhead-adjusted size:      106 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh        100Mbit
  target            5ms
  interval        100ms
  pk_delay          8us
  av_delay          1us
  sp_delay          0us
  backlog            0b
  pkts              737
  bytes          204024
  way_inds            0
  way_miss            8
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1492
  quantum          1514

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 372124 bytes 816 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8015: dev ds-wan_6_4 root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 169194 bytes 663 pkt (dropped 0, overlimits 89 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 13552b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           40 /    1452
 min/max overhead-adjusted size:      114 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh        100Mbit
  target            5ms
  interval        100ms
  pk_delay        167us
  av_delay         17us
  sp_delay          0us
  backlog            0b
  pkts              663
  bytes          169194
  way_inds            0
  way_miss           78
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          4238
  quantum          1514

qdisc ingress ffff: dev ds-wan_6_4 parent ffff:fff1 ----------------
 Sent 335671 bytes 790 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8016: dev ifb4ds-wan_6_4 root refcnt 2 bandwidth 400Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 335671 bytes 790 pkt (dropped 0, overlimits 270 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 8704b of 15140Kb
 capacity estimate: 400Mbit
 min/max network layer size:           40 /    1412
 min/max overhead-adjusted size:      114 /    1486
 average network hdr offset:            0

                  Tin 0
  thresh        400Mbit
  target            5ms
  interval        100ms
  pk_delay         53us
  av_delay          8us
  sp_delay          0us
  backlog            0b
  pkts              790
  bytes          335671
  way_inds            0
  way_miss           77
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1412
  quantum          1514

qdisc cake 801a: dev ifb4pppoe-wan root refcnt 2 bandwidth 400Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 96
 Sent 372124 bytes 816 pkt (dropped 0, overlimits 417 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 66816b of 15140Kb
 capacity estimate: 400Mbit
 min/max network layer size:           72 /    1452
 min/max overhead-adjusted size:      106 /    1486
 average network hdr offset:            0

                  Tin 0
  thresh        400Mbit
  target            5ms
  interval        100ms
  pk_delay        453us
  av_delay         54us
  sp_delay          1us
  backlog            0b
  pkts              816
  bytes          372124
  way_inds            4
  way_miss           78
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1452
  quantum          1514
tc output: IPv4 after
root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 940570489 bytes 1687694 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 4749885 bytes 16964 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 66 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1400010 bytes 7538 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 56 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1136403 bytes 7781 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 99 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth4 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 2695837526 bytes 2198162 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1595 drop_overlimit 0 new_flow_count 55 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth5 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 732633 bytes 7751 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 99 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.10 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.66 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8019: dev pppoe-wan root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 96
 Sent 168855450 bytes 265877 pkt (dropped 0, overlimits 220710 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 30208b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           72 /    1492
 min/max overhead-adjusted size:      106 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh        100Mbit
  target            5ms
  interval        100ms
  pk_delay         13us
  av_delay          1us
  sp_delay          0us
  backlog            0b
  pkts           265877
  bytes       168855450
  way_inds            0
  way_miss           13
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1492
  quantum          1514

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 502681532 bytes 398868 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8015: dev ds-wan_6_4 root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 158209548 bytes 265738 pkt (dropped 62, overlimits 166826 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 1360560b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           40 /    1452
 min/max overhead-adjusted size:      114 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh        100Mbit
  target            5ms
  interval        100ms
  pk_delay        282us
  av_delay         14us
  sp_delay          1us
  backlog            0b
  pkts           265800
  bytes       158296108
  way_inds           13
  way_miss          105
  way_cols            0
  drops              62
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len         18200
  quantum          1514

qdisc ingress ffff: dev ds-wan_6_4 parent ffff:fff1 ----------------
 Sent 486718194 bytes 398827 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8016: dev ifb4ds-wan_6_4 root refcnt 2 bandwidth 400Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 486718194 bytes 398827 pkt (dropped 0, overlimits 621685 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 54Kb of 15140Kb
 capacity estimate: 400Mbit
 min/max network layer size:           40 /    1412
 min/max overhead-adjusted size:      114 /    1486
 average network hdr offset:            0

                  Tin 0
  thresh        400Mbit
  target            5ms
  interval        100ms
  pk_delay        105us
  av_delay         90us
  sp_delay          0us
  backlog            0b
  pkts           398827
  bytes       486718194
  way_inds           13
  way_miss          101
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1412
  quantum          1514

qdisc cake 801a: dev ifb4pppoe-wan root refcnt 2 bandwidth 400Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 96
 Sent 502680080 bytes 398867 pkt (dropped 1, overlimits 621826 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 426240b of 15140Kb
 capacity estimate: 400Mbit
 min/max network layer size:           72 /    1452
 min/max overhead-adjusted size:      106 /    1486
 average network hdr offset:            0

                  Tin 0
  thresh        400Mbit
  target            5ms
  interval        100ms
  pk_delay       2.28ms
  av_delay        2.1ms
  sp_delay          1us
  backlog            0b
  pkts           398868
  bytes       502681532
  way_inds           12
  way_miss          105
  way_cols            0
  drops               1
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1452
  quantum          1514
tc output: IPv6 before speedtest
root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 940570489 bytes 1687694 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 4749885 bytes 16964 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 66 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1400010 bytes 7538 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 56 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1136403 bytes 7781 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 99 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth4 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 2695837526 bytes 2198162 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1595 drop_overlimit 0 new_flow_count 55 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth5 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 732633 bytes 7751 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 99 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.10 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.66 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8019: dev pppoe-wan root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 96
 Sent 168855450 bytes 265877 pkt (dropped 0, overlimits 220710 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 30208b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           72 /    1492
 min/max overhead-adjusted size:      106 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh        100Mbit
  target            5ms
  interval        100ms
  pk_delay         13us
  av_delay          1us
  sp_delay          0us
  backlog            0b
  pkts           265877
  bytes       168855450
  way_inds            0
  way_miss           13
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1492
  quantum          1514

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 502681532 bytes 398868 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8015: dev ds-wan_6_4 root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 158209548 bytes 265738 pkt (dropped 62, overlimits 166826 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 1360560b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           40 /    1452
 min/max overhead-adjusted size:      114 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh        100Mbit
  target            5ms
  interval        100ms
  pk_delay        282us
  av_delay         14us
  sp_delay          1us
  backlog            0b
  pkts           265800
  bytes       158296108
  way_inds           13
  way_miss          105
  way_cols            0
  drops              62
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len         18200
  quantum          1514

qdisc ingress ffff: dev ds-wan_6_4 parent ffff:fff1 ----------------
 Sent 486718194 bytes 398827 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8016: dev ifb4ds-wan_6_4 root refcnt 2 bandwidth 400Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 486718194 bytes 398827 pkt (dropped 0, overlimits 621685 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 54Kb of 15140Kb
 capacity estimate: 400Mbit
 min/max network layer size:           40 /    1412
 min/max overhead-adjusted size:      114 /    1486
 average network hdr offset:            0

                  Tin 0
  thresh        400Mbit
  target            5ms
  interval        100ms
  pk_delay        105us
  av_delay         90us
  sp_delay          0us
  backlog            0b
  pkts           398827
  bytes       486718194
  way_inds           13
  way_miss          101
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1412
  quantum          1514

qdisc cake 801a: dev ifb4pppoe-wan root refcnt 2 bandwidth 400Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 96
 Sent 502680080 bytes 398867 pkt (dropped 1, overlimits 621826 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 426240b of 15140Kb
 capacity estimate: 400Mbit
 min/max network layer size:           72 /    1452
 min/max overhead-adjusted size:      106 /    1486
 average network hdr offset:            0

                  Tin 0
  thresh        400Mbit
  target            5ms
  interval        100ms
  pk_delay       2.28ms
  av_delay        2.1ms
  sp_delay          1us
  backlog            0b
  pkts           398868
  bytes       502681532
  way_inds           12
  way_miss          105
  way_cols            0
  drops               1
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1452
  quantum          1514
tc output: IPv6 after
root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1111049587 bytes 1963825 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 5363071 bytes 19294 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 66 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1501318 bytes 8593 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 56 drop_overlimit 0 new_flow_count 3 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1263990 bytes 8952 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 99 drop_overlimit 0 new_flow_count 3 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth4 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 3196510945 bytes 2600847 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1595 drop_overlimit 0 new_flow_count 68 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth5 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 890692 bytes 9107 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 99 drop_overlimit 0 new_flow_count 3 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.10 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.66 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8019: dev pppoe-wan root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 96
 Sent 333259394 bytes 541974 pkt (dropped 26, overlimits 388800 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 872Kb of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           52 /    1492
 min/max overhead-adjusted size:       96 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh        100Mbit
  target            5ms
  interval        100ms
  pk_delay         41us
  av_delay          1us
  sp_delay          0us
  backlog            0b
  pkts           542000
  bytes       333296314
  way_inds            0
  way_miss           47
  way_cols            0
  drops              26
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         18460
  quantum          1514

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 997966511 bytes 801166 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8015: dev ds-wan_6_4 root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 158654540 bytes 269420 pkt (dropped 62, overlimits 167073 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 1360560b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           40 /    1452
 min/max overhead-adjusted size:      114 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh        100Mbit
  target            5ms
  interval        100ms
  pk_delay         15us
  av_delay          1us
  sp_delay          0us
  backlog            0b
  pkts           269482
  bytes       158741100
  way_inds           39
  way_miss          745
  way_cols            0
  drops              62
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len         18200
  quantum          1514

qdisc ingress ffff: dev ds-wan_6_4 parent ffff:fff1 ----------------
 Sent 487612546 bytes 401971 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8016: dev ifb4ds-wan_6_4 root refcnt 2 bandwidth 400Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 487612546 bytes 401971 pkt (dropped 0, overlimits 622114 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 54Kb of 15140Kb
 capacity estimate: 400Mbit
 min/max network layer size:           40 /    1412
 min/max overhead-adjusted size:      114 /    1486
 average network hdr offset:            0

                  Tin 0
  thresh        400Mbit
  target            5ms
  interval        100ms
  pk_delay          6us
  av_delay          1us
  sp_delay          0us
  backlog            0b
  pkts           401971
  bytes       487612546
  way_inds           21
  way_miss          706
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len          1412
  quantum          1514

qdisc cake 801a: dev ifb4pppoe-wan root refcnt 2 bandwidth 400Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 96
 Sent 997942339 bytes 801149 pkt (dropped 17, overlimits 1313199 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 1128960b of 15140Kb
 capacity estimate: 400Mbit
 min/max network layer size:           56 /    1492
 min/max overhead-adjusted size:       96 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh        400Mbit
  target            5ms
  interval        100ms
  pk_delay        174us
  av_delay        144us
  sp_delay          0us
  backlog            0b
  pkts           801166
  bytes       997966511
  way_inds           37
  way_miss          746
  way_cols            0
  drops              17
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len          1492
  quantum          1514

It might be good to collect some data via tcp-dump to see the actual overhead

Should I just post the whole dump since there are, I think, public IPs in there, or is there a specifc command I should execute?

Edit: Fixed some random quirks

Edit: Also I just noticed that the autogenerated ds-lite "ds-wan_6_4" device has a MTU of 1280 configured, compared to the 1492 of the pppoe-wan device. That seems to be too low or not?

Edit2: The low MTU mentioned in the last edit seems to be a known issue, according to a wiki entry, and can be hotfixed by adding a script to the end of this file:

/etc/hotplug.d/iface/00-netstate
if [ ifup = "$ACTION" ] && [ wan_6_4 = "$INTERFACE" ] ; then
        ip link set ds-wan_6_4 mtu 1452
fi

Last edit: Sorry, I didn't notice until now but I disabled the second SQM pppoe-wan instance at some point. The results are now with it enabled.

Mmm, thanks... we really need to also see what happens on pppoe-wan at the same time...

I guess we could go that way:

  1. log into the router and issue:
tcpdump -i pppoe-wan -s 128 -w dslite.pcap -v -n '(icmp and ip and (ip[1] & 0xfc) >> 2 == 0x2D)' or '(icmp6 and ip6 and (ip6[0:2] & 0xfc0) >> 4  == 180)'

This will only capture ICMP packets (icmp and icmp6) that also carry DSCP decimal 45/hex 0x2D (which corresponds to TOS decimal 180/hex 0xB4), which should result in a decently sparse capture file revealing nothing of concern (which you still should check and confirm before sharing a capture).

You still need to actually create some test traffic:
2) in another shell (leave the tcpdump shell/terminal open) do:

ping -c 3 -Q 0xB4 1.1.1.1
ping -c 3 -Q 0xB4 2606:4700:4700::1111

(you might need to install iputils-ping first opkg update ; opkg install iputils-ping as busybox ping to my knowledge does not support the -Qoption)

  1. then go back to the tcpdump shell and end the capture by pressing ctrl-c

  2. copy the capture file and confirm it is benign, then share with us...

Sorry, I updated my last post.

1 Like

This magically already happened while I typed... :wink: Thanks!

1 Like
Does a txt file work too? Otherwise I would only be comfortable sending it over a direct message.
No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       2a01:::1 2606:4700:4700::1111  ICMPv6   120    Echo (ping) request id=0x207f, seq=1, hop limit=64 (reply in 2)

Frame 1: 120 bytes on wire (960 bits), 120 bytes captured (960 bits)
Linux cooked capture v1
Internet Protocol Version 6, Src: 2a01:::1, Dst: 2606:4700:4700::1111
    0110 .... = Version: 6
    .... 1011 0100 .... .... .... .... .... = Traffic Class: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
    .... 1000 0101 1001 0110 0010 = Flow Label: 0x85962
    Payload Length: 64
    Next Header: ICMPv6 (58)
    Hop Limit: 64
    Source Address: 2a01:::1
    Destination Address: 2606:4700:4700::1111
Internet Control Message Protocol v6
    Type: Echo (ping) request (128)
    Code: 0
    Checksum: 0xd2fb [correct]
    [Checksum Status: Good]
    Identifier: 0x207f
    Sequence: 1
    [Response In: 2]
    Timestamp from Echo data: Apr 25, 2024 18:36:55.942220000 W. Europe Daylight Time
    [Timestamp from Echo data (relative): 0.000038000 seconds]
    Data (40 bytes)

0000  10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f   ................
0010  20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f    !"#$%&'()*+,-./
0020  30 31 32 33 34 35 36 37                           01234567
        Data: 101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
        [Length: 40]

No.     Time           Source                Destination           Protocol Length Info
      2 0.001335       2606:4700:4700::1111  2a01:::1 ICMPv6   120    Echo (ping) reply id=0x207f, seq=1, hop limit=60 (request in 1)

Frame 2: 120 bytes on wire (960 bits), 120 bytes captured (960 bits)
Linux cooked capture v1
Internet Protocol Version 6, Src: 2606:4700:4700::1111, Dst: 2a01:::1
    0110 .... = Version: 6
    .... 1011 0100 .... .... .... .... .... = Traffic Class: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
    .... 0000 1100 1000 0111 1100 = Flow Label: 0x0c87c
    Payload Length: 64
    Next Header: ICMPv6 (58)
    Hop Limit: 60
    Source Address: 2606:4700:4700::1111
    Destination Address: 2a01:::1
Internet Control Message Protocol v6
    Type: Echo (ping) reply (129)
    Code: 0
    Checksum: 0xd1fb [correct]
    [Checksum Status: Good]
    Identifier: 0x207f
    Sequence: 1
    [Response To: 1]
    [Response Time: 1.335 ms]
    Timestamp from Echo data: Apr 25, 2024 18:36:55.942220000 W. Europe Daylight Time
    [Timestamp from Echo data (relative): 0.001373000 seconds]
    Data (40 bytes)

0000  10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f   ................
0010  20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f    !"#$%&'()*+,-./
0020  30 31 32 33 34 35 36 37                           01234567
        Data: 101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
        [Length: 40]

No.     Time           Source                Destination           Protocol Length Info
      3 1.001582       2a01:::1 2606:4700:4700::1111  ICMPv6   120    Echo (ping) request id=0x207f, seq=2, hop limit=64 (reply in 4)

Frame 3: 120 bytes on wire (960 bits), 120 bytes captured (960 bits)
Linux cooked capture v1
Internet Protocol Version 6, Src: 2a01:::1, Dst: 2606:4700:4700::1111
    0110 .... = Version: 6
    .... 1011 0100 .... .... .... .... .... = Traffic Class: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
    .... 1000 0101 1001 0110 0010 = Flow Label: 0x85962
    Payload Length: 64
    Next Header: ICMPv6 (58)
    Hop Limit: 64
    Source Address: 2a01:::1
    Destination Address: 2606:4700:4700::1111
Internet Control Message Protocol v6
    Type: Echo (ping) request (128)
    Code: 0
    Checksum: 0xa5f4 [correct]
    [Checksum Status: Good]
    Identifier: 0x207f
    Sequence: 2
    [Response In: 4]
    Timestamp from Echo data: Apr 25, 2024 18:36:56.943800000 W. Europe Daylight Time
    [Timestamp from Echo data (relative): 0.000040000 seconds]
    Data (40 bytes)

0000  10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f   ................
0010  20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f    !"#$%&'()*+,-./
0020  30 31 32 33 34 35 36 37                           01234567
        Data: 101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
        [Length: 40]

No.     Time           Source                Destination           Protocol Length Info
      4 1.003031       2606:4700:4700::1111  2a01:::1 ICMPv6   120    Echo (ping) reply id=0x207f, seq=2, hop limit=60 (request in 3)

Frame 4: 120 bytes on wire (960 bits), 120 bytes captured (960 bits)
Linux cooked capture v1
Internet Protocol Version 6, Src: 2606:4700:4700::1111, Dst: 2a01:::1
    0110 .... = Version: 6
    .... 1011 0100 .... .... .... .... .... = Traffic Class: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
    .... 0101 1001 0111 0000 0111 = Flow Label: 0x59707
    Payload Length: 64
    Next Header: ICMPv6 (58)
    Hop Limit: 60
    Source Address: 2606:4700:4700::1111
    Destination Address: 2a01:::1
Internet Control Message Protocol v6
    Type: Echo (ping) reply (129)
    Code: 0
    Checksum: 0xa4f4 [correct]
    [Checksum Status: Good]
    Identifier: 0x207f
    Sequence: 2
    [Response To: 3]
    [Response Time: 1.449 ms]
    Timestamp from Echo data: Apr 25, 2024 18:36:56.943800000 W. Europe Daylight Time
    [Timestamp from Echo data (relative): 0.001489000 seconds]
    Data (40 bytes)

0000  10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f   ................
0010  20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f    !"#$%&'()*+,-./
0020  30 31 32 33 34 35 36 37                           01234567
        Data: 101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
        [Length: 40]

No.     Time           Source                Destination           Protocol Length Info
      5 2.003256       2a01:::1 2606:4700:4700::1111  ICMPv6   120    Echo (ping) request id=0x207f, seq=3, hop limit=64 (reply in 6)

Frame 5: 120 bytes on wire (960 bits), 120 bytes captured (960 bits)
Linux cooked capture v1
Internet Protocol Version 6, Src: 2a01:::1, Dst: 2606:4700:4700::1111
    0110 .... = Version: 6
    .... 1011 0100 .... .... .... .... .... = Traffic Class: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
    .... 1000 0101 1001 0110 0010 = Flow Label: 0x85962
    Payload Length: 64
    Next Header: ICMPv6 (58)
    Hop Limit: 64
    Source Address: 2a01:::1
    Destination Address: 2606:4700:4700::1111
Internet Control Message Protocol v6
    Type: Echo (ping) request (128)
    Code: 0
    Checksum: 0x16ed [correct]
    [Checksum Status: Good]
    Identifier: 0x207f
    Sequence: 3
    [Response In: 6]
    Timestamp from Echo data: Apr 25, 2024 18:36:57.945478000 W. Europe Daylight Time
    [Timestamp from Echo data (relative): 0.000036000 seconds]
    Data (40 bytes)

0000  10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f   ................
0010  20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f    !"#$%&'()*+,-./
0020  30 31 32 33 34 35 36 37                           01234567
        Data: 101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
        [Length: 40]

No.     Time           Source                Destination           Protocol Length Info
      6 2.004794       2606:4700:4700::1111  2a01:::1 ICMPv6   120    Echo (ping) reply id=0x207f, seq=3, hop limit=60 (request in 5)

Frame 6: 120 bytes on wire (960 bits), 120 bytes captured (960 bits)
Linux cooked capture v1
Internet Protocol Version 6, Src: 2606:4700:4700::1111, Dst: 2a01:::1
    0110 .... = Version: 6
    .... 1011 0100 .... .... .... .... .... = Traffic Class: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
    .... 0010 0111 0010 0011 1100 = Flow Label: 0x2723c
    Payload Length: 64
    Next Header: ICMPv6 (58)
    Hop Limit: 60
    Source Address: 2606:4700:4700::1111
    Destination Address: 2a01:::1
Internet Control Message Protocol v6
    Type: Echo (ping) reply (129)
    Code: 0
    Checksum: 0x15ed [correct]
    [Checksum Status: Good]
    Identifier: 0x207f
    Sequence: 3
    [Response To: 5]
    [Response Time: 1.538 ms]
    Timestamp from Echo data: Apr 25, 2024 18:36:57.945478000 W. Europe Daylight Time
    [Timestamp from Echo data (relative): 0.001574000 seconds]
    Data (40 bytes)

0000  10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f   ................
0010  20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f    !"#$%&'()*+,-./
0020  30 31 32 33 34 35 36 37                           01234567
        Data: 101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
        [Length: 40]

Edit: Could it be that the command isn't capturing the IPv4 pings?

Edit2: I've captured the IPv4 packets now by changing the listening device to ds-wan_6_4:

Capture
No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       192.0.0.2             1.1.1.1               ICMP     100    Echo (ping) request  id=0x1caa, seq=1/256, ttl=64 (reply in 2)

Frame 1: 100 bytes on wire (800 bits), 100 bytes captured (800 bits)
Linux cooked capture v1
    Packet type: Sent by us (4)
    Link-layer address type: IP6IP6 tunnel (769)
    Link-layer address length: 0
    Unused: 0000000000000000
    Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.0.0.2, Dst: 1.1.1.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
        1011 01.. = Differentiated Services Codepoint: Unknown (45)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 84
    Identification: 0xa784 (42884)
    010. .... = Flags: 0x2, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 64
    Protocol: ICMP (1)
    Header Checksum: 0xd06c [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.0.0.2
    Destination Address: 1.1.1.1
Internet Control Message Protocol

No.     Time           Source                Destination           Protocol Length Info
      2 0.001406       1.1.1.1               192.0.0.2             ICMP     100    Echo (ping) reply    id=0x1caa, seq=1/256, ttl=59 (request in 1)

Frame 2: 100 bytes on wire (800 bits), 100 bytes captured (800 bits)
Linux cooked capture v1
    Packet type: Unicast to us (0)
    Link-layer address type: IP6IP6 tunnel (769)
    Link-layer address length: 0
    Unused: 0000000000000000
    Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 1.1.1.1, Dst: 192.0.0.2
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
        1011 01.. = Differentiated Services Codepoint: Unknown (45)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 84
    Identification: 0x6060 (24672)
    000. .... = Flags: 0x0
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 59
    Protocol: ICMP (1)
    Header Checksum: 0x5c91 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 1.1.1.1
    Destination Address: 192.0.0.2
Internet Control Message Protocol

No.     Time           Source                Destination           Protocol Length Info
      3 1.001230       192.0.0.2             1.1.1.1               ICMP     100    Echo (ping) request  id=0x1caa, seq=2/512, ttl=64 (reply in 4)

Frame 3: 100 bytes on wire (800 bits), 100 bytes captured (800 bits)
Linux cooked capture v1
    Packet type: Sent by us (4)
    Link-layer address type: IP6IP6 tunnel (769)
    Link-layer address length: 0
    Unused: 0000000000000000
    Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.0.0.2, Dst: 1.1.1.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
        1011 01.. = Differentiated Services Codepoint: Unknown (45)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 84
    Identification: 0xa7bb (42939)
    010. .... = Flags: 0x2, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 64
    Protocol: ICMP (1)
    Header Checksum: 0xd035 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.0.0.2
    Destination Address: 1.1.1.1
Internet Control Message Protocol

No.     Time           Source                Destination           Protocol Length Info
      4 1.002487       1.1.1.1               192.0.0.2             ICMP     100    Echo (ping) reply    id=0x1caa, seq=2/512, ttl=59 (request in 3)

Frame 4: 100 bytes on wire (800 bits), 100 bytes captured (800 bits)
Linux cooked capture v1
    Packet type: Unicast to us (0)
    Link-layer address type: IP6IP6 tunnel (769)
    Link-layer address length: 0
    Unused: 0000000000000000
    Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 1.1.1.1, Dst: 192.0.0.2
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
        1011 01.. = Differentiated Services Codepoint: Unknown (45)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 84
    Identification: 0xf1cd (61901)
    000. .... = Flags: 0x0
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 59
    Protocol: ICMP (1)
    Header Checksum: 0xcb23 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 1.1.1.1
    Destination Address: 192.0.0.2
Internet Control Message Protocol

No.     Time           Source                Destination           Protocol Length Info
      5 2.002766       192.0.0.2             1.1.1.1               ICMP     100    Echo (ping) request  id=0x1caa, seq=3/768, ttl=64 (reply in 6)

Frame 5: 100 bytes on wire (800 bits), 100 bytes captured (800 bits)
Linux cooked capture v1
    Packet type: Sent by us (4)
    Link-layer address type: IP6IP6 tunnel (769)
    Link-layer address length: 0
    Unused: 0000000000000000
    Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.0.0.2, Dst: 1.1.1.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
        1011 01.. = Differentiated Services Codepoint: Unknown (45)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 84
    Identification: 0xa7ee (42990)
    010. .... = Flags: 0x2, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 64
    Protocol: ICMP (1)
    Header Checksum: 0xd002 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.0.0.2
    Destination Address: 1.1.1.1
Internet Control Message Protocol

No.     Time           Source                Destination           Protocol Length Info
      6 2.004048       1.1.1.1               192.0.0.2             ICMP     100    Echo (ping) reply    id=0x1caa, seq=3/768, ttl=59 (request in 5)

Frame 6: 100 bytes on wire (800 bits), 100 bytes captured (800 bits)
Linux cooked capture v1
    Packet type: Unicast to us (0)
    Link-layer address type: IP6IP6 tunnel (769)
    Link-layer address length: 0
    Unused: 0000000000000000
    Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 1.1.1.1, Dst: 192.0.0.2
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xb4 (DSCP: Unknown, ECN: Not-ECT)
        1011 01.. = Differentiated Services Codepoint: Unknown (45)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 84
    Identification: 0x5375 (21365)
    000. .... = Flags: 0x0
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 59
    Protocol: ICMP (1)
    Header Checksum: 0x697c [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 1.1.1.1
    Destination Address: 192.0.0.2
Internet Control Message Protocol

So this is what dslite does it transports IPv4 inside IPv6, so on pppoe-wan we expect to only see IPv6 headers, so we need to look deeper into the captured IPv6 payload...
For this sharing the capture would be better... maybe you upload the capture to a hoster like

and then post the link either in the forum, or as a private message to me, then we can try to figure out what is inside...

Yeah, but this is less interesting than pppoe-wan, the question is what overhead do we need to account for on pppoe-wan

Edit: Also I just noticed that the autogenerated ds-lite "ds-wan_6_4" device has a MTU of 1280 configured, compared to the 1492 of the pppoe-wan device. That seems to be too low or not?

What MTU did you end up setting? I would guess with PPPoE overhead and the 40 byte IPv6 header you should not set a MTU higher than 1500-8-40 = 1452 on ds-wan_6_4....

Send a DM.

What MTU did you end up setting? I would guess with PPPoE overhead and the 40 byte IPv6 header yiu should not set a MTU higher than 1500-8-40 = 1452 on ds-wan_6_4....

1452 is what I've set it to now.

Thanks!, I now see my mistake (why the IPv4 pings are missing) the filter required ICMP6 for IPv6 but the encapsulated packets do not carry that, so likely the following invocation should work (relaying on the DSCP):

tcpdump -i pppoe-wan -s 128 -w dslite.pcap -v -n '(ip6 and (ip6[0:2] & 0xfc0) >> 4  == 180)'

1 Like

Thanks, according to the capture, I guess you should instantiate sqm on pppoe-wan and use
34+40 = 74 bytes as overhead, so exactly the value you already figured out for ds-wan_6_4.
The reason is simple, for sqm to control bufferbloat it needs to see all relevant traffic over the interface, so either you use a single shaper* or two shapers, but with two (or more) shapers the sum of the configured shaper rates needs to stay below the true bottleneck rate and that means that no shaper will be able to use the full capacity.
*) EDIT: two or more shapers are fine if they are in series, the 'see all relevant traffic' problem only occurs if shapers are in parallel and hence no single shaper sees all traffic.
The only downside to this solution is, that you will assume a somewhat too large per-packet overhead for true IPv6 data packets, but that should only be a little reduction in potential throughput.

Side note, wireshark is really nice and drills down into the encapsulated headers... thanks for sharing these...

My pleasure, it's nice to learn about these tools and methods along the way too.

Unfortunately my issues aren't yet fully resolved because if SQM is only enabled for the pppoe-wan interface with an overhead of 74, then bufferbloat is again present during the upload portion on IPv4 traffic and maybe also IPv6.

IPv4 test

IPv6 test

The test shows an increase by 4ms... that is not really something to complain about, cake will be default allow a standing queue of 5ms, so these two test IMHO do not indicate actionable bufferbloat neither for IPv6 nor for IPv4. I am not saying that you do not see problems, just that the two screenshots do not show anything unexpected.

Agreed, but the issue is there are spikes up to >40ms and cloudflares test filters them out for some reason. The waveform test showed these more clearly but their tool isn't loading for me currently.

For comparison how the test looks with the two SQM instances:

Both of these tests, suffer from the fact that modern browsers are excellent tools, but pretty mediocre environments to perform precise and accurate measurement in (case in point, your browser only allows to measure the delay quantised to full milliseconds)...

But by all means if two shaper perform better, there is no reason not to run two shapers that are in series like yours (parallel shapers are problematic, but in your case these are in series...)

P.S.: For the cloudflare tests, is is helpful in discussions to see the full page not just some cut and pasted pieces. If you want to react the map and/or IP address, fine, but all the rest can be helpful in diagnosing link problems...

1 Like

Regarding these 40ms spikes, please try to tickle these in a test and immediately when you notice one, exceute tc -s qdisc in an (already open terminal shell on the router) and post to this thread, if that delay happens inside of cake we might see it in the stats.

New results:
root@OpenWrt:/var/run/sqm# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 3893508000 bytes 6812799 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 114876612 bytes 164838 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 431 drop_overlimit 0 new_flow_count 23 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 1298276162 bytes 964578 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1506 drop_overlimit 0 new_flow_count 109 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 10358030 bytes 61356 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 99 drop_overlimit 0 new_flow_count 9 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth4 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 11272250511 bytes 9197791 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1595 drop_overlimit 0 new_flow_count 280 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth5 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 24041374 bytes 77004 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 317 drop_overlimit 0 new_flow_count 8 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.10 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.66 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8061: dev pppoe-wan root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 379533237 bytes 653420 pkt (dropped 159, overlimits 418775 requeues 0)
 backlog 61920b 43p requeues 0
 memory used: 1144320b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           60 /    1492
 min/max overhead-adjusted size:      134 /    1566
 average network hdr offset:            0

                  Tin 0
  thresh        100Mbit
  target            5ms
  interval        100ms
  pk_delay       5.42ms
  av_delay       4.98ms
  sp_delay       4.68ms
  backlog        61920b
  pkts           653622
  bytes       379824117
  way_inds            0
  way_miss           33
  way_cols            0
  drops             159
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         18720
  quantum          1514

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 1277335341 bytes 996307 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev ds-wan_6_4 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8062: dev ifb4pppoe-wan root refcnt 2 bandwidth 400Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 74 mpu 96
 Sent 1277329373 bytes 996303 pkt (dropped 4, overlimits 1723715 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 988416b of 15140Kb
 capacity estimate: 400Mbit
 min/max network layer size:           60 /    1492
 min/max overhead-adjusted size:      134 /    1566
 average network hdr offset:            0

                  Tin 0
  thresh        400Mbit
  target            5ms
  interval        100ms
  pk_delay          4us
  av_delay          0us
  sp_delay          0us
  backlog            0b
  pkts           996307
  bytes      1277335341
  way_inds        17866
  way_miss          177
  way_cols            0
  drops               4
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1492
  quantum          1514

By the way is the set MPU value of 96 correct?

So I've hosted some p2p matches while running long speed tests in the background and noticed no issues (complaints) with one SQM instance on the pppoe-wan interface. However, whatever is happening during these bufferbloat tests they do seem to favor the 2xSQM setup with which I also noticed no issues:

Waveform results

As it stands I'm going to go with the 2xSQM variant even if it's only due to placebo.

I've also now done tcpdumbs on each interface device and analyzed them with Wireshark to determine the smallest length packets for the MPU values. At least I assumed that's how it's supposed to be done :rofl:

Overall, this is now my SQM config:
config queue 'eth1'
	option enabled '1'
	option interface 'ds-wan_6_4'
	option download '400000'
	option upload '100000'
	option qdisc 'cake'
	option script 'piece_of_cake.qos'
	option linklayer 'ethernet'
	option debug_logging '0'
	option verbosity '5'
	option qdisc_advanced '1'
	option squash_dscp '1'
	option squash_ingress '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'ECN'
	option qdisc_really_really_advanced '1'
	option iqdisc_opts 'mpu 56 nat dual-dsthost ingress'
	option eqdisc_opts 'mpu 56 nat dual-srchost'
	option overhead '74'

config queue
	option enabled '1'
	option interface 'pppoe-wan'
	option download '400000'
	option upload '100000'
	option debug_logging '0'
	option verbosity '5'
	option qdisc 'cake'
	option script 'piece_of_cake.qos'
	option qdisc_advanced '1'
	option squash_dscp '1'
	option squash_ingress '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option qdisc_really_really_advanced '1'
	option iqdisc_opts 'mpu 76 nat dual-dsthost ingress'
	option eqdisc_opts 'mpu 76 nat dual-srchost'
	option linklayer 'ethernet'
	option overhead '34'

But the most important thing is to thank you for all the help. This issue has been bugging me for months, and I myself wouldn't have come anywhere close to solving it without some remaining doubt of something being silently broken. Thank you :grin:

1 Like