SQM Bufferbloat Issues

Mmmh, maybe post:

  1. capture output of tc -s qdisc

  2. run a speedtest: https://speed.cloudflare.com

  3. capture output of tc -s qdisc again, immediately after the test (or even at the end of the test)

  4. generate a screenshot of the results page (if desired redact IP address and location, assuming the locationis a) close enough, and b) sensitive)

  5. post the results of 1), 3), and 5) to this thread...

SQM off:

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 16730700032 bytes 11723944 pkt (dropped 0, overlimits 0 requeues 1588)
 backlog 0b 0p requeues 1588
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 5906580396 bytes 4214737 pkt (dropped 0, overlimits 0 requeues 787)
 backlog 0b 0p requeues 787
  maxpacket 25738 drop_overlimit 0 new_flow_count 15048 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 3680568914 bytes 2530515 pkt (dropped 0, overlimits 0 requeues 285)
 backlog 0b 0p requeues 285
  maxpacket 25738 drop_overlimit 0 new_flow_count 1049 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 4632660863 bytes 3196902 pkt (dropped 0, overlimits 0 requeues 284)
 backlog 0b 0p requeues 284
  maxpacket 25738 drop_overlimit 0 new_flow_count 1907 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 2510889859 bytes 1781790 pkt (dropped 0, overlimits 0 requeues 232)
 backlog 0b 0p requeues 232
  maxpacket 24224 drop_overlimit 0 new_flow_count 924 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root
 Sent 273554230 bytes 691803 pkt (dropped 0, overlimits 0 requeues 445)
 backlog 0b 0p requeues 445
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 120547208 bytes 170594 pkt (dropped 0, overlimits 0 requeues 107)
 backlog 0b 0p requeues 107
  maxpacket 17061 drop_overlimit 0 new_flow_count 125 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 108615475 bytes 154714 pkt (dropped 0, overlimits 0 requeues 156)
 backlog 0b 0p requeues 156
  maxpacket 24718 drop_overlimit 0 new_flow_count 133 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 23041785 bytes 184990 pkt (dropped 0, overlimits 0 requeues 85)
 backlog 0b 0p requeues 85
  maxpacket 15547 drop_overlimit 0 new_flow_count 183 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 21349762 bytes 181505 pkt (dropped 0, overlimits 0 requeues 97)
 backlog 0b 0p requeues 97
  maxpacket 15547 drop_overlimit 0 new_flow_count 109 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.99 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

Before (SQM On):

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 17307058546 bytes 12138979 pkt (dropped 0, overlimits 0 requeues 1628)
 backlog 0b 0p requeues 1628
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 5930597339 bytes 4233871 pkt (dropped 0, overlimits 0 requeues 796)
 backlog 0b 0p requeues 796
  maxpacket 25738 drop_overlimit 0 new_flow_count 15057 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 3686920001 bytes 2536652 pkt (dropped 0, overlimits 0 requeues 293)
 backlog 0b 0p requeues 293
  maxpacket 25738 drop_overlimit 0 new_flow_count 1055 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 4660501491 bytes 3219506 pkt (dropped 0, overlimits 0 requeues 286)
 backlog 0b 0p requeues 286
  maxpacket 25738 drop_overlimit 0 new_flow_count 1912 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 3029039715 bytes 2148950 pkt (dropped 0, overlimits 0 requeues 253)
 backlog 0b 0p requeues 253
  maxpacket 24224 drop_overlimit 0 new_flow_count 943 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8013: dev eth1 root refcnt 5 bandwidth 50Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 44
 Sent 67139290 bytes 286350 pkt (dropped 6, overlimits 720392 requeues 2)
 backlog 0b 0p requeues 2
 memory used: 1921280b of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       72 /    1544
 average network hdr offset:           14

                  Tin 0
  thresh         50Mbit
  target            5ms
  interval        100ms
  pk_delay       2.99ms
  av_delay        790us
  sp_delay          2us
  backlog            0b
  pkts           286356
  bytes        67144518
  way_inds          152
  way_miss          664
  way_cols            0
  drops               6
  marks               0
  ack_drop            0
  sp_flows            3
  bk_flows            1
  un_flows            0
  max_len         33442
  quantum          1514

qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
 Sent 533803011 bytes 398829 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.99 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8014: dev ifb4eth1 root refcnt 2 bandwidth 1Gbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 44
 Sent 552069093 bytes 398827 pkt (dropped 2, overlimits 416652 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2650313b of 15140Kb
 capacity estimate: 1Gbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       90 /    1544
 average network hdr offset:           14

                  Tin 0
  thresh          1Gbit
  target            5ms
  interval        100ms
  pk_delay         29us
  av_delay          4us
  sp_delay          0us
  backlog            0b
  pkts           398829
  bytes       552072121
  way_inds          449
  way_miss          638
  way_cols            0
  drops               2
  marks               0
  ack_drop            0
  sp_flows            5
  bk_flows            1
  un_flows            0
  max_len         45420
  quantum          1514

Test Results:



Post-test tc -s qdisc:

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 17848271010 bytes 12526043 pkt (dropped 0, overlimits 0 requeues 1647)
 backlog 0b 0p requeues 1647
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 5930867105 bytes 4235029 pkt (dropped 0, overlimits 0 requeues 796)
 backlog 0b 0p requeues 796
  maxpacket 25738 drop_overlimit 0 new_flow_count 15057 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 3688559132 bytes 2539151 pkt (dropped 0, overlimits 0 requeues 298)
 backlog 0b 0p requeues 298
  maxpacket 25738 drop_overlimit 0 new_flow_count 1059 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 4660936981 bytes 3221452 pkt (dropped 0, overlimits 0 requeues 287)
 backlog 0b 0p requeues 287
  maxpacket 25738 drop_overlimit 0 new_flow_count 1913 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 3567907792 bytes 2530411 pkt (dropped 0, overlimits 0 requeues 266)
 backlog 0b 0p requeues 266
  maxpacket 24224 drop_overlimit 0 new_flow_count 956 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8013: dev eth1 root refcnt 5 bandwidth 50Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 44
 Sent 133126969 bytes 566634 pkt (dropped 7, overlimits 1434953 requeues 2)
 backlog 0b 0p requeues 2
 memory used: 1921280b of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       72 /    1544
 average network hdr offset:           14

                  Tin 0
  thresh         50Mbit
  target            5ms
  interval        100ms
  pk_delay       3.43ms
  av_delay        315us
  sp_delay          0us
  backlog            0b
  pkts           566641
  bytes       133133711
  way_inds          239
  way_miss          960
  way_cols            0
  drops               7
  marks               0
  ack_drop            0
  sp_flows            3
  bk_flows            1
  un_flows            0
  max_len         33442
  quantum          1514

qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
 Sent 1057275566 bytes 786956 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.99 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8014: dev ifb4eth1 root refcnt 2 bandwidth 1Gbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 44
 Sent 1093446324 bytes 786953 pkt (dropped 3, overlimits 827092 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2736668b of 15140Kb
 capacity estimate: 1Gbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       90 /    1544
 average network hdr offset:           14

                  Tin 0
  thresh          1Gbit
  target            5ms
  interval        100ms
  pk_delay        102us
  av_delay         24us
  sp_delay          0us
  backlog            0b
  pkts           786956
  bytes      1093450866
  way_inds          777
  way_miss          899
  way_cols            0
  drops               3
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len         45420
  quantum          1514

Test results (SQM Off):



Hopefully this also will help.

The post test tc -s qdisc has only 1 drop more (per direction) than the pre test one, that is highly unlikely, something there went wrong...

But the packet counters still went up:
Download:
pkts 786956-398829 = 388127

The fact that the the test without SQM did not reach the expected 1000 download, indicates that this might not be a good test/site for your ISP...
Comparing the test with and without SQM I would say SQM helps somewhat (just look at the cloudflare latency during uploads...) but this is still quite odd.
Could you repeat these measurements outside of the primetime, so at a time you expect less/no congestion in the ISP's network or their peerings?

bytes (1093450866-552072121)/1000^2 = 541.378745 MB

2 Likes

tc -s qdisc before:

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 43073142089 bytes 33462134 pkt (dropped 0, overlimits 0 requeues 18512)
 backlog 0b 0p requeues 18512
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 9997652467 bytes 7881521 pkt (dropped 0, overlimits 0 requeues 4050)
 backlog 0b 0p requeues 4050
  maxpacket 25738 drop_overlimit 0 new_flow_count 18447 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 12418143445 bytes 9544619 pkt (dropped 0, overlimits 0 requeues 7858)
 backlog 0b 0p requeues 7858
  maxpacket 25738 drop_overlimit 0 new_flow_count 8251 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 11328467518 bytes 8783153 pkt (dropped 0, overlimits 0 requeues 3334)
 backlog 0b 0p requeues 3334
  maxpacket 25738 drop_overlimit 0 new_flow_count 4694 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 9328878659 bytes 7252841 pkt (dropped 0, overlimits 0 requeues 3270)
 backlog 0b 0p requeues 3270
  maxpacket 25738 drop_overlimit 0 new_flow_count 3395 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8017: dev eth1 root refcnt 5 bandwidth 50Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 44
 Sent 188503 bytes 784 pkt (dropped 0, overlimits 191 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 9984b of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       72 /    1544
 average network hdr offset:           13

                  Tin 0
  thresh         50Mbit
  target            5ms
  interval        100ms
  pk_delay        169us
  av_delay         15us
  sp_delay          0us
  backlog            0b
  pkts              784
  bytes          188503
  way_inds            0
  way_miss           81
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len          4076
  quantum          1514

qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
 Sent 3903583 bytes 3358 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.99 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8018: dev ifb4eth1 root refcnt 2 bandwidth 1Gbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 44
 Sent 3950887 bytes 3358 pkt (dropped 0, overlimits 4832 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 86528b of 15140Kb
 capacity estimate: 1Gbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       90 /    1544
 average network hdr offset:           14

                  Tin 0
  thresh          1Gbit
  target            5ms
  interval        100ms
  pk_delay        205us
  av_delay         72us
  sp_delay          2us
  backlog            0b
  pkts             3358
  bytes         3950887
  way_inds            0
  way_miss           71
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            3
  bk_flows            1
  un_flows            0
  max_len          5207
  quantum          1514

tc -s qdisc after:

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 44160369377 bytes 34209243 pkt (dropped 0, overlimits 0 requeues 18570)
 backlog 0b 0p requeues 18570
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 10022244797 bytes 7901620 pkt (dropped 0, overlimits 0 requeues 4063)
 backlog 0b 0p requeues 4063
  maxpacket 25738 drop_overlimit 0 new_flow_count 18457 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 13394598998 bytes 10205686 pkt (dropped 0, overlimits 0 requeues 7887)
 backlog 0b 0p requeues 7887
  maxpacket 25738 drop_overlimit 0 new_flow_count 8286 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 11408231239 bytes 8842815 pkt (dropped 0, overlimits 0 requeues 3342)
 backlog 0b 0p requeues 3342
  maxpacket 25738 drop_overlimit 0 new_flow_count 4699 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 9335294343 bytes 7259122 pkt (dropped 0, overlimits 0 requeues 3278)
 backlog 0b 0p requeues 3278
  maxpacket 25738 drop_overlimit 0 new_flow_count 3402 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8017: dev eth1 root refcnt 5 bandwidth 50Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 44
 Sent 76258134 bytes 439826 pkt (dropped 4, overlimits 739509 requeues 8)
 backlog 0b 0p requeues 8
 memory used: 538880b of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       72 /    1544
 average network hdr offset:           14

                  Tin 0
  thresh         50Mbit
  target            5ms
  interval        100ms
  pk_delay        567us
  av_delay         31us
  sp_delay          0us
  backlog            0b
  pkts           439830
  bytes        76262768
  way_inds           98
  way_miss          377
  way_cols            0
  drops               4
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len         33442
  quantum          1514

qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
 Sent 1055930703 bytes 751137 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.99 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8018: dev ifb4eth1 root refcnt 2 bandwidth 1Gbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 44
 Sent 1091330169 bytes 751131 pkt (dropped 6, overlimits 792931 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2814671b of 15140Kb
 capacity estimate: 1Gbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       90 /    1544
 average network hdr offset:           14

                  Tin 0
  thresh          1Gbit
  target            5ms
  interval        100ms
  pk_delay        140us
  av_delay         68us
  sp_delay          0us
  backlog            0b
  pkts           751137
  bytes      1091339253
  way_inds          319
  way_miss          365
  way_cols            0
  drops               6
  marks               0
  ack_drop            0
  sp_flows            3
  bk_flows            1
  un_flows            0
  max_len         46934
  quantum          1514

Test Results (sqm enabled):



Without SQM:



Speedtest ran directly on router without SQM on:

SpeedTest++ version 1.20.3
Speedtest.net command line interface
Info: https://github.com/oskarirauta/speedtestcpp
Author: Francesco Laurita <francesco.laurita@gmail.com>
Co-authored-by: Oskari Rauta <oskari.rauta@gmail.com>

IP: [REDACTED] ( Atlantic Broadband ) Location: [37.7149, -76.3806]
Finding fastest server...
10 Servers online
..
Server #60961: Richmond, VA speedtest.internetsubway.com.prod.hosts.ooklaserver.net:8080 by The Internet Subway (52 km from you): 15 ms
Ping: 15 ms.
Jitter: 1 ms.

server selected profile detected: profile selected from server configuration

Download: 1241.28 Mbit/s
Upload: 52.44 Mbit/s

Speedtest ran directly on router with SQM:

SpeedTest++ version 1.20.3
Speedtest.net command line interface
Info: https://github.com/oskarirauta/speedtestcpp
Author: Francesco Laurita <francesco.laurita@gmail.com>
Co-authored-by: Oskari Rauta <oskari.rauta@gmail.com>

IP: [REDACTED] ( Atlantic Broadband ) Location: [37.7149, -76.3806]
Finding fastest server...
10 Servers online
..
Server #1770: Richmond, VA stosat-smil-01.sys.comcast.net.prod.hosts.ooklaserver.net:8080 by Comcast (52 km from you): 18 ms
Ping: 18 ms.
Jitter: 1 ms.

server selected profile detected: profile selected from server configuration

Download: 914.55 Mbit/s
Upload: 43.63 Mbit/s

Bufferbloat test with SQM: https://www.waveform.com/tools/bufferbloat?test-id=43cf7e02-1097-4c27-913e-e9334ff3ce87

Bufferbloat test without SQM: https://www.waveform.com/tools/bufferbloat?test-id=ac17b1af-aa2e-4ecd-b0c0-99ef328fc9d7

Mmmh, the tc -s qdisc output still does not show the expected number of drops/marls for a multisecond capacity test run, it is if something else was limiting the throughput and not your two cake instances... as cake's interventions would show up as drops or marks in cake's statistics output.
However the speedtests indicate that there must be a difference between with SQM and without SQM runs but these do not really materialize in cake's own counters, which is really odd.

1 Like

Agreed, I'm confused as well. What else on OpenWRT would limit the throughput? There's a Cat6a cable between the Arris S33v2 and the eth1 interface on OpenWRT, no black box or anything in between.

Hrm... This smells a bit like IRQ affinity to me. SQM is single threaded, so sits on a single core; if that core is also being used by some of the kernel's network internals then that would slow things down.

I'm no expert on what could/can be moved but I'd look at shifting some of the IRQs to a different core.

2 Likes

I'm currently using irqbalance. Should I turn this off and manually set the IRQ affinities?

If you can, do some tests with irqbalance and SQM on and off, and look at /proc/interrupts after each one. Ideally reboot between each run.

That should prove whether it's a balancing problem - again, I'm no expert here but others are.

But the phenotype for cake running out of CPU is that the throughput stays close to expected, while the latency noticeably increases, while HTB+fq_codel tend to show decent low latency while the throughput goes down noticeably... (so you could test simple.qos/fq_codel to see whether it gives the same capacity test results). But you are on a DOCSIS network, so the whole issue might be that docsis microslot scheduling might not like your own traffic shaper (say if your segment is getting congested)?

Even when /etc/init.d/packet_steering is disabled, irqbalance may fail to recognize more than two cores, typically either overlooking them or failing to balance them properly. In a typical SOHO all-in-one WiFi router, this results in only a few IRQs being redistributed, while the wireless interfaces remain on a single core. Manually pinning the IRQs to specific cores is recommended for optimal performance.

1 Like