SQM (cake) does not achieve fairness

Installing and Using OpenWrt Network and Wireless Configuration
1

I have a personal server behind an (Internet-facing) OpenWrt router. The uplink connection is PPPoE (but not *DSL), with ~46-48 Mbps goodput (symmetric).

The router is configured to use the SQM GUI, with the layer_cake.qos script. I tried to use both the default settings and ingress nat dual-dsthost / nat dual-srchost for resp. ingress/egress qdiscs (according to the SQM article on the OpenWrt wiki). The link-layer overhead is set to overhead 46 mpu 84 noatm and the gross shaper rate is set to 50000.

/etc/config/sqm 
config queue 'eth1'
        option interface 'pppoe-wan'
        option debug_logging '0'
        option qdisc 'cake'
        option verbosity '8'
        option qdisc_advanced '1'
        option squash_dscp '1'
        option squash_ingress '1'
        option ingress_ecn 'ECN'
        option egress_ecn 'NOECN'
        option qdisc_really_really_advanced '1'
        option iqdisc_opts 'ingress nat dual-dsthost'
        option eqdisc_opts 'nat dual-srchost'
        option linklayer 'ethernet'
        option overhead '46'
        option linklayer_advanced '1'
        option tcMTU '2047'
        option tcTSIZE '128'
        option tcMPU '84'
        option linklayer_adaptation_mechanism 'default'
        option download '50000'
        option upload '50000'
        option enabled '1'
        option script 'layer_cake.qos'
`tc qdisc` 
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.1 root refcnt 2 
qdisc noqueue 0: dev eth0.2 root refcnt 2 
qdisc noqueue 0: dev br-guest root refcnt 2 
qdisc noqueue 0: dev site2site root refcnt 2 
qdisc cake 80a3: dev pppoe-wan root refcnt 2 bandwidth 50Mbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 46 mpu 84 
qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc fq_codel 0: dev tailscale0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
qdisc cake 80a4: dev ifb4pppoe-wan root refcnt 2 bandwidth 50Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 46 mpu 84

However, when I try to download some Linux packages from the server (which is basically 5 simultaneous HTTPS streams), this happens:

Снимок экрана от 2024-02-07 00-29-56
Снимок экрана от 2024-02-07 00-29-562557×274 52.7 KB

Note that the last download (which started later) is completely starved by the first four.

Clarification: in this scenario, the server behind the OpenWrt router is serving the packages, while the client is on a different network altogether.

`tc -s qdisc` (idle) 
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 1869388694475 bytes 1681337968 pkt (dropped 92391, overlimits 0 requeues 8703) 
 backlog 0b 0p requeues 8703
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 1869388694475 bytes 1681337968 pkt (dropped 92391, overlimits 0 requeues 8703) 
 backlog 0b 0p requeues 8703
  maxpacket 1502 drop_overlimit 0 new_flow_count 277976 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 277898532918 bytes 1193615462 pkt (dropped 0, overlimits 0 requeues 314) 
 backlog 0b 0p requeues 314
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 277898532918 bytes 1193615462 pkt (dropped 0, overlimits 0 requeues 314) 
 backlog 0b 0p requeues 314
  maxpacket 1514 drop_overlimit 0 new_flow_count 29458453 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev site2site root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 80a3: dev pppoe-wan root refcnt 2 bandwidth 50Mbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 46 mpu 84 
 Sent 20048266 bytes 17983 pkt (dropped 1, overlimits 16237 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 96320b of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           30 /    1480
 min/max overhead-adjusted size:       84 /    1526
 average network hdr offset:            0

                   Bulk  Best Effort        Voice
  thresh       3125Kbit       50Mbit    12500Kbit
  target         5.78ms          5ms          5ms
  interval        101ms        100ms        100ms
  pk_delay        296us        546us          4us
  av_delay         33us        215us          0us
  sp_delay          6us         20us          0us
  backlog            0b           0b           0b
  pkts              948        17031            5
  bytes          153664     19895613          393
  way_inds           10            2            0
  way_miss          402          568            2
  way_cols            0            0            0
  drops               0            1            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            6            6            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len           652         2960           89
  quantum           300         1514          381

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ---------------- 
 Sent 5470430 bytes 10800 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev tailscale0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 912831 bytes 12675 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 73 drop_overlimit 0 new_flow_count 7 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 304 bytes 4 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 80a4: dev ifb4pppoe-wan root refcnt 2 bandwidth 50Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 46 mpu 84 
 Sent 5467470 bytes 10798 pkt (dropped 2, overlimits 5305 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 106Kb of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           40 /    1480
 min/max overhead-adjusted size:       86 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh         50Mbit
  target            5ms
  interval        100ms
  pk_delay       1.94ms
  av_delay       1.76ms
  sp_delay          7us
  backlog            0b
  pkts            10800
  bytes         5470430
  way_inds           17
  way_miss         1068
  way_cols            0
  drops               2
  marks               0
  ack_drop            0
  sp_flows           12
  bk_flows            1
  un_flows            0
  max_len          1480
  quantum          1514
`tc -s qdisc` (loaded) 
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 1869539836697 bytes 1681448564 pkt (dropped 92391, overlimits 0 requeues 8703) 
 backlog 0b 0p requeues 8703
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 1869539836697 bytes 1681448564 pkt (dropped 92391, overlimits 0 requeues 8703) 
 backlog 0b 0p requeues 8703
  maxpacket 1502 drop_overlimit 0 new_flow_count 277976 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 277904631054 bytes 1193671519 pkt (dropped 0, overlimits 0 requeues 314) 
 backlog 0b 0p requeues 314
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 277904631054 bytes 1193671519 pkt (dropped 0, overlimits 0 requeues 314) 
 backlog 0b 0p requeues 314
  maxpacket 1514 drop_overlimit 0 new_flow_count 29458519 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev site2site root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 80a3: dev pppoe-wan root refcnt 2 bandwidth 50Mbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 46 mpu 84 
 Sent 168757376 bytes 128579 pkt (dropped 1094, overlimits 120116 requeues 0) 
 backlog 33396b 23p requeues 0
 memory used: 798464b of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           30 /    1480
 min/max overhead-adjusted size:       84 /    1526
 average network hdr offset:            0

                   Bulk  Best Effort        Voice
  thresh       3125Kbit       50Mbit    12500Kbit
  target         5.78ms          5ms          5ms
  interval        101ms        100ms        100ms
  pk_delay       3.31ms       47.3ms         41us
  av_delay        506us       24.9ms          0us
  sp_delay         22us        920us          0us
  backlog            0b       33396b           0b
  pkts             2343       127343           10
  bytes          347783    170029759          950
  way_inds           21           51            0
  way_miss         1012         1450            5
  way_cols            0            0            0
  drops               0         1094            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            4           11            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len           657         4313          180
  quantum           300         1514          381

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ---------------- 
 Sent 20770188 bytes 73198 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev tailscale0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 913836 bytes 12689 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 73 drop_overlimit 0 new_flow_count 7 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 304 bytes 4 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 80a4: dev ifb4pppoe-wan root refcnt 2 bandwidth 50Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 46 mpu 84 
 Sent 20756868 bytes 73189 pkt (dropped 9, overlimits 27764 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 214208b of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           36 /    1480
 min/max overhead-adjusted size:       84 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh         50Mbit
  target            5ms
  interval        100ms
  pk_delay        192us
  av_delay         27us
  sp_delay          6us
  backlog            0b
  pkts            73198
  bytes        20770188
  way_inds          171
  way_miss         2828
  way_cols            0
  drops               9
  marks               0
  ack_drop            0
  sp_flows           12
  bk_flows            1
  un_flows            0
  max_len          1480
  quantum          1514
`tc -s qdisc` (affected) 
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 1869762979384 bytes 1681609003 pkt (dropped 92391, overlimits 0 requeues 8703) 
 backlog 0b 0p requeues 8703
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 1869762979384 bytes 1681609003 pkt (dropped 92391, overlimits 0 requeues 8703) 
 backlog 0b 0p requeues 8703
  maxpacket 1502 drop_overlimit 0 new_flow_count 277976 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 277912850585 bytes 1193755150 pkt (dropped 0, overlimits 0 requeues 314) 
 backlog 0b 0p requeues 314
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 277912850585 bytes 1193755150 pkt (dropped 0, overlimits 0 requeues 314) 
 backlog 0b 0p requeues 314
  maxpacket 1514 drop_overlimit 0 new_flow_count 29458579 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev site2site root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 80a3: dev pppoe-wan root refcnt 2 bandwidth 50Mbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 46 mpu 84 
 Sent 388370405 bytes 289018 pkt (dropped 2471, overlimits 269333 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 798464b of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           30 /    1480
 min/max overhead-adjusted size:       84 /    1526
 average network hdr offset:            0

                   Bulk  Best Effort        Voice
  thresh       3125Kbit       50Mbit    12500Kbit
  target         5.78ms          5ms          5ms
  interval        101ms        100ms        100ms
  pk_delay       2.04ms       1.03ms        116us
  av_delay        437us        401us          2us
  sp_delay         11us         32us          2us
  backlog            0b           0b           0b
  pkts             4176       287298           15
  bytes          636609    391317538         1686
  way_inds           32          131            0
  way_miss         1719         2552            7
  way_cols            0            0            0
  drops               0         2471            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            4           15            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len           657         4356          188
  quantum           300         1514          381

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ---------------- 
 Sent 37180252 bytes 162570 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev tailscale0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 913836 bytes 12689 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 73 drop_overlimit 0 new_flow_count 7 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 304 bytes 4 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 80a4: dev ifb4pppoe-wan root refcnt 2 bandwidth 50Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 46 mpu 84 
 Sent 37153612 bytes 162552 pkt (dropped 18, overlimits 53750 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 214208b of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           36 /    1480
 min/max overhead-adjusted size:       84 /    1526
 average network hdr offset:            0

                  Tin 0
  thresh         50Mbit
  target            5ms
  interval        100ms
  pk_delay        671us
  av_delay         40us
  sp_delay          7us
  backlog            0b
  pkts           162570
  bytes        37180252
  way_inds          348
  way_miss         5155
  way_cols            0
  drops              18
  marks               0
  ack_drop            0
  sp_flows           20
  bk_flows            1
  un_flows            0
  max_len          1480
  quantum          1514

What am I doing wrong? Is cake even supposed to prevent these kinds of issues, or am I understanding "fairness" incorrectly?

2

For troubleshooting (i.e., for excluding the mirror problem), could you please switch to a different mirror and try to reproduce the issue again? I am asking because some unofficial Arch mirrors are actually configured as caching proxies and downloading rarely-used or too-new packages from them is indeed slow, because their upstream is slow.

Also - does the problem still happen if you severely overshape the connection - e.g., to 30000 kbps?

3

Sorry, I was not clear enough. The packages in question are being served from my personal server that's behind the router, and the client (the machine the screenshot was taken on) is on a different network altogether (in another country, in fact). The problem in question is egress fairness.

4

Indeed, I have looked at the wrong places of the output. CAKE does something both in the "loaded" and "affected" cases, but the affected-case statistics look suspiciously close to the idle ones. I think that CAKE does not fully realize that the link is loaded.

Indeed, in the correct configuration, the shaper bandwidth must be below the available one, set to something like 90%, so that CAKE becomes a bottleneck. According to the numbers you shared (48 Mbps goodput, 50 Mbps target), this is not the case. So that's why the recommendation to start with a severely overshaped (to 30 Mbps) link, check that the issue no longer exists, and increase the shaper bandwidth until the issue starts appearing again still stands.

However, all of the above should be ignored because you have control over the server, and it is in a much better position to ensure a fair distribution of traffic between connections.

The updated recommendation is to try BBR as the TCP congestion control algorithm. Here is how:

modprobe tcp_bbr
sysctl -w net.ipv4.tcp_congestion_control=bbr

Or, to make settings persistent (assuming that your Linux distribution did not do that already):

echo tcp_bbr > /etc/modules-load.d/tcp.conf
echo net.ipv4.tcp_congestion_control=bbr > /etc/sysctl.d/91-tcp.conf

I repeat: this has to be done on the server, not on the router.

More details: https://patrakov.blogspot.com/2018/02/a-case-of-network-throughput.html

5

According to the numbers you shared (48 Mbps goodput, 50 Mbps target), this is not the case. So that's why the recommendation to start with a severely overshaped (to 30 Mbps) link, check that the issue no longer exists

That target was picked with consideration for non-zero overhead compensation I have enabled. Regardless, I tried to reduce the bandwidth limit to 30Mbps with no effect.

The updated recommendation is to try BBR as the TCP congestion control algorithm

I am using BBRv3 already.

6

Thanks for testing. I don't see any misconfiguration. At this point, one more test would be to remove dual-dsthost and dual-srchost and try again. Then try to replace both of these keywords with flows and try again.

If the above tests also don't yield fair traffic distribution among the connections, I guess it becomes a topic for discussion on the BBR lists.

7

Sadly, removing flow isolation parameters (which means cake defaults to triple-isolate) or specifying flows in both directions also have produced no effect.

If the above tests also don't yield fair traffic distribution among the connections, I guess it becomes a topic for discussion on the BBR lists.

BBR, or cake? Am I incorrect in understanding that cake is the issue here?

8

BBR. It is supposed to fight bufferbloat and ensure traffic fairness by itself. Assuming that there is nothing else behind the router, you don't need CAKE here. After all, what you were trying to do is to convince BBR to send traffic at equal speeds, by dropping or delaying packets with CAKE. But BBR already has access to all the round-trip delays that CAKE tries to optimize, so, in some sense, they might be fighting.

Maybe try reverting to BBRv1, or (just to test) try cubic + CAKE, because cubic reacts to packet loss that CAKE introduces? Obviously, switching to cubic would decrease goodput, so it is not recommended except for testing.

9

Try removing the "ingress" keyword... ingress will also account dropped packetsa as part of a flows share, if a flow is unresponsive and stays above its fair share, ingress will assure that that flow is dropped much harder and hence achieves less throughput... Not sure that this is the issue, but it might, if say that flow sets ECT(0) or ECT(1) accidentally and cake tries without success to reign it in with CE marks, before bringing out the big hammer...

10

Will that even do anything considering that the ingress direction is not the one being loaded? I apologize for a possibly unclear starting post; the OpenWrt router hosts the server of these files, and the client that's downloading them (the machine where the screenshot was taken) is on a different network which is not the bottleneck.

11

er, um, BBR is known to be quite unfair to itself, which could be where you are going south.