SQM fine in 17.01.4, can't keep up with line speed in 18.06 (Netgear R7800)

tl;dr: What changed in 18.06 to kill performance on this hardware for SQM: really bad for cake/piece_of_cake, and bad for fq_codel/simple?

My computer is connected by gigabit ethernet to the router, which uses a cable modem through RCN (Boston area).

Here's the DSLreports speed test result running 18.06 with no SQM: 278.2/17.1 Mbps:



The R7800 running 17.01.4 kept up with my ISP line speed (nominal 250Mbps/15Mbps) when configured to use SQM with cake/piece_of_cake, per https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm

I think based on my browser/DSLreports history that this is an example report from running 17.01 configured with cake/piece_of_cake and the same upload/download limits: 223.1/14.71 Mbps:




which looks reasonable based on the configured download/upload limits

A few days ago I installed 18.06 on my R7800 router, configured SQM to
use cake/piece_of_cake, and it can't keep up with the line speed,
here's what I get: 95.8/14.16 Mbps:




Here's the config, and tc -d/tc -s output from before & after the test using cake/piece_of_cake:

root@LEDE:~# cat /etc/config/sqm

config queue 'eth1'
	option qdisc_advanced '0'
	option verbosity '5'
	option upload '15000'
	option linklayer 'ethernet'
	option overhead '18'
	option debug_logging '1'
	option download '232750'
	option interface 'eth0.2'
	option enabled '1'
	option qdisc 'cake'
	option script 'piece_of_cake.qos'

(eth0.2 is the VLAN for wan/wan6, but the results are very similar if I use plain eth0)

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.1 root refcnt 2 
qdisc cake 8012: dev eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate split-gso rtt 100.0ms noatm overhead 18 
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-guest root refcnt 2 
qdisc noqueue 0: dev br-iot root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc cake 8013: dev ifb4eth0.2 root refcnt 2 bandwidth 232750Kbit besteffort triple-isolate wash split-gso rtt 100.0ms noatm overhead 18 

root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 310439874 bytes 1941890 pkt (dropped 0, overlimits 0 requeues 62196) 
 backlog 0b 0p requeues 62196
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 310439874 bytes 1941890 pkt (dropped 0, overlimits 0 requeues 62196) 
 backlog 0b 0p requeues 62196
  maxpacket 1514 drop_overlimit 0 new_flow_count 57761 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 4236913098 bytes 3046781 pkt (dropped 0, overlimits 0 requeues 101975) 
 backlog 0b 0p requeues 101975
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 4236913098 bytes 3046781 pkt (dropped 0, overlimits 0 requeues 101975) 
 backlog 0b 0p requeues 101975
  maxpacket 1514 drop_overlimit 0 new_flow_count 127972 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8012: dev eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate split-gso rtt 100.0ms noatm overhead 18 
 Sent 51838 bytes 351 pkt (dropped 0, overlimits 19 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 3456b of 4Mb
 capacity estimate: 15Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       46 /    1518
 average network hdr offset:           10

                  Tin 0
  thresh         15Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        627us
  av_delay         25us
  sp_delay          8us
  backlog            0b
  pkts              351
  bytes           51838
  way_inds            4
  way_miss           70
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            2
  un_flows            0
  max_len          1923
  quantum           457

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 99707 bytes 1045 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2015423 bytes 2972 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 228 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8013: dev ifb4eth0.2 root refcnt 2 bandwidth 232750Kbit besteffort triple-isolate wash split-gso rtt 100.0ms noatm overhead 18 
 Sent 114901 bytes 1047 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 6296b of 11637500b
 capacity estimate: 232750Kbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh     232750Kbit
  target          5.0ms
  interval      100.0ms
  pk_delay         79us
  av_delay          9us
  sp_delay          3us
  backlog            0b
  pkts             1047
  bytes          114901
  way_inds            5
  way_miss           65
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len          4294
  quantum          1514

And after running the speed test:

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.1 root refcnt 2 
qdisc cake 8012: dev eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate split-gso rtt 100.0ms noatm overhead 18 
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-guest root refcnt 2 
qdisc noqueue 0: dev br-iot root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc cake 8013: dev ifb4eth0.2 root refcnt 2 bandwidth 232750Kbit besteffort triple-isolate wash split-gso rtt 100.0ms noatm overhead 18 

root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 351709498 bytes 2034499 pkt (dropped 0, overlimits 0 requeues 65185) 
 backlog 0b 0p requeues 65185
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 351709498 bytes 2034499 pkt (dropped 0, overlimits 0 requeues 65185) 
 backlog 0b 0p requeues 65185
  maxpacket 1514 drop_overlimit 0 new_flow_count 60774 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 4385184093 bytes 3164614 pkt (dropped 0, overlimits 0 requeues 109126) 
 backlog 0b 0p requeues 109126
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 4385184093 bytes 3164614 pkt (dropped 0, overlimits 0 requeues 109126) 
 backlog 0b 0p requeues 109126
  maxpacket 1514 drop_overlimit 0 new_flow_count 137321 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8012: dev eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate split-gso rtt 100.0ms noatm overhead 18 
 Sent 41321462 bytes 92960 pkt (dropped 919, overlimits 39456 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 194432b of 4Mb
 capacity estimate: 15Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       46 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh         15Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        672us
  av_delay        211us
  sp_delay         11us
  backlog            0b
  pkts            93879
  bytes        42712828
  way_inds            6
  way_miss          392
  way_cols            0
  drops             919
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         12112
  quantum           457

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 143870988 bytes 121001 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2015423 bytes 2972 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 228 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8013: dev ifb4eth0.2 root refcnt 2 bandwidth 232750Kbit besteffort triple-isolate wash split-gso rtt 100.0ms noatm overhead 18 
 Sent 148570176 bytes 120706 pkt (dropped 297, overlimits 622 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 661056b of 11637500b
 capacity estimate: 232750Kbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh     232750Kbit
  target          5.0ms
  interval      100.0ms
  pk_delay        245us
  av_delay         45us
  sp_delay          4us
  backlog            0b
  pkts           121003
  bytes       149019834
  way_inds            6
  way_miss          384
  way_cols            0
  drops             297
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         12112
  quantum          1514

So I tried fq_codel/simple, and that is better but still unable to keep up with the line speed.

Here's that configuration, and before/after of tc -d/tc-s:

root@LEDE:~# cat /etc/config/sqm

config queue 'eth1'
	option qdisc_advanced '0'
	option verbosity '5'
	option upload '15000'
	option linklayer 'ethernet'
	option overhead '18'
	option debug_logging '1'
	option download '232750'
	option interface 'eth0.2'
	option enabled '1'
	option qdisc 'fq_codel'
	option script 'simple.qos'

Before:

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.1 root refcnt 2 
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 18 direct_packets_stat 0 ver 3.17 direct_qlen 1000
 linklayer ethernet overhead 18 
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-guest root refcnt 2 
qdisc noqueue 0: dev br-iot root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 16 direct_packets_stat 0 ver 3.17 direct_qlen 32
 linklayer ethernet overhead 18 
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 352900550 bytes 2042566 pkt (dropped 0, overlimits 0 requeues 65252) 
 backlog 0b 0p requeues 65252
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 352900550 bytes 2042566 pkt (dropped 0, overlimits 0 requeues 65252) 
 backlog 0b 0p requeues 65252
  maxpacket 1514 drop_overlimit 0 new_flow_count 60823 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 4393401923 bytes 3174464 pkt (dropped 0, overlimits 0 requeues 109372) 
 backlog 0b 0p requeues 109372
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 4393401923 bytes 3174464 pkt (dropped 0, overlimits 0 requeues 109372) 
 backlog 0b 0p requeues 109372
  maxpacket 1514 drop_overlimit 0 new_flow_count 137536 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 18 direct_packets_stat 0 direct_qlen 1000
 Sent 205813 bytes 1666 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 202840 bytes 1639 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 1875 drop_overlimit 0 new_flow_count 221 ecn_mark 0
  new_flows_len 1 old_flows_len 2
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2973 bytes 27 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 199 drop_overlimit 0 new_flow_count 27 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 1133663 bytes 2141 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2015423 bytes 2972 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 228 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 16 direct_packets_stat 0 direct_qlen 32
 Sent 1190531 bytes 2143 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 1190531 bytes 2143 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 11668 drop_overlimit 0 new_flow_count 369 ecn_mark 0
  new_flows_len 1 old_flows_len 2

and the DSLreports with this enabled: 159.4/14.64 Mbps:




and the tc -d/tc-s:

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.1 root refcnt 2 
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 18 direct_packets_stat 0 ver 3.17 direct_qlen 1000
 linklayer ethernet overhead 18 
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-guest root refcnt 2 
qdisc noqueue 0: dev br-iot root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 16 direct_packets_stat 0 ver 3.17 direct_qlen 32
 linklayer ethernet overhead 18 
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 401580004 bytes 2171059 pkt (dropped 0, overlimits 0 requeues 70147) 
 backlog 0b 0p requeues 70147
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 401580004 bytes 2171059 pkt (dropped 0, overlimits 0 requeues 70147) 
 backlog 0b 0p requeues 70147
  maxpacket 1514 drop_overlimit 0 new_flow_count 65676 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 4688540505 bytes 3395775 pkt (dropped 0, overlimits 0 requeues 115844) 
 backlog 0b 0p requeues 115844
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 4688540505 bytes 3395775 pkt (dropped 0, overlimits 0 requeues 115844) 
 backlog 0b 0p requeues 115844
  maxpacket 1514 drop_overlimit 0 new_flow_count 147930 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 18 direct_packets_stat 0 direct_qlen 1000
 Sent 50270181 bytes 130159 pkt (dropped 1157, overlimits 44671 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 50235423 bytes 129831 pkt (dropped 1157, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 11688 drop_overlimit 0 new_flow_count 44941 ecn_mark 0
  new_flows_len 0 old_flows_len 1
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 34758 bytes 328 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 225 drop_overlimit 0 new_flow_count 327 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 289853249 bytes 229260 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2015423 bytes 2972 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 228 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 16 direct_packets_stat 0 direct_qlen 32
 Sent 290763815 bytes 228003 pkt (dropped 350, overlimits 56864 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 290763815 bytes 228003 pkt (dropped 350, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 11752 drop_overlimit 0 new_flow_count 23025 ecn_mark 0
  new_flows_len 0 old_flows_len 1

You might want to post to the OpenWrt and/or see if you can track down David Taht from (now defunct) CeroWrt who did a lot of work on the bufferbloat stuff and these modules to find out if he has any idea.s

@dtaht, any thoughts?

Mmmh, since htb/fq_codel is a bit less computationally demanding the fact that you can't reach the configured rate with simple.qos/fq_codel indoicates that something in the kernel got significantly slower (cake also got a bit fatter, but that only explains the difference between HTB+fq_codel and cake, not the full difference between 17.01.4 and 18.06).

Best Regards

Could be another instance of this regular network config regression in the bug tracker:
https://bugs.openwrt.org/index.php?do=details&task_id=1598

@hnyman, any experience to lend on network slowdown in 18.06 vs. 17.01.4?

I haven't really tested network speeds lately, as my own connection is too slow, only 100+ Mbit/s

But in late 2016 (when I got R7800) there was discussion about HTB qdisc causing SQM simple.qos to miscalculate some delays for dual-core CPUs (like in R7800). Discussion starts at


But like I mention in the discussion, those were then mitigated in the upstream Linux.

It was also discussed with SQM & cake authors in https://github.com/tohojo/sqm-scripts/issues/48

But at that time, cake performed better than simple/fq_codel.

My input to the discussion is that

  • it is quite possible that there is some regression.
  • it might be also in the upstream Linux kernel.
  • it might also affect just certain devices like single/dual-core CPUs ARM/MIPS/X86, so debugging can be difficult
  • it might not be the "main" qdisc like cake or fw_codel, but the helpers like HTB (or TBF)

Hopefully those earlier discussions from 2016-2017 help in debugging.

Hmm, interesting. Not sure what to make of all of that but it did prompt me to play around with CPU scaling.
If I put the CPUs into performance mode (echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor, same for cpu1), fq_codel/simple can keep up with line speed. If the CPUs are in ondemand mode, it cannot.
Interestingly, during a DSLreports download test, in ondemand mode, the CPUs don't both top out at top speed, the highest they get is:

root@LEDE:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq
1000000
1725000

so it seems the issue isn't raw CPU power, it's something else (perhaps synchronization latency, which I took as a hint from reading the referred threads).

Take a look at this: R7800 performance

Thanks, some good hints in that thread. Setting the ondemand control values helped a lot on its own. Adding in software flow offloading and I'm up to line speeds for download testing and I get lower CPU frequency when idle, for fq_codel/simple. cake/piece_of_cake still aren't up to line speeds.

root@LEDE:~# cat /etc/rc.local 
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
echo 35 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
exit 0
root@LEDE:~# cat /etc/config/sqm
config queue 'eth1'
	option qdisc_advanced '0'
	option verbosity '5'
	option upload '15000'
	option linklayer 'ethernet'
	option overhead '18'
	option debug_logging '1'
	option interface 'eth0'
	option enabled '1'
	option qdisc 'fq_codel'
	option script 'simple.qos'
	option download '250000'
root@LEDE:~# grep offload /etc/config/firewall
	option flow_offloading '1'
root@LEDE:~# 
1 Like

Thanks all, this settings work for me :slight_smile: