SQM fine in 17.01.4, can't keep up with line speed in 18.06 (Netgear R7800)

[atownlede]

tl;dr: What changed in 18.06 to kill performance on this hardware for SQM: really bad for cake/piece_of_cake, and bad for fq_codel/simple?

My computer is connected by gigabit ethernet to the router, which uses a cable modem through RCN (Boston area).

Here's the DSLreports speed test result running 18.06 with no SQM: 278.2/17.1 Mbps:

Speed test result

The R7800 running 17.01.4 kept up with my ISP line speed (nominal 250Mbps/15Mbps) when configured to use SQM with cake/piece_of_cake, per https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm

I think based on my browser/DSLreports history that this is an example report from running 17.01 configured with cake/piece_of_cake and the same upload/download limits: 223.1/14.71 Mbps:

Speed test result

which looks reasonable based on the configured download/upload limits

A few days ago I installed 18.06 on my R7800 router, configured SQM to
use cake/piece_of_cake, and it can't keep up with the line speed,
here's what I get: 95.8/14.16 Mbps:

Speed test result

Here's the config, and tc -d/tc -s output from before & after the test using cake/piece_of_cake:

root@LEDE:~# cat /etc/config/sqm

config queue 'eth1'
	option qdisc_advanced '0'
	option verbosity '5'
	option upload '15000'
	option linklayer 'ethernet'
	option overhead '18'
	option debug_logging '1'
	option download '232750'
	option interface 'eth0.2'
	option enabled '1'
	option qdisc 'cake'
	option script 'piece_of_cake.qos'

(eth0.2 is the VLAN for wan/wan6, but the results are very similar if I use plain eth0)

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.1 root refcnt 2 
qdisc cake 8012: dev eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate split-gso rtt 100.0ms noatm overhead 18 
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-guest root refcnt 2 
qdisc noqueue 0: dev br-iot root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc cake 8013: dev ifb4eth0.2 root refcnt 2 bandwidth 232750Kbit besteffort triple-isolate wash split-gso rtt 100.0ms noatm overhead 18 

root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 310439874 bytes 1941890 pkt (dropped 0, overlimits 0 requeues 62196) 
 backlog 0b 0p requeues 62196
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 310439874 bytes 1941890 pkt (dropped 0, overlimits 0 requeues 62196) 
 backlog 0b 0p requeues 62196
  maxpacket 1514 drop_overlimit 0 new_flow_count 57761 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 4236913098 bytes 3046781 pkt (dropped 0, overlimits 0 requeues 101975) 
 backlog 0b 0p requeues 101975
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 4236913098 bytes 3046781 pkt (dropped 0, overlimits 0 requeues 101975) 
 backlog 0b 0p requeues 101975
  maxpacket 1514 drop_overlimit 0 new_flow_count 127972 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8012: dev eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate split-gso rtt 100.0ms noatm overhead 18 
 Sent 51838 bytes 351 pkt (dropped 0, overlimits 19 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 3456b of 4Mb
 capacity estimate: 15Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       46 /    1518
 average network hdr offset:           10

                  Tin 0
  thresh         15Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        627us
  av_delay         25us
  sp_delay          8us
  backlog            0b
  pkts              351
  bytes           51838
  way_inds            4
  way_miss           70
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            2
  un_flows            0
  max_len          1923
  quantum           457

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 99707 bytes 1045 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2015423 bytes 2972 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 228 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8013: dev ifb4eth0.2 root refcnt 2 bandwidth 232750Kbit besteffort triple-isolate wash split-gso rtt 100.0ms noatm overhead 18 
 Sent 114901 bytes 1047 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 6296b of 11637500b
 capacity estimate: 232750Kbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh     232750Kbit
  target          5.0ms
  interval      100.0ms
  pk_delay         79us
  av_delay          9us
  sp_delay          3us
  backlog            0b
  pkts             1047
  bytes          114901
  way_inds            5
  way_miss           65
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len          4294
  quantum          1514

And after running the speed test:

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.1 root refcnt 2 
qdisc cake 8012: dev eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate split-gso rtt 100.0ms noatm overhead 18 
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-guest root refcnt 2 
qdisc noqueue 0: dev br-iot root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc cake 8013: dev ifb4eth0.2 root refcnt 2 bandwidth 232750Kbit besteffort triple-isolate wash split-gso rtt 100.0ms noatm overhead 18 

root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 351709498 bytes 2034499 pkt (dropped 0, overlimits 0 requeues 65185) 
 backlog 0b 0p requeues 65185
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 351709498 bytes 2034499 pkt (dropped 0, overlimits 0 requeues 65185) 
 backlog 0b 0p requeues 65185
  maxpacket 1514 drop_overlimit 0 new_flow_count 60774 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 4385184093 bytes 3164614 pkt (dropped 0, overlimits 0 requeues 109126) 
 backlog 0b 0p requeues 109126
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 4385184093 bytes 3164614 pkt (dropped 0, overlimits 0 requeues 109126) 
 backlog 0b 0p requeues 109126
  maxpacket 1514 drop_overlimit 0 new_flow_count 137321 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8012: dev eth0.2 root refcnt 2 bandwidth 15Mbit besteffort triple-isolate split-gso rtt 100.0ms noatm overhead 18 
 Sent 41321462 bytes 92960 pkt (dropped 919, overlimits 39456 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 194432b of 4Mb
 capacity estimate: 15Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       46 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh         15Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        672us
  av_delay        211us
  sp_delay         11us
  backlog            0b
  pkts            93879
  bytes        42712828
  way_inds            6
  way_miss          392
  way_cols            0
  drops             919
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         12112
  quantum           457

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 143870988 bytes 121001 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2015423 bytes 2972 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 228 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8013: dev ifb4eth0.2 root refcnt 2 bandwidth 232750Kbit besteffort triple-isolate wash split-gso rtt 100.0ms noatm overhead 18 
 Sent 148570176 bytes 120706 pkt (dropped 297, overlimits 622 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 661056b of 11637500b
 capacity estimate: 232750Kbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh     232750Kbit
  target          5.0ms
  interval      100.0ms
  pk_delay        245us
  av_delay         45us
  sp_delay          4us
  backlog            0b
  pkts           121003
  bytes       149019834
  way_inds            6
  way_miss          384
  way_cols            0
  drops             297
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         12112
  quantum          1514

So I tried fq_codel/simple, and that is better but still unable to keep up with the line speed.

Here's that configuration, and before/after of tc -d/tc-s:

root@LEDE:~# cat /etc/config/sqm

config queue 'eth1'
	option qdisc_advanced '0'
	option verbosity '5'
	option upload '15000'
	option linklayer 'ethernet'
	option overhead '18'
	option debug_logging '1'
	option download '232750'
	option interface 'eth0.2'
	option enabled '1'
	option qdisc 'fq_codel'
	option script 'simple.qos'

Before:

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.1 root refcnt 2 
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 18 direct_packets_stat 0 ver 3.17 direct_qlen 1000
 linklayer ethernet overhead 18 
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-guest root refcnt 2 
qdisc noqueue 0: dev br-iot root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 16 direct_packets_stat 0 ver 3.17 direct_qlen 32
 linklayer ethernet overhead 18 
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 

root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 352900550 bytes 2042566 pkt (dropped 0, overlimits 0 requeues 65252) 
 backlog 0b 0p requeues 65252
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 352900550 bytes 2042566 pkt (dropped 0, overlimits 0 requeues 65252) 
 backlog 0b 0p requeues 65252
  maxpacket 1514 drop_overlimit 0 new_flow_count 60823 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 4393401923 bytes 3174464 pkt (dropped 0, overlimits 0 requeues 109372) 
 backlog 0b 0p requeues 109372
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 4393401923 bytes 3174464 pkt (dropped 0, overlimits 0 requeues 109372) 
 backlog 0b 0p requeues 109372
  maxpacket 1514 drop_overlimit 0 new_flow_count 137536 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 18 direct_packets_stat 0 direct_qlen 1000
 Sent 205813 bytes 1666 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 202840 bytes 1639 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 1875 drop_overlimit 0 new_flow_count 221 ecn_mark 0
  new_flows_len 1 old_flows_len 2
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2973 bytes 27 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 199 drop_overlimit 0 new_flow_count 27 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 1133663 bytes 2141 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2015423 bytes 2972 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 228 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 16 direct_packets_stat 0 direct_qlen 32
 Sent 1190531 bytes 2143 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 1190531 bytes 2143 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 11668 drop_overlimit 0 new_flow_count 369 ecn_mark 0
  new_flows_len 1 old_flows_len 2

and the DSLreports with this enabled: 159.4/14.64 Mbps:

Speed test result

and the tc -d/tc-s:

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.1 root refcnt 2 
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 18 direct_packets_stat 0 ver 3.17 direct_qlen 1000
 linklayer ethernet overhead 18 
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-guest root refcnt 2 
qdisc noqueue 0: dev br-iot root refcnt 2 
qdisc noqueue 0: dev wlan0 root refcnt 2 
qdisc noqueue 0: dev wlan1 root refcnt 2 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 16 direct_packets_stat 0 ver 3.17 direct_qlen 32
 linklayer ethernet overhead 18 
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 

root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 401580004 bytes 2171059 pkt (dropped 0, overlimits 0 requeues 70147) 
 backlog 0b 0p requeues 70147
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 401580004 bytes 2171059 pkt (dropped 0, overlimits 0 requeues 70147) 
 backlog 0b 0p requeues 70147
  maxpacket 1514 drop_overlimit 0 new_flow_count 65676 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 4688540505 bytes 3395775 pkt (dropped 0, overlimits 0 requeues 115844) 
 backlog 0b 0p requeues 115844
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 4688540505 bytes 3395775 pkt (dropped 0, overlimits 0 requeues 115844) 
 backlog 0b 0p requeues 115844
  maxpacket 1514 drop_overlimit 0 new_flow_count 147930 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 18 direct_packets_stat 0 direct_qlen 1000
 Sent 50270181 bytes 130159 pkt (dropped 1157, overlimits 44671 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 50235423 bytes 129831 pkt (dropped 1157, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 11688 drop_overlimit 0 new_flow_count 44941 ecn_mark 0
  new_flows_len 0 old_flows_len 1
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 34758 bytes 328 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 225 drop_overlimit 0 new_flow_count 327 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 289853249 bytes 229260 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev ovpns0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 2015423 bytes 2972 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev ovpns1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 228 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 16 direct_packets_stat 0 direct_qlen 32
 Sent 290763815 bytes 228003 pkt (dropped 350, overlimits 56864 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 290763815 bytes 228003 pkt (dropped 350, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 11752 drop_overlimit 0 new_flow_count 23025 ecn_mark 0
  new_flows_len 0 old_flows_len 1

[cshoredaniel]

You might want to post to the OpenWrt and/or see if you can track down David Taht from (now defunct) CeroWrt who did a lot of work on the bufferbloat stuff and these modules to find out if he has any idea.s

[atownlede]

@dtaht, any thoughts?

[moeller0]

Mmmh, since htb/fq_codel is a bit less computationally demanding the fact that you can't reach the configured rate with simple.qos/fq_codel indoicates that something in the kernel got significantly slower (cake also got a bit fatter, but that only explains the difference between HTB+fq_codel and cake, not the full difference between 17.01.4 and 18.06).

Best Regards

[atownlede]

Could be another instance of this regular network config regression in the bug tracker:
https://bugs.openwrt.org/index.php?do=details&task_id=1598

[atownlede]

@hnyman, any experience to lend on network slowdown in 18.06 vs. 17.01.4?

[hnyman]

I haven't really tested network speeds lately, as my own connection is too slow, only 100+ Mbit/s

But in late 2016 (when I got R7800) there was discussion about HTB qdisc causing SQM simple.qos to miscalculate some delays for dual-core CPUs (like in R7800). Discussion starts at

Netgear R7800 exploration (IPQ8065, QCA9984)

I got a nice suggestion from tohojo at the SQM scripts development repo, where I opened an issue about this. It may actually be how the HTB (hierarchical token bucket) qdisc perfoms on this CPU... Well in that case it sounds like it has something to do with the way HTB runs on that CPU. You could try if TBF has the same behaviour; enable sqm, then issue the following commands to replace the configured qdiscs with a TBF-based one (provided TBF is in LEDE; not sure if it is): tc qdisc del de…

But like I mention in the discussion, those were then mitigated in the upstream Linux.

It was also discussed with SQM & cake authors in https://github.com/tohojo/sqm-scripts/issues/48

But at that time, cake performed better than simple/fq_codel.

My input to the discussion is that

• it is quite possible that there is some regression.
• it might be also in the upstream Linux kernel.
• it might also affect just certain devices like single/dual-core CPUs ARM/MIPS/X86, so debugging can be difficult
• it might not be the "main" qdisc like cake or fw_codel, but the helpers like HTB (or TBF)

Hopefully those earlier discussions from 2016-2017 help in debugging.

[atownlede]

Hmm, interesting. Not sure what to make of all of that but it did prompt me to play around with CPU scaling.
If I put the CPUs into performance mode (echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor, same for cpu1), fq_codel/simple can keep up with line speed. If the CPUs are in ondemand mode, it cannot.
Interestingly, during a DSLreports download test, in ondemand mode, the CPUs don't both top out at top speed, the highest they get is:

root@LEDE:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq
1000000
1725000

so it seems the issue isn't raw CPU power, it's something else (perhaps synchronization latency, which I took as a hint from reading the referred threads).

[fantom-x]

Take a look at this: R7800 performance

[atownlede]

Thanks, some good hints in that thread. Setting the ondemand control values helped a lot on its own. Adding in software flow offloading and I'm up to line speeds for download testing and I get lower CPU frequency when idle, for fq_codel/simple. cake/piece_of_cake still aren't up to line speeds.

root@LEDE:~# cat /etc/rc.local 
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
echo 35 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
exit 0
root@LEDE:~# cat /etc/config/sqm
config queue 'eth1'
	option qdisc_advanced '0'
	option verbosity '5'
	option upload '15000'
	option linklayer 'ethernet'
	option overhead '18'
	option debug_logging '1'
	option interface 'eth0'
	option enabled '1'
	option qdisc 'fq_codel'
	option script 'simple.qos'
	option download '250000'
root@LEDE:~# grep offload /etc/config/firewall
	option flow_offloading '1'
root@LEDE:~# 

[enri]

@atownlede

echo 35 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
..
option flow_offloading '1'

Thanks all, this settings work for me