SQM QoS changes from 19.07.0-rc2 to 19.07.0 must worse results

Just updated to 19.07.0 from rc2 and noticed that my bufferbloat results using the same settings as I did under rc2 are MUCH worse.

R7800

fq_codel
simplest.qos
download = 215000
upload = 12000
per packet overhead = 22

If I run the test while watching htop on the R7800, the most CPU% I see is 33% on one core. Under RC2 it was in the ~90%. Although the overall speed acheived is similar (rc2 vs final), the bufferbloat rating was B/overall was A under rc2. Now overall is C and bufferbloat is F.

I'm wondering if the SQM code or some internal changed from rc2 --> final that might account for this?

The "Idle" bufferbloat is too high. That's weird. Maybe your internet was not at it's best at that time?

It was fine before I updated (ran it as a control). I don't want to downgrade to rc2.

It would be interesting to see and compare the output of tc -s qdisc from before and after running a Speedtest preferably both for 19.07.0 and the rcN that had better results. Thanks.

Interestingly, I repeated the test this morning and observed CPU usage in the high 90% which is consistent with rc2. As well I measured values from the bufferbloat test consistent with rc2. I cannot explain.

So just to make it explicit, the observation is now that under 19.07.0 things either behave like under rc2 or occasionally much worse? That probably still is a real issue (albeit probably not a sqm-scripts issue, in the end sqm-scripts just configures the qdiscs that are part of the kernel, could still be bugs somewhere in the scripts part, but if the kernel components misbehave, that is a kernel bug :wink: )

Just let me know if I can do anything for you in this matter.

Thanks for the advice. I will repeat the test a few times over this week and watch for issues. What I can't understand is why the CPU% was so low when I got the really poor results. I literally did nothing to the OW device.

This is why the output of tc -s qdisc would be interesting, as we could see whether both sqm shaper instances where actually set-up correctly. If that would be the issue your speedtest above should be similar to a speedtest with sqm disabled, which might or might not be true...

Gotcha. Here is the output after normal web usage:

# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 1554777506 bytes 5534948 pkt (dropped 0, overlimits 0 requeues 91793) 
 backlog 0b 0p requeues 91793
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 1554777506 bytes 5534948 pkt (dropped 0, overlimits 0 requeues 91793) 
 backlog 0b 0p requeues 91793
  maxpacket 1514 drop_overlimit 0 new_flow_count 63520 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 16129484071 bytes 11793994 pkt (dropped 0, overlimits 0 requeues 239317) 
 backlog 0b 0p requeues 239317
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 16129484071 bytes 11793994 pkt (dropped 0, overlimits 0 requeues 239317) 
 backlog 0b 0p requeues 239317
  maxpacket 1514 drop_overlimit 0 new_flow_count 243170 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.3 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc tbf 1: dev eth0.2 root refcnt 2 rate 12Mbit burst 1747b lat 300.0ms 
 Sent 42885375 bytes 124308 pkt (dropped 571, overlimits 34398 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev eth0.2 parent 1: limit 1001p flows 1024 quantum 1518 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 42885375 bytes 124308 pkt (dropped 571, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 1748 drop_overlimit 0 new_flow_count 66083 ecn_mark 0
  new_flows_len 1 old_flows_len 2
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 388740395 bytes 293740 pkt (dropped 1, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc tbf 1: dev ifb4eth0.2 root refcnt 2 rate 215Mbit burst 26848b lat 300.0ms 
 Sent 388106046 bytes 290400 pkt (dropped 1561, overlimits 98233 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1: limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 388106046 bytes 290400 pkt (dropped 1561, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 11692 drop_overlimit 650 new_flow_count 24618 ecn_mark 0 drop_overmemory 64
  new_flows_len 1 old_flows_len 5

And here is after the test which scored pretty well.

# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 1591699824 bytes 5630535 pkt (dropped 0, overlimits 0 requeues 93818) 
 backlog 0b 0p requeues 93818
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 1591699824 bytes 5630535 pkt (dropped 0, overlimits 0 requeues 93818) 
 backlog 0b 0p requeues 93818
  maxpacket 1514 drop_overlimit 0 new_flow_count 65152 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 16429442438 bytes 12009716 pkt (dropped 0, overlimits 0 requeues 244873) 
 backlog 0b 0p requeues 244873
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 16429442438 bytes 12009716 pkt (dropped 0, overlimits 0 requeues 244873) 
 backlog 0b 0p requeues 244873
  maxpacket 1514 drop_overlimit 0 new_flow_count 251858 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.3 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc tbf 1: dev eth0.2 root refcnt 2 rate 12Mbit burst 1747b lat 300.0ms 
 Sent 81666825 bytes 219895 pkt (dropped 1520, overlimits 70259 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev eth0.2 parent 1: limit 1001p flows 1024 quantum 1518 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 81666825 bytes 219895 pkt (dropped 1520, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 1748 drop_overlimit 0 new_flow_count 132216 ecn_mark 0
  new_flows_len 1 old_flows_len 1
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 683823196 bytes 511351 pkt (dropped 1, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc tbf 1: dev ifb4eth0.2 root refcnt 2 rate 215Mbit burst 26848b lat 300.0ms 
 Sent 682739439 bytes 505182 pkt (dropped 2796, overlimits 189257 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1: limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 682739439 bytes 505182 pkt (dropped 2796, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 11692 drop_overlimit 1229 new_flow_count 47829 ecn_mark 0 drop_overmemory 192
  new_flows_len 0 old_flows_len 1
root@serenity:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 1592445716 bytes 5631074 pkt (dropped 0, overlimits 0 requeues 93819) 
 backlog 0b 0p requeues 93819
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 1592445716 bytes 5631074 pkt (dropped 0, overlimits 0 requeues 93819) 
 backlog 0b 0p requeues 93819
  maxpacket 1514 drop_overlimit 0 new_flow_count 65152 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 16429481951 bytes 12010052 pkt (dropped 0, overlimits 0 requeues 244873) 
 backlog 0b 0p requeues 244873
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 16429481951 bytes 12010052 pkt (dropped 0, overlimits 0 requeues 244873) 
 backlog 0b 0p requeues 244873
  maxpacket 1514 drop_overlimit 0 new_flow_count 251858 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.3 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc tbf 1: dev eth0.2 root refcnt 2 rate 12Mbit burst 1747b lat 300.0ms 
 Sent 82414147 bytes 220434 pkt (dropped 1520, overlimits 70699 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev eth0.2 parent 1: limit 1001p flows 1024 quantum 1518 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 82414147 bytes 220434 pkt (dropped 1520, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 1748 drop_overlimit 0 new_flow_count 132284 ecn_mark 0
  new_flows_len 0 old_flows_len 5
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 683846965 bytes 511644 pkt (dropped 1, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc tbf 1: dev ifb4eth0.2 root refcnt 2 rate 215Mbit burst 26848b lat 300.0ms 
 Sent 682773684 bytes 505475 pkt (dropped 2796, overlimits 189257 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1: limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 682773684 bytes 505475 pkt (dropped 2796, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 11692 drop_overlimit 1229 new_flow_count 47879 ecn_mark 0 drop_overmemory 192
  new_flows_len 1 old_flows_len 5

Based on my SQM experience, you did not have the QoS engaged at that time, which is why the low CPU usage and poor test results. The output of tc -s qdisc would be definitive here, but my hunch is sqm was not engaged.

So your quest is to figure out why it did not start the sqm qdsc when you thought it should have.
Maybe the command line for the init.d/sqm restart was not quite right?

1 Like

That hypothesis is also easy to test, just disable sqm and run a Speedtest and post the results here, maybe have a look at https://forum.openwrt.org/t/sqm-qos-recommended-settings-for-the-dslreports-speedtest-bufferbloat-testing/2803 for how to get the most out of a dslteports Speedtest for bufferbloat assessment.