LEDE Luci QOS and SQM at same time

Installing and Using OpenWrt
21

ISP: Comcast/Xfinity
Modem: SURFBoard 5B6183
Router: Netgear R8000

root@LEDE:~# cat /etc/config/sqm
config queue 'eth1'
option qdisc_advanced '0'
option linklayer 'none'
option upload '5000'
option debug_logging '0'
option verbosity '5'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option download '130000'
option enabled '1'
option interface 'eth0.2'

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 ta rget 5.0ms interval 100.0ms ecn
qdisc cake 801c: dev eth1 root refcnt 2 bandwidth 5Mbit besteffort triple-isolat e rtt 100.0ms raw total_overhead 14 hard_header_len 14
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
qdisc fq_codel 0: dev wlan0 root refcnt 2 limit 10240p flows 1024 quantum 1514 t arget 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev wlan1 root refcnt 2 limit 10240p flows 1024 quantum 1514 t arget 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev wlan2 root refcnt 2 limit 10240p flows 1024 quantum 1514 t arget 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc cake 801f: dev eth0.2 root refcnt 2 bandwidth 5Mbit besteffort triple-isol ate rtt 100.0ms raw total_overhead 18 hard_header_len 18
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
qdisc fq_codel 0: dev ifb0 root refcnt 2 limit 10240p flows 1024 quantum 1514 ta rget 5.0ms interval 100.0ms ecn
qdisc cake 801d: dev ifb4eth1 root refcnt 2 bandwidth 130Mbit besteffort triple- isolate wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
qdisc cake 8020: dev ifb4eth0.2 root refcnt 2 bandwidth 130Mbit besteffort tripl e-isolate wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 ta rget 5.0ms interval 100.0ms ecn
Sent 2379508346 bytes 5683296 pkt (dropped 1316, overlimits 0 requeues 1852)
backlog 0b 0p requeues 1852
maxpacket 25738 drop_overlimit 0 new_flow_count 232623 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 801c: dev eth1 root refcnt 2 bandwidth 5Mbit besteffort triple-isolat e rtt 100.0ms raw total_overhead 14 hard_header_len 14
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 0b of 4Mb
capacity estimate: 5Mbit
Tin 0
thresh 5Mbit
target 5.0ms
interval 100.0ms
pk_delay 0us
av_delay 0us
sp_delay 0us
pkts 0
bytes 0
way_inds 0
way_miss 0
way_cols 0
drops 0
marks 0
sp_flows 0
bk_flows 0
un_flows 0
max_len 0

qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev wlan0 root refcnt 2 limit 10240p flows 1024 quantum 1514 t arget 5.0ms interval 100.0ms ecn
Sent 258123260 bytes 525661 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 1514 drop_overlimit 0 new_flow_count 362 ecn_mark 0
new_flows_len 1 old_flows_len 7
qdisc fq_codel 0: dev wlan1 root refcnt 2 limit 10240p flows 1024 quantum 1514 t arget 5.0ms interval 100.0ms ecn
Sent 10037708465 bytes 7467362 pkt (dropped 31, overlimits 0 requeues 11)
backlog 0b 0p requeues 11
maxpacket 3028 drop_overlimit 0 new_flow_count 12554 ecn_mark 0
new_flows_len 0 old_flows_len 3
qdisc fq_codel 0: dev wlan2 root refcnt 2 limit 10240p flows 1024 quantum 1514 t arget 5.0ms interval 100.0ms ecn
Sent 1366703393 bytes 1301596 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 3028 drop_overlimit 0 new_flow_count 3832 ecn_mark 0
new_flows_len 1 old_flows_len 16
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 801f: dev eth0.2 root refcnt 2 bandwidth 5Mbit besteffort triple-isol ate rtt 100.0ms raw total_overhead 18 hard_header_len 18
Sent 41150595 bytes 435526 pkt (dropped 2689, overlimits 231307 requeues 0)
backlog 0b 0p requeues 0
memory used: 706Kb of 4Mb
capacity estimate: 5Mbit
Tin 0
thresh 5Mbit
target 5.0ms
interval 100.0ms
pk_delay 2.3ms
av_delay 374us
sp_delay 4us
pkts 438231
bytes 44868928
way_inds 18451
way_miss 915
way_cols 0
drops 2689
marks 0
sp_flows 2
bk_flows 1
un_flows 0
max_len 10976

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 735381311 bytes 506341 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev ifb0 root refcnt 2 limit 10240p flows 1024 quantum 1514 ta rget 5.0ms interval 100.0ms ecn
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 801d: dev ifb4eth1 root refcnt 2 bandwidth 130Mbit besteffort triple- isolate wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 0b of 6500000b
capacity estimate: 130Mbit
Tin 0
thresh 130Mbit
target 5.0ms
interval 100.0ms
pk_delay 0us
av_delay 0us
sp_delay 0us
pkts 0
bytes 0
way_inds 0
way_miss 0
way_cols 0
drops 0
marks 0
sp_flows 0
bk_flows 0
un_flows 0
max_len 0

qdisc cake 8020: dev ifb4eth0.2 root refcnt 2 bandwidth 130Mbit besteffort tripl e-isolate wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
Sent 743784433 bytes 505884 pkt (dropped 457, overlimits 142601 requeues 0)
backlog 0b 0p requeues 0
memory used: 2428056b of 6500000b
capacity estimate: 130Mbit
Tin 0
thresh 130Mbit
target 5.0ms
interval 100.0ms
pk_delay 492us
av_delay 49us
sp_delay 4us
pkts 506341
bytes 744476301
way_inds 8163
way_miss 870
way_cols 0
drops 457
marks 0
sp_flows 1
bk_flows 1
un_flows 0
max_len 19682

22

Can anyone help with this issue?

23

This looks odd, why do you have shapers on both eth0.2 and eth1, but only have a sqm config for eth0.2?

Also I would recommend to set overhead to 18, and follow the instructions on the sqm howto on making sqm sing and dance (dual-scrhost/dual-dsthost and nat). And then try isolating the problem again...

24

Not sure, I am, that I really understand your issue to begin with (and that makes helping somewhat hard).

25

Unfortunately the WAN interface is 0.2 eth and that is what I set for sqm.

26

Well, in that case I would start with just a shaper instance on eth0.2 and none on eth1. I wonder why your tc output shows a running cake instance on eth1 as well. Could you post the output of "ls -all /var/run/sqm"

27

am not sure why it would be running on eth1 at the same time when WAN is only selected in SQM which is 0.2 eth, am not sure if this is what causing the issue with bufferbloat, currently am back and forth with all kind of firmware's, it seems games were less laggy even with the stock firmware but bufferbloat grades were still C on WIFI unless i use qos to reduce the download speed with MAC priority which i was able to do with the DDWRT firmware, i will install the LEDE firmware again and give it another go.

28

So the first step, IMHO, should be to see whether you get decent performance with gaming and concurrent use over ethernet first. Wifi has its own set of issues that are probably better to deal with after getting sqm configured properly.

29

as mentioned in my very first few posts, the exist in both cases, wifi or not, same issue.

30

Sure, I do not doubt that. To be able to actually correlate performance/behavior of sqm with your load we need to be able to "poke" at the instantiated sqm, so I would heavily recommend to start first with getting the wired configuration up and running, and then take it from there.

Question: How many different devices do you have concurrently active in your network and how do these typically connect to your router?

31 
root@LEDE:~# ls -all /var/run/sqm
drwxr-xr-x    3 root     root            80 Mar 29 23:14 .
drwxr-xr-x    6 root     root           420 Mar 29 23:14 ..
drwxr-xr-x    2 root     root            80 Mar 29 22:54 available_qdiscs
-rw-r--r--    1 root     root           501 Mar 29 23:14 eth0.2.state

root@LEDE:~# cat /etc/config/sqm

config queue 'eth1'
        option enabled '1'
        option interface 'eth0.2'
        option download '135000'
        option debug_logging '0'
        option verbosity '5'
        option qdisc 'cake'
        option script 'piece_of_cake.qos'
        option linklayer 'ethernet'
        option overhead '18'
        option qdisc_advanced '1'
        option squash_dscp '1'
        option squash_ingress '1'
        option ingress_ecn 'ECN'
        option egress_ecn 'NOECN'
        option qdisc_really_really_advanced '1'
        option iqdisc_opts 'nat dual-dsthost'
        option eqdisc_opts 'nat dual-srchost'
        option upload '5000'

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 ta                                                                                                                                                             rget 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev wlan0 root refcnt 2 limit 10240p flows 1024 quantum 1514 t                                                                                                                                                             arget 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev wlan1 root refcnt 2 limit 10240p flows 1024 quantum 1514 t                                                                                                                                                             arget 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev wlan2 root refcnt 2 limit 10240p flows 1024 quantum 1514 t                                                                                                                                                             arget 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc cake 8010: dev eth0.2 root refcnt 2 bandwidth 5Mbit besteffort dual-srchos                                                                                                                                                             t nat rtt 100.0ms raw total_overhead 18 hard_header_len 18
 linklayer ethernet overhead 18
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
qdisc cake 8011: dev ifb4eth0.2 root refcnt 2 bandwidth 135Mbit besteffort dual-                                                                                                                                                             dsthost nat wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
 linklayer ethernet overhead 18
root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 ta                                                                                                                                                             rget 5.0ms interval 100.0ms ecn
 Sent 1258862847 bytes 1425729 pkt (dropped 0, overlimits 0 requeues 124)
 backlog 0b 0p requeues 124
  maxpacket 27252 drop_overlimit 0 new_flow_count 100264 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan0 root refcnt 2 limit 10240p flows 1024 quantum 1514 t                                                                                                                                                             arget 5.0ms interval 100.0ms ecn
 Sent 670520316 bytes 450306 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 740 ecn_mark 0
  new_flows_len 0 old_flows_len 1
qdisc fq_codel 0: dev wlan1 root refcnt 2 limit 10240p flows 1024 quantum 1514 t                                                                                                                                                             arget 5.0ms interval 100.0ms ecn
 Sent 76549983 bytes 62063 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 216 ecn_mark 0
  new_flows_len 0 old_flows_len 1
qdisc fq_codel 0: dev wlan2 root refcnt 2 limit 10240p flows 1024 quantum 1514 t                                                                                                                                                             arget 5.0ms interval 100.0ms ecn
 Sent 225737 bytes 1597 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8010: dev eth0.2 root refcnt 2 bandwidth 5Mbit besteffort dual-srchos                                                                                                                                                             t nat rtt 100.0ms raw total_overhead 18 hard_header_len 18
 Sent 25921037 bytes 159519 pkt (dropped 11748, overlimits 126233 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2932Kb of 4Mb
 capacity estimate: 5Mbit
                  Tin 0
  thresh          5Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        6.3ms
  av_delay        786us
  sp_delay         10us
  pkts           171267
  bytes        28377370
  way_inds            0
  way_miss         1194
  way_cols            0
  drops           11748
  marks               0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         14672

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
 Sent 853681527 bytes 592830 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8011: dev ifb4eth0.2 root refcnt 2 bandwidth 135Mbit besteffort dual-                                                                                                                                                             dsthost nat wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
 Sent 875116329 bytes 592454 pkt (dropped 375, overlimits 301094 requeues 0)
 backlog 1532b 1p requeues 0
 memory used: 1273484b of 6750000b
 capacity estimate: 135Mbit
                  Tin 0
  thresh        135Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        3.8ms
  av_delay        1.6ms
  sp_delay         17us
  pkts           592830
  bytes       875677771
  way_inds            0
  way_miss         1216
  way_cols            0
  drops             375
  marks               0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         47868

dslreport%20errorinterface

32

sqm%20setting

i also forgot to mention, that speed test is without sqm and it ends with that error with or without sqm, this issue only occur with LEDE firmware

33

Thanks a lot. It seems you reconfigured sqm and/or rebooted your router since the last "tc -s qdisc" output. That means that the "ls -all /var/run/sqm" does not help understand why you had shapers on eth1 and eth0.2 (but since the eth1 shaper is gone now, let's just not care).

This indicates that during the copy and paste some of the output was lost (I am missing at least "t nat"). Could you try to confirm this? Otherwise, if this is the real output from tc we have much bigger problems than sqm not behaving as expected.

From the luci pages it is clear that eth0.2 seems to be the WAN interface.

The dslreports error is odd, I certainly am able to get successful speedtests from LEDE (or rather openwrt after the re-merge the name of master reverted back to openwrt). But could you also try to post the result from a speedtest with sqm active?

34

qdisc

35

I think that error is dslreports related. I get is on multiple different connections, firmwares, etc. But it is random if I get it. There are periods where the uploading of the grades is fine.

36

Thanks, unfortunately, the speedtest.net speedtest does not do any latency under load measurements so is not really that useful. You might have better luck with flent's rrul test (https://flent.org)

37

Ah, now I seem to remember that I had this transiently a few times, but I could not find a system behind the problems, so am not convinced that is is related to lede/openwrt per se. Do you know whether the CLI version (https://www.dslreports.com/forum/speedtestbinary) of the test always works okay?

38

You might take a look at @richb-hanover's betterspeedtest...

39

Sure, but that exercises netperf just as flent's rrul test, but it does not offer as much bells and whistles...

40

What it offers is a way for non-Linux users to run performance tests.