SQM and streaming

Installing and Using OpenWrt
1

I have a problem with SQM.
My brother would like to stream to YT his gaming experience (he's not very good but still...) and he can't. The FPS with relatively low bitrate is always red dropping frames like crazy. I have BitTorrent client running on another machine. It wasn't an issue before on different router.

With WR1043NDv4 the problem didn't exist (I don't have it anymore). With C2600 it's significant. The only difference between configuration on both those machines are:

  1. Fast Path installed. Without it there is no real difference.
  2. The Internet speed is higher. Before there was 120/10 Mb/s, now 250/20 Mb/s.
  3. The LEDE 17.01-SNAPSHOT is newer then on WR1043NDv4.
  4. I added mpu 64 to the SQM config.
  5. I changed the link overhead to 18 as per updated guidance.

Here's my SQM config:

config queue 'eth0'
	option enabled '1'
	option interface 'eth0'
	option download '237500'
	option upload '19000'
	option debug_logging '0'
	option verbosity '5'
	option qdisc 'cake'
	option script 'layer_cake.qos'
	option qdisc_advanced '1'
	option squash_dscp '1'
	option squash_ingress '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option qdisc_really_really_advanced '1'
	option iqdisc_opts 'nat dual-dsthost diffserv4 rtt 300ms mpu 64'
	option eqdisc_opts 'nat dual-srchost diffserv4 rtt 300ms mpu 64'
	option linklayer 'ethernet'
	option overhead '18'
	option linklayer_advanced '1'
	option tcMTU '2047'
	option tcTSIZE '128'
	option tcMPU '64'
	option linklayer_adaptation_mechanism 'cake'
2

Mmh, fast path is incompatible with ingress sqm, so if fast path really is actie sqm will not. One of the fast-path community builds will disable fast path if sqm is in use. Could you post the output of "tc -s qdisc" before and after running a dslreoprts speedtest, please. Also please report the link to the detailed results page for that speedtest. The idea is to check whether cake's byte and packet counters show approximately the right increases for the speedtest (the results page contains information about the total transmitted data per direction).

Best Regards

3

Are you sure about the ingress part?
I thought it was fixed in dissent1 version of the patch.

Also when you say ingress, you mean download, right?
By streaming I mean upload. Sending data from my computer to YouTube or Twitch servers.

https://github.com/lede-project/source/pull/1269#issuecomment-321060451

4

here is from dissents patch (https://github.com/dissent1/r7800/commit/bc187a8dda624579a6fd16197100fc75ea1bc552):
+#ifdef CONFIG_NET_CLS_ACT

  • /*
    • If ingress Qdisc configured, and packet not processed by ingress Qdisc yet
    • We cannot accelerate this packet.
  • */
  • if (dev->ingress_queue && !(skb->tc_verd & TC_NCLS)) {
  •  goto rx_exit;
  • }
    +#endif

so if you use that fast path simply will be disabled. So do you have ffast-path enabled or not?

Best Regards

5

I only installed this: opkg install kmod-fast-classifier kmod-shortcut-fe.
The logread said at startup that it's enabled.

6

Ah, I guess there must be a way to query how any flows and/or packets have been "fast-pathed" that would be interesting to look at in combination with the tc -s qdisc counters before and after a speedtest...

7

I will do those test you asked about tomorrow.

8

Before:

root@C2600v1_LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 800a: dev eth0 root refcnt 2 bandwidth 19Mbit diffserv4 dual-srchost nat rtt 300.0ms noatm overhead 18 via-ethernet mpu 64
 Sent 809614439455 bytes 599197325 pkt (dropped 3273934, overlimits 1305927563 requeues 209)
 backlog 0b 0p requeues 209
 memory used: 2773797b of 4Mb
 capacity estimate: 19Mbit
                 Bulk   Best Effort      Video       Voice
  thresh      1187Kbit      19Mbit    9500Kbit    4750Kbit
  target        15.3ms      15.0ms      15.0ms      15.0ms
  interval     300.3ms     300.0ms     300.0ms     300.0ms
  pk_delay         0us      10.8ms       1.6ms       391us
  av_delay         0us       2.9ms       498us        36us
  sp_delay         0us         9us        16us         6us
  pkts               0   602355318       76776       39181
  bytes              0814367655375    97456448     4310912
  way_inds           0    18231781           0           2
  way_miss           0    12874502         437        2550
  way_cols           0           3           0           0
  drops              0     3273932           0           2
  marks              0           0           0           0
  sp_flows           0           1           0           0
  bk_flows           0           0           0           0
  un_flows           0           0           0           0
  max_len            0       51476        1294         590

qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
 Sent 69513007617 bytes 430280650 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 47817158910 bytes 408650053 pkt (dropped 2, overlimits 0 requeues 1896)
 backlog 0b 0p requeues 1896
  maxpacket 1514 drop_overlimit 0 new_flow_count 20078705 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev wlan1 root
 Sent 13960724893 bytes 10687067 pkt (dropped 246, overlimits 0 requeues 136)
 backlog 0b 0p requeues 136
qdisc fq_codel 0: dev wlan1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 1123186 bytes 5947 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan1 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan1 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 13959601707 bytes 10681120 pkt (dropped 246, overlimits 0 requeues 136)
 backlog 0b 0p requeues 136
  maxpacket 1514 drop_overlimit 0 new_flow_count 62195 ecn_mark 0
  new_flows_len 1 old_flows_len 2
qdisc fq_codel 0: dev wlan1 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev wlan0 root
 Sent 27091119598 bytes 20941085 pkt (dropped 0, overlimits 0 requeues 6)
 backlog 0b 0p requeues 6
qdisc fq_codel 0: dev wlan0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 38991 bytes 242 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 27091080607 bytes 20940843 pkt (dropped 0, overlimits 0 requeues 6)
 backlog 0b 0p requeues 6
  maxpacket 1514 drop_overlimit 0 new_flow_count 16756 ecn_mark 0
  new_flows_len 1 old_flows_len 1
qdisc fq_codel 0: dev wlan0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 800b: dev ifb4eth0 root refcnt 2 bandwidth 237500Kbit diffserv4 dual-dsthost nat wash rtt 300.0ms noatm overhead 18 via-ethernet mpu 64
 Sent 76131243971 bytes 430280017 pkt (dropped 633, overlimits 9784003 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 4617920b of 15140Kb
 capacity estimate: 237500Kbit
                 Bulk   Best Effort      Video       Voice
  thresh     14843Kbit  237500Kbit  118750Kbit   59375Kbit
  target        15.0ms      15.0ms      15.0ms      15.0ms
  interval     300.0ms     300.0ms     300.0ms     300.0ms
  pk_delay       921us       190us       2.5ms        12us
  av_delay        29us        58us        71us         7us
  sp_delay        17us         9us        18us         2us
  pkts            5252   421662502     1921235     6691661
  bytes         466107 75073609726   656278435   401669417
  way_inds           0    36060884       18285           0
  way_miss         165    16392329      104283         483
  way_cols           0           0           0           0
  drops              0         621          10           2
  marks              0           2           0           0
  sp_flows           0           0           0           1
  bk_flows           0           0           0           0
  un_flows           0           0           0           0
  max_len         1514       63588       56018         679

After:
root@C2600v1_LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 800a: dev eth0 root refcnt 2 bandwidth 19Mbit diffserv4 dual-srchost nat rtt 300.0ms noatm overhead 18 via-ethernet mpu 64
Sent 809705801995 bytes 599554680 pkt (dropped 3276003, overlimits 1306218367 requeues 209)
backlog 0b 0p requeues 209
memory used: 2773797b of 4Mb
capacity estimate: 19Mbit
Bulk Best Effort Video Voice
thresh 1187Kbit 19Mbit 9500Kbit 4750Kbit
target 15.3ms 15.0ms 15.0ms 15.0ms
interval 300.3ms 300.0ms 300.0ms 300.0ms
pk_delay 0us 1.3ms 1.4ms 4.1ms
av_delay 0us 787us 456us 128us
sp_delay 0us 5us 12us 6us
pkts 0 602713109 76815 40775
bytes 0814461827747 97506914 4499678
way_inds 0 18246049 0 13
way_miss 0 12874744 437 3273
way_cols 0 3 0 0
drops 0 3276001 0 2
marks 0 0 0 0
sp_flows 0 0 0 0
bk_flows 0 1 0 0
un_flows 0 0 0 0
max_len 0 51476 1294 590

qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
 Sent 70348003636 bytes 430901791 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 48675758810 bytes 409265357 pkt (dropped 2, overlimits 0 requeues 1961)
 backlog 0b 0p requeues 1961
  maxpacket 1514 drop_overlimit 0 new_flow_count 20170324 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev wlan1 root
 Sent 13960751674 bytes 10687378 pkt (dropped 246, overlimits 0 requeues 136)
 backlog 0b 0p requeues 136
qdisc fq_codel 0: dev wlan1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 1124160 bytes 5951 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan1 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan1 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 13959627514 bytes 10681427 pkt (dropped 246, overlimits 0 requeues 136)
 backlog 0b 0p requeues 136
  maxpacket 1514 drop_overlimit 0 new_flow_count 62195 ecn_mark 0
  new_flows_len 1 old_flows_len 2
qdisc fq_codel 0: dev wlan1 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev wlan0 root
 Sent 27091139835 bytes 20941342 pkt (dropped 0, overlimits 0 requeues 6)
 backlog 0b 0p requeues 6
qdisc fq_codel 0: dev wlan0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 38991 bytes 242 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 27091100844 bytes 20941100 pkt (dropped 0, overlimits 0 requeues 6)
 backlog 0b 0p requeues 6
  maxpacket 1514 drop_overlimit 0 new_flow_count 16756 ecn_mark 0
  new_flows_len 1 old_flows_len 1
qdisc fq_codel 0: dev wlan0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 800b: dev ifb4eth0 root refcnt 2 bandwidth 237500Kbit diffserv4 dual-dsthost nat wash rtt 300.0ms noatm overhead 18 via-ethernet mpu 64
 Sent 76990311050 bytes 430900959 pkt (dropped 832, overlimits 9917564 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 4617920b of 15140Kb
 capacity estimate: 237500Kbit
                 Bulk   Best Effort      Video       Voice
  thresh     14843Kbit  237500Kbit  118750Kbit   59375Kbit
  target        15.0ms      15.0ms      15.0ms      15.0ms
  interval     300.0ms     300.0ms     300.0ms     300.0ms
  pk_delay       921us       149us       2.5ms        29us
  av_delay        29us        33us        70us         7us
  sp_delay        17us         8us        13us         4us
  pkts            5252   422280955     1921238     6694346
  bytes         466107 75932813391   656278863   401830517
  way_inds           0    36064118       18285           0
  way_miss         165    16394848      104285         483
  way_cols           0           0           0           0
  drops              0         820          10           2
  marks              0           2           0           0
  sp_flows           0           0           0           1
  bk_flows           0           0           0           0
  un_flows           0           0           0           0
  max_len         1514       51590       56018         679

For this test I disconnected the BitTorrent server from the network.
Speedtest:

For this test I reinstalled LEDE without SFE (Shortcut Forward Engine). I think it will be easier to diagnose the problem.

For this test I connected the BitTorrent server. The Internet speed has lowered without SFE.

Another one:

One thing is sure. Without SFE the download is hurting.

9

The last two are exactly what you'd expect since torrents are taking bandwidth away from the speedtest.

The ping time has not increased with torrents running, which is the main purpose of SQM.

Set a bandwidth limit on the torrent server itself if you want to have some reserved for users to send video.

10

Before cake bytes (after - before):
Egress: 814461827747 - 814367655375 = 94172372 / 1000^2 = 94.172372 Mbps
Ingress: 75932813391- 75073609726 = 859203665 / 1000^2 = 859.203665 Mbps

Speedtest 1 (SFE, no BitTorrent, SQM?): Total megabytes consumed: 844.5 (down:778.6 up:65.9)
Speedtest 2 (no SFE, no BitTorrent, SQM?): Total megabytes consumed: 815.3 (down:750.5 up:64.8)
Speedtest3 (no SFE, BitTorrent, SQM?): Total megabytes consumed: 564.6 (down:531 up:33.6)
Speedtest4 (no SFE, BitTorrent, SQM?): Total megabytes consumed: 515.5 (down:482.7 up:32.8)

Could you clarify which tests were performed with cake active, please

Now, if the before and after were from around only speedtest1, I would say the cake byte counter increases look reasonable, but if the before after would cover all 4 speedtests, I would say we have a problem...
Now it would e great to also see the SFE counters from before and after. AND report the output of ifconfig from just before and after the speedtest (these also contain byte counters; the goal here is to figure out which counters are well correlated).
Finally the simplest test for ingress SQM would be to set the ingress/download shaper to, say 80 Mbps (to pick a number noticeably lower than what you currently got with your slowest speedtest, but still high enough to not run into lack of bandwidth issues).

But in the end I agree with @mk24, concurrent bit torrent tradffic will reduce the goodput measurable in a speedtest. Assuming the bit torrent coputer is different from the machine you use for speedtesting, you could instal iftop on your router (opkg update ; opkg install iftop) and run "iftop -i eth0" during a combined speedtest bit-torrent experiment to monitor how many/which flows actually are concurrently active and how much total bandwidth they consume. (One problem would be that measuring this on the router will cost CPU cycles and if you are already CPU limited this will result in slightly higher latency under load numbers).

Best Regards

11

@mk24 - You are right and wrong at the same time. You are right about limiting the BitTorrent server. It would solve the issue. However I don't want to do that because it sets a fixed limit. I want BitTorrent to use as much of my Internet as possible but not at the costs of any other user of my network. The fq_codel wasn't up to the task but cake with layer_cake configured as shown can manage Internet connection as needed.

Example:
2 a.m. - BitTorrent uses whole network
12 a.m. - Users are actively using the network, SQM is limiting BitTorrent.

@moeller0 - Yes, in all of those tests SQM was active, configured as said above.
I've actually fixed it. It turns out SFE was causing all of those problems. I got rid of it and my brother tested streaming. He said everything works great. It seems SFE doesn't play well with SQM no matter what the devs are saying. It seems to be working to some extend but many of SQM (cake) features are handicapped by SFE. The NAT and dual-dsthost/dual-srchost doesn't seem to work at all with SFE.

The above also proves that I don't need to set a fixed limit on BitTorrent. SQM is more then capable of handling it when properly configured. Maybe users that are relying on SQM and are wondering about SFE can take a note from this thread.

12

Well these are interrelated, if the NAT lookup is not successful all the dual-XXXhost will see is the router's external address for both srchost and dsthost, in that case the dual isolations degrade back to basically simple per-flow fairness. But good to know that SFE seems still not 100% compatible with sqm/cake.

13

Maybe someone can use my case to further improve SFE. I still believe it's a great feature, it just needs a bit more work.