QoS, SQM & latency in online games

Hi, after reading through a bunch of related topics on this amazing forum i couldn't find any solution to my problem so here it goes.

My problem is: 3 people using the network and i have terrible ping spikes, packet drops, stuttering/lagging in all online games when anyone opens a youtube video or any streaming service. The lag happens immediately when the video loads and then after 10-15s it gets back to normal.

What i have: UBNT-ERX router with OpenWrt 19.07.2, ADSL2 through copper telephone line at around 10Mb/s down / 0.5Mb/s up speed.

Things that i tried so far:

  • Setting up sqm according to SQM Details with layer_cake / piece_of_cake, with and without the per-host isolation commands. I tried these in every possible combination. Tried to actually measure the ATM overhead (result was 40) and using that or the default 44bytes. Didn't make any difference.
  • Limiting every other device (except gaming one) on the network with Qos over nftables and static limits with and without sqm. I tested this with only 2 devices on the network (1 gaming pc, 1 laptop) and limiting the laptop down to 3Mbit or not limiting it at all didn't make any difference to latency, lagging.
  • Setting up dscp marks with layer cake with diffserv4 which actually marked my gaming packets properly with EF PHB and youtube frames with AF41 (using wireshark) but it didn't make any difference in terms of latency and lagging in game.

So right now i'm just out of ideas and i have a very limited knowledge about networks at this level. But one thing was obvious: bandwidth limit doesn't make any difference so somehow i'd need to prioritize all traffic not by ports and services but by device. So my overall question is how can i prioritize 1 device by ip or mac address against all of the others? Can someone point me to a general right direction? Or is my internet so bad that it won't even matter?:smiley: Here are some outputs and numbers:

SQM OFF:

SQM ON with layer_cake

SQM ON with piece_of_cake

etc/config/sqm:


config queue 'eth1'
	option interface 'pppoe-wan'
	option debug_logging '0'
	option verbosity '5'
	option qdisc_advanced '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option qdisc_really_really_advanced '1'
	option linklayer 'atm'
	option download '8500'
	option iqdisc_opts 'nat dual-dsthost ingress'
	option eqdisc_opts 'nat dual-srchost'
	option qdisc 'cake'
	option enabled '1'
	option squash_dscp '1'
	option overhead '44'
	option script 'piece_of_cake.qos'
	option squash_ingress '1'
	option upload '425'

'tc -s qdisc' before loading a youtube video and gaming:

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 148046589 bytes 197203 pkt (dropped 290, overlimits 0 requeues 24)
 backlog 0b 0p requeues 24
  maxpacket 3028 drop_overlimit 0 new_flow_count 304 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 801f: dev pppoe-wan root refcnt 2 bandwidth 425Kbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms atm overhead 44
 Sent 2789095 bytes 15940 pkt (dropped 1861, overlimits 17152 requeues 0)
 backlog 128b 2p requeues 0
 memory used: 177408b of 4Mb
 capacity estimate: 425Kbit
 min/max network layer size:           40 /    1492
 min/max overhead-adjusted size:      106 /    1696
 average network hdr offset:            0

                  Tin 0
  thresh        425Kbit
  target         42.7ms
  interval      137.7ms
  pk_delay       43.4ms
  av_delay        3.5ms
  sp_delay         79us
  backlog          128b
  pkts            17803
  bytes         3479868
  way_inds          436
  way_miss          426
  way_cols            0
  drops            1861
  marks               0
  ack_drop            0
  sp_flows            5
  bk_flows            1
  un_flows            0
  max_len          5840
  quantum           300

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 22046451 bytes 21227 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8020: dev ifb4pppoe-wan root refcnt 2 bandwidth 8500Kbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100.0ms atm overhead 44
 Sent 21697960 bytes 20987 pkt (dropped 240, overlimits 15615 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 104832b of 4Mb
 capacity estimate: 8500Kbit
 min/max network layer size:           38 /    1492
 min/max overhead-adjusted size:      106 /    1696
 average network hdr offset:            0

                  Tin 0
  thresh       8500Kbit
  target          5.0ms
  interval      100.0ms
  pk_delay        340us
  av_delay         20us
  sp_delay         12us
  backlog            0b
  pkts            21227
  bytes        22046451
  way_inds          511
  way_miss          425
  way_cols            0
  drops             240
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1492
  quantum           300


'tc -s qdisc' after loading a youtube video and gaming:

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 225384759 bytes 321768 pkt (dropped 290, overlimits 0 requeues 24)
 backlog 0b 0p requeues 24
  maxpacket 3028 drop_overlimit 0 new_flow_count 408 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 801f: dev pppoe-wan root refcnt 2 bandwidth 425Kbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms atm overhead 44
 Sent 6848901 bytes 58285 pkt (dropped 3551, overlimits 52494 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 463680b of 4Mb
 capacity estimate: 425Kbit
 min/max network layer size:           40 /    1492
 min/max overhead-adjusted size:      106 /    1696
 average network hdr offset:            0

                  Tin 0
  thresh        425Kbit
  target         42.7ms
  interval      137.7ms
  pk_delay       13.4ms
  av_delay        2.5ms
  sp_delay         24us
  backlog            0b
  pkts            61836
  bytes         7762147
  way_inds         3050
  way_miss          691
  way_cols            0
  drops            3551
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          5840
  quantum           300

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 94813529 bytes 104548 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8020: dev ifb4pppoe-wan root refcnt 2 bandwidth 8500Kbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100.0ms atm overhead 44
 Sent 92884570 bytes 103164 pkt (dropped 1359, overlimits 72013 requeues 0)
 backlog 34450b 25p requeues 0
 memory used: 497952b of 4Mb
 capacity estimate: 8500Kbit
 min/max network layer size:           38 /    1492
 min/max overhead-adjusted size:      106 /    1696
 average network hdr offset:            0

                  Tin 0
  thresh       8500Kbit
  target          5.0ms
  interval      100.0ms
  pk_delay       44.8ms
  av_delay       25.8ms
  sp_delay        3.6ms
  backlog        34450b
  pkts           104548
  bytes        94813529
  way_inds         4699
  way_miss          691
  way_cols            0
  drops            1359
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1492
  quantum           300

in-game latency before and then after loading a video on another device:
wo_youtube
w_youtube

0.5 Mbps up is going to be a challenge regardless of the SQM type settings. Here's how that works out..

A full size packet is 1500 bytes, so 1500*8 bits. At 0.5 Mbps it takes 1500*8/5e5 = 0.024 = 24ms to send.

it seems like you have really more like 350kbps, so even more spikes just from 1 packet.

So, you have to assume that even with just one packet in a queue ahead of you you will get 20-50ms of ping spike on a regular basis. It would not be atypical to get several packets in a queue temporarily... so you should expect 100s of milliseconds of latency variation. Sometimes it will be fine, and then 3 packets get ahead of you... and 100 ms... etc

1 Like