Sqm doesnt play nice together with stadia?

mezo · December 3, 2020, 8:05pm

hey,
i was super fine with sqm till "today" when i noticed even a slow download (1/2 of my connetion) cause stadia to stutter.

im running OpenWrt 19.07.4 on a ZyXEL NBG6617.

here are some infos:

Mon Sep 7 02:36:13 2020 daemon.info pppd[8047]: Remote message: SRU=41296#SRD=109974#


config queue 'eth1'
	option interface 'pppoe-wan'
	option debug_logging '0'
	option verbosity '5'
	option qdisc 'cake'
	option script 'piece_of_cake.qos'
	option linklayer 'ethernet'
	option overhead '34'
	option qdisc_advanced '1'
	option squash_dscp '1'
	option squash_ingress '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option qdisc_really_really_advanced '1'
	option enabled '1'
	option download '98976'
	option upload '40883'
	option iqdisc_opts 'diffserv4 nat dual-dsthost ingress'
	option eqdisc_opts 'diffserv4 nat dual-srchost'

qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth1.7 root refcnt 2 
qdisc cake 8031: dev pppoe-wan root refcnt 2 bandwidth 40883Kbit diffserv4 dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 34 
qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ---------------- 
qdisc cake 8032: dev ifb4pppoe-wan root refcnt 2 bandwidth 98976Kbit diffserv4 dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 34

qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 3989931935710 bytes 3210730344 pkt (dropped 159, overlimits 0 requeues 3573) 
 backlog 0b 0p requeues 3573
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 75129428 bytes 405240 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 1828256686075 bytes 1408559667 pkt (dropped 158, overlimits 0 requeues 2734) 
 backlog 0b 0p requeues 2734
  maxpacket 1514 drop_overlimit 0 new_flow_count 2160 ecn_mark 0
  new_flows_len 1 old_flows_len 16
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 1144271842886 bytes 966054880 pkt (dropped 0, overlimits 0 requeues 529) 
 backlog 0b 0p requeues 529
  maxpacket 3028 drop_overlimit 0 new_flow_count 1502 ecn_mark 0
  new_flows_len 0 old_flows_len 2
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 1017328277321 bytes 835710557 pkt (dropped 1, overlimits 0 requeues 310) 
 backlog 0b 0p requeues 310
  maxpacket 1514 drop_overlimit 0 new_flow_count 1779 ecn_mark 0
  new_flows_len 1 old_flows_len 24
qdisc mq 0: dev eth1 root 
 Sent 217896645257 bytes 1719160849 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 35257641 bytes 380436 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 50978630783 bytes 383593218 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 58714865001 bytes 474800081 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 108167891832 bytes 860387114 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 164 drop_overlimit 0 new_flow_count 2 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.7 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8031: dev pppoe-wan root refcnt 2 bandwidth 40883Kbit diffserv4 dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 34 
 Sent 1359784600 bytes 13624863 pkt (dropped 217, overlimits 168124 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 155008b of 4Mb
 capacity estimate: 40883Kbit
 min/max network layer size:           28 /    1492
 min/max overhead-adjusted size:       62 /    1526
 average network hdr offset:            0

                   Bulk  Best Effort        Video        Voice
  thresh       2555Kbit    40883Kbit    20441Kbit    10220Kbit
  target          7.1ms        5.0ms        5.0ms        5.0ms
  interval      102.1ms      100.0ms      100.0ms      100.0ms
  pk_delay          0us        687us        193us        1.6ms
  av_delay          0us        112us         87us         76us
  sp_delay          0us         16us         20us         15us
  backlog            0b           0b           0b           0b
  pkts                0     13623219          414         1447
  bytes               0   1359733177        31464       269511
  way_inds            0       218604            0            0
  way_miss            0       104327          238          268
  way_cols            0            0            0            0
  drops               0          217            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            2            1            1
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len             0        24820           76          576
  quantum           300         1247          623          311

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ---------------- 
 Sent 37365882268 bytes 31369039 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8032: dev ifb4pppoe-wan root refcnt 2 bandwidth 98976Kbit diffserv4 dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 34 
 Sent 37321282381 bytes 31337775 pkt (dropped 31264, overlimits 15642863 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 2112Kb of 4948800b
 capacity estimate: 98976Kbit
 min/max network layer size:           28 /    1492
 min/max overhead-adjusted size:       62 /    1526
 average network hdr offset:            0

                   Bulk  Best Effort        Video        Voice
  thresh       6186Kbit    98976Kbit    49488Kbit    24744Kbit
  target          5.0ms        5.0ms        5.0ms        5.0ms
  interval      100.0ms      100.0ms      100.0ms      100.0ms
  pk_delay          0us        1.2ms          0us        4.7ms
  av_delay          0us        150us          0us        243us
  sp_delay          0us         21us          0us         23us
  backlog            0b           0b           0b           0b
  pkts                0     31358309            0        10730
  bytes               0  37362815869            0      3066399
  way_inds            0       328716            0            0
  way_miss            0       115424            0           78
  way_cols            0            0            0            0
  drops               0        31264            0            0
  marks               0            2            0            0
  ack_drop            0            0            0            0
  sp_flows            0            1            0            1
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len             0         1492            0         1383
  quantum           300         1514         1510          755

dlakelan · December 3, 2020, 8:46pm

Your peak delay in the qdisc was 4.7ms and avg was 0.24 ms so I'm guessing the stutter was somewhere upstream of you. Drop your speed settings by 10% and see if it helps

mezo · December 5, 2020, 11:43am

I tried... Even way lower. It has no effect

moeller0 · December 5, 2020, 8:15pm

Let me guess, a Deutsche Telekom VDSL2 100/40 (L) link?

Well, how does it go, if you disable SQM? (Ssh into the router and issue /etc/init.d/sqm stop on the command line, use /etc/init.d/sqm start to start again).

Also, you have per internal IP fairness configured, how many concurrently downloading machines do you have in your home network? And which DNS servers are you using?

mezo · December 7, 2020, 4:22pm

yes

same if i do that

1

the one from the provider

moeller0 · December 7, 2020, 4:29pm

Sorry, I wad imprecise in my question, if you plat without SQM things are just as bad as when playing with SQM enabled?

In that case internal IP fairness reduces to flow fairness, so not your issue.

Mmmh, maybe try 8.8.8.8, google's own DNS servers? I would assume they would be worse, but trying should not be too hard...

mezo · December 21, 2020, 1:23pm

sorry for the late answer.

its waaaay worse without sqm
honestly i was not able to reproduce the lags today, i will check more soon.

set this.

mezo · December 21, 2020, 6:14pm

i can still confirm this. maybe 1/2 of my connection is not enough for smooth 4k streaming? i sadly dont know the average usage of this 4k stream. maybe it exceed what sqm can promise me?

moeller0 · December 21, 2020, 6:33pm

Maybe the CPU is not fast enough? Sure it is a quad core, but only running at ~700 MHz, and traffic shaping really needs loads of CPU. Have a look at the SQM FAQ's Measured goodput in speed tests with SQM is considerably lower than without for quick and dirty CPU load assessments. Let me know how this goes... (also, pleas post the output of cat /proc/interrupts)

mezo · December 22, 2020, 12:51pm

when i run a speedtest on speedtest.net

root@OpenWrt:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       
 18:   28355775   28355784   28355784   28355784     GIC-0  20 Level     arch_timer
 22:       9354          0          0          0     GIC-0 270 Level     bam_dma
 23:      97365          0          0          0     GIC-0 127 Level     78b5000.spi
 24:          0          0          0          0     GIC-0 239 Level     bam_dma
 25:          5          0          0          0     GIC-0 139 Level     msm_serial0
 42:   16630374          0          0          0     GIC-0 200 Level     ath10k_ahb
 59:   15767353          0          0          0     GIC-0 201 Level     ath10k_ahb
 60:   40111729          0          0          0     GIC-0  97 Edge      edma_eth_tx0
 61:       7830          0          0          0     GIC-0  98 Edge      edma_eth_tx1
 62:   50628093          0          0          0     GIC-0  99 Edge      edma_eth_tx2
 63:        561          0          0          0     GIC-0 100 Edge      edma_eth_tx3
 64:   25186254          0          0          0     GIC-0 101 Edge      edma_eth_tx4
 65:        303          0          0          0     GIC-0 102 Edge      edma_eth_tx5
 66:     266159          0          0          0     GIC-0 103 Edge      edma_eth_tx6
 67:         23          0          0          0     GIC-0 104 Edge      edma_eth_tx7
 68:   31744873          0          0          0     GIC-0 105 Edge      edma_eth_tx8
 69:         13          0          0          0     GIC-0 106 Edge      edma_eth_tx9
 70:     555735          0          0          0     GIC-0 107 Edge      edma_eth_tx10
 71:         21          0          0          0     GIC-0 108 Edge      edma_eth_tx11
 72:      12568          0          0          0     GIC-0 109 Edge      edma_eth_tx12
 73:          6          0          0          0     GIC-0 110 Edge      edma_eth_tx13
 74:       9758          0          0          0     GIC-0 111 Edge      edma_eth_tx14
 75:          4          0          0          0     GIC-0 112 Edge      edma_eth_tx15
 76:  116845801          0          0          0     GIC-0 272 Edge      edma_eth_rx0
 78:    7285453          0          0          0     GIC-0 274 Edge      edma_eth_rx2
 80:   15145919          0          0          0     GIC-0 276 Edge      edma_eth_rx4
 82:   19477300          0          0          0     GIC-0 278 Edge      edma_eth_rx6
 92:          1          0          0          0   msmgpio   2 Edge      keys
 93:          1          0          0          0   msmgpio  63 Edge      keys
 94:          1          0          0          0   msmgpio   4 Edge      keys
 95:          0          0          0          0     GIC-0 164 Level     xhci-hcd:usb1
 96:          0          0          0          0     GIC-0 168 Level     xhci-hcd:usb3
IPI0:          0          0          0          0  CPU wakeup interrupts
IPI1:          0          0          0          0  Timer broadcast interrupts
IPI2:     224178    2595469    5364711    2649064  Rescheduling interrupts
IPI3:         60   66276755   20018416   28404175  Function call interrupts
IPI4:          0          0          0          0  CPU stop interrupts
IPI5:    2556158    2558480    3796494    3535712  IRQ work interrupts
IPI6:          0          0          0          0  completion interrupts
Err:          0

dlakelan · December 22, 2020, 2:33pm

Do you have 4 cores? Because 35% sirq it could be one core saturated plus extra... SQM is single threaded

Never mind I see the proc interrupts... So YES you probably are saturating core 0

You could try the easy way and install irqbalance and maybe at least the wifi interrupts will not interfere with the SQM

moeller0 · December 22, 2020, 2:50pm

Yes, good thought.

Please do opkg upgrade ; opkg install htop, and then run htop, which will show a graphical representation of load per CPU. If one CPU is overloaded (as likely given that most IRQ processing happens on CPU0) then I concur with the irqbalance recommendation.

mezo · December 30, 2020, 2:17pm

again sorry for the late answer.

i tried this and it change absolutly nothing.

dlakelan · December 30, 2020, 2:35pm

What speeds are you shaping? Also what bandwidth is being used by the stream? Take a screenshot of the LUCI bandwidth graph for WAN during streaming?

mezo · December 31, 2020, 1:32pm

connection is 100/40 mbit

dlakelan · December 31, 2020, 2:35pm

my impression is this router should handle 100Mbps no problem. i mean i had a MIPS device from Buffalo that would do that in 2012 or so... im confused.

moeller0 · December 31, 2020, 2:35pm

mmmh, this from during gameplay? Looks a bit odd that it drops <= 5 Mbps for extended multi-second periods, no? But maybe these corresspond to periods in the game with no changing visual content?

One other thing I wonder, your ISP is notorious for trying to incentivize content-providers like Google to maintain paid peering links into its network (AS3320) by letting its peerings with the other T1-ISP run way hotter than customary between T1-ISPs (actually Deutsche Telekom is not alone, AT&T, as well as Verizon and ComCast are known to behave similarly). A consequence of this is, if google sends those streams via its transit links these might simply encounter overloaded routers and hence show everything undesirable like frame drops and jitter. The typical way to confirm this hypothesis is to use a VPN to route all relevant traffic around DT's overloaded peerings.

dlakelan · December 31, 2020, 3:08pm

I'm not super familiar with stadia, though I'm beginning to like the idea. But can you reduce resolution and see if it behaves similarly? I'm wondering if maybe your display device overheats and throttles or something similar rather than a network issue?

mezo · January 1, 2021, 12:52pm

its during gameplay. its not that odd, because the drops are loading screens etc. where nothing rly happens.

wouldn't that mean, it would also lag if there is only the game stream running, which works fine?

i will try this next time. but im almost sure it will improve the situation if i drop resolution from 4k to 1080p.

moeller0 · January 1, 2021, 1:07pm

Thanks, that explains that pattern effortlessly.

Mostly yes, there are a few (hypothetical scenarios I can think of where that could be different (like when parallel to the game stream you also have say video streaming traffic over the same congested peering point, but that is really unlikely).

Yes, good test!.

P.S.: I note that the LuCI GUI plots like the bandwidth plots you show are quite CPU intensive and might cause issues if the router is actually hard working at the same time (now with 4 cores one would expect that the be a non-issue, but even with a dual core router, shaping close to my router's limit I could see the effect of the GUI). I note that OpenWrt seems to be shifting to client side rendering, which might, or might not ameliorate that issue.