SQM for FTTH, overhead issue i guess?

qdisc cake 808f: dev veth0 root refcnt 2 bandwidth 22Mbit diffserv4 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 40 mpu 64
 Sent 23606608 bytes 18893 pkt (dropped 53, overlimits 32153 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 649152b of 4Mb
 capacity estimate: 22Mbit
 min/max network layer size:           28 /    1476
 min/max overhead-adjusted size:       68 /    1516
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh       1375Kbit       22Mbit       11Mbit     5500Kbit
  target         13.2ms        5.0ms        5.0ms        5.0ms
  interval      108.2ms      100.0ms      100.0ms      100.0ms
  pk_delay       15.9ms          0us       23.4ms        242us
  av_delay        353us          0us       15.3ms         44us
  sp_delay        160us          0us         66us         24us
  backlog            0b           0b           0b           0b
  pkts               22            0        18225          699
  bytes           19576            0     23533956       126852
  way_inds            0            0            0            0
  way_miss            2            0           69            6
  way_cols            0            0            0            0
  drops               0            0           53            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            0            0            1
  bk_flows            0            0            1            0
  un_flows            0            0            0            0
  max_len          1490            0         1490         1454
  quantum           300          671          335          300

again i see 20+ ms on video....

i wander if my overhead is wrong?

i called to ask my isp about it, they had no idea about it... all they could say is they are using GPON...

the thing is i have been testing on different values, i couldnt find one satisfying, if i go higher number than 41 i will start having Upload spikes...

that's odd. overhead doesn't really work that way so it's maybe just coincidence? what overhead does is it's a number that cake adds to the packet size to figure out how much bandwidth you're using. so high overheads should just effectively lower your bandwidth a bit.

For GPON try 50, it's almost sure to be an overestimate, which is better than underestimate.

What I think is really going on is that you're far from servers, and so they don't react quickly to signalling congestion, so stuff piles up in a queue somewhere upstream of you, so cake doesn't really have a chance to do a good job.

Just to see how things work, try dropping your speed to 10Mbps and tell us what that looks like.

1 Like

no luck, i dropped my speed to 10 Mbps and also changed my overhead to 50
for me, it seems like changing the overhead value to higher one gives me more spikes on upload portion and sometimes on download...

qdisc cake 809b: dev veth0 root refcnt 2 bandwidth 10Mbit diffserv4 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 50 mpu 64
 Sent 1083042 bytes 799 pkt (dropped 18, overlimits 1574 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 179424b of 4Mb
 capacity estimate: 10Mbit
 min/max network layer size:           40 /    1378
 min/max overhead-adjusted size:       90 /    1428
 average network hdr offset:           13

                   Bulk  Best Effort        Video        Voice
  thresh        625Kbit       10Mbit        5Mbit     2500Kbit
  target         29.1ms        5.0ms        5.0ms        7.3ms
  interval      124.1ms      100.0ms      100.0ms      102.3ms
  pk_delay          8us          0us       65.5ms          0us
  av_delay          0us          0us       53.6ms          0us
  sp_delay          0us          0us        7.1ms          0us
  backlog            0b           0b           0b           0b
  pkts                1            0          816            0
  bytes              66            0      1108032            0
  way_inds            0            0            0            0
  way_miss            1            0           10            0
  way_cols            0            0            0            0
  drops               0            0           18            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            0            1            0
  bk_flows            0            0            1            0
  un_flows            0            0            0            0
  max_len            66            0         1392            0
  quantum           300          305          300          300

This is 50 overhead

this is 40 overhead
(i dont understand i limited upload to 10000 but still it went above 10 mbps)

Hmm... those delay time graphs actually look excellent, except for the isolated events... what device are you running on? I think this is nothing to do with SQM and to do instead with some ethernet/switch/hardware/driver issue that causes delays in kernel processing.

You can see that the bandwidth is oscillating up and down... this is caused by your high ping time to the servers. I wouldn't worry about the 11Mbps vs 10 that's probably measurement error more than anything else.

Set your speed back to the 22Mbps and set overhead of 50 and show us results of that speed test

i m using TP-link Archer c20 v4 for router, and for PC i m using an old dell laptop(inspiron 15 3000series?) with ethernet cable...
i do these dslreports test on ipad pro 3rd gen on 5 ghz wifi too

on 22mbps with overhead 50

qdisc cake 80a7: dev veth0 root refcnt 2 bandwidth 22Mbit diffserv4 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 50 mpu 64
 Sent 91122396 bytes 79603 pkt (dropped 2109, overlimits 125260 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 1215648b of 4Mb
 capacity estimate: 22Mbit
 min/max network layer size:           28 /    1476
 min/max overhead-adjusted size:       78 /    1526
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh       1375Kbit       22Mbit       11Mbit     5500Kbit
  target         13.2ms        5.0ms        5.0ms        5.0ms
  interval      108.2ms      100.0ms      100.0ms      100.0ms
  pk_delay         47us         14us       32.9ms        481us
  av_delay         26us          0us       16.6ms        111us
  sp_delay         13us          0us        2.9ms         14us
  backlog            0b           0b           0b           0b
  pkts            65459            2        15661          590
  bytes        75864263         2046     18339840        40605
  way_inds            0            0          240            0
  way_miss           36            1          125            2
  way_cols            0            0            0            0
  drops            2049            0           60            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows           16            0            2            1
  bk_flows            0            0            0            0
  un_flows            0            0            0            0
  max_len          1490         1392         1490          183
  quantum           300          671          335          300

qdisc noqueue 0: dev veth1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 80a4: dev pppoe-wan root refcnt 2 bandwidth 22Mbit diffserv4 dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 50 mpu 64
 Sent 38927023 bytes 60243 pkt (dropped 416, overlimits 51740 requeues 0)
 backlog 56342b 39p requeues 0
 memory used: 497952b of 4Mb
 capacity estimate: 22Mbit
 min/max network layer size:           40 /    1476
 min/max overhead-adjusted size:       90 /    1526
 average network hdr offset:            0

                   Bulk  Best Effort        Video        Voice
  thresh       1375Kbit       22Mbit       11Mbit     5500Kbit
  target         13.2ms        5.0ms        5.0ms        5.0ms
  interval      108.2ms      100.0ms      100.0ms      100.0ms
  pk_delay       45.0ms        219us        3.0ms        379us
  av_delay       17.1ms          6us        952us         95us
  sp_delay        3.4ms          6us         40us         32us
  backlog        56342b           0b           0b           0b
  pkts            50834           28         9221          615
  bytes        33321310        23390      6209369        37304
  way_inds            0            0            0            0
  way_miss           35            4          124           33
  way_cols            0            0            0            0
  drops             396            0           20            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            1            1            1
  bk_flows           16            0            0            0
  un_flows            0            0            0            0
  max_len          1476         1378         2152          798
  quantum           300          671          335          300

my problem with these is its never consistent for me...

For me, problem comes while gaming, like lack of hit registrations and sometime like lag all over the place

and also, i m using this on sysctl, is that the cause?

net.core.default_qdisc=cake
net.ipv4.tcp_congestion_control=cubic
net.core.rmem_max = 1010944
net.core.wmem_max = 1010944
net.ipv4.tcp_rmem = 8192        87380   1010944
net.ipv4.tcp_wmem = 8192        87380   1010944
net.ipv4.route.flush=1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 2500
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_max_tw_buckets = 288000
net.ipv4.tcp_tw_reuse = 1
net.netfilter.nf_conntrack_acct=1
net.netfilter.nf_conntrack_checksum=0
net.netfilter.nf_conntrack_max=16384
net.netfilter.nf_conntrack_tcp_timeout_established=7440
net.netfilter.nf_conntrack_udp_timeout=60
net.netfilter.nf_conntrack_udp_timeout_stream=180

Both of your speed tests indicate that SQM is performing optimally. Your problem is likely that you're very far from the gaming servers. It's like trying to play jazz music when you're a mile away from someone else... they play some notes, but have already moved on to the next notes by the time you hear the notes... so you're behind them and then they hear your notes... and they don't mesh with what they are playing because your notes are based on what they were playing 4 or 5 notes ago... so the whole thing breaks down.

Add in that you've got some undersea cables, and etc and you'll have packet loss and latency even if SQM is doing its job. You can't get away from the fact that when you ping singapore it takes 200ms or whatever. If the game server is in singapore, then the soonest after your key-press the server can know what you've done is 0.2 seconds later which is pretty long.

Sorry I said some stuff that's wrong about music notes... I'll recalculate that...

Ok, for the music analogy ... 100 beats per minute, so 100/60 beats per second, 1/8 note is 1/8 of a beat... so 100/60/8 = 0.208 so in fact I was right, a 200ms delay is like 1 full 1/8th note behind in music.

i understood ur point but, i get avg of 70ms on the games i play... the problem i face is like, same ammount of hit registers and kills a player that is when i check on dslreports and the uploads are fine... and i go play on other times when uploads arent fine it takes like 2-3 times the normal ammount... it's just frustrating... also when this happens i see problems in like messaging apps too...

i barely see any download issues, like text comes in, games load, but when the upload side of mine is fluctuating, the text i send takes like a sec or 2 to be sent and like games show's the network errors and all

is it possible that it is an issue of upload side from isp side?

This may be oversubscription of your ISP... Maybe certain times of the day people are sending a bunch of data... and so your packets are lost and delayed not in your router/ONT but somewhere inside the ISP with their congestion.

If you are seeing problems now... try running mtr against something like the game servers you try to play on.

you can get MTR for windows here: https://sourceforge.net/projects/winmtr/

you can also install the command line mtr package on the router and run it from ssh

i downloaded it for pc, i dont know what i m looking at, i m not that much technically educated on these things....

i checked and i dont know if its the actual game server but the results were like this...
p.s i mostly play games on ipad pro 3, on 5 ghz wifi, usually games like Clash royale, pubg mobile etc

|------------------------------------------------------------------------------------------|

|                                      WinMTR statistics                                   |

|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |

|------------------------------------------------|------|------|------|------|------|------|

|                             OpenWrt.lan -    0 |  122 |  122 |    0 |    0 |    1 |    0 |

|                           103.10.30.153 -    0 |  122 |  122 |    1 |    2 |   26 |    1 |

|                             103.10.29.4 -    0 |  122 |  122 |    1 |    3 |   57 |    2 |

|                   ae0-bg1.vianet.com.np -    0 |  122 |  122 |    1 |    3 |   59 |   10 |

|abts-north-static-153.224.246.61.airtelbroadband.in -    0 |  122 |  122 |    2 |    7 |   78 |    2 |

|                           116.119.68.60 -    0 |  122 |  122 |  172 |  175 |  214 |  173 |

|             ldn-b2-link.ip.twelve99.net -    0 |  122 |  122 |  179 |  179 |  181 |  180 |

|            ldn-bb1-link.ip.twelve99.net -    0 |  122 |  122 |  304 |  304 |  306 |  305 |

|            nyk-bb2-link.ip.twelve99.net -    3 |  110 |  107 |  276 |  276 |  278 |  277 |

|            ash-bb2-link.ip.twelve99.net -   19 |   71 |   58 |  282 |  282 |  284 |  282 |

|             mai-b2-link.ip.twelve99.net -    0 |  122 |  122 |  303 |  304 |  305 |  305 |

|akamai-svc069986-lag003206.ip.twelve99-cust.net -    0 |  122 |  122 |  298 |  306 |  383 |  299 |

|       ae12.nota-mia7.netarch.akamai.com -    0 |  122 |  122 |  292 |  318 |  557 |  294 |

|a96-16-98-73.deploy.static.akamaitechnologies.com -    0 |  122 |  122 |  299 |  299 |  301 |  300 |

|________________________________________________|______|______|______|______|______|______|

   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

about '50 overhead' , i thought only EPON had higher overhead value?

i m suffering on upload side only mostly, at 7pm to 10 pm usually i have ping spike on upload side only, the download remains normal...

The MTR results indicate that from the hop 116.119.68.60 onward ping times were very high (175 ms and up). Gaming with those ping times will just always be bad. It really has nothing to do with SQM and bufferbloat.

Most likely that node is the far side of a trans-oceanic cable. Since the hop before

|abts-north-static-153.224.246.61.airtelbroadband.in -    0 |  122 |  122 |    2 |    7 |   78 |    2 |

showed ping times avg 7ms and worst case was 78ms... Sure the 78 ms is bad, but it's nothing like ~ 300 avg to the final destination

i see, i guess i will test 50 overhead for a while and see how it goes....

thank you for ur time

1 Like

No problem. If you can find some friends in the same country (even better same general town/county/region) and play a game that is self-hosted on your computer you will likely enjoy it a lot more than a game hosted across a trans-ocean cable.

kind a hard to find such, and there arent much gamer from here...
and no server is nearby for any game, it sucks....

anyways thanks again for ur time

1 Like

No worries. with COVID I have been trying to get friends I know to play stuff, because otherwise I don't see or do anything with them ... but it's not easy. Good luck!

1 Like

Well the ping on upload started again...

while this happens, i feel the hit registration delay

From the details on that page:

Speed	RTT / Jitter                          Avg	Re-xmit Avg	Cwnd Avg
Tokyo, Japan (amazon)	d3	1.47 Mb/s	317.3±52.6ms	7.7%	71
Singapore (softlayer)	d6	1.28 Mb/s	204.2±89.1ms	3%	12
Tokyo, Japan (amazon)	u3	1.09 Mb/s	144±62.3ms	-	10
Singapore (softlayer)	u8	1.65 Mb/s	73.5±30.3ms	-	10

You'll see that to one tokyo server you were dropping 7.7% of packets, and to Singapore you were dropping 3% of packets

But:

  drops               0            0           18            0

so you weren't dropping them in cake, they were getting dropped (along the way). This probably indicates your problem is an ISP or trans-oceanic issue. There is nothing much SQM can do about it.

i see, i wanted to change my isp, but dad kept internet for a year, thus stuck with this isp for a while...

will getting static ip help?

No static ip won't help. Sometimes using a VPN can help because the VPN provider uses different routes across different links

What pops out is that for both Tokyo and Singapore, the RTT for download is double that as the RTT for upload to the same target. In theory that could be the result of asymmetric paths and hot-potato routing, but it also could indicate simply congestion in the download direction (which fits with the high Re-xmit numbers).

The unoaded RTTs were:

05.67s 75ms Singapore
05.67s 151ms Tokyo, Japan
05.67s 178ms Amsterdam, Netherlands, EU
05.67s 189ms Zurich, Switzerland, EU

So it seems clear that the download paths between you and the Tokyo and Singapore servers are overloaded...

I would also say that the 260ms RTT you configured initially was a bit generous, I would at best go for 150-180 ms RTT for cake (and only if you want/need to transfer data at high rates to remote places, otherwise a lower RTT will allow lower latency at a slight decrease in throughput).

2 Likes