SQM cake: traffic prioritisation

Gigabit · December 11, 2020, 4:39pm

@dlakelan try this, mtr running from my Mac rather than OpenWrt. Wired in via GigE. Unloaded:

                                                                                   Packets               Pings
 Host                                                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    openwrt.lan (192.168.1.1)                                            0.0%    65    0.4   0.4   0.3   0.7   0.1
 2. AS???    192.168.0.254 (192.168.0.254)                                        0.0%    65    1.3   1.3   1.0   4.4   0.4
 3. AS???    172.16.14.147 (172.16.14.147)                                        0.0%    65   15.0   5.8   5.0  15.6   1.8
 4. AS2856   31.55.186.177 (31.55.186.177)                                       90.8%    65    7.5   7.4   7.1   7.5   0.2
 5. AS2856   31.55.186.176 (31.55.186.176)                                        0.0%    65    7.3   7.6   6.9  18.6   1.5
 6. AS2856   core3-hu0-16-0-1.faraday.ukcore.bt.net (195.99.127.196)              0.0%    65    7.6   7.6   7.1  14.1   0.9
 7. AS2856   62.172.103.224 (62.172.103.224)                                      0.0%    65   67.9  15.9   7.3  67.9  14.1
 8. (waiting for reply)
 9. AS54113  151.101.0.81 (151.101.0.81)                                          0.0%    65    7.9   8.2   7.4  21.7   2.0

dlakelan · December 11, 2020, 4:42pm

great, very readable. Is this unloaded? I see you already answered that... let's see during the speed test!

Gigabit · December 11, 2020, 4:42pm

@dlakelan loaded, with fast.com speed test:

                                                                                   Packets               Pings
 Host                                                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    openwrt.lan (192.168.1.1)                                            0.0%   136    0.3   0.4   0.3   0.8   0.1
 2. AS???    192.168.0.254 (192.168.0.254)                                        0.0%   136    1.7   1.5   0.9  18.5   1.6
 3. AS???    172.16.14.147 (172.16.14.147)                                        0.0%   136    5.3   6.1   4.9  17.2   2.0
 4. AS2856   31.55.186.181 (31.55.186.181)                                       92.6%   136   11.9   7.8   6.9  11.9   1.5
 5. AS2856   31.55.186.180 (31.55.186.180)                                        0.0%   136    8.8   8.5   7.0  37.7   3.2
 6. AS2856   195.99.127.96 (195.99.127.96)                                        0.0%   136    8.1   8.4   6.9  23.3   2.3
 7. AS2856   194.72.16.48 (194.72.16.48)                                          0.0%   136    8.3  17.1   7.4 101.5  16.0
 8. (waiting for reply)
 9. AS54113  151.101.192.81 (151.101.192.81)                                      0.0%   136    8.2   8.9   7.5  19.7   2.1

dlakelan · December 11, 2020, 4:43pm

Ok, so this shows at least at the moment that the same situation that was occurring before is not occurring now. Is this with your full speed set on SQM?

Gigabit · December 11, 2020, 4:45pm

This is SQM configured as follows:

dlakelan · December 11, 2020, 4:57pm

try to give it 95% of what you get for a speedtest without any SQM, see if your problem returns?

Gigabit · December 11, 2020, 5:01pm

Torrents seem to cause the biggest problem, I'll torrent something now and run the same test.

Gigabit · December 11, 2020, 5:07pm

@dlakelan

With a torrent running on another client wired in via GigE, 7 peers it says.

Haven't made any changes to SQM yet.

                                                                                   Packets               Pings
 Host                                                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    openwrt.lan (192.168.1.1)                                            0.0%   151    0.4   0.4   0.3   0.6   0.1
 2. AS???    192.168.0.254 (192.168.0.254)                                        1.3%   151    1.1   2.9   0.9  31.2   5.1
 3. AS???    172.16.14.147 (172.16.14.147)                                        0.0%   151    5.3   7.5   4.9  48.1   5.3
 4. AS2856   31.55.186.181 (31.55.186.181)                                       94.6%   150    7.1  11.7   7.0  29.7   8.3
 5. AS2856   31.55.186.180 (31.55.186.180)                                        0.0%   150    7.4  10.2   7.0  72.1   6.9
 6. AS2856   195.99.127.96 (195.99.127.96)                                        0.0%   150    6.9   9.2   6.6  31.7   3.9
 7. AS2856   194.72.16.48 (194.72.16.48)                                          0.0%   150    7.9  17.0   7.2 109.9  16.8
 8. (waiting for reply)
 9. AS54113  151.101.192.81 (151.101.192.81)                                      0.0%   150    8.1  10.3   7.4  37.0   5.4

dlakelan · December 11, 2020, 5:11pm

as you can see the Wrst column now goes to 31.2 at the second hop, which is your modem I guess? and variation goes to 5.1 ms StDev, and then for all steps beyond that it has much higher variation (StdDev) and Wrst case ping.

This indicates your modem (or whatever device has IP 192.168.0.254 and is directly connected to your router) is causing the problem

Gigabit · December 11, 2020, 5:16pm

Hi @dlakelan that's my modem correct, so what can I do about this now?

I thought the point of bufferbloat was to work around issues in the modem.

dlakelan · December 11, 2020, 5:22pm

It's a good question. Do you have a cable modem? Does it have a Puma chipset? These are known to have problems.

Gigabit · December 11, 2020, 5:23pm

No I am on VDSL, my "modem" is the BT Smart Hub.

dlakelan · December 11, 2020, 5:31pm

I wonder if there isn't some kind of prioritization or something going on in the BT Smart Hub that is on-purpose slowing down all your traffic when you torrent.

which torrent client are you using? does it set a DSCP tag on its packets?

Gigabit · December 11, 2020, 5:31pm

qBittorent.

dlakelan · December 11, 2020, 6:14pm

Clearly something special is going on for bittorrent type traffic, and it's happening inside your modem. I didn't see in the online information any info about what qBittorrent does in terms of marking packets. I don't think this is just about queueing issues in your modem, it's probably more like some kind of throttling or maybe CPU limitations when there are a lot of connections, since bittorrent makes tens or hundreds of connections simultaneously.

also I think modern bittorrent uses UDP? And that might be part of the issue as well.

Gigabit · December 11, 2020, 6:27pm

Thanks for your help, however I want to ensure I am also seeing no issues with normal traffic.

I ran the same fast.com test and pinged an IP from OpenWrt:

64 bytes from 8.8.8.8: icmp_req=1 ttl=116 time=8.03 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=116 time=20.9 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=116 time=8.13 ms
64 bytes from 8.8.8.8: icmp_req=4 ttl=116 time=7.92 ms
64 bytes from 8.8.8.8: icmp_req=5 ttl=116 time=7.87 ms
64 bytes from 8.8.8.8: icmp_req=6 ttl=116 time=8.58 ms
64 bytes from 8.8.8.8: icmp_req=7 ttl=116 time=11.3 ms
64 bytes from 8.8.8.8: icmp_req=8 ttl=116 time=7.91 ms
64 bytes from 8.8.8.8: icmp_req=9 ttl=116 time=8.13 ms
64 bytes from 8.8.8.8: icmp_req=10 ttl=116 time=9.63 ms
64 bytes from 8.8.8.8: icmp_req=11 ttl=116 time=8.37 ms
64 bytes from 8.8.8.8: icmp_req=12 ttl=116 time=8.10 ms
64 bytes from 8.8.8.8: icmp_req=13 ttl=116 time=7.86 ms
64 bytes from 8.8.8.8: icmp_req=14 ttl=116 time=8.37 ms
64 bytes from 8.8.8.8: icmp_req=15 ttl=116 time=7.88 ms
64 bytes from 8.8.8.8: icmp_req=16 ttl=116 time=8.80 ms
64 bytes from 8.8.8.8: icmp_req=17 ttl=116 time=8.89 ms
64 bytes from 8.8.8.8: icmp_req=18 ttl=116 time=7.63 ms
64 bytes from 8.8.8.8: icmp_req=19 ttl=116 time=7.64 ms
64 bytes from 8.8.8.8: icmp_req=20 ttl=116 time=8.44 ms
64 bytes from 8.8.8.8: icmp_req=21 ttl=116 time=8.62 ms
64 bytes from 8.8.8.8: icmp_req=22 ttl=116 time=9.14 ms
64 bytes from 8.8.8.8: icmp_req=23 ttl=116 time=7.83 ms
64 bytes from 8.8.8.8: icmp_req=24 ttl=116 time=9.41 ms
64 bytes from 8.8.8.8: icmp_req=25 ttl=116 time=8.63 ms
64 bytes from 8.8.8.8: icmp_req=26 ttl=116 time=8.15 ms
64 bytes from 8.8.8.8: icmp_req=27 ttl=116 time=7.60 ms
64 bytes from 8.8.8.8: icmp_req=28 ttl=116 time=7.88 ms
64 bytes from 8.8.8.8: icmp_req=29 ttl=116 time=54.1 ms
64 bytes from 8.8.8.8: icmp_req=30 ttl=116 time=9.61 ms
64 bytes from 8.8.8.8: icmp_req=31 ttl=116 time=8.12 ms
64 bytes from 8.8.8.8: icmp_req=32 ttl=116 time=9.93 ms
64 bytes from 8.8.8.8: icmp_req=33 ttl=116 time=7.59 ms
64 bytes from 8.8.8.8: icmp_req=34 ttl=116 time=8.89 ms
64 bytes from 8.8.8.8: icmp_req=35 ttl=116 time=8.34 ms
64 bytes from 8.8.8.8: icmp_req=36 ttl=116 time=8.81 ms
64 bytes from 8.8.8.8: icmp_req=37 ttl=116 time=8.38 ms
64 bytes from 8.8.8.8: icmp_req=38 ttl=116 time=7.63 ms
64 bytes from 8.8.8.8: icmp_req=39 ttl=116 time=7.86 ms
64 bytes from 8.8.8.8: icmp_req=40 ttl=116 time=8.34 ms

I see these random spikes upwards still. Now I know you'll say it's just a one off but I monitor my connection consistently with a ping monitor and I see these kind of spikes throughout the day but only when the connection is under some kind of load.

What exactly is happening here, I just wonder if it's the same symptom but the torrents just make it more prevalent with the increased number of connections.

dlakelan · December 11, 2020, 6:33pm

I know that occasional spikes were associated with particular driver/hardware issues in the r7800 which is nearly identical internally to the ZyXEL. I don't know if that's been really fixed or not.

Gigabit · December 11, 2020, 6:38pm

Fair point, I feel as if I have got the router wrong every time

Archer C7 seems to run out of out CPU when doing torrents. X86 with Intel NIC is apparently creating lag caused by flow offloading (?)

I wish I could just find a router that was good to go, do you have any suggestions?

dlakelan · December 11, 2020, 7:18pm

you can disable those packet offloads, and in fact I think SQM does so automatically. If you have an x86 I would use that, and choose a decent switch with VLANs and QoS: I use a ZyXEL gs1900-24e and a few TP-Link sg108e...

With my whole family at home, zooming, gaming, youtubing etc we can have easily 8 to 10 devices active at once, including 2 or 3 lines of VOIP calls and kids gaming on minecraft etc, as well as at least 3 machines using an NFS server for home directories. it all works well. Of course I'm lucky enough to get gigabit fiber from ATT, but the thread on extreme gaming stability shows that with appropriate settings in customized scripts, it's possible to get rock steady game latency even at 700kbps. Since you have well over 3000kbps each direction, you should be able to get very good latency, less than 15ms round trip time even with regular SQM, and probably better than that if you mark VOIP packets etc and use diffserv4

HOWEVER that will be just latency on exit of your router heading towards the modem. If the modem is borked and fluttering around with insufficient CPU or intentional packet shaping issues to discourage torrenting, etc then all bets are off.

EDIT:

When it comes to latency issues, there are two very very sensitive applications: Gaming and VOIP. I think SQM doesn't quite get to the state where gaming and VOIP are comfortable at speeds less than about 3000kbps but this is more or less "to be expected". However, above that speed, with SQM and diffserv4 I'd be surprised if at least VOIP wasn't very comfortable. Again, assuming your modem doesn't bork things.

moeller0 · December 11, 2020, 9:38pm

Nope, SQM doesn't, but cake automatically splits GSO type meta packets on links slower than 1 Gbps. SQM does have some example function for how to disable offloads, but most often it is not needed.

For VoIP I am not so sure, if egress VoIP packets are dscp marked into the high priority tier. VoIP takes ~ 100 Kbps per direction and cake's 1/4 of the total for the highest tin will give sufficient bandwidth for a single VoIP flow down to a net 400 Kbps link. And that tier is sufficiently prioritized to actually give ViOP packets a short circuit!.
For gaming I agree, at low rates things get very iffy for fatter flows than VoIP. But my hunch is using diffserv4's 2nd highest priority tin (which gets up to 50% of the total link) exclusively for gaming packets (and conpletely sparing the highest tin) should go a long way! Sure not as refined and bespoke as you fine script in the other thread, but for games with not total-crap dejitter buffers it could be pretty okay (as long as the link is not totally sluggish). But judging from the other thread de-jitter buffers seems to be super sensitive in many games posing the question with what type of links game developers actually test their games' playability, but I digress...