SQM, Flow Offloading, VLAN Tagging and Gaming

lezz · November 15, 2018, 1:55am

This acutally seems a rather important issue, as I had a constant loss of sync when I left it blank or put it on 1500. Does this make sense to you? Or was it just a really really bad coincedence with the ISP being bad (I normally have a very stable sync).? I am kinda afraid of going back to leave it blank as it took kinda all afternoon to break it down to the mtu change that is causing the sync loss.

Regarding the new values for igress and egress: they seem to be working good as well. I see constantly an A+ on bufferbloat. The quality grade sometimes go to A but I think this is due to the ISP or dslreports servers and not the router settings. So I guerss right that your next suggestion/tuning would be to look into internal IP isolation right? Or is there anything beforehand to tune? In the end, I really want to have the best settings and router possible.....

moeller0 · November 15, 2018, 9:48am

Intresting, could you please blank this field again, reconnect the wan interface and then do a "logread | grep -e pppd" and post the REDACTED (the debug output might contain clear text PPPoE username & password which you should not post here) output here? In my case pppd takes care of setting the MTU to 1492.

There is also the method described in https://en.code-bude.net/2017/02/20/how-to-read-calculate-and-set-mtu-in-windows-linux-and-osx/ to empirically check the current effective MTU, could you run this test as well (with the MTU field blanked out).

Wrong MTU on IPv4 might cause fragmentation and on IPv6 it might cause packet loss (if pMTUd does not work correctly) but it should have zero effect on the DSL-synchronisation, so what errors did you encounter exactly?

Great, now could you test this again with your game, just to confirm that this alone is not as good as you like it, espcially when the other user(s) start to concurrently use the link?

Yes, basically I would propose you replace your curent /etc/config/sqm with:

config queue
        option debug_logging '0'
        option verbosity '5'
        option qdisc_advanced '1'
        option squash_dscp '0'
        option squash_ingress '0'
        option ingress_ecn 'ECN'
        option egress_ecn 'ECN'
        option qdisc_really_really_advanced '1'
        option linklayer 'ethernet'
        option overhead '34'
        option linklayer_advanced '1'
        option tcMTU '2047'
        option tcTSIZE '128'
        option tcMPU '64'
        option linklayer_adaptation_mechanism 'default'
        option iqdisc_opts 'nat dual-dsthost ingress'
        option eqdisc_opts 'nat dual-srchost'
        option interface 'pppoe-wan'
        option download '22300'
        option enabled '1'
        option upload '4950'
        option qdisc 'cake'
        option script 'layer_cake.qos'

And test this, if this is still not as good as with internal DSCPs (which would amaze my, as typically the upstream is somewhat more sensitive to DSCPs,) switch back to your dual sqm-instances, on-onWAN and one-on-LAN approach.

Have a look at https://serverfault.com/questions/769843/cannot-set-dscp-on-windows-10-pro-via-group-policy for potential ways to convince Windows 10 to expose its per application dscp marking capabilities even on non-domain controller/AD managed networks. This should help in marking your games (unless you play on a console, as I assume it will be super hard to configure this there). Heck you might even contact the game studio and ask nicely to make the DSCPs configurable....

lezz · November 15, 2018, 10:50am

Sure thing. First here is the determined MTU size with the method described above. 1464

Here the output of logread:

So the disconnects or loss of sync even happen with the set MTU of 1492. I have the fields empty at the moment. Fun fact: I think my ISP lowered my downlink (see SRD value in the screenshot above). Is that because I asked them to check the connection because of the disconnects? Are these frequent disconnects my mistake in the router config or is it the ISP?

OK, thought so. This is in line with my thoughts as mentioned above.

At the moment it is kinda unplayable as I have so many disconnects... but I will enter your SQM proposal in 20 min and get back to you.

Thank you so much, I really appreciate this kind of help!

moeller0 · November 15, 2018, 11:08am

This indicates an MTU of 1464 + 28 = 1492 so PPPoE does the right thing already, also notes how pppd complains about your MTU of 1492, as in that case it will need to fit its 8 byte header into the 1492 bytes resulting in an usable MTU to the internet of 1492-8 = 1484 Bytes...

Have a look at the output of the sync values in your modem, I expect that the Sync actually fell. If the sync value (or rather the maximum allowed by the DSLAM) end up being a muktiple of 1000Kbps it is very likely that DTAG's latest "toy" got activated on your link; DTAG now introduces Assia's DSLExpresse to manage problematic lines, you line with its relatively frequent losses might have triggered DSLExpresses intervention, which mostly shows as a fast reduction of the sync bandwidth that will, assuming your link stays error free enough, slowly be increased again. The idea is to trade a little bandwidth for more stability. In general not a bad idea,, but certainly something to keep in mind when using a traffic shaper on your router...

First thanks for all the testing, secondly, as long as your line keeps loosing sync no amount of sqm-goodness is going to save your bacon while gaming, this needs to be solved first, think about a "Stoerungsmeldung" with your ISP.

lezz · November 15, 2018, 11:35am

So I inserted the SQM settings as above. One question is regarding the custom firewall rules I insterted earlier with the iptables: Do I need to remove them or shall they stay?

moeller0 · November 15, 2018, 2:55pm

For initial testing I would remove them, then if the simpler cake-only mode (with somewhat optimized cake settings) still does not work well enough with your gaming (which I assume tp be the case) you should reintroduce the iptable commands (and potentially also the split of the shaper into egress shaping on WAN and LAN).

lezz · November 15, 2018, 3:34pm

Ok will do that.

When I go back to WAN LAN option, shall I keep the advanced settings you gave me?

And as you expected it, it is not good without the iptables.

moeller0 · November 15, 2018, 3:52pm

Well these will to be adapted a bit for the LAN part, but let's tackle this once we get there...

This leaves me at a bit puzzled, do you run any heavy concurrent traffic while gaming?

BTW setting ICMP to CS5 seems slightly wrong, unless you exclusively want tp use ICMP packets to probe how your game traffic would behave (but please note that at least on egress unix ping will allow you to specify the dscp/tos marking on the packet, so you do not necessarily treat all icmp packets as CS5).

lezz · November 15, 2018, 3:58pm

Ok then what do you suggest I put into the iptables?

No I run basicly nothing while gaming.

dlakelan · November 15, 2018, 5:37pm

I wonder if DSCP on egress causes the ISP to handle your packets better. I'd assume most ISPs ignore DSCP but you never know.

EDIT:

when you say it's "not good without the iptables" what do you mean? Can you be specific? Also, did you fix the sync issue? Without a good sync it's pointless to test.

lezz · November 15, 2018, 6:53pm

The sync seems to be fixed and I have a stable sync since 5 h. It s also back to 23mbit downlink.

With "not good" I mean that without your entries in the firewall, the hitreg with SQM is really bad (shots just go through etc.). As I am on a very high level in that game I can judge, feel and see the difference quite precise.

So right now I am back to the WAN LAN SQM solution. @moeller0: Does it make sense to insert the advanced settings at least into the WAN SQM?

One point where Im still clueless: With SQM activated the game sometimes have little micro stutters. It does not when I turn it off (tested it for 2-3 times).

moeller0 · November 15, 2018, 7:28pm

Have a look at the dsl errors, especially CRC errors as these always incurr packet loss...

Sure, on WAN these can stay as is, just on LAN you need to exchange the content of iqdisc_opts and eqdisc_opts as the directionality changes (so dual-srchost on LAN ingress and dual-dsthost on LAN egress).

I do not doubt your judgement there, it is just on a quit network these iptable rules should have not much effect (unless you use a wifi ling with WMM in that case CS5 should buy you some latency at a rather steep bandwidth cost...)

Well, if cake does not work for you, you might as well try the more conventional but computationally cheaper simple.qos/fq_codel combination (just make sure to clear the eqdisc_opts and iqdisc_opts as trying to feed cake keywords into fq_codel will lead to a silently aborted sqm start).

lezz · November 15, 2018, 7:57pm

Where can I check CRC errors in openwrt?

moeller0 · November 15, 2018, 9:27pm

You would need to look into the error-counters of your dsl-modem, on your router those CRC errors on the dsl link will only show up as TCP-retransmits or UDP packet loss (but the router does not know about any of these).

dlakelan · November 15, 2018, 10:27pm

fq_codel does not have the "layer" aspect (ie. DSCP bins) so I think that will not be a good solution for this case as the DSCP tags seem to make a difference.

If the ISP is doing something with the DSCP tags it might improve hitreg. Specifically perhaps the ISP is less likely to drop CS5 tagged UDP packets sent out of your network, and therefore the game server doesn't miss shots etc because the DSCP tags on outbound packets.

moeller0 · November 15, 2018, 11:15pm

moeller0:

Well, if cake does not work for you, you might as well try the more
conventional but computationally cheaper simple.qos/fq_codel
combination

fq_codel does not have the "layer" aspect (ie. DSCP bins) so I think
that will not be a good solution for this case as the DSCP tags seem to
make a difference.

Simple.qos uses a three tier HTB hierarchy that does support dscps. But you are right fq_codel alone does not.

moeller0:

I do not doubt your judgement there, it is just on a quit network these
iptable rules should have not much effect (unless you use a wifi ling
with WMM in that case CS5 should buy you some latency at a rather steep
bandwidth cost...)

If the ISP is doing something with the DSCP tags it might improve

When I measured some time ago, I saw traces of dscps being conserved over IPv6 but nothing on IPv4.

hitreg. Specifically perhaps the ISP is less likely to drop CS5 tagged
UDP packets sent out of your network, and therefore the game server
doesn't miss shots etc because the DSCP tags on outbound packets.

Could be, true. I would be amazed though if the relative low bandwidth game traffic would actually need such special treatment. BUT the fact that dspc work better than no dscps clearly shows that my intuition in this case is wrong, so I want to learn why....

lezz · November 16, 2018, 12:32am

Thx Again for your replies. Of Courage i also tried fq codel with simple.qos, it is worse than Layer cake.

I also tried again without The dscp in The policy Based qos and it Makes definitely a
Difference.

Another question: is there a possibility to turn of The Firewall comepletely? Just for sone testing. Behause when i Disabled it under Startup i coulldnt use The Internet.

dlakelan · November 16, 2018, 12:54am

Did you figure out how to put DSCP on the packets sent by your computer using Windows policy or is the router the only thing doing packet marking?

I too am interested in why it is that DSCP seems to help. I wonder if it isn't something to do with Windows network stack and latency in processing packets within the gaming computer? Perhaps windows sees DSCP on packets and causes delivery to happen more quickly to the game program.

lezz · November 16, 2018, 7:07am

Yes, it use policy based qos in Windows to Set cs5 Flags on udp Traffic for Ports 27000-27100. therefore also The outgoing packets Are Marked. Should I also Mark some icmp packets within The policy based qos?

Regarding The Windows Network Stack: what Could it Be?

moeller0 · November 16, 2018, 8:21am

Okay, how about you try the following combinations trying to figure out why and where dscps matter:

#baselines

base-line: no iptables (on the LAN interface), no windows policy no sqm
baseline with load: same as 1) but also run a speedtest and/or stream video on another computer

dscp only

like 1) but with your iptables and windows policy enabled
like 2) but with your iptables and windows policy enabled
Please note that if this is actually better than 1) or 2) this would implicate the windows network stack and you should try to separate out the effect of ingress and egress marking, by repeating these tests once with only windows policy egress dscps and once with only iptables

sqm only, we should be able to skip these since you did this test

like 1), but with sqm enabled on WAN (use the bidirection shaper)
like 2), but with sqm enabled on WAN (use the bidirection shaper)

split sqm only, we should be able to skip these since you did this test

like 1), but with split sqm enabled on WAN & LAN (two unidirectional shapers)
like 2), but with split sqm enabled on WAN & LAN (two unidirectional shapers)

sqm & dscp

like 5), but with iptables and windows policy
like 6), but with iptables and windows policy

sqm & dscp

like 7), but with iptables and windows policy
like 8), but with iptables and windows policy

And if the earlier no-sqm + dscp experiments did not already reveal which direction is more important for dscps you should try to separate out windows policy egress dscps from iptables ingress dscps.

After these tests, you should have a pretty good handle about the importance of the individual components of your set-up. Now, I am not sure how easy this is going to be, after all you would need to qualify all of these combinations while playing, and that might lead to less than ideal in-game performance (after all you set out to optimize these settings to avoid artifical handicaps while playing)...