Need help regarding SQM

Installing and Using OpenWrt Network and Wireless Configuration
1

Router: TP-Link TL-WR841N/ND v8 4MB ROM with 32 MB RAM (Using as a switch)
Firmware Version: OpenWrt 18.06.5
Kernel Version: 4.9.198
I play games on both LAN & Wi-Fi. I've set up SQM with respect to ipset.
I followed this guide: Ultimate SQM settings: Layer_cake + DSCP marks
My current configs are:
dnsmasq.conf:

ipset=/igamecj.com/gcloudcs.com/qos.gcloud.qq.com/tencentgames.helpshift.com/latsens
ipset=/.googlevideo.com/youtube.com/streaming
ipset=/live.mp3quran.net/tiktokv.com/media-imdb.com/v.redd.it/last.fm/sndcdn.com/deezer.com/audio-fa.scdn.cot/vevo.com/ttvnw.net/s3.ll.dash.row.aiv-cdn.net/d25xi40x97liuc.cloudfront.net/aiv-delivery.net/nflxvideo.net/av.alarabiya.net/streaming
ipset=/.fbcdn.net/.whatsapp.net/.whatsapp.com/usrcdn
ipset=/lol.secure.dyn.riotcdn.net/lol.dyn.riotcdn.net/lol-promos.s3.amazonaws.com/.amazonaws.com/l3cdn.riotgames.com/.akamaihd.net/ucy.ac.cy/hwcdn.net/windowsupdate.com/update.microsoft.com/share.box/akamaized.net/usrcdn
ipset=/.google.com/googletagmanager.com/.googleusercontent.com/google.com/.googleapis.com/1e100.net/usrcdn
ipset=/dropbox.com/dropboxstatic.com/dropbox-dns.com/log.getdropbox.com/gs2.ww.prod.dl.playstation.net/steamcontent.com/download.qq.com/bulk

firewall.user:

#!/bin/sh
IPT="iptables"
WANIF="pppoe-wan" #wan interface

##ipset for streming sites, etc; they are bening filled by dnsmasq
ipset create streaming hash:ip
ipset create usrcdn hash:ip
ipset create bulk hash:ip
ipset create latsens hash:ip

$IPT -t mangle -N dscp_mark > /dev/null 2>&1
$IPT -t mangle -F dscp_mark
## check if POSTROUTING already exits then jumps to our tables if not, add them

$IPT -t mangle -L POSTROUTING -n | grep dscp_mark || $IPT -t mangle -A POSTROUTING -j dscp_mark

iptmark(){
    $IPT -t mangle -A dscp_mark "$@"
}

## start by washing the dscp to CS0

iptmark -j DSCP --set-dscp 0

#A robust 2 rules to detect realtime traffic

# mark connections that go over 115 packets per second, not prioritized
iptmark -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 115/sec --hashlimit-burst 50 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"

# unmarked UDP streams with small packets get CS6
iptmark -p udp -m connmark ! --mark 0x55 -m multiport ! --ports 22,25,53,67,68,123,143,161,162,514,5353,80,443,8080,60001 -m connbytes --connbytes 0:940 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS6 -m comment --comment "small udp connection gets CS6"

#large udp streams like video call get AF41
iptmark -p udp -m connmark ! --mark 0x55 -m multiport ! --ports 22,25,53,67,68,123,143,161,162,514,5353,80,443,8080,60001 -m connbytes --connbytes 940:1500 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class AF41 -m comment --comment "large udp connection gets AF41"

########
##Browsing
########
## medium priority for browsing
iptmark -p tcp -m multiport --ports 80,443,8080 -j DSCP --set-dscp-class CS3 -m comment --comment "Browsing at CS3"

##################
#TCP SYN,ACK flows
##################
#Make sure ACK,SYN packets get priority (to avoid upload speed limiting our download speed)
iptmark -p tcp --tcp-flags ALL ACK -m length --length :128 -j DSCP --set-dscp-class CS3
iptmark -p tcp --tcp-flags ALL SYN -m length --length :666 -j DSCP --set-dscp-class CS3

#Small packet is probably interactive or flow control
iptmark -m dscp ! --dscp  24 -m dscp ! --dscp  18 -m dscp ! --dscp  34 -m dscp ! --dscp  40 -m dscp ! --dscp  48 -m length --length 0:500 -j DSCP --set-dscp-class CS3

#Small packet connections: multi purpose (don't harm since not maxed out)
iptmark -m dscp ! --dscp  24 -m dscp ! --dscp  18 -m dscp ! --dscp  34 -m dscp ! --dscp  40 -m dscp ! --dscp  48 -m connbytes --connbytes 0:250 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS3


########################################
# Streaming Media (videos/audios)
########################################
#Known video streams sites like netflix
iptmark -m set --match-set streaming src,dst -j DSCP --set-dscp-class AF41 -m comment --comment "video audio stream ipset"

# some iptv provider's use this port
iptmark -p tcp -m multiport --ports 1935,9982 -j DSCP --set-dscp-class AF41 -m comment --comment "some iptv streaming service"

#known usrcdn like google or akamai

iptmark -m set --match-set usrcdn src,dst -j DSCP --set-dscp-class AF21 -m comment --comment "usrcdn ipset"

#########################################
# Background Traffic (Bulk/file transfer)
#########################################
#bulk traffic ipset, like windows udates and steam updates/downloads
iptmark -p tcp -m set --match-set bulk src,dst -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"
iptmark -p udp -m set --match-set bulk src,dst -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"
iptmark -p tcp -m connbytes --connbytes 350000: --connbytes-dir both --connbytes-mode bytes -m dscp --dscp-class CS0 -j DSCP --set-dscp-class CS1 -m comment --comment "Downgrade CS0 to CS1 for bulk tcp traffic"
iptmark -p tcp -m connbytes --connbytes 350000: --connbytes-dir both --connbytes-mode bytes -m dscp --dscp-class CS3 -j DSCP --set-dscp-class CS1 -m comment --comment "Downgrade CS3 to CS1 for bulk tcp traffic"
iptmark -p udp -m multiport --port 60001 -j DSCP --set-dscp-class CS1 -m comment --comment "bulk torrent port UDP"

########################################
# Latency Sensitive (gaming/voip)
########################################
##ICMP, to prioritize pings
iptmark -p icmp -j DSCP --set-dscp-class CS5 -m comment --comment "ICMP-pings"

#DNS traffic both udp and tcp
iptmark -p udp -m multiport --port 53,5353,8888 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS udp"
iptmark -p tcp -m multiport --port 53,5353,8888 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS tcp"

#NTP
iptmark -p udp -m multiport --port 123 -j DSCP --set-dscp-class CS6 -m comment --comment "NTP udp"

#High priority ipset
iptmark ! -p tcp -m set --match-set latsens src,dst -j DSCP --set-dscp-class CS6 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset,udp

iptmark -p tcp -m set --match-set latsens src,dst -j DSCP --set-dscp-class CS5 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset

#########################################
# Rainbow Six Siege
#########################################
iptmark -p udp -m multiport --port 3074,6015,10000:10099 -j DSCP --set-dscp-class EF -m comment --comment "Rainbow Six Siege UDP" 
#########################################
# COD Mobile
#########################################
#iptmark -p udp -m multiport --port 88,500,3074,3544,3478:3479,4380,4500,27000-27036 -j DSCP --set-dscp-class CS6 -m comment --comment "COD" 
iptmark -p udp -m multiport --port 7500:7700 -j DSCP --set-dscp-class CS7 -m comment --comment "COD Ports"

#iptmark -p udp -m iprange --src-range 124.156.50.0-124.156.70.0 -j DSCP --set-dscp-class CS7 -m comment --comment "TEST"
#iptmark -p udp -m iprange --dst-range 124.156.50.0-124.156.70.0 -j DSCP --set-dscp-class CS7 -m comment --comment "TEST"

SQM:

config queue 'eth1'
	option linklayer 'none'
	option debug_logging '0'
	option verbosity '5'
	option qdisc_advanced '1'
	option egress_ecn 'NOECN'
	option qdisc 'cake'
	option qdisc_really_really_advanced '1'
	option upload '4900'
	option enabled '1'
	option interface 'pppoe-wan'
	option download '0'
	option squash_dscp '1'
	option squash_ingress '1'
	option iqdisc_opts 'diffserv8 nat ack-filter datacentre'
	option eqdisc_opts 'diffserv8 nat ack-filter datacentre'
	option ingress_ecn 'NOECN'
	option script 'layer_cake.qos'

config queue
	option debug_logging '0'
	option verbosity '5'
	option linklayer 'none'
	option qdisc 'cake'
	option script 'layer_cake.qos'
	option qdisc_advanced '1'
	option qdisc_really_really_advanced '1'
	option ingress_ecn 'NOECN'
	option egress_ecn 'ECN'
	option enabled '1'
	option interface 'br-lan'
	option download '0'
	option upload '9800'
	option squash_dscp '1'
	option squash_ingress '1'
	option iqdisc_opts 'diffserv8 nat ack-filter datacentre'
	option eqdisc_opts 'diffserv8 ingress nat ack-filter datacentre'

I've passed "datacentre" string in both ingress & egress advanced options. It gives me be better & faster peek/hitreg while playing FPS games.
So I got few questions/doubts.

  1. How this SQM can further be optimized theoretically in terms of least latency for playing games both on LAN & Wi-Fi. (I don't mind creating two separate optimized SQM files enabled one at a time specifically for either LAN or Wi-Fi).
  2. Does the following mean it disables SQM & it's advance parameters completely for ingress?

Download speed (kbit/s) (ingress) set to 0 to selectively disable ingress shaping

That means I don't have to pass any advanced option string in ingress although SQM will avoid it right?
3. I put upload speed in WAN egress. For this instance only, which queue setup script is better? layer_cake.qos OR piece_of_cake.qos
4. For both PPPOE-WAN & br-lan SQM instances what should I put?

Squash DSCP on inbound packets (ingress) = ?
Ignore DSCP on ingress = ?
Explicit congestion notification (ECN) status on inbound packets (ingress) = ?
Explicit congestion notification (ECN) status on outbound packets (egress) = ?
  1. Could someone explain me about this field called Hard limit on ingress/egress queues how can this affect packets in gaming? Will setting these fields to 18 result in better hitreg theoretically? (Does lower value affect wireless UDP connections?)
  2. Does changing RTT affect UDP traffic? is 200ms RTT too much? When I set it to 200 it gives me horrible peek/hitreg lag in LAN games. Not sure about Wi-Fi.
  3. In the guide I mentioned, will tagging gaming packets to CS7 work for the highest packet prioritization? OR It should be CS6 only? I mean 802.11 standard does specify CS7.

Thanks in advance & Merry Christmas to everyone!

2

Kindly and Respectfully,

Have you read the sqm documents provided on the main openwrt index?

3

Yes I've along with other posts. But the documentation doesn't say much when we create two instances of SQM to take advantage of ipset.

4

So that device is nearing its useful service life with recent OpenWrts, both due to low flash and low ram. But more importantly, what do you mean with "using as switch". Specifically which ports of the device are you using, which are connected to the real "modem/ISP-router" and which interfaces are connected to how many client devices?

For anything related to that post, you might want to bring in the experts, @dlakelan and @hisham2630 , I will be able to give a few generic answers for the SQM related questions, but anything detailed about the actual detailed qos scheme I will need to pass.

Your first sqm instance will only control your upload (as you set option download '0', which will not instantiate sqm for the download direction at all, as you can see from the lack of a cake instance for ifb4pppoe-wan in the output of tc -s qdisc), as you expect the whole option iqdisc_opts 'diffserv8 nat ack-filter datacentre' line becomes moot.
Regarding your [i|e]qdisc_opts' "ack_filter" really only makes sense on egress/upload if at all on ingress/download we recommend to leave ACK packets alone; "datacentre" will set both interval (and indirectly target) to values appropriate for very low RTT links, as one might encounter inside of a data center, for your typical WAN RTT in the 10-150ms range this will cost you throughput per individual flow. In effect datacentre will give you 5µs taget and 100µs interval instead of the typical internet-suitable 5ms and 100ms, so a factor of 1000 difference. If this has any effect on your gaming performance than by massively throttling all TCP flows, leaving sparse UDP flows less affected (I have a hard time believing that this has a measurable effect, as cake's flow-queueing and sparse flow boosting should already prioritize gaming traffic versus background TCP flows, unless due to your low total bandwidth the gaming traffic is not treated as sparse). option ingress_ecn 'NOECN' on ingress packets will already have passed the real bottleneck link before being handled by cake, dropping them at that point effectively wastes bandwidth, so that is why the default is to enable ECN on ingress, BUT cake will ignore the ECN settings anyway...
For the second instance on option interface 'br-lan' I note that typically sqm will not work well on a bridge interface at all, so maybe you want to spend time https://forum.openwrt.org/t/ultimate-sqm-settings-layer-cake-dscp-marks/ thread to learn about the dual veth approach to control ingress traffic while still allowing iptables. The comments about the xqdisc_opts from above apply here as well.

Do you have a way to actually quantify this?

Typically sqm should not be needed inside your LAN at all (unless you have a LAN internal bottleneck, like say a powerline link between two gigabit switches) and for WAN you need to optimize things as only you know which parameters you actually want to optimize.

Yes, the advances parameters are used to configure a specific sqm instance, if that instance is disabled, these parameters are not applied to anything. You can easily test this by comparing the output of tc -s qdisc with option download '0' versus say option download '9800' on pppoe-wan.

Correct.

There is no better, it depends on what you want to achieve, but in your case you should follow the advice in https://forum.openwrt.org/t/ultimate-sqm-settings-layer-cake-dscp-marks/ as mixing different QoS recipes typically is possible but requires careful evaluation to make things work as desired, so better to stick to one recipe initially and only start modifying that once you have a baseline performance established.

First, you probably should not instantiate sqm on br-lan at all.

This will re-map DSCP markings on incoming packets to 0, which will not work well with https://forum.openwrt.org/t/ultimate-sqm-settings-layer-cake-dscp-marks as there the goal is to properly mark packets with meaningfu DSCP marks before handing them to cake. In typical simple sqm configurations we typically do not trust the ISP to pass in meaningful DSVP marks and since mismarking can have undesirable side effects (thing 802.11 WMM prioritization) we default to re-map to zero.

This basically controls whether the ingress cake instance will actually differentiate between packets based on DSCP at all, setting this to 1 will change the ingress cake instance to single-tin best-effort, which seems not compatible with your larger QoS scheme.

These do not matter since cake does not allow to manually configure ECN (you can always disable ECN at your client machines at home, it will only be used if both ends of a connection support it).

This is a safety valve to avoid sqm eating too much memory under load, there should be no need to change that unless you encounter massive packet loss on a fast link (so not your's) ad even then you would only increase it. Again a word of caution about hitreg, if you have a way to hard and cold quantify hitreg you can run your own experiments (and I am sure some folks in this forum would be interested in empiric data) answering such questions for your self and your specific situation.

There is no RTT parameter in the sqm confifuration, so I am at a loss. But effectively fq_codel's/cake's interval parameter should be tailored to your typical RTT (roughly this is more an order of magnitude thing than a precise value), the default of 100ms work typically pretty well for real RTTs in the range of say 20 to 200ms. Have a look at https://tools.ietf.org/html/rfc8289 for some details about the theory behind target and interval in codel (which also effectively is also applicable to codel's descendants fq_codel and cake).

You should probably as that in the thread for that guide? Personally I would use neither CS6 or CS7 for game traffic, as on wifi with WMM (so all modern wifi) this will map into the AC_VO traffic class, which has low latency but also enough side-effects that i always would see wether AC_VI might not be good enough first, before resorting to AC_VO. If you do not use wifi at all this probably does not matter too much, assuming the priority tin in question has enough bandwidth for your traffic.

I would also respectfully endorse @mindwolf's idea of carefully going through the available documentation ...

3 Likes
5

Because that is not in the scope of sqm-scripts :wink: The idea behind sqm-scripts was and is not to create the most bad-asses custom tailored QoS scheme on the planet, but really just create a relative simple QoS scheme that does the right thing for most users, so the explicit goal is "good enough" not optimal. (One of the reasons is that we can create a generic "good enough" solution, but optimal solutions will always depend on the circumstances and individual preferences.

1 Like
6

Final note, with your apparent 10/5 Mbps link you are in a regime where things can become tight easily, sqm will not be able to perform miracles, so keep that in mind.

Also WiFi can be tricky and cause issues, say if your wifi connected laptop periodically tries to scan all other channels you might see periodic latency increases, so preferably game vie wired machines.

2 Likes
7

i suggest you:

  • enable sqm on ingress
  • not use the datacentre keyword
  • run sqm only on the pppoe (wan) interface
  • use a wired connection for gaming (srsly!)
2 Likes
8

I would also change the xqdisc_opts to:

	option iqdisc_opts 'nat dual-dsthost ingress'
	option eqdisc_opts 'nat dual-srchost'

and make sure to configure the correct per packet overhead.

9

I honestly can't thank you guys enough for your in-depth answers especially moeller0. As soon as I get free time I'll reply to you all! :smiley:

10

What game title are you referring to for hit registration?
Most game developers do not consult network engineers when creating these algorithms for lag compensation. Heck some of them couldn't answer what udp stands for. I recently throttled my connection to a 56k bandwidth limit to experiment. This particular title still allowed me to play instead of denying or kicking me from the server! My supposed connection bars stayed 4 bars and didn't flinch. The ignorant eye would assume my connection was decent. On my end it was a terrible experience. This is part of the bigger problem!

2 Likes
11

AFAIK if you're living near game servers then whatever connection speed/bandwidth you have it doesn't matter. People who are far away & get frequent network jitters & higher ping than 60ms it's better to consider network optimization & DSCP tagging approach. Hell I give it to game latency compensation but it has limitation. Compensation won't work unless you got ping less than 60ms. I play Rainbow Six Siege (ping ~70ms) & Call of Duty (ping ~40ms-120ms). So its unfortunate for people like us got nothing else other than optimizing on current region network e.g. SQM which helps a lot at least go against pro players to their levels.

12

Isn't default "triple-isolate" a better choice since it automatically handles fairness for both source & destination. tc-cake says

triple-isolate (default):
Flows are defined by the 5-tuple, and fairness is applied over source and destination addresses intelligently (ie. not merely by host-pairs), and also over individual flows. Use this if you're not certain whether to use dual-srchost or dual-dsthost; it'll do both jobs at once, preventing any one host on either side of the link from monopolising it with a large number of flows.

So how come specifying dual-srchost & dual-dsthost could be a better choice here?

13

So for most mixed multi-flow use cases triple and dual will be quite similar in fairness and triple does not need to factor the direction into the equation. The dualmodes however need to take direction into account, hence dual-srchost and dual-dsthost. The advantage of the dual configuration is that it can give you strict per-internal-IP address fairness, while triple only approximates that. So in corner cases it is not straight forward to predict how triple is going to behave. But in the end this is a policy decision you, as admin of your network/router, will have to make.

1 Like
14

It’s switch0 setup. Internet<---->THIS MAIN ROUTER (OpenWrt) <----> Wifi-router(Default TP-link firwmare)
VLAN: Port status 1/ eth1 tagged/ LAN1-LAN4 untagged
Connection type: PPPoE (So I guess there is no need to specify overhead. Cake handles it automatically?)

Correct. So ack-filter only for egress.

I agree default RTT 100ms is the best choice overall & I shouldn't change it. But the thing is that even without load due to default 100ms RTT enemies are able to peek & shoot me before I could even react. I don't have empirical statistics to prove but I did a lot of tests while playing lan game & found putting less than 30ms RTT makes it a fair match. I think it has to do with tight target so SQM has to send packets without waiting for 5ms more?

Got it :smiley:

Alright so for this setup Squash DSCP on inbound packets (ingress) should be SQUASH.

But here we're avoiding DSCP tags on ingress flow (ingress shaping is disabled on both SQM instances) hence this option won't affect result at all right?

So go with default options for both ingress/egress?

sch_cake: Add DiffServ handling document suggests to tag gaming packets to CS4. I tried that but the end result was horrible even in unloaded network. CS6/CS7 works better for me.

15

No not at all, you always need to specify the overhead, otherwise you will only get what the kernel might automatically add (0 bytes for pppoe). What kind of link technology are you using?

So depending on where you live in relation to the game servers it might make sense to change these, but given your numbers I would leave this at 100ms.

That is not how this parameter works, you basically just tell cake how long it should give each isolated flow to respond to a drop or ECN mark before dropping/marking more, if this is below the real RTT the flow will never reach full speed even if it would be the only active flow, setting it to high is more benign for well responding flows and for non-responsive flows the fq (flow queueing) isolation will make sure, that the increased latency coming from non responsive flows is only experienced by those flows themselves.

SQM will not really wait on purpose, these 5ms is just what cake will acccept as standing queue under saturating load (in cake you can not set this directly, so if you really need a shorter target you also need to reduce interval, which you will pay for with reduced throughput for TCP flows with longer RTT than interval, again a policy decision).

Policy decision, you need to make, but unless testing what shenanigans my ISP does to DSCP marks, I normally squash them.

It will make all the special DSCP assignments in your firewall.user moot for ingress...

Sure.

I would assume this comes from your firewall.user. It appears that you are trying to combine a standard sqm-scripts installation with the bespoke qos-scheme from https://forum.openwrt.org/t/ultimate-sqm-settings-layer-cake-dscp-marks/ that is a bit tricky.

I guess you need to decide whether you want to go all in with the ultimate scheme or start (at least initially) with a more run-of the mill "white-bread" sqm-scripts install ;). I would personally start with the white-bread approach, if only to be able to appreciate all the goodness the ultimate-scheme buys you (as its complexity cost is obvious). But again, your network, your decisions.

16

ISP have setup their switch at our house. Till switch fiber connection followed by cat5e Ethernet connection to my router. I'll need help with this one too to calculate overhead then.

So basically it'll only matter under load understood :smiley:

I'm honestly confused on this one. I made this easy diagram to represent packets flow through SQM instances & firewall. Please correct me.

Setup Diagram
Setup Diagram1102×203 22.1 KB

Firewall rules only apply right after SQM correct? The whole reason I went with this approach is that once incoming packets passes through either one of the SQM instances (ingress shaping selectively disabled) firewall rules (DSCP tagging) applies to all packets then goes to next SQM instance, here they are prioritize based on tags (thanks to layer.cake). Whereas, normally with basic single SQM instance for both ingress & egress prioritization of packets never happens inside the queue since iptable rules apply after SQM & not before it. Correct?

17

Your diagram is more or less correct, but ... and this is key... br-lan itself doesn't do any queueing, only the individual ethernet and wifi interfaces queue. So putting a SQM instance on it doesn't help.

in order to avoid this problem, I invented the veth combination. One of the veths is in the bridge, and the other is a kind of "injector" that can queue. So you modify the inbound route to send packets to veth0 where they queue, and then are sent to veth1 which receives them in the bridge and br-lan bridges them to the ethernet or wifi.

@moeller0, I am not really up for creating a package to instantiate my hfsc tuned QoS with firewall rules etc and a LUCI config... but if you know someone who has the skills and time to do that, I'd be happy to provide consulting assistance. Any ideas of someone we should tag?

2 Likes
18

But in Ultimate SQM settings: Layer_cake + DSCP marks OP says at the bottom

For routers that have Switch0 interface, you don't need to use veth method!

Mine is switch0 interface

Capture
Capture971×468 36.7 KB

19

it's not the name switch0 that is the issue, he's talking about DSA architecture for switch, which is different from your device.

2 Likes
20

Oh thanks for letting me know :smiley: I'll build a new image asap

1 Like