Ultimate SQM settings: Layer_cake + DSCP marks

Yea true, gaming packets and voip usually dont max out the payload. Perhaps just tag packets based on its payload size would actually work well with QoS cake? Dunno, just thinking out loud here :stuck_out_tongue:

I'm just wondering how you could make general rules and optimize it all for general use without having to resort to specifically aiming at applications/games separately.

This will make rules more effective also you will end have a few rules to maintain and understand.more rules
will use more cpu and maybe will make it useless.
it's just the knowledge and testing (try/error), i do packet capturing to see how it's work!

Hello everyone! I came across this post not too long ago and read a couple of the posts in the thread and I have became very interested in setting up dscp markings for my network. But I have no idea whatsoever on how to begin, like which packages to download and how to start marking the packets. Please if you guys can help me out on setting up dscp marking I would really appreciate it. And thanks @dlakelan and @hisham2630 for helping everyone out and contributing to help people achieve the best latency in their networks! Keep up the awesome work!

Hi @Ernieelias welcome to the party :slightly_smiling_face: first thing you should do is install SQM and tune it to your connection speed. Already this will go a long way toward making games and VOIP work better. Everything else here is second order improvement.

Start by running a speed test on dslreports and post the link to the results page here. Then using the Luci app for sqm enable an sqm on your wan interface with about 90% of measured download speed and 95% of measured upload. Redo the test and post that. Once you are at that point we can discuss next steps.

Thanks! Ah yes sorry, I have forgot to mention that I do have sqm installed on Wrt3200acm and I have a wrt1900acv2 setup as a dumb ap. Did you still want me to post the settings of sqm?

Ok, post a speed test result with your current SQM.

If you do not mind also post the output of:

  1. cat /etc/config/sqm

  2. tc -s qdisc #(directly before a speedtest)

  3. tc -s qdisc #(directly after the same speedtest)

Reading @easyteacher's rule it seems to be connmarking connections based on them having a rate between 15 and 128 packets per second. I'm not quite clear on how it prevents connections from bouncing back and forth if they're near to the edge of those ranges, but maybe it doesn't matter. Doesn't seem to take packet size into account at all which could be fine too as long as you're in that range of packets per second the max bandwidth is 1500 bytes * 128 packets/s = 1.5 Mbps. So if you have at least 15Mbps then you're probably fine ignoring packet size. Only issue is if you get a LAN party going with 20 people all wanting to play CS:GO and wind up clogging your entire pipe :slight_smile:

Here is without sqm on

Here is my result for the speed test with sqm on

Now I'm not sure what's going on with the fluctuations in the speed but typically without sqm on I would hit around 800 mbps for download and around higher 800 mbps for upload. I'm not sure if its the ethernet cable is to blame or not. and unfortunately I don't have another pc or laptop to test it on...

What is your speed setting for sqm I agree your results are strange.

Speed setting for download is set to 0
Upload speed is set to 20000kbps

I suspect the wrt3200 will handle around 250Mbps to 300Mbps shaping without running out of CPU. Try setting it to up and down speeds of 200 and watch top -d 1 for the percentage idle, it should stay above say 10% to get good performance, from there bump it up in speed until you get to around 10% idle and then show that speed test result.

20000 is 20 Mbps you need an extra zero :grinning:

Set 200000 both directions. Download may be suffering due to bufferbloat.

Connbyte with avgpkt has already taken packet length into account. Specifically, upload and download traffic need to be separately considered.

Take LOL for example.
ipv4 2 udp 17 179 src=192.168.1.121 dst=182.254.35.166 sport=8786 dport=7080 packets=55483 bytes=4841699 src=182.254.35.166 dst=172.27.77.225 sport=7080 dport=8786 packets=65694 bytes=12491560 [ASSURED] mark=80 zone=0 use=2

Average upload packet length: 4841699/55483≈87
Average download packet length: 12491560 / 65694≈190
Games like DNF will have larger download packet length around 300-400.

I am using connmark to avoid those connections whose rates are bouncing back and forth. The following two rules are used to save CPU cycles by only checking connections with no connmark and remove connmark with a probability of 0.5% in order to let some connections checked again later.

$ipt -A PREROUTING -m conntrack --ctstate RELATED,ESTABLISHED -m connmark ! --mark 0xf0/0xf0 -m connmark ! --mark 0x60/0xf0 -m connmark ! --mark 0x70/0xf0 -m statistic --mode random --probability 0.005 -j CONNMARK --and-mark 0xff0f
$ipt -A PREROUTING -i br-lan -m connmark --mark 0x0/0xf0 -j mark_DSCP_ul

That rule seems reasonably good provided you've tested it and it works with your games. It seems pretty specialized, the one @hisham2630 and I tried is going to be somewhat more generic, will catch VOIP, most games, and probably catch a few other things, maybe some of which shouldn't be gotten, but I think in general it's pretty robust. Yours seems to be tuned more tightly.

The idea of a low-probability of resetting the connmark is maybe a good one in case stuff hits a threshold briefly, and never will come back. probability 0.005 gives you something like a 50% probability of getting reset after 1 second at 100 pps. So that seems reasonable you won't bounce back and forth too much.

@dlakelan
Hi
Have a look here, he used connection rate and connection bytes for some games, messengers and other services; this may help us
detect more traffic correctly!

I think that qualifies as too baroque for my tastes :wink: I really like our rate/size filter, and a few ipsets or proxy rules... at this point because I have much faster bandwidth than you it's probably just fine for me. I understand when you have limited bandwidth it can be worth it to try to squeeze better responsiveness out of it. I still don't think anything like that ruleset is a good idea, all those manually entered ip addresses etc... not good.

In your current system, what issues do you still face? delay? stuttering in games/streams? voip breakup?

2 Likes

I'm like you, i don't like a lot of rules(why i should have a bunch of rules, when one rule can do everything).
i like keep things simple and tidy, ip and port based prioritization is not good most of times and eat a lot
of cpu, also it's hard to maintain. but what i want to say from that link is to get some idea about how he did
used connbytes and connection rate to prioritize important services and some games without using ip' or
ports.

Games and Voip's is fine, even i don't have a problem with two voip running at the same time while download
and upload is running, the only problem is delay when do web browsing or when opening some apps
like youtube, it will take some time to load, i think you didn't had a look at this post
download won't fall into bulk connbytes (i see only a few packets).
i tried to use those two rules to speed up browsing and other things:

$IPT -t mangle -A PREROUTING -m length --length 0:500 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -m connbytes --connbytes 0:250 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS3

It's not specific to tcp or udp, i think it's need some tuning, also some VOIP will fall into the second rule!

@dlakelan I see it's better to isolate rules, so they won't interact with each other, because i see that sometime youtube packets tagged with CS1 instead of AF41, to fix the problem i will use something like this:

$IPT -t mangle -A PREROUTING -m set --match-set vidstream src,dst -j CONNMARK --set-mark 0x41 -m comment --comment "vidstream ipset"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x41 -j DSCP --set-dscp-class AF41

$IPT -t mangle -A PREROUTING -p tcp -m connmark ! --mark 0x41 -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887,27014:27050 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x41 -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1

Yes, I couldn't make heads or tails of those images on my phone :wink: screen too small. I'm on my computer now, but still dont' know what I'm looking for. Do I understand that you have a rule to down-prioritize large total byte flows like CS1 for background downloads and it's not matching? This would be in the last image right?

Your image: https://forum.openwrt.org/uploads/default/original/2X/3/3aa2dfa0c51a46808ffefa40bb8b3e97bad2b06c.png

Seems to indicate about 300 MB of total traffic marked using those rules, so what was it you were thinking should have happened?

I notice you're marking 1.3 GB of TCP flags, so that seems probably not ideal. but it may be remarked by later rules anyway, like the rule that marks port 80/443 etc "priority for faster browsing-tcp"

Here's what I suggest: simplify your rule set for testing down to this:


########################################
# Latency Sensitive (gaming/voip)
########################################
##ICMP, to prioritize pings
$IPT -t mangle -A PREROUTING -p icmp -j DSCP --set-dscp-class CS5 -m comment --comment "ICMP-pings"

#DNS traffic both udp and tcp
$IPT -t mangle -A PREROUTING -p udp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS udp"
$IPT -t mangle -A PREROUTING -p tcp -m multiport --port 53,5353 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS tcp"

#High priority ipset, i use for pubgM
$IPT -t mangle -A PREROUTING -p udp -m set --match-set latsens src -j DSCP --set-dscp-class CS6  -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset

#A robust 2 rules to detect realtime traffic
$IPT -t mangle -A PREROUTING -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 120/sec --hashlimit-burst 50 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x55 -m conntrack --ctorigsrc 192.168.1.0/24 -m multiport ! --ports 53,5353,80,443,8080,60887 -m connbytes --connbytes 0:700 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS6 -m comment --comment "small udp connection"

########################################
# Streaming Media (videos/audios)
########################################
#Know video streams sites like youtube,netflix
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src,dst -j CONNMARK --set-mark 0x41 -m comment --comment "vidstream ipset"
$IPT -t mangle -A PREROUTING -m connmark --mark 0x41 -j DSCP --set-dscp-class AF41

##set dscp tag for our ipset usrcdn #those cdn's have an isolated speeds by my ISP
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src -j DSCP --set-dscp-class AF21 -m comment --comment "usrcdn ipset"

# some iptv provider's use this port
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 1935 -j DSCP --set-dscp-class AF41 -m comment --comment "some iptv streaming service"

#########################################
# Background Traffic (Bulk/file transfer)
#########################################
#bulk traffic ipset, like windows udates and steam updates/downloads
$IPT -t mangle -A PREROUTING -m set --match-set bulk src -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"

#A robust rule to detect downloads/uploads and torrents!

$IPT -t mangle -A PREROUTING -p tcp -m connmark ! --mark 0x41 -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887,27014:27050 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m connmark ! --mark 0x41 -m multiport --ports 21,25,80,81,443,444,554,8000,8080,8409,60887 -m connbytes --connbytes 500000: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1

I just removed some of the ack rules and browsing rules... I included your modifications to the connmark to prevent downgrading YouTube etc.

I think upgrading browsing and acks... may be a bad idea. Those seem like they should be "best effort" unless you are really starving acks, but even so I would use CS3 for acks not higher if you're using diffserv8.

1 Like

Np, i don't like small screens, that's why i always answering from computer.:upside_down_face:
Those large downloads rules is not working properly sometimes and only a few MB fall into connbytes TCP,
while udp connbytes, i don't know why it's have a big MB cause downloads usually use TCP (maybe youtube)
fallen into connbytes udp.
I think it's a bad idea to marking big tcp flags, that's why i comment it.

Thanks, but i'm planing to give a browsing some priority like CS2 or CS3, this will help speedup browsing and
opening apps login's etc ?!.
I will prioritize only small ACK's !, and i think it's better to use connmark !--mark xxx to avoid downgrading and
interact between rules :wink: