Ultimate SQM settings: Layer_cake + DSCP marks

should i delete the /root/net.sh if /root/qos.sh script is active? in my /etc/config/sqm should i change my Overhead to 18 for all the queue? because i have a cable connection from Xfinity/Comcast. after adding all these scripts, DSLreport test only gives less than 15 mbps download speed, upload speed is fine but there are still some spikes on the bufferbloat on the upload speed.

i only have limited knowledge on these things, i was in touch with hisham2630 and he have been helpful answering all my question, i thought i will share things here, as this could benefit others as well. please take a look at my setting and see if am doing this all wrong the whole time. my Wan/Wan6 is on eth1.2, my internet speed is 60 Down/ 5 Up.

/root/net.sh

IPT="iptables"

####Latency Sensitive Ports
##league of legends,PUBGM,some voip service like whatsapp.
sport1="37,53,81,88,123,500,1500,1863,3074:3076,3478:3481,3544,3659,4244"
sport2="4379,4380,4500,5222,5223,5228,5242,5000:5500,7985,8000,8888,9339,9785,34784"
sport3="10000:10300,19132,25565,27000:27030,28960,59172,39499,34141,59437:59581,45395,50318,59234"
dport1="37,53,81,88,123,500,1500,1863,3005,3074:3076,3101,3478:3481,3544"
dport2="3659,4244,4379,4380,4500,5222,5223,5228,5242,5000:5500,7985,8000,8888,34784"
dport3="9339,9785,10000:10300,19132,25565,27000:27030,28960,59172,39499,34141,59437:59581,45395"
dport4="1119,3724,7000:7999,8011,8081:8090,9030,17000,50318,59234"

## set up a 2 pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth

## get new veth interfaces up
ip link set veth0 up
ip link set veth1 up

## trun on promisc mode,sometimes it's needed to make bridge work
ip link set veth1 promisc on

## add veth1,veth3 to bridge
brctl addif br-lan veth1

## just to make sure there's nothing inside those 2 tables
ip rule del priority 100
ip route flush table 100

##ipset for streming sites.they are bening filled by dnsmasq
ipset create vidstream hash:ip
ipset create usrcdn hash:ip
ipset create bulk hash:ip
ipset create latsens hash:ip

## flush mangle table
$IPT -t mangle -F PREROUTING

## add routing for veth0 this will handle all slow traffic
ip route add default dev veth0 table 100
ip rule add iif pppoe-wan table 100 priority 100

#Clear interface dscp marks, we don't trust ISP marks(also to use our own marks).
$IPT -t mangle -A PREROUTING -i pppoe-wan -j DSCP --set-dscp 0

########################################
# Latency Sensitive (gaming/voip)
########################################
##Latency Sensitive
$IPT -t mangle -A PREROUTING -p icmp -j DSCP --set-dscp-class CS6 ##dscp tag for ping packets

$IPT -t mangle -A PREROUTING -m set --match-set latsens src -j DSCP --set-dscp-class CS6 ## set dscp tag for our Latency Sensitive (latsens) ipset

$IPT -t mangle -A PREROUTING -p tcp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4" #i don't have PS4, change the ip according to your ip setting.
$IPT -t mangle -A PREROUTING -p udp -m conntrack --ctorigsrc 192.168.1.150 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4"

$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport1 -j DSCP --set-dscp-class CS6 
$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport2 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport3 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p tcp -m multiport --sport 22,2099,4244,5222,5223,5228,5242,8088,8393:8400,50318,59234 -j DSCP --set-dscp-class CS6

$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport1 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport2 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport3 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport4 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 22,2099,3013,4244,5222:5228,5242,10012,17000,17500,8088,8393:8400,50318,59234 -j DSCP --set-dscp-class CS6

########################################
# Streaming Media (videos/audios)
########################################
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src -j DSCP --set-dscp-class AF41 ##set dscp tag for our ipset vidstearm
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src -j DSCP --set-dscp-class AF21 ##set dscp tag for our ipset usrcdn #those cdn's are isolated by my ISP.
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 1935 -j DSCP --set-dscp-class AF41

########################################
# Best Effort (browsing/services)
########################################
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 80,443,853,3455,8080,27015 -j DSCP --set-dscp-class AF13
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 80,443,853,3455,8080 -j DSCP --set-dscp-class AF13

#########################################
# Background Traffic (Bulk/file transfer)
#########################################
$IPT -t mangle -A PREROUTING -m set --match-set bulk src -j DSCP --set-dscp-class CS1 ##set dscp tag for our bulk ipset
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 21,80,443,8080,60887,27014:27050 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 21,80,443,8080,60887 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1


/root/qos.sh

#!/bin/bash -x
IPT="iptables"
WANIF="eth1.2" #wan interface

tc qdisc add dev wlan0 root mq #setup multi queue for wifi device
## set up a pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth

## get new veth interfaces up
ip link set veth0 up
ip link set veth1 up

## trun on promisc mode,sometimes it's needed to make bridge work
ip link set veth1 promisc on

## add veth1 to bridge
brctl addif br-lan veth1

## just to make sure there's nothing inside those 2 tables
ip rule del priority 100
ip route flush table 100

##ipset for streming sites.they are bening filled by dnsmasq
ipset create streaming hash:ip
ipset create usrcdn hash:ip
ipset create bulk hash:ip
ipset create latsens hash:ip


## add routing for veth0 this will handle all slow traffic
ip route add default dev veth0 table 100
ip rule add iif $WANIF table 100 priority 100

$IPT -t mangle -N dscp_mark > /dev/null 2>&1
$IPT -t mangle -F dscp_mark
## check if POSTROUTING already exits then jumps to our tables if not, add them

$IPT -t mangle -L POSTROUTING -n | grep dscp_mark || $IPT -t mangle -A POSTROUTING -j dscp_mark

iptmark(){
    $IPT -t mangle -A dscp_mark "$@"
}

## start by washing the dscp to CS0

iptmark -j DSCP --set-dscp 0

#A robust 2 rules to detect realtime traffic

# mark connections that go over 115 packets per second, not prioritized
iptmark -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 115/sec --hashlimit-burst 50 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"

# unmarked UDP streams with small packets get CS6
iptmark -p udp -m connmark ! --mark 0x55 -m multiport ! --ports 22,25,53,67,68,123,143,161,162,514,5353,80,443,8080,60001 -m connbytes --connbytes 0:940 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS6 -m comment --comment "small udp connection gets CS6"

#large udp streams like video call get AF41
iptmark -p udp -m connmark ! --mark 0x55 -m multiport ! --ports 22,25,53,67,68,123,143,161,162,514,5353,80,443,8080,60001 -m connbytes --connbytes 940:1500 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class AF41 -m comment --comment "large udp connection gets AF41"

########################################
# Latency Sensitive (gaming/voip)
########################################
##ICMP, to prioritize pings
iptmark -p icmp -j DSCP --set-dscp-class CS5 -m comment --comment "ICMP-pings"

#DNS traffic both udp and tcp
iptmark -p udp -m multiport --port 53,5353,8888 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS udp"
iptmark -p tcp -m multiport --port 53,5353,8888 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS tcp"

#NTP
iptmark -p udp -m multiport --port 123 -j DSCP --set-dscp-class CS6 -m comment --comment "NTP udp"

#High priority ipset, i use for pubgM
iptmark ! -p tcp -m set --match-set latsens src,dst -j DSCP --set-dscp-class CS6 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset,udp

iptmark -p tcp -m set --match-set latsens src,dst -j DSCP --set-dscp-class CS5 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset

########
##Browsing
########
## medium priority for browsing
iptmark -p tcp -m multiport --ports 80,443,8080 -j DSCP --set-dscp-class CS3 -m comment --comment "Browsing at CS3"

##################
#TCP SYN,ACK flows
##################
#Make sure ACK,SYN packets get priority (to avoid upload speed limiting our download speed)
iptmark -p tcp --tcp-flags ALL ACK -m length --length :128 -j DSCP --set-dscp-class CS3
iptmark -p tcp --tcp-flags ALL SYN -m length --length :666 -j DSCP --set-dscp-class CS3

#Small packet is probably interactive or flow control
iptmark -m dscp ! --dscp  24 -m dscp ! --dscp  18 -m dscp ! --dscp  34 -m dscp ! --dscp  40 -m dscp ! --dscp  48 -m length --length 0:500 -j DSCP --set-dscp-class CS3

#Small packet connections: multi purpose (don't harm since not maxed out)
iptmark -m dscp ! --dscp  24 -m dscp ! --dscp  18 -m dscp ! --dscp  34 -m dscp ! --dscp  40 -m dscp ! --dscp  48 -m connbytes --connbytes 0:250 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS3


########################################
# Streaming Media (videos/audios)
########################################
#Known video streams sites like netflix
iptmark -m set --match-set streaming src,dst -j DSCP --set-dscp-class AF41 -m comment --comment "video audio stream ipset"

# some iptv provider's use this port
iptmark -p tcp -m multiport --ports 1935,9982 -j DSCP --set-dscp-class AF41 -m comment --comment "some iptv streaming service"

#known usrcdn like google or akamai

iptmark -m set --match-set usrcdn src,dst -j DSCP --set-dscp-class AF21 -m comment --comment "usrcdn ipset"

#########################################
# Background Traffic (Bulk/file transfer)
#########################################
#bulk traffic ipset, like windows udates and steam updates/downloads
iptmark -p tcp -m set --match-set bulk src,dst -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"
iptmark -p udp -m set --match-set bulk src,dst -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"
iptmark -p tcp -m connbytes --connbytes 350000: --connbytes-dir both --connbytes-mode bytes -m dscp --dscp-class CS0 -j DSCP --set-dscp-class CS1 -m comment --comment "Downgrade CS0 to CS1 for bulk tcp traffic"
iptmark -p tcp -m connbytes --connbytes 350000: --connbytes-dir both --connbytes-mode bytes -m dscp --dscp-class CS3 -j DSCP --set-dscp-class CS1 -m comment --comment "Downgrade CS3 to CS1 for bulk tcp traffic"
iptmark -p udp -m multiport --port 60001 -j DSCP --set-dscp-class CS1 -m comment --comment "bulk torrent port UDP"


#tcpdump rule, copy and paste this rule into terminal, this rule is used to capture realtime traffic, you can change ip to what you like
#tcpdump -i br-lan host 192.168.1.126 and udp and portrange 1-65535 and !port


 /etc/dnsmasq.conf

ipset=/igamecj.com/gcloudcs.com/qos.gcloud.qq.com/tencentgames.helpshift.com/latsens
ipset=/.googlevideo.com/youtube.com/streaming
ipset=/live.mp3quran.net/tiktokv.com/media-imdb.com/v.redd.it/last.fm/sndcdn.com/deezer.com/audio-fa.scdn.cot/vevo.com/ttvnw.net/s3.ll.dash.row.aiv-cdn.net/d25xi40x97liuc.cloudfront.net/aiv-delivery.net/nflxvideo.net/av.alarabiya.net/streaming
ipset=/.fbcdn.net/.whatsapp.net/.whatsapp.com/usrcdn
ipset=/lol.secure.dyn.riotcdn.net/lol.dyn.riotcdn.net/lol-promos.s3.amazonaws.com/.amazonaws.com/l3cdn.riotgames.com/.akamaihd.net/ucy.ac.cy/hwcdn.net/windowsupdate.com/update.microsoft.com/share.box/akamaized.net/usrcdn
ipset=/.google.com/googletagmanager.com/.googleusercontent.com/google.com/.googleapis.com/1e100.net/usrcdn
ipset=/dropbox.com/dropboxstatic.com/dropbox-dns.com/log.getdropbox.com/gs2.ww.prod.dl.playstation.net/steamcontent.com/download.qq.com/bulk


/etc/sysctl.conf 

# Defaults are configured in /etc/sysctl.d/* and can be customized in this file
#new
net.ipv6.conf.default.forwarding=1
net.ipv6.conf.all.forwarding=1

net.netfilter.nf_conntrack_acct=1
net.netfilter.nf_conntrack_checksum=0
net.netfilter.nf_conntrack_max=16384
net.netfilter.nf_conntrack_tcp_timeout_established=7440
net.netfilter.nf_conntrack_udp_timeout=60
net.netfilter.nf_conntrack_udp_timeout_stream=180

#new settings to improve ping response and lowering ping spikes
net.core.netdev_budget=300
net.core.netdev_max_backlog = 1000

#ecn
net.ipv4.tcp_ecn=2
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_dsack=1


/etc/iproute2/rt_dsfield

# Differentiated field values
# These include the DSCP and unused bits
0x0     default
# Newer RFC2597 values
0x28    AF11
0x30    AF12
0x38    AF13
0x48    AF21
0x50    AF22
0x58    AF23
0x68    AF31
0x70    AF32
0x78    AF33
0x88    AF41
0x90    AF42
0x98    AF43


/etc/config/sqm

config queue 'wan'
	option ingress_ecn 'ECN'
	option egress_ecn 'ECN'
	option enabled '1'
	option debug_logging '0'
	option verbosity '5'
	option qdisc 'cake'
	option qdisc_advanced '1'
	option qdisc_really_really_advanced '1'
	option squash_dscp '0'
	option squash_ingress '0'
	option download '0'
	option script 'layer_cake.qos'
	option linklayer 'ethernet'
	option linklayer_advanced '1'
	option tcMTU '2047'
	option tcTSIZE '128'
	option tcMPU '64'
	option interface 'eth1.2'
	option upload '10000'
	option eqdisc_opts 'diffserv4 nat dual-srchost rtt 170ms'
	option overhead '50'
	option linklayer_adaptation_mechanism 'cake'
	option iqdisc_opts 'diffserv4 nat dual-dsthost rtt 170ms autorate-ingress'

config queue
	option debug_logging '0'
	option verbosity '5'
	option ingress_ecn 'ECN'
	option tcMTU '2047'
	option tcTSIZE '128'
	option enabled '1'
	option download '0'
	option qdisc 'cake'
	option qdisc_advanced '1'
	option squash_dscp '0'
	option squash_ingress '0'
	option egress_ecn 'ECN'
	option qdisc_really_really_advanced '1'
	option linklayer 'ethernet'
	option linklayer_advanced '1'
	option tcMPU '64'
	option linklayer_adaptation_mechanism 'cake'
	option script 'layer_cake.qos'
	option interface 'veth0'
	option upload '30000'
	option eqdisc_opts 'diffserv4 nat dual-dsthost rtt 170ms'
	option iqdisc_opts 'diffserv4 nat dual-srchost rtt 170ms autorate-ingress'
	option overhead '54'

config queue
	option debug_logging '0'
	option verbosity '5'
	option qdisc 'cake'
	option qdisc_advanced '1'
	option ingress_ecn 'ECN'
	option qdisc_really_really_advanced '1'
	option enabled '1'
	option squash_dscp '0'
	option squash_ingress '0'
	option egress_ecn 'ECN'
	option interface 'eth0'
	option upload '16000'
	option script 'piece_of_cake.qos'
	option iqdisc_opts 'dual-srchost'
	option eqdisc_opts 'dual-dsthost'
	option linklayer 'ethernet'
	option overhead '8'
	option linklayer_advanced '1'
	option tcMTU '2047'
	option tcTSIZE '128'
	option tcMPU '64'
	option linklayer_adaptation_mechanism 'cake'
	option download '0'

This is a highly customized firewall built with iptables and I am assuming this was built in a linux environment so I cannot offer assistance with this. There a lot of the modules but certain things like this udp packet prioritization I can offer some suggestion - after some thought and consideration as @moeller0 has stated it is easier to get lost in the lines than to actually produce results especially on a custom firewall which is not stable yet. I would suggest just to avoid this for now because it definately requires some testing and imho is too precise for general usage and could break tins.

About the iptables rules there are so many - why not attempt to organize by tin instead? Just some thoughts, but if building in a linux environment and working some experimental variables - why not keep things simple first? I should mention I am not a fan of 8 tin systems for latency or entire configs with every aspect of the network with its own iptable rule applied.

Don't use the old script which is mainly based on port prioritization, so keep only the new one qos.sh

Select the overhead that suite's your connection type!

As i said before use the new script that i shared with you!

1 Like

got this error:

sysctl: error: 'net.bridge/bridge-nf-call-arptables' is an unknown key
sysctl: error: 'net.bridge/bridge-nf-call-ip6tables' is an unknown key
sysctl: error: 'net.bridge/bridge-nf-call-iptables' is an unknown key
sysctl: error: 'net.bridge/bridge-nf-filter-vlan-tagged' is an unknown key
sysctl: error: 'net.bridge/bridge-nf-filter-eth1/2-tagged' is an unknown key

this is not a problem, you have those in your sysctl.conf?
just skip those errors!

Yes they are in sysctl.conf, I followed few other steps to install other scripts that were mentioned by the person who had the same issue. I also made the same firewall rules from that picture. Skip? Am assuming just delete these scripts and keep the other one in there?

Thanks

now just try qos.sh (which is not based on ports!)
use dnsmasq.conf
tell me about results!

The cake qdisc keyword "wash" = "squash"?

Don't know if anyone is following/using my 'store DSCP into firewall marks on egress and restore on ingress to exercise CAKE's diffserv tins ctinfo/CONNMARK' tweaks but I fell over and solved a problem with the tc filter incantation in my script yesterday.

Due to incorrectly using 'flowid 1:1', flow isolation on egress was broken. The challenges of copy/pasting code! Code offered in case it helps someone out.

Edited later to add some more info:

Symptoms of the error were interesting/odd. The visible symptom was that downloads weren't going as quickly as I'd expect in the face of simultaneous uploads (in this case to Onedrive) My NAS was doing a backup and in the process creating 5 upstream flows, downloads were being restricted to just over half download bandwidth. All the flows were being classified as 'bulk' as per my rules. Downloads that weren't classified as bulk went at full speed...but in the face of no competing traffic everything should have just run at full rate. Technical symptoms were displayed in the tc -s qdisc show dev stats, namely that on the egress interface bulk flow counts were only indicating '1' flow per tin, when clearly there was more than 1. This only affected the egress side of things. Removing the 'flowid 1:1' or indeed changing it to 'flowid 1:' stopped tc filter overriding the flow ID, which meant cake could identify each individual flow, restore it's bulk/sparse flow accounting and just start doing things properly.

Flow handling & isolation is now much, much better!

3 Likes

Why not use cake's new fwmark feature and skip the dscp to connmark part?

Because some applications are already DSCP aware, ie. they set a suitable DSCP value that I would like to retain. To do so you have to map the DSCP to tin value and store that in the fwmark, ie. you have to know the internals of CAKE, which change depending on diffserv mode.

I wanted to retain the DSCP value as the 'key' into CAKE's existing DSCP>tin mapping.

Good point but I wouldn't rely on applications setting the right dscp value.
It also allows users on your network to "abuse" dscp values.

Thanks a lot for high-lighting this forgotten corner of the actual sqm scripts! I will have a look at this issue and run a few tests whether 'flowid 1:1 is actually ever required for any of the scripts, this comes from early times and I for one simply copied the if instantiation parts over into new scripts as I was happy to accept this as settled and side-effects free. Now, it turns out that the second is not true, so a re-visit seems merited. I will also look at the priority assignment while I am at it.
Now, I will not promise an ETA since I am constantly out of time...

1 Like

Well, in a semi-trusted environment like the home network, accepting endpoint set DSCPs as probably correct and intended seems as a first measure defensible to me, especially since it is not that hard to actually enforce/sanitize internally set DSCPs before cake sees them.

Comes down to personal preference, I guess.

By design cake puts all ARP packets into the high prio tin.
On docsis, there can be a lot of arp packets.
Doesn't this screw things up a bit?
Those packets get dropped anyway but will first go through cake.

When experimenting with the fwmark feature,
I made a simple patch to let cake ignore the dscp values (and ARP packets)
but 4 packets hit the high prio tin. How is this possible?

If any help 'flowid 1:' works and eliminates 'flowid ???' in tc filter show dev eth'n' output.

will do, i still use all the other scripts such as sqm etc? or you really meant to say is to use qos.sh and dnsmasq.conf only and delete all the other older scripts that you have mentioned in your first post. currently am running these two scripts as well as Sqm scripts, so far i have had mixed results and still trying to figure out the best setting for sqm, i seems to notice more changes to my gaming when i adjust the sqm setting. tell me more about diffserv4 vs diffserv8, do i need to change anything special here or can i just delete the 4 and replace it with 8?

i sometime seems to get better results with fq_codel compare to Cake, am not sure the exact reason but in most cases the game seems to play smooth.

can i change these for this test:
option qdisc 'cake' to option qdisc 'fq_codel'
option script 'simple.qos'
option linklayer_adaptation_mechanism 'default
'

you can just play with sqm settings.
my script will work with cake and layer_cake, not sure how it will act with fq_codel

yes you can try that.

try to play with sqm until you find the best settings for you!

So I was reading a bit and can this maybe be helpful for ingress burst size?

tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null

tc qdisc add dev $DEV handle ffff: ingress

tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src
192.168.1.1 police rate ${34860}kbit burst 10k drop flowid :1

Sorry for the months late answer @hisham2630 the wifi is working great on my phones though I made some changes, for example went back to have all the sqm settings in the pppoe-wan (upload and download), I had upload only in wan and the download settings for the upload on the veth0, also deleted some of the advanced settings of sqm.

Something I have not solved is how to make my openvpn server work with dscp, if I have your script on, I can only ssh into my router using openvpn everything else (like browsing, etc) do not work, to use my vpn from work I have to disable your scrip and restart the router first. Any ideas??

1 Like