Ultimate SQM settings: Layer_cake + DSCP marks

hisham2630 · November 24, 2018, 6:40pm

Hi again!
I switched back from fireqos to SQM, because my router (TP-LINK W8970) have low power to handle fireqos config.
also it's better idea to control both bufferbloat + doing prioritization using a AIO solution.

I started to read and play with SQM settings, DSCP marks,etc...
and the result was amazing!, it's like having SQM + adaptive QOS, in one package simple and easy to config.
so i made a script that will handle DSCP marks and traffic routing using veth method like how @dlakelan taught me, now i can play an online (LOL) game on my pc while download + seeding a torrent without lag,
ping is very stable with rare ping spike (due to bad bandwidth settings and bad isp's network), also i can
download + browsing internet without delay (download will suffer when i browsing or download on my phone),
bandwidth fairness is really good, no more streaming lags.

this script still need to be optimization, correction to handle everything, after that i want to integrate it to layer_cake script, so it will be more easier for basic users to handle a simple configs.
to make this happen i want the valued help from this great forum members/modders.

some tests:

SQM off:
a.unloaded network:

League of legends game ping is 90~100ms.
whatsapp voice calls,viber voice calls= few lags and delay.

b.loaded network:

League of legends game ping is 200~400ms.
whatsapp voice calls,viber voice calls= more lags and delay, sometimes other person cant's hear clearly.

SQM on with DSCP tags:
a. unloaded network:

League of legends game ping is 100~117ms.
whatsapp voice calls,viber voice calls= no lags or delay.

b.loaded network:

League of legends game ping is 109~124ms.
whatsapp voice calls,viber voice calls= very rare lag, no delay.

This is ISP bandwidth splitting speeds:
download:
services= google ggc,cache,games updates,some CDN's.==8 mbps upto 12 mbps.
youtube= 8mbps upto 12 mbps.
facebook= 4 mbps.
main download speed=1.1 mbps, and 8mbps from 2am to 10am.

upload:
between 1.5mbps ~ 7mbps differ from site to site.but if i disable upload shaping bufferbloat is too low for upload, only my download have spikes!, and can be fixed by set download limit to 1000kbit.

*all of those traffic is isolated from each other!.they use dscp tag on youtube traffic (can't remember which value exactly but i think its 0x60 or 0x30).

My setup:
i used this guide to have extroot--> https://www.pcsuggest.com/configuring-extroot-with-openwrt-on-tp-link-mr-3220/

use this command to backup your installed packages names to file:

opkg list-installed | cut -f 1 -d ' ' > /etc/config/packages.list
and use this to reinstall your packages after sysupgrade
opkg install $(cat /etc/config/packages.list)

needed packages:
first run "opkg update", without quotes, then

opkg install kmod-veth kmod-netem kmod-ipt-ipopt kmod-nf-nathelper-extra kmod-sched kmod-sched-cake ip-full ipset iptables-mod-conntrack-extra iptables-mod-extra iptables-mod-ipopt iptables-mod-nat-extra
opkg remove dnsmasq ; opkg install dnsmasq-full

optional:

opkg install tcpdump -->to dump packets on interface for debugging.
opkg install kmod-tcp-bbr -->to enable BBR tcp congestion control,it's really improve network response.

then go to /etc/sysctl.conf and add those two lines:

net.core.default_qdisc=fq_codel
net.ipv4.tcp_congestion_control=bbr

I named my script net.sh and place it in /root/net.sh

IPT="iptables"

####Latency Sensitive Ports
##league of legends,PUBGM,some voip service like whatsapp.
sport1="37,53,81,88,123,500,1500,1863,3074:3076,3478:3481,3544,3659,4244"
sport2="4379,4380,4500,5222,5223,5228,5242,5000:5500,7985,8000,8888,9339,9785,34784"
sport3="10000:10300,19132,25565,27000:27030,28960,59172,39499,34141,59437:59581,45395,50318,59234"
dport1="37,53,81,88,123,500,1500,1863,3005,3074:3076,3101,3478:3481,3544"
dport2="3659,4244,4379,4380,4500,5222,5223,5228,5242,5000:5500,7985,8000,8888,34784"
dport3="9339,9785,10000:10300,19132,25565,27000:27030,28960,59172,39499,34141,59437:59581,45395"
dport4="1119,3724,7000:7999,8011,8081:8090,9030,17000,50318,59234"

## set up a 2 pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth

## get new veth interfaces up
ip link set veth0 up
ip link set veth1 up

## trun on promisc mode,sometimes it's needed to make bridge work
ip link set veth1 promisc on

## add veth1,veth3 to bridge
brctl addif br-lan veth1

## just to make sure there's nothing inside those 2 tables
ip rule del priority 100
ip route flush table 100

##ipset for streming sites.they are bening filled by dnsmasq
ipset create vidstream hash:ip
ipset create usrcdn hash:ip
ipset create bulk hash:ip
ipset create latsens hash:ip

## flush mangle table
$IPT -t mangle -F PREROUTING

## add routing for veth0 this will handle all slow traffic
ip route add default dev veth0 table 100
ip rule add iif pppoe-wan table 100 priority 100

#Clear interface dscp marks, we don't trust ISP marks(also to use our own marks).
$IPT -t mangle -A PREROUTING -i pppoe-wan -j DSCP --set-dscp 0

########################################
# Latency Sensitive (gaming/voip)
########################################
##Latency Sensitive
$IPT -t mangle -A PREROUTING -p icmp -j DSCP --set-dscp-class CS6 ##dscp tag for ping packets

$IPT -t mangle -A PREROUTING -m set --match-set latsens src -j DSCP --set-dscp-class CS6 ## set dscp tag for our Latency Sensitive (latsens) ipset

$IPT -t mangle -A PREROUTING -p tcp -m conntrack --ctorigsrc 192.168.1.150 -m multiport ! --ports 80,443 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4" #i don't have PS4, change the ip according to your ip setting.
$IPT -t mangle -A PREROUTING -p udp -m conntrack --ctorigsrc 192.168.1.150 -j DSCP --set-dscp-class CS6 -m comment --comment "PS4"

$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport1 -j DSCP --set-dscp-class CS6 
$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport2 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --sport $sport3 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p tcp -m multiport --sport 22,2099,4244,5222,5223,5228,5242,8088,8393:8400,50318,59234 -j DSCP --set-dscp-class CS6

$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport1 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport2 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport3 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport $dport4 -j DSCP --set-dscp-class CS6
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 22,2099,3013,4244,5222:5228,5242,10012,17000,17500,8088,8393:8400,50318,59234 -j DSCP --set-dscp-class CS6

########################################
# Streaming Media (videos/audios)
########################################
$IPT -t mangle -A PREROUTING -m set --match-set vidstream src -j DSCP --set-dscp-class AF41 ##set dscp tag for our ipset vidstearm
$IPT -t mangle -A PREROUTING -m set --match-set usrcdn src -j DSCP --set-dscp-class AF21 ##set dscp tag for our ipset usrcdn #those cdn's are isolated by my ISP.
$IPT -t mangle -A PREROUTING -p tcp -m multiport --dport 1935 -j DSCP --set-dscp-class AF41

########################################
# Best Effort (browsing/services)
########################################
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 80,443,853,3455,8080,27015 -j DSCP --set-dscp-class AF13
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 80,443,853,3455,8080 -j DSCP --set-dscp-class AF13

#########################################
# Background Traffic (Bulk/file transfer)
#########################################
$IPT -t mangle -A PREROUTING -m set --match-set bulk src -j DSCP --set-dscp-class CS1 ##set dscp tag for our bulk ipset
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 21,80,443,8080,60887,27014:27050 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1 #60887 bittorrent
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 21,80,443,8080,60887 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1

Q1. Is it needed to tag sport?, if yes then how ?
Q2. Should i route my fast traffic to another pair of veth?
Q3. How to benefit from ECN ?
Q4. How to wmm qos queue ?
Q5. Is it needed to add -w option to iptables to get rid of xt_table lock issue?

My dnsmasq.conf ipset's /etc/dnsmasq.conf :

##Latency Sensitive (gaming/voip)
ipset=/*.igamecj.com/igamecj.com/latsens

##video/audio streams
# Youtube is also isolated by my isp
ipset=/googlevideo.com/*.googlevideo.com/vidstream
# NetFlix
ipset=/nflxvideo.net/vidstream
# AmazonVideo
ipset=/s3.ll.dash.row.aiv-cdn.net/d25xi40x97liuc.cloudfront.net/aiv-delivery.net/vidstream
# Facebook
ipset=/fbcdn.net/vidstream
# Twitch
ipset=/ttvnw.net/vidstream
# VeVo
ipset=/vevo.com/vidstream
# Spotify
ipset=/audio-fa.scdn.cot/vidstream
# Deezer
ipset=/deezer.com/vidstream
# SoundCloud
ipset=/sndcdn.com/vidstream
# last.fm
ipset=/last.fm/vidstream

##i have isolated speed for those cdn's 
ipset=/googletagmanager.com/googleusercontent.com/*.googleusercontent.com/google.com/fbcdn.net/*.fbcdn.net/akamaihd.net/*.akamaihd.net/whatsapp.net/*.whatsapp.net/whatsapp.com/*.whatsapp.com/www-cdn.whatsapp.net/googleapis.com/*.googleapis.com/ucy.ac.cy/1e100.net/hwcdn.net/usrcdn

## Bulk downloads
#qq download
ipset=/download.qq.com/bulk
# Steam Download
ipset=/steamcontent.com/bulk
# PSN Download
ipset=/gs2.ww.prod.dl.playstation.net/bulk
# DropBox
ipset=/dropbox.com/dropboxstatic.com/dropbox-dns.com/log.getdropbox.com/bulk
# Google Drive
ipset=/drive.google.com/drive-thirdparty.googleusercontent.com/bulk
# Google Docs
ipset=/docs.google.com/docs.googleusercontent.com/bulk
# PlayStore Download
ipset=/gvt1.com/bulk
# WhatsApp Files
ipset=/mmg-fna.whatsapp.net/bulk
# Youtube Upload
ipset=/upload.youtube.com/upload.video.google.com/bulk
# WindowsUpdate
ipset=/windowsupdate.com/update.microsoft.com/bulk

My /etc/config/sqm :

config queue 'wan'
	option ingress_ecn 'ECN'
	option egress_ecn 'ECN'
	option enabled '1'
	option debug_logging '0'
	option verbosity '5'
	option qdisc 'cake'
	option qdisc_advanced '1'
	option qdisc_really_really_advanced '1'
	option squash_dscp '0'
	option squash_ingress '0'
	option linklayer 'ethernet'
	option linklayer_advanced '1'
	option tcMTU '2047'
	option tcTSIZE '128'
	option tcMPU '64'
	option linklayer_adaptation_mechanism 'cake'
	option download '0'
	option interface 'veth0'
	option shaper_burst '1'
	option script 'layer_cake.qos'
	option ilimit '18'
	option elimit '18'
	option iqdisc_opts 'overhead 28 diffserv4 nat dual-srchost rtt 200ms'
	option eqdisc_opts 'overhead 35 diffserv4 nat dual-dsthost rtt 200ms'
	option overhead '0'
	option upload '16000'

config queue
	option debug_logging '0'
	option verbosity '5'
	option ingress_ecn 'ECN'
	option tcMTU '2047'
	option tcTSIZE '128'
	option enabled '1'
	option download '0'
	option qdisc 'cake'
	option qdisc_advanced '1'
	option squash_dscp '0'
	option squash_ingress '0'
	option egress_ecn 'ECN'
	option qdisc_really_really_advanced '1'
	option linklayer 'ethernet'
	option linklayer_advanced '1'
	option tcMPU '64'
	option linklayer_adaptation_mechanism 'cake'
	option shaper_burst '1'
	option upload '7000'
	option script 'layer_cake.qos'
	option ilimit '18'
	option elimit '18'
	option interface 'pppoe-wan'
	option overhead '0'
	option iqdisc_opts 'diffserv4 nat dual-dsthost rtt 200ms'
	option eqdisc_opts 'overhead 18 diffserv4 nat dual-srchost rtt 200ms'

config queue
	option debug_logging '0'
	option verbosity '5'
	option qdisc 'cake'
	option qdisc_advanced '1'
	option ingress_ecn 'ECN'
	option qdisc_really_really_advanced '1'
	option enabled '1'
	option squash_dscp '0'
	option squash_ingress '0'
	option egress_ecn 'ECN'
	option shaper_burst '1'
	option interface 'eth0'
	option script 'test_LAN_triple-isolate__piece_of_cake.qos'
	option linklayer 'ethernet'
	option overhead '8'
	option linklayer_advanced '1'
	option tcMTU '2047'
	option tcTSIZE '128'
	option tcMPU '64'
	option linklayer_adaptation_mechanism 'cake'
	option ilimit '18'
	option elimit '18'
	option iqdisc_opts 'mpu 64 nat dual-srchost'
	option eqdisc_opts 'mpu 64 nat dual-dsthost'
	option download '58120'
	option upload '0'

I used diffserv4 for layer cake:

you can edit this in /usr/lib/sqm/defaults.sh
replace diffserv3 with diffserv4

I'm using "Linux OpenWrt 4.14.79 #0 SMP Mon Nov 12 16:09:09 2018 mips GNU/Linux
- r8450", tomorrow i will upgrade to latest snapshot.
*note: software offloading is off!.

bufferbloat test:
sqm off:

sqm on:

Many thanks to openwrt dev's/modder's for making openwrt great and better.
Many thanks to @dlakelan for taught me and help me to configure fireqos.
Many thanks to @shm0 for your script, i took some codes from your script!

*NOTE:
Here is the newest script, don't use old script it's a port based and outed dated!

QOS script

#!/bin/bash -x
IPT="iptables"
WANIF="pppoe-wan" #wan interface

tc qdisc add dev wlan0 root mq #setup multi queue for wifi device(optional)

####veth start's here####

## set up a pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth

## get new veth interfaces up
ip link set veth0 up
ip link set veth1 up

## trun on promisc mode,sometimes it's needed to make bridge work
ip link set veth1 promisc on

## add veth1 to bridge
brctl addif br-lan veth1

## just to make sure there's nothing inside those 2 tables
ip rule del priority 100
ip route flush table 100

## add routing for veth0 this will handle all traffic
ip route add default dev veth0 table 100
ip rule add iif $WANIF table 100 priority 100

###veth end ###

##ipset for streming sites, etc; they are bening filled by dnsmasq
ipset create streaming hash:ip
ipset create usrcdn hash:ip
ipset create bulk hash:ip
ipset create latsens hash:ip

$IPT -t mangle -N dscp_mark > /dev/null 2>&1
$IPT -t mangle -F dscp_mark
## check if POSTROUTING already exits then jumps to our tables if not, add them

$IPT -t mangle -L POSTROUTING -n | grep dscp_mark || $IPT -t mangle -A POSTROUTING -j dscp_mark

iptmark(){
    $IPT -t mangle -A dscp_mark "$@"
}

## start by washing the dscp to CS0

iptmark -j DSCP --set-dscp 0

#A robust 2 rules to detect realtime traffic

# mark connections that go over 115 packets per second, not prioritized
iptmark -p udp -m hashlimit --hashlimit-name udp_high_prio --hashlimit-above 115/sec --hashlimit-burst 50 --hashlimit-mode srcip,srcport,dstip,dstport -j CONNMARK --set-mark 0x55 -m comment --comment "connmark for udp"

# unmarked UDP streams with small packets get CS6
iptmark -p udp -m connmark ! --mark 0x55 -m multiport ! --ports 22,25,53,67,68,123,143,161,162,514,5353,80,443,8080,60001 -m connbytes --connbytes 0:940 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS6 -m comment --comment "small udp connection gets CS6"

#large udp streams like video call get AF41
iptmark -p udp -m connmark ! --mark 0x55 -m multiport ! --ports 22,25,53,67,68,123,143,161,162,514,5353,80,443,8080,60001 -m connbytes --connbytes 940:1500 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class AF41 -m comment --comment "large udp connection gets AF41"

########################################
# Latency Sensitive (gaming/voip)
########################################
##ICMP, to prioritize pings
iptmark -p icmp -j DSCP --set-dscp-class CS5 -m comment --comment "ICMP-pings"

#DNS traffic both udp and tcp
iptmark -p udp -m multiport --port 53,5353,8888 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS udp"
iptmark -p tcp -m multiport --port 53,5353,8888 -j DSCP --set-dscp-class CS5 -m comment --comment "DNS tcp"

#NTP
iptmark -p udp -m multiport --port 123 -j DSCP --set-dscp-class CS6 -m comment --comment "NTP udp"

#High priority ipset, i use for pubgM
iptmark ! -p tcp -m set --match-set latsens src,dst -j DSCP --set-dscp-class CS6 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset,udp

iptmark -p tcp -m set --match-set latsens src,dst -j DSCP --set-dscp-class CS5 -m comment --comment "latency sensitive ipset" ## set dscp tag for Latency Sensitive (latsens) ipset

########
##Browsing
########
## medium priority for browsing
iptmark -p tcp -m multiport --ports 80,443,8080 -j DSCP --set-dscp-class CS3 -m comment --comment "Browsing at CS3"

##################
#TCP SYN,ACK flows
##################
#Make sure ACK,SYN packets get priority (to avoid upload speed limiting our download speed)
iptmark -p tcp --tcp-flags ALL ACK -m length --length :128 -j DSCP --set-dscp-class CS3
iptmark -p tcp --tcp-flags ALL SYN -m length --length :666 -j DSCP --set-dscp-class CS3

#Small packet is probably interactive or flow control
iptmark -m dscp ! --dscp  24 -m dscp ! --dscp  18 -m dscp ! --dscp  34 -m dscp ! --dscp  40 -m dscp ! --dscp  48 -m length --length 0:500 -j DSCP --set-dscp-class CS3

#Small packet connections: multi purpose (don't harm since not maxed out)
iptmark -m dscp ! --dscp  24 -m dscp ! --dscp  18 -m dscp ! --dscp  34 -m dscp ! --dscp  40 -m dscp ! --dscp  48 -m connbytes --connbytes 0:250 --connbytes-dir both --connbytes-mode avgpkt -j DSCP --set-dscp-class CS3


########################################
# Streaming Media (videos/audios)
########################################
#Known video streams sites like netflix
iptmark -m set --match-set streaming src,dst -j DSCP --set-dscp-class AF41 -m comment --comment "video audio stream ipset"

# some iptv provider's use this port
iptmark -p tcp -m multiport --ports 1935,9982 -j DSCP --set-dscp-class AF41 -m comment --comment "some iptv streaming service"

#known usrcdn like google or akamai

iptmark -m set --match-set usrcdn src,dst -j DSCP --set-dscp-class AF21 -m comment --comment "usrcdn ipset"

#########################################
# Background Traffic (Bulk/file transfer)
#########################################
#bulk traffic ipset, like windows udates and steam updates/downloads
iptmark -p tcp -m set --match-set bulk src,dst -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"
iptmark -p udp -m set --match-set bulk src,dst -j DSCP --set-dscp-class CS1 -m comment --comment "bulk traffic ipset"
iptmark -p tcp -m connbytes --connbytes 350000: --connbytes-dir both --connbytes-mode bytes -m dscp --dscp-class CS0 -j DSCP --set-dscp-class CS1 -m comment --comment "Downgrade CS0 to CS1 for bulk tcp traffic"
iptmark -p tcp -m connbytes --connbytes 350000: --connbytes-dir both --connbytes-mode bytes -m dscp --dscp-class CS3 -j DSCP --set-dscp-class CS1 -m comment --comment "Downgrade CS3 to CS1 for bulk tcp traffic"
iptmark -p udp -m multiport --port 60001 -j DSCP --set-dscp-class CS1 -m comment --comment "bulk torrent port UDP"


#tcpdump rule, copy and paste this rule into terminal, this rule is used to capture realtime traffic, you can change ip to what you like
#tcpdump -i br-lan host 192.168.1.126 and udp and portrange 1-65535 and !port

Dnsmasq, dnsmasq.conf:

dnsmasq.conf

ipset=/igamecj.com/gcloudcs.com/qos.gcloud.qq.com/tencentgames.helpshift.com/latsens
ipset=/.googlevideo.com/youtube.com/streaming
ipset=/live.mp3quran.net/tiktokv.com/media-imdb.com/v.redd.it/last.fm/sndcdn.com/deezer.com/audio-fa.scdn.cot/vevo.com/ttvnw.net/s3.ll.dash.row.aiv-cdn.net/d25xi40x97liuc.cloudfront.net/aiv-delivery.net/nflxvideo.net/av.alarabiya.net/streaming
ipset=/.fbcdn.net/.whatsapp.net/.whatsapp.com/usrcdn
ipset=/lol.secure.dyn.riotcdn.net/lol.dyn.riotcdn.net/lol-promos.s3.amazonaws.com/.amazonaws.com/l3cdn.riotgames.com/.akamaihd.net/ucy.ac.cy/hwcdn.net/windowsupdate.com/update.microsoft.com/share.box/akamaized.net/usrcdn
ipset=/.google.com/googletagmanager.com/.googleusercontent.com/google.com/.googleapis.com/1e100.net/usrcdn
ipset=/dropbox.com/dropboxstatic.com/dropbox-dns.com/log.getdropbox.com/gs2.ww.prod.dl.playstation.net/steamcontent.com/download.qq.com/bulk

**Use an overhead that will suit your connection type!
***I used sqm on eth0 for test purpose and maybe sometimes fix spikes!
****For routers that have Switch0 interface(DSA architecture switch), you don't need to use veth method!

hisham2630 · November 24, 2018, 6:48pm

I forget to say that my connection type now is FTTH using EPON modem. my connection RTT is between 150ms-200ms.

My /etc/sysctl.conf :

# Defaults are configured in /etc/sysctl.d/* and can be customized in this file
#new
net.ipv6.conf.default.forwarding=1
net.ipv6.conf.all.forwarding=1

net.netfilter.nf_conntrack_acct=1
net.netfilter.nf_conntrack_checksum=0
net.netfilter.nf_conntrack_max=16384
net.netfilter.nf_conntrack_tcp_timeout_established=7440
net.netfilter.nf_conntrack_udp_timeout=60
net.netfilter.nf_conntrack_udp_timeout_stream=180

#new settings to improve ping response and lowering ping spikes
net.core.netdev_budget=300
net.core.netdev_max_backlog = 1000

#ecn
net.ipv4.tcp_ecn=2
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_dsack=1

moeller0 · November 24, 2018, 7:34pm

First, impressive!

This seems odd, typically networks use symmetric overhead in both directions and recent cake learned how to undo the variable overhead part the kernel adds dependent on the type of interface. Could you post the output of "tc -s qdisc" so we can figure out whether your cake is recent enough?

moeller0 · November 24, 2018, 7:36pm

This will only affect traffic that is served from your router, so if you do not serve TCP-connections from your browser this will have very little effect.

hisham2630 · November 25, 2018, 9:32am

i was just playing with overheads,but still can't find the best value.
my connection type is pppoe.

this is tc -s qdisc:

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8007: dev eth0 root refcnt 2 bandwidth 58120Kbit besteffort dual-dsthost nat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 8 mpu 64
 Sent 36284638 bytes 178510 pkt (dropped 0, overlimits 53 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 25152b of 4Mb
 capacity estimate: 58120Kbit
 min/max network layer size:            6 /    1488
 min/max overhead-adjusted size:       64 /    1496
 average network hdr offset:           14

                  Tin 0
  thresh      58120Kbit
  target          5.0ms
  interval      100.0ms
  pk_delay         36us
  av_delay         19us
  sp_delay         13us
  backlog            0b
  pkts           178510
  bytes        36284638
  way_inds         6618
  way_miss        10185
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            6
  bk_flows            1
  un_flows            0
  max_len          1502
  quantum          1514

qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.4 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev wlan1 root
 Sent 144423068 bytes 124690 pkt (dropped 0, overlimits 0 requeues 162)
 backlog 0b 0p requeues 162
qdisc fq_codel 0: dev wlan1 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 101269688 bytes 82764 pkt (dropped 0, overlimits 0 requeues 109)
 backlog 0b 0p requeues 109
  maxpacket 1494 drop_overlimit 0 new_flow_count 60 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan1 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 42059490 bytes 40485 pkt (dropped 0, overlimits 0 requeues 52)
 backlog 0b 0p requeues 52
  maxpacket 1494 drop_overlimit 0 new_flow_count 27 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan1 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 1093890 bytes 1441 pkt (dropped 0, overlimits 0 requeues 1)
 backlog 0b 0p requeues 1
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8009: dev pppoe-wan root refcnt 2 bandwidth 7Mbit diffserv4 dual-srchost nat nowash no-ack-filter split-gso rtt 200.0ms noatm overhead 18 mpu 64
 Sent 32050269 bytes 166437 pkt (dropped 6, overlimits 25346 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 142528b of 4Mb
 capacity estimate: 7Mbit
 min/max network layer size:           40 /    1480
 min/max overhead-adjusted size:       64 /    1498
 average network hdr offset:            0

                   Bulk  Best Effort        Video        Voice
  thresh      437496bit        7Mbit     3500Kbit     1750Kbit
  target         41.3ms       10.0ms       10.0ms       10.3ms
  interval      231.3ms      200.0ms      200.0ms      200.3ms
  pk_delay       27.1ms        1.1ms         26us        942us
  av_delay        5.6ms        153us          4us         55us
  sp_delay        1.3ms         22us          4us         24us
  backlog            0b           0b           0b           0b
  pkts            61871        93137           45        11390
  bytes        18048123     12155738         3420      1851748
  way_inds         6637         1337            0            5
  way_miss         1093         7820           45          287
  way_cols            3            0            0            0
  drops               0            6            0            0
  marks               0            4            0            0
  ack_drop            0            0            0            0
  sp_flows           19            4            1            5
  bk_flows            1            0            0            0
  un_flows            0            0            0            0
  max_len          1456         1480           76         1480
  quantum           300          300          300          300

qdisc noqueue 0: dev wlan0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 800b: dev veth0 root refcnt 2 bandwidth 16Mbit diffserv4 dual-dsthost nat nowash no-ack-filter split-gso rtt 200.0ms noatm overhead 35 mpu 64
 Sent 218590656 bytes 201369 pkt (dropped 28, overlimits 97689 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 187200b of 4Mb
 capacity estimate: 16Mbit
 min/max network layer size:           28 /    1487
 min/max overhead-adjusted size:       64 /    1522
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh          1Mbit       16Mbit        8Mbit        4Mbit
  target         18.2ms       10.0ms       10.0ms       10.0ms
  interval      208.2ms      200.0ms      200.0ms      200.0ms
  pk_delay        1.3ms        298us        915us        475us
  av_delay        1.2ms         40us         57us         56us
  sp_delay         17us         21us         21us         20us
  backlog            0b           0b           0b           0b
  pkts            66757       123074         5294         6272
  bytes        85714453    125375695      2399370      5137749
  way_inds            0         2176           77            6
  way_miss           30         2607           27          287
  way_cols            0            0            0            0
  drops               3           23            0            2
  marks               0            1            0            0
  ack_drop            0            0            0            0
  sp_flows            1           18            1            4
  bk_flows            0            2            0            0
  un_flows            0            0            0            0
  max_len          1494         1501         1484         1494
  quantum           300          488          300          300

qdisc noqueue 0: dev veth1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

hisham2630 · November 25, 2018, 9:33am

but i see there's some improvement to network!

hisham2630 · November 25, 2018, 9:38am

I forgot that this file is needed /etc/iproute2/rt_dsfield

# Differentiated field values
# These include the DSCP and unused bits
0x0     default
# Newer RFC2597 values
0x28    AF11
0x30    AF12
0x38    AF13
0x48    AF21
0x50    AF22
0x58    AF23
0x68    AF31
0x70    AF32
0x78    AF33
0x88    AF41
0x90    AF42
0x98    AF43
# Older values RFC2474
0x20    CS1
0x40    CS2
0x60    CS3
0x80    CS4
0xA0    CS5
0xC0    CS6
0xE0    CS7
# RFC 2598
0xB8    EF

This is important to make (--set-dscp-class) work.

moeller0 · November 25, 2018, 10:19am

Well, what kind of link layer are you actually using, PPPoE is going tp add 8 bytes (6 for PPP and 2 foe the oE part) but that is only part of the story.
Ah, I see EPON/FTTH, I have so far not looked at the framing on EPON, even though I naively expect pure ethernet overhead (in your case plus PPPoE). I will have a look at that later this week.

moeller0 · November 25, 2018, 10:27am

Well, that either means you are terminating a sufficiently large amount of TCP connections at the router (say if you would run a squid proxy there you can expect to see effects of the router's TCP CC algorithm) or there are some unexplainable side-effects of setting the router itself to BBR.
How do you measure the improvements by the way?

hisham2630 · November 25, 2018, 1:59pm

Hmm!
let me explain how networks is working here:

Main upstream provider-->sub isp-->repater-->manager(can have a service from two sub-isp)-->user.
we can't get internet directly from main isp, there's four main internet providers here:

https://ipinfo.io/AS21277
https://ipinfo.io/AS44217
https://ipinfo.io/AS39216 (used as backup)
https://ipinfo.io/AS50597

connection between user and sub isp is bridged, manager use vlans to isolate isp's if he subscribed to 2 isp's or more!
*most of times connection between user and sub-isp is wireless mesh network!, but in my case it's fiber connection via a single core cable with EPON modem.
for now i assume 8bytes(pppoe)+4bytes(vlan for isolating the pppoe connection from the 2 sub-isp)=12 overhead.

hisham2630 · November 25, 2018, 2:03pm

i didn't made any measurements by numbers, but when i was on cubic i see sometimes webpages loading slowly on unloaded network,sometimes i have to refresh page to get it displayed. but after i used this tcp BBR
it's loading normally. i think there's some unexplained good side-effects from using BBR!

Emtee · November 25, 2018, 2:41pm

Hi guys! Sorry to chime in but I'm trying to get a good QoS setup going as well, but all these threads about it make me somewhat confused on the inner workings.

I'm currently using windows to mark my game's outbound packets with DSCP 46, confirmed with Wireshark and all seems to be working well.

Additionally enabled windows QoS to disallow any app/service to tag their own packets (so solely my game gets priority. As for my Wifi I disabled WMM.

Combined with proper settings and layer_cake all should work well I assume? Or am i missing something? I disabled ECN on my network adapters btw, since i dont believe its importent for UDP data stream?

But now I'm trying to also tag incoming packets (ingress) from WAN side, but somewhat confused on how to proceed.

Thanks!

moeller0 · November 25, 2018, 2:42pm

Well since EPON is a variant of ethernet (for all intents and purposes) I would assume that a competent ISP will account for the full ethernet overhead in its shapers, so in your case that would be 38 (full effective ethernet overhead) + 8 (PPPoE) + 4 (VLAN) = 38+8+4 = 50 Bytes.
But I would be amazed if most ISPs would actually look that closely...

hisham2630 · November 25, 2018, 2:48pm

now i set the overhead to 50bytes, let me test and see!
this is really big overhead. BTW : isp's here is using mikrotk router to handle all the traffic shaping and splitting.
they use something like simple queue+pcq or fq.

hisham2630 · November 25, 2018, 3:00pm

HI
No problem, i happy to help you getting this to work as you wish, but let ask you something;
Q1, your download/upload speeds?
Q2. your /etc/config/sqm settings?
Q3. which games do you play?

tagging game packet with dscp tags without any prioritization is useless!
It's better to have wireless WMM on to get benefit from wireless queue.

Yeah it should work very well, but i'd like to know which version of openwrt you are using also which router?
ECN is not important for UDP, but for TCP. Have ECN on will not harm UDP!
It would be easy to configure sqm and dscp tags, if you provide the required info.

Emtee · November 25, 2018, 3:38pm

Hi thanks again for your swift response!

I think I managed to at least receive altered (With DSCP Expedited forwarding 46) packets from the port range I want now. At least wireshark shows me these packets now from the port range. (Whether they are actually shaped by Cake, I dont know)

Linksys WRT1900ACv2
OpenWrt 18.06.1 r7258-5eb055306f

SQM config:


config queue 'eth1'
	option interface 'eth1.2'
	option debug_logging '0'
	option verbosity '5'
	option qdisc 'cake'
	option script 'layer_cake.qos'
	option qdisc_advanced '1'
	option squash_dscp '1'
	option squash_ingress '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option linklayer 'ethernet'
	option overhead '18'
	option enabled '1'
	option download '30000'
	option upload '10000'
	option qdisc_really_really_advanced '1'
	option iqdisc_opts 'nat dual-dsthost'
	option eqdisc_opts 'nat dual-srchost'

config queue
	option debug_logging '0'
	option verbosity '5'
	option squash_dscp '1'
	option squash_ingress '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option enabled '1'
	option interface 'br-lan'
	option download '30000'
	option upload '10000'
	option qdisc 'cake'
	option script 'layer_cake.qos'
	option qdisc_advanced '1'
	option linklayer 'ethernet'
	option overhead '18'
	option qdisc_really_really_advanced '1'
	option iqdisc_opts 'nat dual-dsthost'
	option eqdisc_opts 'nat dual-srchost'

My Iptables:

iptables -t mangle -N dscp_mark
iptables -t mangle -F dscp_mark

iptables -t mangle -A FORWARD -j dscp_mark

## icmp gets CS5
iptables -t mangle -A dscp_mark  -p icmp -j DSCP --set-dscp-class CS5

## inbound game traffic for MWO
iptables -t mangle -A dscp_mark  -p udp --sport 30000:60000 -j DSCP --set-dscp-class EF
iptables -t mangle -A dscp_mark  -p udp --dport 30000:60000 -j DSCP --set-dscp-class EF

The game uses port around 45000~ udp for actual gameplay traffic, but it varies so I took a slightly wide range. The only other application I think uses this range at this point (looking at wireshark) is Voice for Discord, which wouldnt really be a problem altho not highest priority for me personally, but cant hurt.

I have SQM applied to both WAN and LAN separately now.
My internet is actually 4G with modem (Roughly 45/30 Mbit bandwidth) and latency is low enough for competitive gameplay (20/30ms to first hop) and very little jitter since I attach my modem to only 1 channel, i think jitter and packet loss is even more important than latency, since most games have some mechanic to compensate for latency anyhow.

I put my limits quite conservative atm because 4G bandwidth fluctuates a bit more than a fixed line, but so far no indication of any buffer bloat

So since outgoing and incoming DSCP packets seem to work now according to Wireshark from my PC, the only question is does SQM really shape my outgoing packets, or even better, my incoming packets which are now altered by iptables!

(I think Wifi no longer gets shaped in this situation I think, should I also apply a SQM instance to my Wifi interface?)

dlakelan · November 25, 2018, 3:52pm

If you use layer cake it should give priority to your tagged packets. I imagine there is some tc command to see how the layers work but I don't know off top of my head.

hisham2630 · November 25, 2018, 3:56pm

real life test+ wire shark will prove if it's work!

Emtee · November 25, 2018, 4:00pm

Yea thats true I'll give it some real life tests!

However, now my Download speed is capped at the upload limit i set the two SQM's at.

I assume I now have to 'reverse' the values of one of the two? Makes sense somehow.

dlakelan · November 25, 2018, 4:04pm

Both shapers should shape egress only. The LAN shaper should be set to the bandwidth of your download.