Using DSCP for QoS

Hi!

Im trying to set dscp flags for some packets and make cake use those.
Since sqm-scripts use ifb setting dscp flags on ingress will not work.
So i modded the script to use imq instead and also added imq back to the kernel/iptables.
Currently im using this approach.

###################################################################
# Create DSCP Marking Groups for
# Cakes DiffServ4 Implementation
# Since we issue ACCEPT after Match 
# make sure DSCP Rules are issued last
###################################################################

$IPT -t mangle -N sqm
$IPT -t mangle -N dscp_cs0
$IPT -t mangle -N dscp_cs1
$IPT -t mangle -N dscp_cs3
$IPT -t mangle -N dscp_cs7

# Only set DSCP Flags on Packets that are going to be forwarded to the internet
$IPT -t mangle -A FORWARD -i br-lan -o eth1 -j sqm
$IPT -t mangle -A FORWARD -i br-isolated -o eth1 -j sqm
# Cant use FORWARD on ingress because of imq/cake
$IPT -t mangle -A PREROUTING -i eth1 -j sqm

# Setup dscp set flag chains
$IPT -t mangle -A dscp_cs0 -j DSCP --set-dscp-class CS0 -m comment --comment "CS0 Best Effort"
$IPT -t mangle -A dscp_cs0 -j ACCEPT
$IPT -t mangle -A dscp_cs1 -j DSCP --set-dscp-class CS1 -m comment --comment "CS1 Background"
$IPT -t mangle -A dscp_cs1 -j ACCEPT
$IPT -t mangle -A dscp_cs3 -j DSCP --set-dscp-class CS3 -m comment --comment "CS3 Streaming"
$IPT -t mangle -A dscp_cs3 -j ACCEPT
$IPT -t mangle -A dscp_cs7 -j DSCP --set-dscp-class CS7 -m comment --comment "CS7 Latency Sensitive"
$IPT -t mangle -A dscp_cs7 -j ACCEPT

###################################################################
# Latency Sensitive (Voice Tin)
###################################################################

# Generic
$IPT -t mangle -A sqm -m ndpi --NTP -j dscp_cs7
# Gaming
$IPT -t mangle -A sqm -p tcp -m conntrack --ctorigsrc 10.0.1.60 -m multiport ! --ports 80,443 -j dscp_cs7 -m comment --comment "PS4"
$IPT -t mangle -A sqm -p udp -m conntrack --ctorigsrc 10.0.1.60 -j dscp_cs7 -m comment --comment "PS4"
$IPT -t mangle -A sqm -p udp -m multiport --ports 5000:5500 -j dscp_cs7 -m comment --comment "Leauge of Legends"
$IPT -t mangle -A sqm -m ndpi --CSGO -j dscp_cs7
$IPT -t mangle -A sqm -m ndpi --WorldOfWarcraft -j dscp_cs7
# Voice
$IPT -t mangle -A sqm -m ndpi --TeamSpeak -j dscp_cs7
$IPT -t mangle -A sqm -m ndpi --WhatsAppVoice -j dscp_cs7
$IPT -t mangle -A sqm -m ndpi --SIP -j dscp_cs7

###################################################################
# Streaming Media (Video Tin)
###################################################################

$IPT -t mangle -A sqm -m ndpi --YouTube -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --NetFlix -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --AmazonVideo -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --Vevo -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --Twitch -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --GoogleHangout -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --Spotify -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --Deezer -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --SoundCloud -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --LastFM -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --Skype -j dscp_cs3
# Remote Administration
$IPT -t mangle -A sqm -m ndpi --TeamViewer -j dscp_cs3
$IPT -t mangle -A sqm -m ndpi --VNC -j dscp_cs3

###################################################################
# Background Traffic (Bulk Tin)
###################################################################

# Mail
$IPT -t mangle -A sqm -m ndpi --SMTP -j dscp_cs1					# Maybe remove, since i block all insecure SMTP traffic anyway (used for spam)
$IPT -t mangle -A sqm -m ndpi --SMTPS -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --IMAP -j dscp_cs1					# Maybe also remove and block instead since insecure
$IPT -t mangle -A sqm -m ndpi --IMAPS -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --POP3 -j dscp_cs1					# Maybe also remove and block instead since insecure
$IPT -t mangle -A sqm -m ndpi --POPS -j dscp_cs1
# P2P
$IPT -t mangle -A sqm -m ndpi --BitTorrent -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --eDonkey -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --Thunder -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --AppleJuice -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --Soulseek -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --Gnutella -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --DirectConnect -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --FastTrack -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --Stealthnet -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --Filetopia -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --Usenet -j dscp_cs1
# Cloud
$IPT -t mangle -A sqm -m ndpi --Dropbox -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --GoogleDrive -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --MS_OneDrive -j dscp_cs1
# Other
$IPT -t mangle -A sqm -m ndpi --Steam -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --PlayStore -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --GoogleDocs -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --YouTubeUpload -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --WhatsAppFiles -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --FTP_DATA -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --Git -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --RSYNC -j dscp_cs1
$IPT -t mangle -A sqm -m ndpi --WindowsUpdate -j dscp_cs1

# Consider "large" HTTP/S traffic (out/in) as Bulk
# Need some better solution for this
# But should work for most use cases
$IPT -t mangle -A sqm -p tcp -m connbytes --connbytes 10485760 --connbytes-dir reply --connbytes-mode bytes -m multiport --sports 80,443 -j dscp_cs1
$IPT -t mangle -A sqm -p tcp -m connbytes --connbytes 2097152 --connbytes-dir original --connbytes-mode bytes -m multiport --dports 80,443 -j dscp_cs1

###################################################################
# Best Effort (Best Effort Tin (Default))
###################################################################
# Since we dont know if LAN Users are messing around with DSCP Flags
# And my ISP is doing weird Stuff with DSCP Flags too
# Simply default all Traffic to BE and overwrite later on

$IPT -t mangle -A sqm -j dscp_cs0 -m comment --comment "CS0 Best Effort (Default)"

It works but was thinking about something better.
One problem is that it has to match packets/connection on both sides.
Maybe something like, only do the classification on egress side and mark those packets/connections with a connmark.
And use that mark on ingress side.
That should reduce cpu load?

I have somewhat working solution which utilize this.
Do i have to use conntrack with ctstate (NEW,ESTABLISHED,RELATED) when using connmark?
Someone has a better idea?

This was proposed for addition in cake.
https://lists.bufferbloat.net/pipermail/cake/2017-February/002421.html

http://models.street-artists.org/2017/12/11/inbound-qos-with-virtual-ethernet-and-policy-routing/

So he using a virtual interface, bridges that into his lan and uses routes to make traffic flow to the virtual interface. Why this makes no sense to me. also sounds like a bad solution.

and i think if the sqm scripts sets up ifb and redirect all traffic from wan to the ifb, this will also not work.

Why is everyone hating about imq?

Im the only one who thinks ifb, tc and so on sucks hard times?

from https://wiki.linuxfoundation.org/networking/ifb :
"The Intermediate Functional Block device is the successor to the IMQ iptables module that was never integrated. Advantage over current IMQ; cleaner in particular in SMP; with a lot less code"

surely not :wink: but while iptables processing is more convenient and capable, it is also significantly slower.
afaik veth and ifb avoid calling into higher-level filtering for this reason.
this does not matter for a few mbps, but for a few hundred it probably will.

maybe bpfilter can make a difference here?

Addon:
i did not want to imply that you are in the wrong using imq for netfilter capability. especially if it works for you ™ ...
The author of ifb allegedly also came around and regreted this design aspect, but as with the integration in cake, nobody came around to do/change it yet .

I'm the blog author. The point of the veth solution was simply that ifb gets the packets before iptables runs, so you need to queue packets after iptables runs. Queuing on output of veth solves this problem

I will check the veth solution later.

Here is my other approach using connmarks.
When using imq, i think this is the better solution.
Since traffic only needs to be classified on the egress side.
After classifying set some connmarks.
use those marks on egress and ingress to set the appropriate dscp flags.

###################################################################
# Make use of Cake's DiffServ4 Implementation
# by dscp flags
###################################################################
# Setup some custom chains
$IPT -t mangle -N sqm
$IPT -t mangle -N sqm_mark_cs0
$IPT -t mangle -N sqm_mark_cs1
$IPT -t mangle -N sqm_mark_cs3
$IPT -t mangle -N sqm_mark_cs7
$IPT -t mangle -N sqm_set_dscp

# Since we dont know if lan users are messing around with dscp flags
# and my isp is doing weird stuff with dscp flags too
# default all packets to cs0 (BE) and override later on
$IPT -t mangle -A PREROUTING -j DSCP --set-dscp-class CS0 -m comment --comment "Set CS0 as default"

# Only try to match traffic thats is going to be forwarded to internet
$IPT -t mangle -A FORWARD -o eth1 -j sqm
$IPT -t mangle -A FORWARD -o eth1 -j sqm_set_dscp
# Match marks on ingress and set dscp flags
$IPT -t mangle -A PREROUTING -i eth1 -j sqm_set_dscp

# match packet mark and set appropriate dscp flag
$IPT -t mangle -A sqm_set_dscp -j CONNMARK --restore-mark
$IPT -t mangle -A sqm_set_dscp -m mark --mark 0 -j DSCP --set-dscp-class CS0 -m comment --comment "CS0 Best Effort"
$IPT -t mangle -A sqm_set_dscp -m mark --mark 0 -j RETURN
$IPT -t mangle -A sqm_set_dscp -m mark --mark 1 -j DSCP --set-dscp-class CS1 -m comment --comment "CS1 Background"
$IPT -t mangle -A sqm_set_dscp -m mark --mark 1 -j RETURN
$IPT -t mangle -A sqm_set_dscp -m mark --mark 3 -j DSCP --set-dscp-class CS3 -m comment --comment "CS3 Streaming"
$IPT -t mangle -A sqm_set_dscp -m mark --mark 3 -j RETURN
$IPT -t mangle -A sqm_set_dscp -m mark --mark 7 -j DSCP --set-dscp-class CS7 -m comment --comment "CS7 Latency Sensitive"
$IPT -t mangle -A sqm_set_dscp -m mark --mark 7 -j RETURN

# set connection mark
# return back to main table/chain
$IPT -t mangle -A sqm_mark_cs1 -j CONNMARK --set-mark 1
$IPT -t mangle -A sqm_mark_cs1 -j RETURN
$IPT -t mangle -A sqm_mark_cs3 -j CONNMARK --set-mark 3
$IPT -t mangle -A sqm_mark_cs3 -j RETURN
$IPT -t mangle -A sqm_mark_cs7 -j CONNMARK --set-mark 7
$IPT -t mangle -A sqm_mark_cs7 -j RETURN

# Restore connmark to packet mark
# check if packet has already been marked
# if true return to back to main table/chain
$IPT -t mangle -A sqm -j CONNMARK --restore-mark
$IPT -t mangle -A sqm -m mark ! --mark 0 -j RETURN

###################################################################
# Latency Sensitive (Voice Tin)
###################################################################
# Generic
$IPT -t mangle -A sqm -m ndpi --NTP -g sqm_mark_cs7
# Gaming
$IPT -t mangle -A sqm -s 10.0.1.60 -p udp -m multiport ! --dports 9000  -g sqm_mark_cs7 -m comment --comment "PS4 UDP"
$IPT -t mangle -A sqm -s 10.0.1.60 -p tcp -m multiport ! --dports 80,443  -g sqm_mark_cs7 -m comment --comment "PS4 TCP"
$IPT -t mangle -A sqm -p udp -m multiport --dports 5000:5500 -g sqm_mark_cs7 -m comment --comment "League of Legends"
$IPT -t mangle -A sqm -m ndpi --CSGO -g sqm_mark_cs7
$IPT -t mangle -A sqm -m ndpi --WorldOfWarcraft -g sqm_mark_cs7
# Voice
$IPT -t mangle -A sqm -m ndpi --TeamSpeak -g sqm_mark_cs7
$IPT -t mangle -A sqm -m ndpi --WhatsAppVoice -g sqm_mark_cs7

###################################################################
# Streaming Media (Video Tin)
###################################################################

$IPT -t mangle -A sqm -m ndpi --YouTube -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --NetFlix -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --AmazonVideo -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Vevo -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Twitch -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --GoogleHangout -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Spotify -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Deezer -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --SoundCloud -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --LastFM -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Skype -g sqm_mark_cs3
# Remote Administration
$IPT -t mangle -A sqm -m ndpi --TeamViewer -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --VNC -g sqm_mark_cs3

###################################################################
# Background Traffic (Bulk Tin)
###################################################################
# Mail
$IPT -t mangle -A sqm -m ndpi --SMTP -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --SMTPS -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --IMAP -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --IMAPS -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --POP3 -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --POPS -g sqm_mark_cs1
# P2P
$IPT -t mangle -A sqm -m ndpi --BitTorrent -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --eDonkey -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Thunder -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --AppleJuice -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Soulseek -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Gnutella -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --DirectConnect -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --FastTrack -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Stealthnet -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Filetopia -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Usenet -g sqm_mark_cs1
# Cloud
$IPT -t mangle -A sqm -m ndpi --Dropbox -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --GoogleDrive -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --MS_OneDrive -g sqm_mark_cs1
# Other
$IPT -t mangle -A sqm -m ndpi --Steam -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --PlayStore -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --GoogleDocs -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --YouTubeUpload -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --WhatsAppFiles -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --FTP_DATA -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Git -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --RSYNC -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --WindowsUpdate -g sqm_mark_cs1

# Consider "large" HTTP/S traffic (out/in) as Bulk
# Need some better solution for this
# But should work for most use cases
$IPT -t mangle -A sqm -p tcp -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -m multiport --dports 80,443 -g sqm_mark_cs1

how fast is your connection and what router-device are you using?

150/15 Mbit/s on a wrt1200.

I edited my post above.
To make the rules use goto instead of jump.

Can someone explain this behavior to me:

When i remove the
$IPT -t mangle -A sqm_mark_cs<n> -j sqm_set_dscp
lines (or switch them to -j RETURN),
the sqm_set_dscp fails to mark some packets.

//edit
Okay, when i use jump here obviously the packet will go straight into sqm_set_dscp.
So the traffic/packet counter will be the same as for the sqm_mark_cs mark group.
When using return, the packet seems actually really going back to the main table/chain (mangle/forward)
But then somehow only a small number of packets seem to be marked in sqm_set_dscp chain.
So something seems of here...
Someone has an idea?
//edit end

According to man iptables:

-g, --goto chain
This specifies that the processing should continue in a user specified chain. Unlike the --jump option return will not continue processing in this chain but instead in the chain that called us via --jump.

So when only using gotos and the chain comes along at a return target (e.g. -j RETURN or an extension that is returning) it goes back to the chain that called via -jump.
What happens when there are no -jumps ? I would suspect the packet goes back to main table/chain (e.g. mangle/forward)?
What happens if a packet hits the end of a custom chain? I guess, its the same as hitting a return?

and for jump:
return does nothing? Is this correct?
e.g. When using jump to chain and the packets hits a RETURN there, nothing will happen and the packets travels further along the chain until hitting the end of chain.
Then it goes back to the previous chain?

//edit 2
This is when using return:
Chain sqm_mark_cs1
52209 74.17 MB CONNMARK
52209 74.17 MB RETURN

So roughly 75mb matched and marked.

But the sqm_set_dscp chain shows:
24639 992.52 KB DSCP

maybe i have to switch to mark and use connmark --save-mark and --restore-mark?

//edit 3
Hmm. actually it makes sense...
Since connmark only marks connections and mark marks packets.

//edit 4
Updated the script again.

  1. Restore connmark to packet mark
  2. check if the packet has already been marked
  3. If true, set a connmark according to the iptables rules and go back to main chain
  • If false, go back to main
  1. Packet goes now into the dscp class set chain
  • do also a connmark restore here (since we use this chain also on ingress)
  • match packet mark and set proper dscp class

I think this saves some cpu time?!

Regarding goto/jump
end of chain:
jump:
goes back to previous chain
goto:
goes not back to previous chain.
policy of the main chain is applied. (?)

return:
jump:
goes back to previous chain.
goto:
goes back to chain that called via jump.

is this correct?

Here is a solution using ifb and a combination of tc and iptables.
Logic is almost the same.

egress side: iptables only.
Classify traffic on egress.
Set connmark.
Restore connmark to packet marks.
Set dscp flags on egress packets..

firewall script:
(Not all chains are used in the sqm part, but i use them for other things)

###################################################################
###  Create Custom Chains (Mangle)
###################################################################

$IPT -t mangle -F

### Prerouting
$IPT -t mangle -N zone_lan_prerouting
$IPT -t mangle -N zone_isolated_prerouting
$IPT -t mangle -N zone_wan_prerouting

$IPT -t mangle -A PREROUTING -i br-lan -j zone_lan_prerouting
$IPT -t mangle -A PREROUTING -i br-isolated -j zone_isolated_prerouting
$IPT -t mangle -A PREROUTING -i eth1 -j zone_wan_prerouting

### Forward
$IPT -t mangle -N zone_lan_forward
$IPT -t mangle -N zone_isolated_forward

$IPT -t mangle -A FORWARD -i br-lan -j zone_lan_forward
$IPT -t mangle -A FORWARD -i br-isolated -j zone_isolated_forward
###################################################################
### Make use of Cake's DiffServ4 Implementation
### by using dscp flags
###################################################################

### Setup some custom chains
$IPT -t mangle -N sqm
$IPT -t mangle -N sqm_mark_cs1
$IPT -t mangle -N sqm_mark_cs3
$IPT -t mangle -N sqm_mark_cs7
$IPT -t mangle -N sqm_set_dscp

### Since we dont know if lan users are messing around with dscp flags
### default all packets to cs0 (BE) and override later on
$IPT -t mangle -I zone_lan_prerouting -j DSCP --set-dscp-class CS0 -m comment --comment "Set DSCP CS0 as default"
$IPT -t mangle -I zone_isolated_prerouting -j DSCP --set-dscp-class CS0 -m comment --comment "Set DSCP CS0 as default"

### Only try to match traffic thats is going to be forwarded to internet
$IPT -t mangle -A zone_lan_forward -o eth1 -j sqm
$IPT -t mangle -A zone_lan_forward -o eth1 -j sqm_set_dscp
$IPT -t mangle -A zone_isolated_forward -o eth1 -j sqm
$IPT -t mangle -A zone_isolated_forward -o eth1 -j sqm_set_dscp

### match packet mark and set appropriate dscp flag
$IPT -t mangle -A sqm_set_dscp -m connmark --mark 1 -j DSCP --set-dscp-class CS1 -m comment --comment "CS1 Background"
$IPT -t mangle -A sqm_set_dscp -m connmark --mark 1 -j RETURN
$IPT -t mangle -A sqm_set_dscp -m connmark --mark 3 -j DSCP --set-dscp-class CS3 -m comment --comment "CS3 Streaming"
$IPT -t mangle -A sqm_set_dscp -m connmark --mark 3 -j RETURN
$IPT -t mangle -A sqm_set_dscp -m connmark --mark 7 -j DSCP --set-dscp-class CS7 -m comment --comment "CS7 Latency Sensitive"
$IPT -t mangle -A sqm_set_dscp -m connmark --mark 7 -j RETURN

### set connection mark
### return back to main table/chain
$IPT -t mangle -A sqm_mark_cs1 -j CONNMARK --set-mark 1
$IPT -t mangle -A sqm_mark_cs1 -j RETURN
$IPT -t mangle -A sqm_mark_cs3 -j CONNMARK --set-mark 3
$IPT -t mangle -A sqm_mark_cs3 -j RETURN
$IPT -t mangle -A sqm_mark_cs7 -j CONNMARK --set-mark 7
$IPT -t mangle -A sqm_mark_cs7 -j RETURN

### check if packet has already been marked
### if true return to back to main table/chain
### if false set connmark based on iptables rules below
$IPT -t mangle -A sqm -m connmark ! --mark 0 -j RETURN

###################################################################
# Latency Sensitive (Voice Tin)
###################################################################

### Generic
$IPT -t mangle -A sqm -m ndpi --NTP -g sqm_mark_cs7
### Gaming
$IPT -t mangle -A sqm -m ndpi --CSGO -g sqm_mark_cs7
$IPT -t mangle -A sqm -s 10.0.1.60 -p udp -m multiport ! --dports 9000  -g sqm_mark_cs7 -m comment --comment "PS4 UDP"
$IPT -t mangle -A sqm -s 10.0.1.60 -p tcp -m multiport ! --dports 80,443  -g sqm_mark_cs7 -m comment --comment "PS4 TCP"
$IPT -t mangle -A sqm -p udp -m multiport --dports 5000:5500 -g sqm_mark_cs7 -m comment --comment "League of Legends"
$IPT -t mangle -A sqm -m ndpi --WorldOfWarcraft -g sqm_mark_cs7
### Voice
$IPT -t mangle -A sqm -m ndpi --TeamSpeak -g sqm_mark_cs7
$IPT -t mangle -A sqm -m ndpi --WhatsAppVoice -g sqm_mark_cs7

###################################################################
### Streaming Media (Video Tin)
###################################################################

$IPT -t mangle -A sqm -m ndpi --YouTube -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --NetFlix -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Skype -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --AmazonVideo -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Vevo -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Twitch -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --GoogleHangout -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Spotify -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --Deezer -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --SoundCloud -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --LastFM -g sqm_mark_cs3
### Remote Administration
$IPT -t mangle -A sqm -m ndpi --TeamViewer -g sqm_mark_cs3
$IPT -t mangle -A sqm -m ndpi --VNC -g sqm_mark_cs3

###################################################################
# Background Traffic (Bulk Tin)
###################################################################
### Mail
$IPT -t mangle -A sqm -m ndpi --SMTP -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --SMTPS -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --IMAP -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --IMAPS -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --POP3 -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --POPS -g sqm_mark_cs1
### P2P
$IPT -t mangle -A sqm -m ndpi --BitTorrent -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --eDonkey -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Thunder -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --AppleJuice -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Soulseek -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Gnutella -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --DirectConnect -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --FastTrack -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Stealthnet -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Filetopia -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Usenet -g sqm_mark_cs1
### Cloud
$IPT -t mangle -A sqm -m ndpi --Dropbox -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --GoogleDrive -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --MS_OneDrive -g sqm_mark_cs1
### Other
$IPT -t mangle -A sqm -m ndpi --PlayStore -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --GoogleDocs -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --YouTubeUpload -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --WhatsAppFiles -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --FTP_DATA -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --Git -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --RSYNC -g sqm_mark_cs1
$IPT -t mangle -A sqm -m ndpi --WindowsUpdate -g sqm_mark_cs1

# Consider "large" HTTP/S traffic (out/in) as Bulk
# Need some better solution for this
# But should work for most use cases
$IPT -t mangle -A sqm -p tcp -m connbytes --connbytes 10485760: --connbytes-dir reply --connbytes-mode bytes -m multiport --dports 80,443 -g sqm_mark_cs1
$IPT -t mangle -A sqm -p tcp -m connbytes --connbytes 1048576: --connbytes-dir original --connbytes-mode bytes -m multiport --dports 80,443 -g sqm_mark_cs1

ingress:
restore connmark to fwmark/packet mark
match marks to set tos/dscp flags

this one goes into layer cake script:

	# Set default class to best effort
	$TC filter add dev $IFACE parent ffff: protocol ip prio 10 u32 \
    match u32 0 0 flowid :1 \
	action pedit munge ip tos set 0 \
	pipe csum ip continue
	
    # restore connection's mark value into the packet's fwmark.
	$TC filter add dev $IFACE parent ffff: protocol ip prio 20 u32 \
    match u32 0 0 flowid :1 \
	action connmark continue
	
	# match fwmarks and set tos flag in ip header
	# fix checksum
	# immediately redirect to ifb4eth1 to avoid further unnecessary packet matching
	
	# CS1
    $TC filter add dev $IFACE parent ffff: protocol ip prio 30 u32 \
    match mark 0x1 0xf flowid :1 \
	action pedit munge ip tos set 32 \
	pipe csum ip \
	pipe mirred egress redirect dev $DEV
	
	# CS3
    $TC filter add dev $IFACE parent ffff: protocol ip prio 40 u32 \
    match mark 0x3 0xf flowid :1 \
	action pedit munge ip tos set 96 \
	pipe csum ip \
	pipe mirred egress redirect dev $DEV
	
	# CS7
    $TC filter add dev $IFACE parent ffff: protocol ip prio 50 u32 \
    match mark 0x7 0xf flowid :1 \
	action pedit munge ip tos set 224 \
	pipe csum ip \
	pipe mirred egress redirect dev $DEV
	
	# Redirect everything else to ifb4eth1
    $TC filter add dev $IFACE parent ffff: protocol all prio 100 u32 \
	match u32 0 0 flowid :1 \
	action mirred egress redirect dev $DEV

To make this work, following changes are needed also:

diff --git a/package/kernel/linux/modules/netsupport.mk b/package/kernel/linux/modules/netsupport.mk
index ff76720636..c245d36678 100644
--- a/package/kernel/linux/modules/netsupport.mk
+++ b/package/kernel/linux/modules/netsupport.mk
@@ -695,7 +695,7 @@ $(eval $(call KernelPackage,mppe))
 
 
 SCHED_MODULES = $(patsubst $(LINUX_DIR)/net/sched/%.ko,%,$(wildcard $(LINUX_DIR)/net/sched/*.ko))
-SCHED_MODULES_CORE = sch_ingress sch_fq_codel sch_hfsc sch_htb sch_tbf cls_fw cls_route cls_flow cls_tcindex cls_u32 em_u32 act_mirred act_skbedit
+SCHED_MODULES_CORE = sch_ingress sch_fq_codel sch_hfsc sch_htb sch_tbf cls_fw cls_route cls_flow cls_tcindex cls_u32 em_u32 act_mirred act_pedit act_csum act_skbedit
 SCHED_MODULES_FILTER = $(SCHED_MODULES_CORE) act_connmark sch_netem
 SCHED_MODULES_EXTRA = $(filter-out $(SCHED_MODULES_FILTER),$(SCHED_MODULES))
 SCHED_FILES = $(patsubst %,$(LINUX_DIR)/net/sched/%.ko,$(filter $(SCHED_MODULES_CORE),$(SCHED_MODULES)))
@@ -704,6 +704,7 @@ SCHED_FILES_EXTRA = $(patsubst %,$(LINUX_DIR)/net/sched/%.ko,$(SCHED_MODULES_EXT
 define KernelPackage/sched-core
   SUBMENU:=$(NETWORK_SUPPORT_MENU)
   TITLE:=Traffic schedulers
+  DEPENDS:=+kmod-lib-crc32c
   KCONFIG:= \
        CONFIG_NET_SCHED=y \
        CONFIG_NET_SCH_HFSC \
@@ -719,6 +720,8 @@ define KernelPackage/sched-core
        CONFIG_NET_CLS_TCINDEX \
        CONFIG_NET_CLS_U32 \
        CONFIG_NET_ACT_MIRRED \
+       CONFIG_NET_ACT_PEDIT \
+       CONFIG_NET_ACT_CSUM \
        CONFIG_NET_ACT_SKBEDIT \
        CONFIG_NET_EMATCH=y \
        CONFIG_NET_EMATCH_U32

1 Like

action connmark is OpenWRT specific, and rejected by the kernel developers as fundamentally breaking the semantics of their system, as it puts the connmark on before the RAW table has a chance to look at the packet, if I remember correctly. Basically, it's always going to be a custom somewhat non-kosher solution.

Of course, if it works for you, go for it, but I think that's why it's not more widely used.

the veth solution allows you to set DSCP with iptables at a fully standard time and place.

Hopefully someday we'll have a good solution to this situation. Currently there's nothing ideal.

Are you sure that action connmark is openwrt specific?

https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/tree/tc/m_connmark.c

Seems like it was accepted upstream?

According to this picture you remembered correctly :wink:
But for me this is not a big deal and agree all solutions i found so far are quite bit suboptimal.

If cake would expose its tins as classes i would be possible to switch over to ip/port (or something else) based matching and put the traffic directly into the tins with tc.

But this is also suboptimal as it doesn't allow more complex matching.

And in this situation (router/nat) i think the best solution is to match traffic on the egress side.
Since all (relevant) connections originate from there (egress/lan side) anyway.

I edited my post above to only use connmark.

That's what I do, and is the purpose of the veth. The veth just provides a place to inject routed traffic into the lan bridge through a qdisc.

It gets complicated if you have vlans. It's possible to put your real wan iface and a veth both into a separate namespace, and treat the other end of the veth as your main router namespaces wan...

The latest cake now supports tc filter, so DSCP mark is no longer needed.

Nice.
But only in the upstream-4.18 branch or?

I just did a test, and currently cake in openwrt still cannot use tc filter. I think your solution is the best.

DSCP marking is still useful as it will affect the use of different priority queues in managed switches and WMM in the APs particularly for larger networks with multiple stand alone Enterprise APs and VoIP installs you would want to get the DSCP marks right.

After looking into WMM (ty for the hint) using CS3 for video seems bad.
AF4x seems better.
Don't use cake wash on ingress so wmm can make use of the dscp flags.
For egress i don't know if the common smart phone OSs use dscp.
Then the default override to CS0 needs to be changed?

So here is an updated version:

  • Switched to port and hostname based matching (to get rid of ndpi)
  • Changed the connmark/mark to use the actual decimal dscp value
    (maybe use hex values instead ?)

/etc/firewall.user

###################################################################
###  Create Custom Mangle Chains
###################################################################

### Flush Mangle Table
$IPT -t mangle -F

### Prerouting
$IPT -t mangle -N zone_lan_prerouting

$IPT -t mangle -A PREROUTING -i br-lan -j zone_lan_prerouting

### Forward
$IPT -t mangle -N zone_lan_forward

$IPT -t mangle -A FORWARD -i br-lan -j zone_lan_forward

###################################################################
### Make use of Cake's DiffServ4 Implementation
### by using dscp flags
###################################################################

### Setup some custom chains
$IPT -t mangle -N dscp_classify
$IPT -t mangle -N dscp_mark_cs6
$IPT -t mangle -N dscp_mark_af41
$IPT -t mangle -N dscp_mark_cs1
$IPT -t mangle -N dscp_set

### Since we dont know if lan users are messing around with dscp flags
### default all packets to cs0 (BE) and override later on
$IPT -t mangle -A zone_lan_prerouting -j DSCP --set-dscp-class CS0 -m comment --comment "Set DSCP CS0/BE as default"

### Setup dscp chain jumps
$IPT -t mangle -A zone_lan_forward -o eth1 -j dscp_classify
$IPT -t mangle -A zone_lan_forward -o eth1 -j dscp_set

### match packet mark and set appropriate dscp flag
$IPT -t mangle -A dscp_set -m connmark --mark 48 -j DSCP --set-dscp-class CS6 -m comment --comment "CS6 Voice Tin"
$IPT -t mangle -A dscp_set -m connmark --mark 48 -j RETURN
$IPT -t mangle -A dscp_set -m connmark --mark 34 -j DSCP --set-dscp-class AF41 -m comment --comment "AF41 Video Tin"
$IPT -t mangle -A dscp_set -m connmark --mark 34 -j RETURN
$IPT -t mangle -A dscp_set -m connmark --mark 8 -j DSCP --set-dscp-class CS1 -m comment --comment "CS1 Bulk Tin"
$IPT -t mangle -A dscp_set -m connmark --mark 8 -j RETURN

### set connection mark
### return back to main table/chain
$IPT -t mangle -A dscp_mark_cs6 -j CONNMARK --set-mark 48
$IPT -t mangle -A dscp_mark_cs6 -j RETURN
$IPT -t mangle -A dscp_mark_af41 -j CONNMARK --set-mark 34
$IPT -t mangle -A dscp_mark_af41 -j RETURN
$IPT -t mangle -A dscp_mark_cs1 -j CONNMARK --set-mark 8
$IPT -t mangle -A dscp_mark_cs1 -j RETURN

### check if packet has already been marked
### if true return to back to main table/chain
### if false set connmark based on iptables rules below
$IPT -t mangle -A dscp_classify -m connmark ! --mark 0 -j RETURN

###################################################################
# Latency Sensitive (Voice Tin)
###################################################################

### IPset (Hostnames)
$IPT -t mangle -A dscp_classify -m set --match-set dscp_cs6 dst -g dscp_mark_ef -m comment --comment "IPSet CS6 Hostname Match"

### Generic
$IPT -t mangle -A dscp_classify -p icmp -m limit --limit 1/sec --limit-burst 30 -g dscp_mark_cs6 -m comment --comment "ICMP"
### Gaming
# Steam Generic
$IPT -t mangle -A dscp_classify -p udp --sport 27000:27015 -g dscp_mark_cs6 -m comment --comment "Steam Generic"
# Steam CS:GO
$IPT -t mangle -A dscp_classify -p udp --dport 27020:27100 -g dscp_mark_cs6 -m comment --comment "CS:GO"
# PS4
$IPT -t mangle -A dscp_classify -p udp -s 10.0.1.60 -m multiport ! --dports 9000 -g dscp_mark_cs6 -m comment --comment "PS4 UDP"
$IPT -t mangle -A dscp_classify -p tcp -s 10.0.1.60 -m multiport ! --dports 80,443 -g dscp_mark_cs6 -m comment --comment "PS4 TCP"
# League of Legends
$IPT -t mangle -A dscp_classify -p udp --dport 5000:5500 -g dscp_mark_cs6 -m comment --comment "League of Legends"
# World Of Warcraft
$IPT -t mangle -A dscp_classify -p tcp -m multiport --dports 1119,3724 -g dscp_mark_cs6 -m comment --comment "World of Warcraft"

###################################################################
### Streaming Media (Video Tin)
###################################################################

# IPset (Hostnames)
$IPT -t mangle -A dscp_classify -m set --match-set dscp_af41 dst -g dscp_mark_af41 -m comment --comment "IPSet AF41 Hostname Match"

###################################################################
# Background Traffic (Bulk Tin)
###################################################################

### IPset (Hostnames)
$IPT -t mangle -A dscp_classify -m set --match-set dscp_cs1 dst -g dscp_mark_cs1 -m comment --comment "IPSet CS1 Hostname Match"

### Mail
# SMTP/S
$IPT -t mangle -A dscp_classify -p tcp -m multiport --dports 25,465,587 -g dscp_mark_cs1 -m comment --comment "SMTP/S"
# IMAP/S
$IPT -t mangle -A dscp_classify -p tcp -m multiport --dports 143,993 -g dscp_mark_cs1 -m comment --comment "IMAP/S"
# POP/S
$IPT -t mangle -A dscp_classify -p tcp -m multiport --dports 110,995 -g dscp_mark_cs1 -m comment --comment "POP3"

# Consider "large" HTTP/S traffic (out/in) as Bulk
# Need some better solution for this
# But should work for most use cases
$IPT -t mangle -A dscp_classify -p tcp -m connbytes --connbytes 10485760: --connbytes-dir reply --connbytes-mode bytes -m multiport --dports 80,443 -g dscp_mark_cs1
$IPT -t mangle -A dscp_classify -p tcp -m connbytes --connbytes 1048576: --connbytes-dir original --connbytes-mode bytes -m multiport --dports 80,443 -g dscp_mark_cs1

/etc/config/firewall

config ipset
	option name 'dscp_cs6'
	option family 'ipv4'
	option storage 'hash'
	option hashsize '64'
	option match 'ip'
	option timeout '86400'

config ipset
	option name 'dscp_af41'
	option family 'ipv4'
	option storage 'hash'
	option hashsize '64'
	option match 'ip'
	option timeout '86400'

config ipset
	option name 'dscp_cs1'
	option family 'ipv4'
	option storage 'hash'
	option hashsize '64'
	option match 'ip'
	option timeout '86400'

/usr/lib/sqm/layer_cake.qos

    # Set default class to best effort
    $TC filter add dev $IFACE parent ffff: protocol ip prio 10 u32 \
    match u32 0 0 flowid :1 \
    action pedit munge ip tos set 0 \
    pipe csum ip continue
	
    # restore connection's mark value into the packet's fwmark.
    $TC filter add dev $IFACE parent ffff: protocol ip prio 20 u32 \
    match u32 0 0 flowid :1 \
    action connmark continue
	
    # match fwmarks and set tos flag in ip header
    # fix checksum
    # immediately redirect to ifb4eth1 to avoid further unnecessary packet matching

    # CS6
    $TC filter add dev $IFACE parent ffff: protocol ip prio 50 u32 \
    match mark 48 0xff flowid :1 \
    action pedit munge ip tos set 192 \
    pipe csum ip \
    pipe mirred egress redirect dev $DEV

    # AF41
    $TC filter add dev $IFACE parent ffff: protocol ip prio 40 u32 \
    match mark 34 0xff flowid :1 \
    action pedit munge ip tos set 136 \
    pipe csum ip \
    pipe mirred egress redirect dev $DEV

    # CS1
    $TC filter add dev $IFACE parent ffff: protocol ip prio 30 u32 \
    match mark 8 0xff flowid :1 \
    action pedit munge ip tos set 32 \
    pipe csum ip \
    pipe mirred egress redirect dev $DEV
	
    # Redirect everything else to ifb4eth1
    $TC filter add dev $IFACE parent ffff: protocol ip prio 100 u32 \
    match u32 0 0 flowid :1 \
    action mirred egress redirect dev $DEV

some hostname matching examples:
dnsmasq needs ipset support
also add /etc/dnsmasq.conf to files that should be saved.

/etc/dnsmasq.conf

### AF41
# Youtube
ipset=/googlevideo.com/dscp_af41
# NetFlix
ipset=/nflxvideo.net/dscp_af41
# AmazonVideo
ipset=/s3.ll.dash.row.aiv-cdn.net/d25xi40x97liuc.cloudfront.net/aiv-delivery.net/dscp_af41
# Facebook
ipset=/fbcdn.net/dscp_af41
# Twitch
ipset=/ttvnw.net/dscp_af41
# VeVo
ipset=/vevo.com/dscp_af41
# Spotify
ipset=/audio-fa.scdn.cot/dscp_af41
# Deezer
ipset=/deezer.com/dscp_af41
# SoundCloud
ipset=/sndcdn.com/dscp_af41
# last.fm
ipset=/last.fm/dscp_af41

### CS1
# Steam Download
ipset=/steamcontent.com/dscp_cs1
# PSN Download
ipset=/gs2.ww.prod.dl.playstation.net/dscp_cs1
# DropBox
ipset=/dropbox.com/dropboxstatic.com/dropbox-dns.com/log.getdropbox.com/dscp_cs1
# Google Drive
ipset=/drive.google.com/drive-thirdparty.googleusercontent.com//dscp_cs1
# Google Docs
ipset=/docs.google.com/docs.googleusercontent.com/dscp_cs1
# PlayStore Download
ipset=/gvt1.com/dscp_cs1
# WhatsApp Files
ipset=/mmg-fna.whatsapp.net/dscp_cs1
# Youtube Upload
ipset=/upload.youtube.com/upload.video.google.com/dscp_cs1
# WindowsUpdate
ipset=/windowsupdate.com/update.microsoft.com/dscp_cs1

1 Like

EF is the standard for voice but unfortunately the Linux device driver writers didn't know this, they use the top 3 bits and so EF winds up in VID WMM queue, because of this I strongly recommend CS6 for voice

1 Like

Can give more detail please?

This entire DSCP to WMM mapping...
Not all manufacturers use 100% the same mapping?

But it seems most use this mapping:
Maps precedence 6 and 7 to WMM Voice.
That would be CS6 and CS7.
EF (precedence 5) would results in the WMM video tin then.

So you right cs6 seems to be a better choice.
Ty =)

//edit
Answering my own question x)
I know what you mean now.
WMM (apparently?) uses precedence 4 and 5 for the "video access category" .
So CS5, VA, EF would all end up in this category.
Thanks again for pointing this out.

For reference what the kernel does, see:
https://wireless.wiki.kernel.org/en/developers/documentation/mac80211/queues

For 802.1p see:
https://en.wikipedia.org/wiki/IEEE_P802.1p

For 802.11e (WMM) see:
https://en.wikipedia.org/wiki/IEEE_802.11e-2005

Note how 802.1p still calls the "CS5" bit pattern VO/Voice, ist is only WMM which gets it "wrong"... well not technically wrong, as there is no hard definition for the meaning of the bits, just different traditions.

The whole thing is a mess, IMHO mainly caused by the unwillingness/inability to re-define DSCP 0 to denote background instead of best-effort, thereby throwing away a simple magnitude to priority mapping that would be understandable.