QoS and nftables ... some findings to share

actually @LeTran, in your case you actually don't need the first script at all, since you are just bridging... that first script is entirely for people who want to have the router do routing and firewalling... you are bridging between eth0.3 and eth1.2 and trying to DSCP tag in the middle... so you have a different requirement and don't need any firewall as there's a separate device doing all the routing and firewalling for you!

let me have a look at the second script and update that a little. You can install just the second script.

OK, here's a script explicitly for your case, where you have a "bump in the wire", two interfaces where you receive packets, tag them, and send them out the other interface...

flush ruleset


define inif1 = eth0.3
define inif2 = eth1.2


table netdev retag {
    chain tagin1 {
    	  ## mangle priority, retag anything coming in this interface
	  
    	  type filter hook ingress device $inif1 priority -149; policy accept;

	  ip dscp set cs0 ## start with everything at cs0
	  ip6 dscp set cs0

	  # tag ntp packets very high priority
	  ip protocol udp udp sport ntp ip dscp set cs6
	  ip6 nexthdr udp udp sport ntp ip6 dscp set cs6


	  ## icmp/icmpv6 gets high priority but you might not want
	  ## this!  it does let you find out what the round trip time
	  ## for high priority packets is by just using ping though
	  ip protocol icmp ip dscp set cs5
	  ip6 nexthdr icmpv6 ip6 dscp set cs5 

	  ## game traffic on ip and ipv6
	  udp dport {7000-9000, 27000-27200} ip dscp set cs5
	  udp sport {7000-9000, 27000-27200} ip dscp set cs5

	  ip6 nexthdr udp udp dport {7000-9000, 27000-27200} ip6 dscp set cs5
	  ip6 nexthdr udp udp sport {7000-9000, 27000-27200} ip6 dscp set cs5

    }


    chain tagin2 {
    	  ## mangle priority, retag anything coming from this interface
	  
    	  type filter hook ingress device $inif2 priority -149; policy accept;

	  ip dscp set cs0 ## start with everything at cs0
	  ip6 dscp set cs0

	  # tag ntp packets very high priority
	  ip protocol udp udp sport ntp ip dscp set cs6
	  ip6 nexthdr udp udp sport ntp ip6 dscp set cs6


	  ## icmp/icmpv6 gets high priority but you might not want
	  ## this!  it does let you find out what the round trip time
	  ## for high priority packets is by just using ping though
	  ip protocol icmp ip dscp set cs5
	  ip6 nexthdr icmpv6 ip6 dscp set cs5 

	  ## game traffic on ip and ipv6
	  udp dport {7000-9000, 27000-27200} ip dscp set cs5
	  udp sport {7000-9000, 27000-27200} ip dscp set cs5

	  ip6 nexthdr udp udp dport {7000-9000, 27000-27200} ip6 dscp set cs5
	  ip6 nexthdr udp udp sport {7000-9000, 27000-27200} ip6 dscp set cs5

    }



}

you can add ports to the port list as needed for your particular games

1 Like

So what i did is replace only your sample port from both tagin1 and tagin2

 ## game traffic on ip and ipv6
	  udp dport {7000-9000, 27000-27200} ip dscp set cs5
	  udp sport {7000-9000, 27000-27200} ip dscp set cs5

	  ip6 nexthdr udp udp dport {7000-9000, 27000-27200} ip6 dscp set cs5
	  ip6 nexthdr udp udp sport {7000-9000, 27000-27200} ip6 dscp set cs5

to this

	  udp dport {3074, 3478, 4379-4380, 27000-27031, 27036} ip dscp set cs5
	  udp sport {3074, 3478, 4379-4380, 27000-27031, 27036} ip dscp set cs5

	  ip6 nexthdr udp udp dport {3074, 3478, 4379-4380, 27000-27031, 27036} ip6 dscp set cs5
	  ip6 nexthdr udp udp sport {3074, 3478, 4379-4380, 27000-27031, 27036} ip6 dscp set cs5

Can I change the cs5 to cs6 or even cs7, I want it as high as possible XD

From https://dl.lochnair.net/Bufferbloat/Cake/tc-cake.8.html

I would try using cake with diffserv4 first, and use dscp tag cs5 which puts your game traffic in the high priority tin, which is guaranteed at least 25% of the bandwidth. If you use diffserv3 it's kind of not fine grained enough, and with diffserv8 it's too fine grained

Cool! I add diffserv4 in advanced option for both eth0.3 and eth1.2. Do i need "nat dual-srchost" and "nat dual-dsthost ingress"?

In your case put "nonat flows" because you're outside the firewall and so you have no information about the NAT at the point where you are queueing outside the firewall...

Btw, talking about dual-srchost and dual-dsthost, i read your link, dual-srchost or dual-dsthost seems give fairness to everybody so if someone torrenting, it will not effect the others? But since we add tag for gaming packet, then do i still need dual-srchost or dual-dsthost? It's ok for everyone else to be slow, because of their torrenting and downloading

the dual-srchost and dual-dsthost will not work the way it usually does because you are outside the firewall. just leave them off and put "nonat flows diffserv4"

1 Like

anyway that I can see my udp packet counting in mangle or somewhere? I just want to make sure my packet is prioritized

easiest thing is to look at tc -s qdisc and see how many packets are going through the high priority tin

I see 4 tins, bulk, best effort, video, voice. I'm not sure if high priority in Voice tin.
I see you set every packets come in is cs0 by this script

## mangle priority, retag anything coming in this interface
	  
    	  type filter hook ingress device $inif1 priority -149; policy accept;

	  ip dscp set cs0 ## start with everything at cs0
	  ip6 dscp set cs0

so it should be 2 tins to be fill up because only 2 value use: cs0 and cs5?(i don't know which tin though). But when i look at all 4 tins, all 4 of the tins fill up it seems to me ips tagging also mess up with our rule(or i do something wrong). Should I set "Squash DSCP on inbound packets (ingress)" to Squash and "Ignore DSCP on ingress" to Ignore for both eth0.3 and eth1.2?

How do i reset the qdisc number of all 4 tins so everything back to 0? it's easier for me to monitor when everything back to 0

reset should happen if you simply restart the WAN interface I think.

@dlakelan Should I set "Squash DSCP on inbound packets (ingress)" to Squash and "Ignore DSCP on ingress" to Ignore for both eth0.3 and eth1.2? I'm afraid doing that will remove your tagging too

Don't do those things. we control dscp in this example entirely by our own tagging, so we want cake to respond to it!

Here is a snapshot that i took. I wonder if your script work as the way you want to or not because i'm not sure. Left is router just restart, middle is 15 mins of netflix, right is 1 hour of gaming. Gaming is not very responsive as before without custom qos. I'm still testing. Hope i can figure out why

What does this script do? Why do i need to find out the round trip time? It help the high priority packets?

## icmp/icmpv6 gets high priority but you might not want
	  ## this!  it does let you find out what the round trip time
	  ## for high priority packets is by just using ping though
	  ip protocol icmp ip dscp set cs5
	  ip6 nexthdr icmpv6 ip6 dscp set cs5 

I'm not sure why the first qdisc is using so much bulk while eth1.2 is going mostly into voice and best effort. I'm also not sure why anything gets into video.

Which file are you using to put the nftables script in? Is it /etc/nftables.conf?

and do you have

nft -f /etc/nftables.conf

in your /etc/rc.local ?

what happens when you run nft list ruleset ? does it show the tables?

This is all the script of nftables.conf i put in /etc/nftables.conf

flush ruleset


define inif1 = eth0.3
define inif2 = eth1.2


table netdev retag {
    chain tagin1 {
    	  ## mangle priority, retag anything coming in this interface
	  
    	  type filter hook ingress device $inif1 priority -149; policy accept;

	  ip dscp set cs0 ## start with everything at cs0
	  ip6 dscp set cs0

	  # tag ntp packets very high priority
	  ip protocol udp udp sport ntp ip dscp set cs6
	  ip6 nexthdr udp udp sport ntp ip6 dscp set cs6


	  ## icmp/icmpv6 gets high priority but you might not want
	  ## this!  it does let you find out what the round trip time
	  ## for high priority packets is by just using ping though
	  ip protocol icmp ip dscp set cs5
	  ip6 nexthdr icmpv6 ip6 dscp set cs5 

	  ## game traffic on ip and ipv6
	  udp dport {3074, 3478, 4379-4380, 27000-27031, 27036} ip dscp set cs5
	  udp sport {3074, 3478, 4379-4380, 27000-27031, 27036} ip dscp set cs5

	  ip6 nexthdr udp udp dport {3074, 3478, 4379-4380, 27000-27031, 27036} ip6 dscp set cs5
	  ip6 nexthdr udp udp sport {3074, 3478, 4379-4380, 27000-27031, 27036} ip6 dscp set cs5

    }


    chain tagin2 {
    	  ## mangle priority, retag anything coming from this interface
	  
    	  type filter hook ingress device $inif2 priority -149; policy accept;

	  ip dscp set cs0 ## start with everything at cs0
	  ip6 dscp set cs0

	  # tag ntp packets very high priority
	  ip protocol udp udp sport ntp ip dscp set cs6
	  ip6 nexthdr udp udp sport ntp ip6 dscp set cs6


	  ## icmp/icmpv6 gets high priority but you might not want
	  ## this!  it does let you find out what the round trip time
	  ## for high priority packets is by just using ping though
	  ip protocol icmp ip dscp set cs5
	  ip6 nexthdr icmpv6 ip6 dscp set cs5 

	  ## game traffic on ip and ipv6
	  udp dport {3074, 3478, 4379-4380, 27000-27031, 27036} ip dscp set cs5
	  udp sport {3074, 3478, 4379-4380, 27000-27031, 27036} ip dscp set cs5

	  ip6 nexthdr udp udp dport {3074, 3478, 4379-4380, 27000-27031, 27036} ip6 dscp set cs5
	  ip6 nexthdr udp udp sport {3074, 3478, 4379-4380, 27000-27031, 27036} ip6 dscp set cs5

    }



}

You write "/usr/sbin/nft -f /etc/nftables.conf" before. Now i have to fix it to
nft -f /etc/nftables.conf

you meant i have to run nft list ruleset in the command line? nothing show up

what I had before is fine, it is a little more specific, better for the script, don't change.

aha! The script has not run. probably a reboot of the device is all that's needed. If you have already rebooted the device then what happens when you run

/etc/rc.local

manually on the command line? errors? also if you run that with no errors, then what shows for nft list ruleset ? if nothing then we have a problem.

you meant cat /etc/rc.local? because /etc/rc.local i got '-ash: /etc/rc.local: Permission denied'
Here is the log of 'cat /etc/rc.local'

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

/usr/sbin/nft -f /etc/nftables.conf

exit 0

I restarted the Wrt already, nothing show up
Capture5

actually it turns out that these commands are sourced by a shell.. you can do

sh /etc/rc.local

from the command line, then nft list ruleset if that shows nothing then we really need to debug something.

this is what i get

root@OpenWrt:~# sh /etc/rc.local
/etc/nftables.conf:12:43-43: Error: syntax error, unexpected '$', expecting string
          type filter hook ingress device $inif1 priority -149; policy accept;
                                             ^
/etc/nftables.conf:41:43-43: Error: syntax error, unexpected '$', expecting string
          type filter hook ingress device $inif2 priority -149; policy accept;