NFtables and QoS in 2021

_FailSafe · November 28, 2021, 4:57am

No doubt it will work as you have written it, I just thought a rule was evaluated sequentially (LTR) and at the first falsy evaluation the engine bails and moves to the next rule. If that is true, as written an IPv4 and IPv6 packet (in the FaceTime port range) would both necessitate the sport and dport evaluations twice per packet:

udp sport $facetime_ports udp dport $facetime_ports ip dscp set af41 counter comment "Facetime AF41"
udp sport $facetime_ports udp dport $facetime_ports ip6 dscp set af41 counter comment "Facetime AF41"

It seems less CPU/time would be consumed if written as:

ip protocol udp udp sport $facetime_ports udp dport $facetime_ports ip dscp set af41 counter comment "Facetime AF41"
ip6 nexthdr udp udp sport $facetime_ports udp dport $facetime_ports ip6 dscp set af41 counter comment "Facetime AF41"

Please know I am not trying to argue this, but am instead trying to exercise my understanding of nftables. If my understanding above is incorrect, please call me out so I can learn

dave14305 · November 28, 2021, 5:02am

inet family tables seem to know what to do.
https://wiki.nftables.org/wiki-nftables/index.php/Nftables_families#inet

Within a table of inet family, both IPv4 and IPv6 packets traverse the same rules. Rules for IPv4 packets don't affect IPv6 packets and vice-versa.

_FailSafe · November 28, 2021, 5:04am

Fair enough! I interpreted that same explanation differently in light of other readings, but you have persuaded me to go rewrite a couple of my other rules to take advantage of this benefit.

anon78773196 · November 28, 2021, 5:04am

after reboot nft list ruleset doesn't work

i must do restart for launch do you has a ideas ?

_FailSafe · November 28, 2021, 5:06am

I had to update & move the hotplug script for my nftables to come up automatically upon reboot.

Note this updated file and location:

root@OpenWrt:~# cat /etc/hotplug.d/iface/13-nfthotplug
#!/bin/sh

[ -n "$DEVICE" ] || exit 0

[ "$ACTION" = ifup ] && /etc/init.d/nftables enabled && {
	/etc/init.d/nftables restart
	logger -t nftables "Restarting nftables due to $ACTION of $INTERFACE ($DEVICE)"
}

@dlakelan If others find this useful, it might be beneficial to update it in your repo as well.

anon78773196 · November 28, 2021, 5:13am

perfect it works

@dave14305

just before that i add ?

lets create a tagger table for more sophisticated tagging that

can't happen at ingress since that's too early in the packet

processing chain

table inet cttags {

    set bulk4 {
            type ipv4_addr
            timeout 1d
            counter
            comment "Bulk IPv4"
    }

or my script begin with the sqm script

dlakelan · November 28, 2021, 5:17am

Thanks! I like that updated hotplug so I pushed that to the repo.

anon78773196 · November 28, 2021, 5:21am

@dave14305

ah sorry it's if only i do the idir script i have to comment ok sorry my understanding of english is sometimes very bad haha

why my Bufferbloat got bad compared to usual with nftables,

_FailSafe · November 28, 2021, 2:17pm

After having this run all night, there is a counter that stands out as peculiar to me.

Ruleset:

Summary

...
chain in_dscp {
		type filter hook postrouting priority filter; policy accept;
		oifname "eth0" ct mark & 0x01c00000 == 0x00000000 counter packets 1173134 bytes 698031751 jump qos_sqm
	}

	chain qos_sqm {
		ct mark & 0x02000000 == 0x00000000 counter packets 12775 bytes 1112306 goto cttags
	}

	chain qos_sqm_remap {
		ct mark set (@nh,8,8 & 252) >> 2 counter packets 264 bytes 17952
		ct mark set @nh,0,16 & 4032 >> 6 counter packets 0 bytes 0
		ct mark set ct mark << 26 | 0x03000000
	}

	chain cttags {
		ip dscp != cs0 counter packets 264 bytes 17952 goto qos_sqm_remap
		ip6 dscp != cs0 counter packets 0 bytes 0 goto qos_sqm_remap
		ip daddr @bulk4 ip dscp set cs1 counter packets 278 bytes 17792 comment "bulk4 to CS1"
		ip6 daddr @bulk6 ip6 dscp set cs1 counter packets 36 bytes 3024 comment "bulk6 to CS1"
		ip daddr @besteffort4 ct mark set 0x01000000 counter packets 19 bytes 1216 comment "besteffort4 to CS0"
		ip6 daddr @besteffort6 ct mark set 0x01000000 counter packets 61 bytes 15861 comment "besteffort6 to CS0"
		ip daddr @video4 ip dscp set cs3 counter packets 4 bytes 256 comment "video4 to CS3"
		ip6 daddr @video6 ip6 dscp set cs3 counter packets 461 bytes 104762 comment "video6 to CS3"
		ip daddr @voice4 ip dscp set cs4 counter packets 0 bytes 0 comment "voice4 to CS4"
		ip6 daddr @voice6 ip6 dscp set cs4 counter packets 0 bytes 0 comment "voice6 to CS4"
		ip daddr 17.0.0.0/8 tcp dport { 993, 5223 } ip dscp set cs0 counter packets 13 bytes 832 comment "Apple Mail and APNS CS0"
		udp sport { 3478-3497, 16384-16387, 16393-16402 } udp dport { 3478-3497, 16384-16387, 16393-16402 } ip dscp set cs4 counter packets 0 bytes 0 comment "Facetime CS4"
		udp sport { 3478-3497, 16384-16387, 16393-16402 } udp dport { 3478-3497, 16384-16387, 16393-16402 } ip6 dscp set cs4 counter packets 0 bytes 0 comment "Facetime CS4"
		ip daddr @zoom4 udp dport 8801-8810 ip dscp set cs4 counter packets 0 bytes 0 comment "Zoom CS4"
		ip6 daddr @zoom6 udp dport 8801-8810 ip6 dscp set cs4 counter packets 0 bytes 0 comment "Zoom CS4"
		udp sport 4500 udp dport 4500 ip dscp set cs4 counter packets 1 bytes 396 comment "VZW WiFi Calling CS4"
		udp sport 4500 udp dport 4500 ip6 dscp set cs4 counter packets 0 bytes 0 comment "VZW WiFi Calling CS4"
		ip daddr @webex4 tcp dport { 444, 5004, 33434 } ip dscp set cs4 counter packets 216 bytes 13824 comment "WebEx TCP CS4"
		ip6 daddr @webex6 tcp dport { 444, 5004, 33434 } ip6 dscp set cs4 counter packets 0 bytes 0 comment "WebEx TCP CS4"
		ip daddr @webex4 udp dport { 5004, 9000, 33434-33598 } ip dscp set cs4 counter packets 0 bytes 0 comment "WebEx UDP CS4"
		ip6 daddr @webex6 udp dport { 5004, 9000, 33434-33598 } ip6 dscp set cs4 counter packets 0 bytes 0 comment "WebEx UDP CS4"
		tcp sport != 2049 ip dscp < cs4 ct bytes >= 35000000 counter packets 0 bytes 0 ip dscp set cs1
		tcp sport != 2049 ip6 dscp < cs4 ct bytes >= 35000000 counter packets 0 bytes 0 ip6 dscp set cs1
		ip dscp < cs5 udp dport != { 53, 80, 443 } udp sport != { 53, 80, 443 } meter udp4meter size 65535 { ip saddr . ip daddr . udp sport . udp dport limit rate over 200/second burst 100 packets } counter packets 0 bytes 0 ct mark set 0x00000055
		ip6 dscp < cs5 udp dport != { 53, 80, 443 } udp sport != { 53, 80, 443 } meter udp6meter size 65535 { ip6 saddr . ip6 daddr . udp sport . udp dport limit rate over 200/second burst 100 packets } counter packets 0 bytes 0 ct mark set 0x00000055
		ct mark 0x00000055 numgen random mod 10000 < 50 ct mark set 0x00000000 counter packets 0 bytes 0 comment "small probability to unmark over-threshold connections"
		ct mark != 0x00000055 ip dscp < cs5 udp dport != { 53, 80, 443 } udp sport != { 53, 80, 443 } ct avgpkt 0-450 counter packets 167 bytes 42581 ip dscp set cs4
		ct mark != 0x00000055 ip6 dscp < cs5 udp dport != { 53, 80, 443 } udp sport != { 53, 80, 443 } ct avgpkt 0-450 counter packets 92 bytes 25045 ip6 dscp set cs4
		ct mark set (@nh,8,8 & 252) >> 2 counter packets 7746 bytes 527040
		ct mark set @nh,0,16 & 4032 >> 6 counter packets 4765 bytes 567314
		ct mark set ct mark << 26 | 0x02000000 counter packets 12511 bytes 1094354
	}

The nonexistent IPv6 remapped packets is what stands out to me. I am wondering if the first of these two lines in cttags is catching IPv6 as well?

ip dscp != cs0 counter packets 264 bytes 17952 goto qos_sqm_remap
ip6 dscp != cs0 counter packets 0 bytes 0 goto qos_sqm_remap

I may try adding meta nfproto to these two lines to see if that helps. But I'm certainly open to other thoughts/suggestions.

On a separate note, given that I run a NextDNS client on my OpenWRT host for all DNS on my subnets, where would be the right place to add a rule to prioritize just its traffic? It is DoH, so I realize that will be prioritizing :443, but ideally it would be just for connections initiated from the OpenWRT box. Would I need to match on lo?

dave14305 · November 28, 2021, 2:17pm

If you don’t use the ldir script with the tc filters for ctinfo to restore the dscp mark, then my config won’t work well for you since only the first packets will get dscp changed and the rest will end up in the wrong tins.

dave14305 · November 28, 2021, 2:19pm

It might just be that you have no locally-generated IPv6 traffic that sends with a DSCP value already set. I find it rare myself.

_FailSafe · November 28, 2021, 9:39pm

Have any of you tried something like this?

dlakelan · November 28, 2021, 9:41pm

I haven't tried that but yes you could match on iif lo or on ip saddr or ip6 saddr

dave14305 · November 28, 2021, 9:56pm

Yeah, what’s the equivalent of the OUTPUT chain? Loopback doesn’t sound right to me.

dlakelan · November 28, 2021, 10:44pm

ah set an output hook...

table foo {
   type filter hook output priority 0; policy accept
   ...
}

_FailSafe · November 28, 2021, 10:56pm

Did you use the iface hotplug script I referred to yesterday?

dlakelan · November 28, 2021, 11:15pm

but if you put it in postrouting it should handle both forwarded and output at the same time, in which case in the postrouting chain you maybe can match tcp dport and ip/ip6 daddr together to hit just traffic to the DNS you use.

dave14305 · November 29, 2021, 12:30am

Getting back to the original conf file…What are the pros and cons of using the ingress hook for tagging LAN traffic, versus a POSTROUTING hook?

dlakelan · November 29, 2021, 12:34am

The pros of an ingress hook are that you get the tags on early enough to go through any IFB device... though on LAN this seems largely unnecessary. The cons are you can't use conntrack.

The pros of postrouting are that you can handle everything there and you can use conntrack for things like rate related rules. The cons being that you can't get to it before an IFB. So it's potentially a bigger con for WAN incoming packets.

On my actual router I think I do everything in postrouting ~~or forward or something like that~~, and I put all my shapers on egress of wan/lan/guest etc.

anon78773196 · November 29, 2021, 12:38am

Yes i has use and he work