NFtables and QoS in 2021

[Lynx]

@dlakelan

Remind me why you want the egress hook?
For tagging in the downstream direction just process the packets in regular forward hook chain. Oh wait I see you want it to go through an iFB, I guess you want to dscp tag it before it hits the IFB?

Exactly - I want one interface for CAKE for upload and one interface for CAKE for download notwithstanding having WAN/WireGuard and br-lan/br-guest.

Presently I achieve this by setting up two IFB interfaces as follows:

That is, I create 'ifb-ul' by mirroring the ingress from br-lan and br-guest and I create 'ifb-dl' by mirroring the egress from br-lan and br-guest. I also take care to skip out the LAN-LAN traffic.

@dlakelan

I guess probably the wire guard doesn't have an ingress?

It does, but it's layer 3:

IFBs in WireGuard Context

No, the problem is that a wireguard interface is layer 3 only, so the packets are missing an Ethernet header. Which means that just redirecting won't work. You could do the redirecting with a BPF program and have that add an Ethernet header first (and remove it again later), but that becomes messy quickly...

And so mirroring/forwarding from the WireGuard interface won't work properly.

Therefore instead I create my IFB for download traffic by taking the egress from br-lan and br-guest.

The tc-based solution for my dual ifb approach looks like this:

    tc qdisc add dev br-lan handle ffff: ingress
    tc qdisc add dev br-guest handle ffff: ingress
	tc qdisc add dev br-lan handle 1: root prio
	tc qdisc add dev br-guest handle 1: root prio

	# capture upload (ingress) on br-lan and br-guest
	tc filter add dev br-lan parent ffff: protocol ip prio 1 u32 match ip dst 192.168.1.0/24 action pass
	tc filter add dev br-lan parent ffff: protocol ip prio 2 matchall action mirred egress redirect dev ifb-ul
	tc filter add dev br-guest parent ffff: protocol ip prio 1 u32 match ip dst 192.168.2.0/24 action pass
	tc filter add dev br-guest parent ffff: protocol ip prio 2 matchall action mirred egress redirect dev ifb-ul
       	
	# capture download (egress) on br-lan and br-guest
	tc filter add dev br-lan parent 1: protocol ip prio 1 u32 match ip src 192.168.1.0/24 action pass
	tc filter add dev br-lan parent 1: protocol ip prio 2 matchall action mirred egress redirect dev ifb-dl
	tc filter add dev br-guest parent 1: protocol ip prio 1 u32 match ip src 192.168.2.0/24 action pass
	tc filter add dev br-guest parent 1: protocol ip prio 2 matchall action mirred egress redirect dev ifb-dl

I think the equivalent in nftables with DSCP marking support looks like this:

table netdev cake {

        chain capture-ul {
                type filter hook ingress devices = { br-lan, br-guest } priority -149; policy accept;
                ip daddr != { 192.168.1.0/24, 192.168.2.0/24 } jump process-ul
        }

        chain process-ul {
                ip dscp set cs3
                ip protocol icmp ip dscp set cs5 counter
                ether type ip fwd to "ifb-ul"
        }

        chain capture-dl {
                type filter hook egress devices = { br-lan, br-guest } priority -149; policy accept;
                ip saddr != { 192.168.1.0/24, 192.168.2.0/24 } jump process-dl
        }

        chain process-dl {
                ip dscp set cs3
                ip protocol icmp ip dscp set cs5 counter
                ether type ip fwd to "ifb-dl"
        }

}

I love how simple and elegant this is and how it facilitates DSCP tagging prior to forwarding to the IFBs.

But sadly whilst the upload part works in respect of ingress from br-lan/br-guest, the download part does not yet work because the egress hook isn't supported until linux kernel 5.16+ - see here:

https://lwn.net/Articles/876497/

And here:

NFTables Egress Hook?

So unless somehow this egress hook functionality is backported such that we can use it in our 5.10 kernel in OpenWrt, this solution won't work until way into the future.