QoS and nftables ... some findings to share

I bought two additional RPi4 one for use at my wife's work and one for use in my original project since I coopted the first one as my main router. maybe I will do some testing with openwrt before converting...

How would I begin this process on a Debian device? I plan on building one in the near future, but would it be like OpenWRT with iptables - I would have to install additional modules for the Debian router to function under nftables? Assuming I begin with a common Debian router build, whatever that is.

I figure if I am going to do nftables, I may as well begin in a Linux environment. The only scripts I would need beside the minimal firewall rules would be something similar in state to

iptables -A FORWARD -s 192.1.2.3 -j ACCEPT
iptables -A FORWARD -d 192.1.2.3 -j ACCEPT

iptables -t mangle -A PREROUTING -s 192.1.2.3 -j DSCP --set-dscp-class CS6
iptables -t mangle -A PREROUTING -d 192.1.2.3 -j DSCP --set-dscp-class CS6

Debian already uses nftables by default, and by default it uses it through an iptables shim... so people who have existing iptables rulesets can continue to use those. basically the iptables commands actually are translator commands that convert things to nftables... So in reality you can just install nftables and create /etc/nftables.conf and ignore iptables entirely on Debian and it works out of the box, after doing.

apt-get update
apt-get install nftables
systemctl enable nftables

put your /etc/nftables.conf in place, and then reboot.

as for doing the routing, you can set up most of that by editing files like /etc/systemd/network/foo.network and foo.netdev. Basically you define a few vlans if needed, and then define some DHCP clients or some static ip addresses on those network devices. Voila you have a router.

What's more complicated is handling WiFi hardware, you have to custom config hostapd.

1 Like

I use the command below in my edgerouter x 19.07.1 to get atomic rule replacement. I can't use "flush ruleset" in the top of the config file or I'll get the same error you get.

nft flush ruleset inet\;flush table ip nat\;include '"/etc/nftables.conf"'

As far as dnsmasq respawning, it does that to me if I dont have the following in my filter input chain

iif "lo" accept

Any update on nftables on OpenWrt at all? was a bug ever filed in a bug tracker or anything like that?

If this question is too far OT or inappropriate for this thread please let me know...

I noticed while playing with sqm-scripts that the simple.qos script uses iptables to set a few rules to the mangle table.

Am I correct in assuming that I'd have to modify this script and change the iptables rule format to a nftables format? (I don't think that be difficult - I'm just making my own plans to migrate to nftables from iptables on a DIY ubuntu router using sqm-scripts).

Yes, under nftables you can usually put your rules into /etc/nftables.conf where they'll be loaded by the operating system during boot.

1 Like

Yes, except most nftables based application should be able to execute legacy iptables commands. But my personal goal is to switch sqm scripts over the nftables, once OpenWrt made the cut. The biggest prize gained by that will be the ability to use nftables on the ingress side without needing addition heroic effort (like routing the ingress traffic through an additional veth pair). But I am not holding my breath...

1 Like

Indeed, but if we can get nftables working appropriately on OpenWrt, even if it isn't the standard, then I can publish some howtos on custom QoS... Unfortunately the 19 series seems to have partially broken nftables compared to the 18 series where it worked fine.

Have to say, I ran out of time to keep digging into the problem with the current openwrt release. I'm still on 19.07.01, and kinda have nftables working - still has the old problems with loading the tables, and occasionally seems to barf up other services.

Its almost certainly connected with how the kernel is compiled, but with several recompiles, I couldn't find what was causing the problem. Suspect I'll probably wait until openwrt version 20 comes out, then look at the problem again.

Oh yes, worth saying yes nftables can be controled like ip tables with commands. To my mind its more flexable than iptables, easier to make sure rules are loaded in the right place. So yes if you wanted a script, e.g. sqm scripts, to change the firewall rules, yes should be possible with nftables; just a case of rewriting everything.

Not sure what the official openwrt approach will be; there was talk of rewriting fw3 as fw4, where fw4 uses nftables. Don't know if any progress was made, its not something I followed.

1 Like

Hi, @dlakelan,

I am interested in moving my current approach to QoS from iptables+ctinfo+DSCP to nftables netdev ingress marking.

For this purpose I installed the master branch in a RPi3 I had laying around. I stopped the firewall, removed any iptables module, including user space utilities. And, right now only these modules are loaded:

nf_conntrack           98304  8 nft_flow_offload,nft_redir,nft_nat,nft_masq,nft_ct,nf_nat,nf_flow_table,nf_conntrack_rtcache
nf_conntrack_rtcache   16384  0 
nf_defrag_ipv4         16384  1 nf_conntrack
nf_defrag_ipv6         16384  1 nf_conntrack
nf_dup_netdev          16384  2 nft_fwd_netdev,nft_dup_netdev
nf_flow_table          24576  5 nft_flow_offload,nf_flow_table_ipv6,nf_flow_table_ipv4,nf_flow_table_inet,nf_flow_table_hw
nf_flow_table_hw       16384  0 
nf_flow_table_inet     16384  0 
nf_flow_table_ipv4     16384  0 
nf_flow_table_ipv6     16384  0 
nf_log_common          16384  2 nf_log_ipv6,nf_log_ipv4
nf_log_ipv4            16384  0 
nf_log_ipv6            16384  0 
nf_nat                 36864  3 nft_redir,nft_nat,nft_masq
nf_reject_ipv4         16384  3 nft_reject_bridge,nft_reject_ipv4,nft_reject_inet
nf_reject_ipv6         16384  3 nft_reject_bridge,nft_reject_ipv6,nft_reject_inet
nf_tables             122880 28 nft_fib_inet,nft_reject_bridge,nft_meta_bridge,nft_flow_offload,nft_fib_ipv6,nft_fib_ipv4,nft_fib,nf_flow_table_ipv6,nf_flow_table_ipv4,nf_flow_table_inet,nft_reject_ipv6,nft_reject_ipv4,nft_reject_inet,nft_reject,nft_redir,nft_quota,nft_objref,nft_numgen,nft_nat,nft_masq,nft_log,nft_limit,nft_hash,nft_fwd_netdev,nft_dup_netdev,nft_ct,nft_counter,nf_tables_set
nf_tables_set          28672  0 
nfnetlink              16384  1 nf_tables
nft_counter            16384  0 
nft_ct                 20480  0 
nft_dup_netdev         16384  0 
nft_fib                16384  3 nft_fib_inet,nft_fib_ipv6,nft_fib_ipv4
nft_fib_inet           16384  0 
nft_fib_ipv4           16384  1 nft_fib_inet
nft_fib_ipv6           16384  1 nft_fib_inet
nft_flow_offload       16384  0 
nft_fwd_netdev         16384  0 
nft_hash               16384  0 
nft_limit              16384  0 
nft_log                16384  0 
nft_masq               16384  0 
nft_meta_bridge        16384  0 
nft_nat                16384  0 
nft_numgen             16384  0 
nft_objref             16384  0 
nft_quota              16384  0 
nft_redir              16384  0 
nft_reject             16384  4 nft_reject_bridge,nft_reject_ipv6,nft_reject_ipv4,nft_reject_inet
nft_reject_bridge      16384  0 
nft_reject_inet        16384  0 
nft_reject_ipv4        16384  0 
nft_reject_ipv6        16384  0 

No iptables or x_tables modules are loaded at all; however, I keep receiving this error when I try to create my nat chains:

Error: Could not process rule: No such file or directory
add chain ip nat PREROUTING { type nat hook prerouting priority 0; policy accept; }
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

And it happens even with your basic configuration script, same error creating nat chain:

hastaloshuevos.conf:85:8-14: Error: Could not process rule: No such file or directory
	chain masqout {
	      ^^^^^^^

In theory, the reason for this error to happen is usually a missed module, but I cannot see which one I am forgetting, to me, all required nat modules are loaded. Do you see something I missed?

Thanks heaps.

Hi. I'm not sure what the issue is, but here is a dump from my router, which is running Raspbian and it all works fine, perhaps you can compare and figure it out?

Module                  Size  Used by
cn                     16384  1
wireguard             143360  0
ip6_udp_tunnel         16384  1 wireguard
udp_tunnel             16384  1 wireguard
nf_conntrack_netlink    40960  0
sch_fq_codel           20480  12
sch_hfsc               28672  3
bnep                   20480  2
hci_uart               40960  1
btbcm                  16384  1 hci_uart
serdev                 20480  1 hci_uart
bluetooth             389120  24 hci_uart,bnep,btbcm
ecdh_generic           28672  1 bluetooth
8021q                  32768  0
garp                   16384  1 8021q
stp                    16384  1 garp
llc                    16384  2 garp,stp
nft_masq_ipv4          16384  1
nft_masq               16384  1 nft_masq_ipv4
nft_redir_ipv6         16384  1
nft_chain_nat_ipv6     16384  1
nf_nat_ipv6            20480  1 nft_chain_nat_ipv6
nft_nat                16384  2
nft_redir_ipv4         16384  1
nft_redir              16384  2 nft_redir_ipv6,nft_redir_ipv4
nft_chain_nat_ipv4     16384  3
nf_nat_ipv4            16384  2 nft_chain_nat_ipv4,nft_masq_ipv4
nf_nat                 36864  5 nft_nat,nft_redir_ipv6,nft_redir_ipv4,nf_nat_ipv6,nf_nat_ipv4
nf_log_ipv6            16384  3
nf_log_ipv4            16384  3
nf_log_common          16384  2 nf_log_ipv6,nf_log_ipv4
nft_log                16384  3
nft_objref             16384  1
nft_numgen             16384  3
nft_limit              16384  74900
vc4                   176128  0
bcm2835_codec          36864  0
bcm2835_v4l2           45056  0
brcmfmac              311296  0
v4l2_mem2mem           24576  1 bcm2835_codec
videobuf2_dma_contig    20480  1 bcm2835_codec
bcm2835_mmal_vchiq     32768  2 bcm2835_codec,bcm2835_v4l2
brcmutil               16384  1 brcmfmac
v4l2_common            16384  1 bcm2835_v4l2
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2         24576  3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common       45056  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
drm_kms_helper        184320  1 vc4
videodev              200704  6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
v3d                    73728  0
media                  36864  3 bcm2835_codec,videodev,v4l2_mem2mem
sha256_generic         20480  0
vc_sm_cma              36864  1 bcm2835_mmal_vchiq
gpu_sched              28672  1 v3d
nft_counter            16384  10
rpivid_mem             16384  0
drm                   442368  5 v3d,vc4,gpu_sched,drm_kms_helper
cfg80211              647168  1 brcmfmac
drm_panel_orientation_quirks    16384  1 drm
rfkill                 28672  4 bluetooth,cfg80211
snd_soc_core          192512  1 vc4
snd_bcm2835            24576  1
snd_compress           20480  1 snd_soc_core
snd_pcm_dmaengine      16384  1 snd_soc_core
snd_pcm               102400  4 vc4,snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
snd_timer              32768  1 snd_pcm
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
raspberrypi_hwmon      16384  0
fb_sys_fops            16384  1 drm_kms_helper
hwmon                  16384  1 raspberrypi_hwmon
snd                    73728  7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
fixed                  16384  0
uio_pdrv_genirq        16384  0
uio                    20480  1 uio_pdrv_genirq
sg                     28672  0
nft_ct                 20480  18
nf_conntrack          135168  11 nft_ct,nft_nat,nft_redir,nft_redir_ipv6,nft_redir_ipv4,nft_masq,nf_conntrack_netlink,nf_nat_ipv6,nft_masq_ipv4,nf_nat_ipv4,nf_nat
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
nf_tables_set          32768  71
nft_quota              16384  1
nf_tables             122880  810 nft_ct,nft_nat,nft_numgen,nft_chain_nat_ipv6,nft_redir,nft_redir_ipv6,nft_chain_nat_ipv4,nft_redir_ipv4,nft_limit,nf_tables_set,nft_masq,nft_objref,nft_masq_ipv4,nft_quota,nft_counter,nft_log
nfnetlink              16384  2 nf_conntrack_netlink,nf_tables
ip_tables              24576  0
x_tables               32768  1 ip_tables
ipv6                  454656  311 wireguard,nf_nat_ipv6

1 Like

@amteza, if you figure out what is needed to make nftables work well on recent OpenWrt please post here! I've got lots of things to say about how to do DSCP based QoS on nftables, and am basically waiting until nftables is a reliable alternative on OpenWrt to start a new/second thread on this topic.

1 Like

@dlakelan, will do! I am giving it a go again this morning. I even had translated all QoS rules in ctinfo_4layercake.qos @ldir's repo, so this is annoying me as it was working in an Ubuntu box.

@dlakelan, sadly I found out what's the problem, I tried a 18.06.8 version in place of a x86-master and this is the result:

As you can see, it works and is not failing, the exact same one in 19.07.3 or master fails with my previous error "Could not process rule: No such file or directory". I do not know what else to do, as the device I have routing my traffic is a RPi4 running master from some days ago.

Any ideas, anyone?

Hi, @dlakelan,

Got it working! I had to compile a 19.07-SNAPSHOT with 19.07.3 options to experiment; and I got it working, removing completely the firewall and a whole bunch of modules before installing nftables and kmod-nft-nat, which seemed to fix my problems:

Okay, let's do this.

Can you put your config here so others can use it as a base?

I'll start a new thread, got a bunch of stuff going on but will be able to get started on the new thread later this week.

Excited. glad you got it working.

1 Like

Sure, let's see, slice by slice as Jack "The Ripper" might say.

List of "firewall" modules initially installed, only:

kmod-nf-conntrack kmod-nf-conntrack6 kmod-nf-nat kmod-nf-reject kmod-nf-reject6 kmod-nfnetlink kmod-nft-core kmod-nft-nat

Additional modules that can be installed without problems:

kmod-nft-arp kmod-nft-bridge kmod-nft-fib kmod-nft-nat6 kmod-nft-netdev kmod-nft-offload

Userland application installed, including dependencies:

nftables libnftnl11

This is how it should look once you finalise installing the basic modules:

nf_conntrack           65536 12 nft_redir_ipv4,nft_redir,nft_nat,nft_masq_ipv4,nft_masq,nft_ct,nf_conntrack_ipv6,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat_ipv4,nf_nat,nf_conntrack_rtcache
nf_conntrack_ipv4      12288  1 
nf_conntrack_ipv6      12288  0 
nf_conntrack_rtcache   12288  0 
nf_defrag_ipv4         12288  1 nf_conntrack_ipv4
nf_defrag_ipv6         12288  1 nf_conntrack_ipv6
nf_nat                 20480  4 nft_nat,nf_nat_redirect,nf_nat_masquerade_ipv4,nf_nat_ipv4
nf_nat_ipv4            12288  1 nft_chain_nat_ipv4
nf_nat_masquerade_ipv4   12288  1 nft_masq_ipv4
nf_nat_redirect        12288  1 nft_redir_ipv4
nf_reject_ipv4         12288  2 nft_reject_ipv4,nft_reject_inet
nf_reject_ipv6         12288  2 nft_reject_ipv6,nft_reject_inet
nf_tables              69632 25 nft_set_rbtree,nft_set_hash,nft_reject_ipv6,nft_reject_ipv4,nft_reject_inet,nft_reject,nft_redir_ipv4,nft_redir,nft_quota,nft_numgen,nft_nat,nft_meta,nft_masq_ipv4,nft_masq,nft_log,nft_limit,nft_exthdr,nft_ct,nft_counter,nft_chain_route_ipv6,nft_chain_route_ipv4,nft_chain_nat_ipv4,nf_tables_ipv6,nf_tables_ipv4,nf_tables_inet
nf_tables_inet         12288  0 
nf_tables_ipv4         12288  0 
nf_tables_ipv6         12288  0 
nfnetlink              12288  1 nf_tables
nft_chain_nat_ipv4     12288  0 
nft_chain_route_ipv4   12288  0 
nft_chain_route_ipv6   12288  0 
nft_counter            12288  0 
nft_ct                 16384  0 
nft_exthdr             12288  0 
nft_limit              12288  0 
nft_log                12288  0 
nft_masq               12288  1 nft_masq_ipv4
nft_masq_ipv4          12288  0 
nft_meta               12288  0 
nft_nat                12288  0 
nft_numgen             12288  0 
nft_quota              12288  0 
nft_redir              12288  1 nft_redir_ipv4
nft_redir_ipv4         12288  0 
nft_reject             12288  3 nft_reject_ipv6,nft_reject_ipv4,nft_reject_inet
nft_reject_inet        12288  0 
nft_reject_ipv4        12288  0 
nft_reject_ipv6        12288  0 
nft_set_hash           16384  0 
nft_set_rbtree          8192  0 

Directory /etc/modules.d/ contains only the next files related to nftables:

 nf-conntrack
-rw-r--r--    1 root     root            33 May 16 18:32 nf-conntrack6
-rw-r--r--    1 root     root            58 May 16 18:32 nf-nat
-rw-r--r--    1 root     root            15 May 16 18:32 nf-reject
-rw-r--r--    1 root     root            15 May 16 18:32 nf-reject6
-rw-r--r--    1 root     root            10 May 16 18:32 nfnetlink
-rw-r--r--    1 root     root           272 May 16 18:32 nft-core
-rw-r--r--    1 root     root            65 May 16 18:32 nft-nat

And this is it, in this installation, no LuCI and no firewall or anything from iptables, not even userland executables.

To compile openwrt I used https://downloads.openwrt.org/releases/19.07-SNAPSHOT/targets/x86/64/ checking out b515edb775 and getting release 19.07.3 configuration file, so I am able to install packages and modules without problems https://downloads.openwrt.org/releases/19.07.3/targets/x86/64/config.buildinfo

Hope this help others getting a machine running. This will work with different targets.

2 Likes

ok... this wasn't catching

#>>> /etc/modules.d/nf-ipt
#>>> /etc/modules.d/nf-ipt6
mv /etc/modules.d/nf-ipt* /etc/modules.d.unused

with those gone... can confirm things play ball :star_struck:

2 Likes

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.