Nft script file vs fw4 include file

Hello,

trying to implement a new logic for pbr where it will create a fw4 include file, however the file which works fine with nft -f file doesn't work when placed in /usr/share/nftables.d/table-post/:

File:

add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 counter mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 counter mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add chain inet fw4 pbr_mark_0x030000
add rule inet fw4 pbr_mark_0x030000 counter mark set mark and 0xff00ffff xor 0x030000
add rule inet fw4 pbr_mark_0x030000 return
add chain inet fw4 pbr_mark_0x040000
add rule inet fw4 pbr_mark_0x040000 counter mark set mark and 0xff00ffff xor 0x040000
add rule inet fw4 pbr_mark_0x040000 return
add chain inet fw4 pbr_mark_0x050000
add rule inet fw4 pbr_mark_0x050000 counter mark set mark and 0xff00ffff xor 0x050000
add rule inet fw4 pbr_mark_0x050000 return
add chain inet fw4 pbr_mark_0x060000
add rule inet fw4 pbr_mark_0x060000 counter mark set mark and 0xff00ffff xor 0x060000
add rule inet fw4 pbr_mark_0x060000 return

The CLI output:

xg-135r3 in /tmp # service firewall reload
Automatically including '/usr/share/nftables.d/table-post/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_forward/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_input/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_output/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_postrouting/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_prerouting/30-pbr.nft'
Hardware flow offloading unavailable, falling back to software offloading
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:1:1-3: Error: syntax error, unexpected add
add chain inet fw4 pbr_mark_0x010000
^^^
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:2:1-3: Error: syntax error, unexpected add
add rule inet fw4 pbr_mark_0x010000 counter mark set mark and 0xff00ffff xor 0x010000
^^^
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:3:1-3: Error: syntax error, unexpected add
add rule inet fw4 pbr_mark_0x010000 return
^^^
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:4:1-3: Error: syntax error, unexpected add
add chain inet fw4 pbr_mark_0x020000
^^^
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:5:1-3: Error: syntax error, unexpected add
add rule inet fw4 pbr_mark_0x020000 counter mark set mark and 0xff00ffff xor 0x020000
^^^
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:6:1-3: Error: syntax error, unexpected add
add rule inet fw4 pbr_mark_0x020000 return
^^^
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:7:1-3: Error: syntax error, unexpected add
add chain inet fw4 pbr_mark_0x030000
^^^
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:8:1-3: Error: syntax error, unexpected add
add rule inet fw4 pbr_mark_0x030000 counter mark set mark and 0xff00ffff xor 0x030000
^^^
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:9:1-3: Error: syntax error, unexpected add
add rule inet fw4 pbr_mark_0x030000 return
^^^
In file included from /dev/stdin:317:2-55:
/usr/share/nftables.d/table-post/30-pbr.nft:10:1-3: Error: syntax error, unexpected add
add chain inet fw4 pbr_mark_0x040000
^^^

Any ideas why? Is a translation needed from nft script file before it can be included with fw4?

The files in those locations are "includes" right into the generated .nft file, so they should be formatted as rules, chains or elements, depending on where they are included.

Here's, for example, my DoH set definitions:

$ head /usr/share/nftables.d/table-pre/10-doh_sets.nft
    set doh_ipv4 {
        typeof ip daddr
        flags timeout
        timeout 7d
        gc-interval 6h
...

And if I look at where it's included, I see:

$ fw4 print | grep -B5 10-doh_sets.nft
        # User includes
        #

        include "/etc/nftables.d/*.nft"

        include "/usr/share/nftables.d/table-pre/10-doh_sets.nft"

In your case, the table-post file should probably just look like this.

chain pbr_mark_0x010000 {
    counter mark set mark and 0xff00ffff xor 0x010000
}
chain pbr_mark_0x020000 {
    counter mark set mark and 0xff00ffff xor 0x020000
}
...
1 Like

Thanks @efahl!

I don't know the inner-workings of fw4, so it was a surprise to me that it accepts only one of the nft-compatible file formats and not the other.

@jow is there a tool/script I could use on OpenWrt 22.03 and up to translate my file in "nft scripted format" to the "nft output format" as defined here?

I've tried looking up, but most of the "translate" references are for migrating from iptables to nftables and not doing what I need.

Maybe just run your nft script, then do nft list ruleset and cherry pick the pieces you want? Or, you can use the -e option on nft and it spits out the resulting item...

You can use a script, but it’s slightly less integrated with nftables atomic rule replacement.

I have a script to delete empty default fw4 chains with hooks defined. I add the include to the config:

config include
        option enabled '1'
        option type 'script'
        option path '/etc/fw4-trim.sh'
        option fw4_compatible '1'

My script has nft commands:

nft delete chain inet fw4 mangle_output
nft delete chain inet fw4 mangle_input
nft delete chain inet fw4 mangle_postrouting
nft delete chain inet fw4 mangle_prerouting
nft delete chain inet fw4 raw_output
nft delete chain inet fw4 raw_prerouting
nft delete chain inet fw4 prerouting

It doesn't spit them in the format fw4 accepts when I load the file, you can try yourself.

If you just remove "nft" from each command, you can load it atomically with nft -f.

I appreciate the replies, but they are not what I asked for.

Move your include under /usr/share/nftables.d/ruleset-post/ so that your commands aren’t interpreted within the context of the fw4 table or any other “in-progress” object being created.

1 Like

Huh, works for me:

$ nft list chain inet fw4 pbr_mark_0x060000
table inet fw4 {
        chain pbr_mark_0x060000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
                return
        }
}

$ nft add rule inet fw4 pbr_mark_0x060000 'counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000'

$ nft list chain inet fw4 pbr_mark_0x060000
table inet fw4 {
        chain pbr_mark_0x060000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
                return
                counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
        }
}

You're not loading the file.

All variants that I try work fine for me. What exactly "doesn't work"?

$ cat x.sh
#!/bin/sh -x

nft -e -f - <<XXX
table inet fw4 {
        chain pbr_mark_0x060000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
                return
        }
}
XXX

nft list chain inet fw4 pbr_mark_0x060000

echo > x.nft <<XXX
table inet fw4 {
        chain pbr_mark_0x060000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
                return
        }
}
XXX

nft -f x.nft

nft list chain inet fw4 pbr_mark_0x060000

Run it:

$ ./x.sh
+ nft -e -f -
add chain inet fw4 pbr_mark_0x060000
add rule inet fw4 pbr_mark_0x060000 counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
add rule inet fw4 pbr_mark_0x060000 return
# new generation 76 by process 2894 (nft)
+ nft list chain inet fw4 pbr_mark_0x060000
table inet fw4 {
        chain pbr_mark_0x060000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
                return
        }
}
+ echo
+ nft -f x.nft
+ nft list chain inet fw4 pbr_mark_0x060000
table inet fw4 {
        chain pbr_mark_0x060000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
                return
        }
}

I’ve now converted my archaic include script to use the native commands in the ruleset-post include. Today, I learned! :nerd_face:

1 Like

Just making sure... You did add your file to /etc/sysupgrade.conf, right? :slight_smile:

1 Like

I build my own images using files/ so I have copied back my changes to my buildroot. When I sysupgrade, I don’t save settings.

1 Like

Try with the file from my original post.

I'm certainly missing something, here's what I tried.

Create x.nft with cut'n'paste from your original post, follow that with commands as echoed by nft in verbose mode. I expect it to create the same rules twice in each chain, since I'm just repeating each command with an input syntax variation.

$ cat x.nft

add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 counter mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 counter mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add chain inet fw4 pbr_mark_0x030000
add rule inet fw4 pbr_mark_0x030000 counter mark set mark and 0xff00ffff xor 0x030000
add rule inet fw4 pbr_mark_0x030000 return
add chain inet fw4 pbr_mark_0x040000
add rule inet fw4 pbr_mark_0x040000 counter mark set mark and 0xff00ffff xor 0x040000
add rule inet fw4 pbr_mark_0x040000 return
add chain inet fw4 pbr_mark_0x050000
add rule inet fw4 pbr_mark_0x050000 counter mark set mark and 0xff00ffff xor 0x050000
add rule inet fw4 pbr_mark_0x050000 return
add chain inet fw4 pbr_mark_0x060000
add rule inet fw4 pbr_mark_0x060000 counter mark set mark and 0xff00ffff xor 0x060000
add rule inet fw4 pbr_mark_0x060000 return

add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000
add rule inet fw4 pbr_mark_0x020000 return
add chain inet fw4 pbr_mark_0x030000
add rule inet fw4 pbr_mark_0x030000 counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
add rule inet fw4 pbr_mark_0x030000 return
add chain inet fw4 pbr_mark_0x040000
add rule inet fw4 pbr_mark_0x040000 counter packets 0 bytes 0 meta mark set meta mark & 0xff04ffff | 0x00040000
add rule inet fw4 pbr_mark_0x040000 return
add chain inet fw4 pbr_mark_0x050000
add rule inet fw4 pbr_mark_0x050000 counter packets 0 bytes 0 meta mark set meta mark & 0xff05ffff | 0x00050000
add rule inet fw4 pbr_mark_0x050000 return
add chain inet fw4 pbr_mark_0x060000
add rule inet fw4 pbr_mark_0x060000 counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
add rule inet fw4 pbr_mark_0x060000 return

Then I run it with the -e verbose mode turned on.

$ nft -e -f x.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
add rule inet fw4 pbr_mark_0x010000 handle 1103 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000
add rule inet fw4 pbr_mark_0x020000 handle 1105 return
add chain inet fw4 pbr_mark_0x030000
add rule inet fw4 pbr_mark_0x030000 counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
add rule inet fw4 pbr_mark_0x030000 handle 1107 return
add chain inet fw4 pbr_mark_0x040000
add rule inet fw4 pbr_mark_0x040000 counter packets 0 bytes 0 meta mark set meta mark & 0xff04ffff | 0x00040000
add rule inet fw4 pbr_mark_0x040000 handle 1109 return
add chain inet fw4 pbr_mark_0x050000
add rule inet fw4 pbr_mark_0x050000 counter packets 0 bytes 0 meta mark set meta mark & 0xff05ffff | 0x00050000
add rule inet fw4 pbr_mark_0x050000 handle 1111 return
add chain inet fw4 pbr_mark_0x060000
add rule inet fw4 pbr_mark_0x060000 counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
add rule inet fw4 pbr_mark_0x060000 handle 1113 return
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 handle 1104 counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 handle 1106 counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000
add rule inet fw4 pbr_mark_0x020000 return
add chain inet fw4 pbr_mark_0x030000
add rule inet fw4 pbr_mark_0x030000 handle 1108 counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
add rule inet fw4 pbr_mark_0x030000 return
add chain inet fw4 pbr_mark_0x040000
add rule inet fw4 pbr_mark_0x040000 handle 1110 counter packets 0 bytes 0 meta mark set meta mark & 0xff04ffff | 0x00040000
add rule inet fw4 pbr_mark_0x040000 return
add chain inet fw4 pbr_mark_0x050000
add rule inet fw4 pbr_mark_0x050000 handle 1112 counter packets 0 bytes 0 meta mark set meta mark & 0xff05ffff | 0x00050000
add rule inet fw4 pbr_mark_0x050000 return
add chain inet fw4 pbr_mark_0x060000
add rule inet fw4 pbr_mark_0x060000 handle 1114 counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
add rule inet fw4 pbr_mark_0x060000 return
# new generation 113 by process 11331 (nft)

Check one of the chains, and see the rules doubled up as expected.

$ nft list chain inet fw4 pbr_mark_0x060000
table inet fw4 {
        chain pbr_mark_0x060000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
                return
                counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000
                return
        }
}

If the formatting you referred to in your first reply was required (and thanks to @dave14305 we now know it is not), then the -e option would not produce it from the file I've posted.

Not sure how that example can be wrong, it's code cut from my edge router running 22.03.5 and has been in production for over a year now.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.