[22.03] Translate extra/raw firewall rules

grrr2 · October 3, 2022, 11:52am

https://wiki.nftables.org/wiki-nftables/index.php/Main_Page

especially for custom config: https://openwrt.org/docs/guide-user/firewall/firewall_configuration#includes_for_2203_and_later_with_fw4

lleachii · October 3, 2022, 12:05pm

~~Ummm...did you mean to link a particular section of the OpenWrt wiki?~~ (disregard, browser issue on a mobile device - I did see this section, thanks)

Also, you posted the main page to the other wiki too...

Both were already posted...

I do appreciate the time - perhaps if not much trouble, you could highlight what syntax I'm supposed to see...where the file should be placed...etc. (i.e. the inquiry)?

Following your link I see nft commands. That's not what's shown even when the developers and other show rules in that thread.

I also see about 7 locations to place an nft file, 1 way to make an nft script and a way to make a script with nft commands inside. At this point I've only seen/demonstrated the latter thanks to dave14305. I assume the thread you linked shows the former...but where's the syntax, how did they generate it from iptables-translate (which I clearly showed is not identical to what jow posted to an inquiry), etc?

I was hoping for a link that would fill-in that missing information.

In a basic question - where is the manual thats showing people to write rules with { } () these marks.

All the wiki pages/example I see just use nft ....

Or...You can do either/or?

Note: I see how to display nft chains etc...but in iptables that display looked like iptables syntax...but the displayed output is not idetical to now the Wiki is showing adding them, though...(how would you re-add the same rules being shown by others then?)

EDIT: I see the OpenWrt section tagged in the URL on a full device, thanks - just need to know where it would go (I can pick a spot, I guess "before fw4") - and more so what syntax/how it's generated

grrr2 · October 3, 2022, 12:28pm

that's all you got am afraid. there is no "the" guide other than the ones already linked (*).
but i think you should reset your mindset iptabels and nftables are not 1:1 mapped, there are different features in both only similar exists in the other, e.g. sets are similar but not exactly the same.
and yes, you have to learn the new syntax: start with the nftables wiki. the linked owrt own wiki explains how you can add custom config to fw4 via *.nft files can be placed in different location depending at which point/level you want to add custom stuff. but in general you can still use luci/uci interface to create rules.

definitely it will require effort to learn nftables, i agree it is not that obvious.

(*) i mean there are others of couse, archlinux is pretty good, but i'm not aware of detailed comprehensive guide for iptables -> nftables transition. there is the tool which (tries to ) translate iptables rule. which is also linked already.

lleachii · October 3, 2022, 12:44pm

Thanks.

(Perhaps for another thread)

Now I wish the big 100k set (it was 1+ MB in ipset memory) set would load faster. I imagine I could somehow rewitre the script to generate one long command with:

nft add element inet fw4 bogons { $ip, $ip ,$ip...}

instead of

nft add element inet fw4 bogons { $ip }
nft add element inet fw4 bogons { $ip }
nft add element inet fw4 bogons { $ip }
...

But I understand there's character limits to the bash "interface", how to properly delineate many per line etc. And, I'm not sure that will even speed it up.

@trendy @dave14305 - thanks to you too!

grrr2 · October 3, 2022, 1:08pm

but why you want to do this way?

you can add custom *.nft, which can be a new chain with all the rules. so create the nft file from script, add your 1M ip addresses (you sure cannot summarize them in CIDR format?) and just import that chain from your nft file

lleachii · October 3, 2022, 1:24pm

They are summarized (nonetheless even if pulled from 2 sources with dupes - I now understand nft aggregates them anyways and produces an add error).

Are you saying there's some "hot plug"?

I mean...

If I'm understanding, just make this an nft file, correct?
Are you saying copying an *.nft file into the correct folder will hot plug it?

and...does it needs to be in the other syntax I donno yet?

grrr2 · October 3, 2022, 1:47pm

no.

you mix nft cli commands with actual nft rules.

nft add element is a cli command.

this is an actual table (called filter) + chain (called output) + chain definition and 3 rules

table filter {
        chain output {
                 type filter hook output priority 0;
                 ip protocol tcp counter packets 190 bytes 21908 # handle 8
                 ip daddr 127.0.0.8 drop # handle 10
                 ip saddr 127.0.0.1 ip daddr 127.0.0.6 drop # handle 7
        }
}

if read through the fw link above on owrt wiki there is this subsection: https://openwrt.org/docs/guide-user/firewall/firewall_configuration#drop-in_includes_for_package_authors

For package authors that need custom firewall rules, it is possible to add nftables snippets in the following directories, depending on the desired position:

so for example if you create a file like /usr/share/nftables.d/chain-post/${chain}/*.nft with a content like above (in nft syntax, the table filter {} stuff), it will be added to your nft config when fw4 starts. obviously using nft cli commands is slower, but if you create bogon.nft and let's say want to add to forward chain after all other rules then you can create /usr/share/nftables.d/chain-post/forward/bogon.nft and populate with nft rules.

check https://openwrt.org/docs/guide-user/advanced/ipset_extras for idea how to use hotplug system, so you can add online "event" when your wan is up do stuff. in your case (re)-create bogon.nft, as when wan is up firewall will restart anyhow. you should not copy ipset stuff, just get the idea how the hotplug system works. if your 1M IP list is static not changing you don't even need to use the hotplug system, just create a bogon.nft.

grrr2 · October 3, 2022, 1:52pm

when you do nft add/delete etc can check the actual "compiled" nft rule with nft list ruleset. probably that's the easiest way to empirically verify how nft cli is translated to nft rule.

lleachii · October 3, 2022, 1:53pm

Sorry, I forgot "hotplug" had another term/definition in OpenWrt.

But I think this covered my inquiry - thanks!

My inquiry was more so - "would loading a *.nft file - scripted to be written in proper nft rule syntax - be loaded faster than nft CLI commands."

I understand you're saying "obviously more so".

And the list will change.

grrr2 · October 3, 2022, 1:57pm

yes, likely so. plus you don't have to worry about command line arguments buffer size and co,

lleachii · October 3, 2022, 2:05pm

So - does anyone have a script that already takes an *.ipset standard (IPs and CIDRs line-by-line, other lines commented) file and converts it into a nftables { $ip, $ip, $ip...} stanza?
I'll see if the provider offers a *.nft version of the file

I guess I could then write it to /tmp/bad_people.nft - then copy it to the correct folder and do an /etc/init.d/firewall reload

grrr2 · October 3, 2022, 2:42pm

add to /etc/config/firewall:

config ipset
        option name test_set
        option enabled 1
        option match 'dest_ip'
        list entry '1.1.1.1'

$ service firewall restart

verify if set is created:
$ nft list ruleset
table inet fw4 {
        set test_set {
                type ipv4_addr
                elements = { 1.1.1.1 }
        }

        chain input { }

so now you know the syntax.
according to https://openwrt.org/docs/guide-user/firewall/firewall_configuration#drop-in_includes_for_package_authors:

/usr/share/nftables.d/table-pre/*.nft 	Included at the beginning of the fw4 table, before any chain definition 

disable ipset in firewall config and create a bogon.nft under /usr/share/nftables.d/table-pre/ with content coming from your ipset using the syntax above.

lleachii · October 3, 2022, 2:46pm

???

I think that's been covered.

I inquiring about an actual *.ipset file that would have 100k lines of CIDR ranges - like a cybersecurity company issues in subscriptions.

There's nothing there to add thousands of lines to the elements = {$ip, $ip, $ip...}

I do appreciate how to script the *.nft file - though!

grrr2 · October 3, 2022, 2:48pm

ok, then am lost. if you have a raw file with ip addresses then you don't need to translate ipset to nft set per se. just add your IP's to bogon.nft using the syntax and done.

lleachii · October 3, 2022, 2:49pm

Taking hours-days instead of 1-2 minutes is the issue.

(and correction - the entries are about 100K - it took 1 MB in memory from the old iptables)

and this is an x86_64 taking that long

grrr2 · October 3, 2022, 2:53pm

but if i recall correctly you use nft cli command to add the IPs, don't you. so that is slow if you iterate over 1M entries and do nft add element. or you are saying you have an nft file which looks like

        set test_set {
                type ipv4_addr
                elements = { 1.1.1.1, 1.1.1.2, 1.1.1.3 }
        }

with your 1M list already added as elements {}, and still slow?

lleachii · October 3, 2022, 2:59pm

Well yes, lemme show you the script, but indeed, it merely runs now in loop:

nft add element inet fw4 x { $ip }
nft add element inet fw4 x { $ip }
nft add element inet fw4 x { $ip }
...

(FYI, there's a thread I made in a higher-level section on this free malware set, I cant search right now.)

Here's the script:

#!/bin/sh
# load block lists into script_block list

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"


cd /tmp || exit 1

###########################

rm /tmp/block_list.txt

while read url; do
  wget "$url" -O ->> /tmp/block_list.txt
done < /etc/config/block_urls.txt

nft flush set inet fw4 script_block 2>/dev/null

while read p; do
  nft add element inet fw4 script_block { $p } 2>/dev/null
done < /tmp/block_list.txt

rm /tmp/block_list.txt

exit 0

The file /etc/config/block_urls.txt contains the URLs of the malware IP block lists you're subscribed to. The downloads are a text line-separate CIDR ranges, usually with the file extension *.ipset

dave14305 · October 3, 2022, 3:08pm

I’ve not tried this myself, but try the loadfile '/tmp/block_list.txt' option for the ipset in /etc/config/firewall.

grrr2 · October 3, 2022, 3:09pm

ok then i understood correctly.

yes, your problem is that you run a loop of nft cli commands on who knows how many times ... that will be surely slow.

what am suggesting to do something like this:

echo > bogon.nft <<EOF
        set test_set {
                type ipv4_addr
                elements = {
EOF

while read p; do
   echo >> bogon.nft <<EOF
$p,
EOF
done < /tmp/block_list.txt

echo >> bogon.nft << EOF
}
EOF

this will create a bogon.nft, put this to the location above. (you have to check if tailing comma is accepted by nft or you need to remove. and it is not tested code, just a quick brainstorming). this bogon.nft will be a "native" nft script instead of looping 100K times nft cli commands, read and processed only once when fw4 starts.

note: i was confused and under impression you have ipset and from there you wanted to extract and copy to nft set, but if you have the IP's in a file you can use sed to convert all \n to coma which is even faster than read loop.

lleachii · October 3, 2022, 3:12pm

I think so - which works if the stanza wasn't used to dynamically add by the call of a daemon - for various other sets also. But at the time - I actually wanted iptables rules (not a set) to see the traffic on each ip (for certain lists). Also, it was developed for a Amateur Radio network community, and ipset was an added module to a community that already configures these for our safety worldwide in communication losses. That set is about 1k entries long and works ok.

(Does that relate to nft, or just making sure I was using iptables best practices in the past?)

Since a module is not needed for nft, this isn't a problem now.

I'll work on something like that and test. Thanks!