[22.03] Translate extra/raw firewall rules

[dave14305]

loadfile will read in the list of IPs from the file when generating the ruleset, and print them in-line as part of the initial set definition. It’s quite neat. Preview it with fw4 print | more after adding the loadfile path to the set.

[dave14305]

When you update your downloaded list, you can force a set update with fw4 reload-sets instead of having to restart the firewall.

[lleachii]

@dave14305

loadfile

Is this a CLI command...sorry, you lost me, I searched.

@dave14305

you can force a set update with fw4 reload-sets instead of having to restart the firewall.

Wow - the firewall probably isn't that huge in my case, but I see how that is cool in nft!

(I'm understanding better how it better processes and the user interacts with it.)

I don't have all the syntax A-Z, and since I'm leanirng it, this is n00b to me. But hey...so was iptables years ago when I first learned Linux/OpenWrt staring at it...before my account in the archive days...and didn't even know HOW to ask a question...

So thanks!

[dave14305]

@lleachii

Is this a CLI command...sorry, you lost me, I searched.

In /etc/config/firewall where you define each of your sets, also include the loadfile option with the path to the list of IPs or CIDRs for that set. fw4 will load those entries from the file so you don’t have to manually load them. Whenever you update the downloaded file, run the reload command in my previous post.

The syntax is the same as fw3 described in the wiki (except for the storage parameter):

IP set examples

[lleachii]

@dave14305

loadfile option with the path to the list of IPs or CIDRs for that set. fw4 will load those entries from the file so you don’t have to manually load them. Whenever you update the downloaded file, run the reload command in my previous post.

Hummm...

I think I tried this years ago. Never worked (wish I had a link to a thread or something...)

Do I have to remove comment lines with:

###this is the White Hat people giving you a bad list####

or something?

And...if it's UCI...how do I load it in via script...every...12 hours or less?

Wait, are you're saying make 100k entries in UCI - a.k.a. /etc/config/firewall on flash/non-volatile space...and reload it?

whoa...that might work...well...I need to verify how UCI handles that...

I assume it too parses the file somehow upon each reload (code?), at minimum...same CPU...will need to test...I cannot have a hang in the firewall for even seconds to reload it, let alone 1-2 minutes or more now... I assume this is the same for all users.

I'll update. Thanks.

• Doesn't the rule fail in fw4 if the file is not present [on boot]? (the data cannot be written to non-volatile flash, it's too big)
• I'd have also dynamically add the rule in script and make sure it's removed on backups...etc.

(Downloading could take ~70 seconds alone...before the firewall has loaded allow traffic rules...) won't work (chicken-or-the-egg paradox)...that workaround was to only enumerate the set in UCI and just pre-seed any [known] IPs needed for "bootstrapping" any software.

@grrr2

what am suggesting to do something like this

I think I'm gong to have to write a script like yours to create a *.nft file and then run.

@dave14305

fw4 reload-sets

As I assume that's the purpose of the *.nft file locations...to reload from those folders, correct?

@dave14305 @grrr2 - all of your options require saving to flash (or re-creating a rule referencing a file in /tmp - which rule writes to flash). Unless I'm grossly mistaken, for large dynamic purposes, that won't work. (*)

one set is a routing table that is saved - called by the daemon (it's a custom RIPv4 routing daemon for Amateur Radio) which has about 1k entries, but the text of the other sets are larger than most flash spaces - it's native output language is not CIDR - it uses a least-specific short CIDR notation (e.g. "10.0.0.0/8" is "10/8" - in most Linux software, it seems to note such a CIDR, you omit the slash - such as in ip route get), so the script similar to above with a stanza still has to be executed in userspace to make an *ipset-compatible file - that that one might stay with an nft loop

[dave14305]

You at least understand some of the new firewall4 features and/or how old features are now implemented. Don’t break anything!

[lleachii]

LOL!

Thanks!

I definitely appreciate everything!

[grrr2]

you are lucky as option loadfile is what it says: you can store the IPs in a file and fw3/4 will load from it into ipt/nft set. config like this works nicely.

config ipset
        option name test_set
        option enabled 1
        option match 'dest_ip'
        option loadfile /tmp/test.list

$ cat /tmp/test.list
1.1.1.1
1.1.1.2
1.1.1.3

$ nft list ruleset
table inet fw4 {
        set test_set {
                type ipv4_addr
                elements = { 1.1.1.1, 1.1.1.2,
                             1.1.1.3 }
        }

[lleachii]

@grrr2

/tmp/test.list

This can't exist on boot...but...I made the rule...

root@OpenWrt:~# /etc/init.d/firewall reload
Unable to load file '/tmp/test.list' for set 'test_set': No such file or directory

Although...

root@OpenWrt:~# nft list set inet fw4 test_set
table inet fw4 {
        set test_set {
                type ipv4_addr
        }
}

It created a blank set!

So...

root@OpenWrt:~# wget "https://www.example.com/list.txt" -O ->> /tmp/test.list
root@OpenWrt:~# ls -l /tmp/test.list 
-rw-r--r--    1 root     root        203862 Oct  3 18:52 /tmp/test.list
root@OpenWrt:~# /etc/init.d/firewall reload
##there was a few seconds pause
##SUCCESS

So....

rm /tmp/test.list
root@OpenWrt:~# wget "https://www0.example.com/list.txt" -O ->> /tmp/test.list && wget "https://www1.example.com/list.txt" -O ->> /tmp/test.list && wget "https://www2.example.com/list.txt" -O ->> /tmp/test.list && wget "https://www3.example.com/list.txt" -O ->> /tmp/test.list && wget "https://www0.example.com/list.txt" -O ->> /tmp/test.list
root@OpenWrt:~# ls -l /tmp/test.list 
-rw-r--r--    1 root     root       2581855 Oct  3 19:05 /tmp/test.list
root@OpenWrt:~#  /etc/init.d/firewall reload

It took 13 seconds!

• Now the script must reload the whole firewall and not just the set in question...with about a ~15-20 second freeze
  • (I wonder if someone could show a link to this code???)
• The files containing the loaded sets must remain present in /tmp in case the firewall is reloaded (taking memory)

These 2 dynamic scripts just load the file into /tmp/*.txt and does't delete it, and the radio route set still uses the nft loop in another call script...so...

@dave14305

Don’t break anything!

netlink: Error: Could not process rule: No buffer space available

Oops!

root@OpenWrt:/etc/config# ls -l /tmp/*.txt
-rw-r--r--    1 root     root       2581747 Oct  3 19:42 /tmp/block_list.txt
-rw-r--r--    1 root     root         11077 Oct  3 19:42 /tmp/geo_list.txt

[dave14305]

@lleachii

Now the script must reload the whole firewall and not just the set in question...with about a ~15-20 second freeze

Why can’t you just run fw4 reload-sets?

[lleachii]

@dave14305

Why can’t you just run fw4 reload-sets?

They remain empty - it seems to do nothing in my case:

root@OpenWrt:~# nft flush set inet fw4 script_block
root@OpenWrt:~# nft flush set inet fw4 geo_block
root@OpenWrt:~# rm /tmp/block_list.txt 
root@OpenWrt:~# rm /tmp/geo_list.txt
root@OpenWrt:~# /etc/init.d/firewall reload
Unable to load file '/tmp/geo_list.txt' for set 'geo_block': No such file or directory
Unable to load file '/tmp/block_list.txt' for set 'script_block': No such file or directory
root@OpenWrt:~# ./add_blocks_tmp.sh ##<--this adds them to /tmp
root@OpenWrt:~# fw4 reload-sets
root@OpenWrt:~# nft list set inet fw4 geo_block
table inet fw4 {
        set geo_block {
                type ipv4_addr
                flags interval
                auto-merge
        }
}

I tested without flushing, same processing time, no changes. In fact another issue, if the one that runs out of buffer remains on the firewall rules, neither set populates.

[lleachii]

@dave14305

can’t you just run fw4 reload-sets?

I need to know what use case this actually works:

Firewall fw4 rule "hits" - how to see

I now see where the hits appear in LuCI too. I now see the total bytes - and you hover over them you additionally see number of packets. Thanks....but Do they reset after n time? Why - I know I had [a lot of] traffic since my router was up on a rule...but... tcp dport xxx counter packets 0 bytes 0 accept comment "!fw4: xxx_in" I tested because I was perplexed: tcp dport xxx counter packets 2 bytes 124 accept comment "!fw4: xx_in" Disregard...I know what's occurring. A script for a nft s…

Because the firewall reload method [obviously] clears my firewall counters now.

[dave14305]

I think there might be a bug in main.uc in reload_sets() function. The loadfile exists, but there are no entries in the definition of the set, it seems to skip the reload process. I think we need @jow to validate the intended logic.

https://git.openwrt.org/?p=project/firewall4.git;a=blob;f=root/usr/share/firewall4/main.uc;hb=HEAD#l21

If I remove the || !length(set.entries) from the line linked above, I see all the flush and add statements generated. I get too easily confused with boolean logic and negation, so I will think about it some more. jow is pretty busy these days.

So when no set entries are defined, the length of set.entries is null, and is inverted to true, so the set is skipped unnecessarily (I think).

[jow]

@dave14305

I think we need @jow to validate the intended logic.

I think we can remove that entire skip condition and simply reload sets unconditionally given that the other restrictions (data type equivalence, set existing) apply.

• If list entry ... in /e/c/firewall is changed and fw4 reload-sets called, skipping a set just because it has no loadfile defined is counter-intuitive.
• The requirement for the reloaded set to have entries already clearly makes no sense

[dave14305]

Thanks. Do you need to account for sets that are only created but not populated during initialization? For example, sets that are populated dynamically from external scripts (nft add element…) or dnsmasq 2.87?

[jow]

@dave14305

Do you need to account for sets that are only created but not populated during initialization?

Hm indeed, that might've been the intention of the original skip statement. Maybe we should skip sets that neither have static entries in uci nor any loadfile option defined.

Or maybe we shouldn't bother and teach users that a set reload will flush all sets and that external filling processes need to be restarted...

Opinions?

[dave14305]

My opinion only reload sets when we know there is a source for loading them (loadfile or entries). Is it just a matter of ANDing the 2 conditions instead or ORing (No loadfile AND no entries)?

If dnsmasq 2.87 efforts find a new owner, there would be an increase in nftables set adoption. Flushing those sets could create unrealized problems.

I seem to get the intended results by changing the || to &&. I also added an if ( set.loadfile ) check before the fw4.parse_setfile(set, printer); line to avoid a warning for null loadfile when only an entry is defined in the ipset config.