FW4 / NFTables 'add element' Speed

pretty new to fw4/nft still, thought I would ask a question here to make sure I'm doing this in the most efficient way. have a list off 17,294 IPs and networks from FireHOL, the level1 + level2 lists combined. have a set that reloads on firewall up and populates the set with the following...

nft add set inet custom mal { type ipv4_addr; flags interval; comment "malicious CIDRs"; }
...
nft flush set inet custom mal
cat /root/malicious.txt |sed -e 's/.*/{&}/' |xargs -n1 nft add element inet custom mal

...the thing is that it's taking an hour to load the 17k lines on a NBG6817.

any way to speed this up?

…I am nft newbee, but my bash performance brain cells get nervous on xargs. My guess is, this is, where your router spends the day, iterating through thousands of lines, calling „nft add“ a few thousand times. My guess is that each „nft add“ takes a few hundred milliseconds for processing each call. Probably go and prefix it with „time …“, to confirm.

quick look in nft doc tells me, there is a

nft -f rules.conf

Again: So far I am the greenest nft newbee of all nft newbees, but I would try to shell-pre-process the nft input file, with the goal to obtain an nft-readable output file and then do 1 single call to nft command to load that generated config file. Avoiding thousands of separate add-calls.

Maybe there are more alternatives, to do a single load of a preprocessed input file. But key from my perspective would be to generate a preprocessed list.

thanks for the idea, that may be a good way to skin this cat. dumping the table to a file it seems to be just a comma separated list 2 items wide so may be do-able without too much hassle. it would mean that the list could not be dynamic and only change when the firewall is initialized but that may be a price worth paying for my use-case. I would worry that the table load may still take an hour and that would delay the other rules from being enforced, will need to test.

I believe it's the 'nft add element' command that is taking by far the most time, I use xargs for a bunch of other things and never noticed it slowing anything down. 3600 seconds by 17k means each command is taking 1/5th of a second roughly, I'm guessing most of that is inserting the element and re-indexing the database or whatever it does internally. guess I was hopping for a mass-add command, or delay re-indexing for X seconds or something along those lines.

cheers.

so a little testing and it looks like a config file with just the set loaded doesn't overwrite the entire table, just overwrites the set and loads much quicker. thanks for the idea, @Pico

table inet custom {
        set mal {
                type ipv4_addr
                flags interval
                comment "malicious CIDRs"
                elements = { 0.0.0.0/8, 1.0.171.2, <etc> }
        }
}

fw4 can load a ipset specified in /etc/config/firewall using the option loadfile '/path/file' syntax. If you download a new file, you can the. run fw4 reload-sets to update the set without reloading the full ruleset.

I've seen reports that this method isn't working, bug that was patched?

It was patched after 22.03.2 was tagged, but should be OK if you have updated firewall4 since then (disclaimer: I don’t use this loadfile option at all).