Non-trivial means synchronous change in multiple places potentially breaking something else.
Also interesting is this option thatās mentioned in comment 7:
--debug=netlink
If you run the nft add element command with this option you can see that it does indeed get the contents of every set, not just the one youāre updating. Seems bonkers.
1 Like
It is this same Anton over there 
Actually, I have to correct my statement above:
The reported bug concerns adding new sets
I forgot that the test script actually adds elements to sets, so it does in fact concern updating sets. At the time of filing this bug report, I was not aware that there was a better method of populating sets, which is basically piping the complete nft set definition along with elements into STDIN of nft -f -. This method was hinted at by @dibdot after I filed the bug, and I subsequently adopted it in geoip-shell. This avoids most of the memory and performance penalty of modifying sets via add element.
Unfortunately, the bug title does not reflect the substance of the bug. I guess I should post another comment there asking to rename the bug? And stressing the point of memory consumption spike.
I use that approach for initial population of sets, but I also have dynamic sets where I want to add new elements at arbitrary times. In addition, some of these sets have timeouts, so recreating the whole set each time I want to add a new element isnāt an option (because doing so would reset all of the entriesā timeouts).
Iāve modified each of my scripts to write all their nft commands to a temporary file, and then pass that file into nft -f at the end. This doesnāt eliminate the add element performance problem, but if multiple elements need to be added to multiple sets, collecting the commands into a single nft script and only running nft once does at least seem to mean that the performance problem only occurs once - i.e. it occurs once for each execution of nft -f, rather than for each add element command in the script.
Depends what they want I guess. Iām happy for the nft add element performance issue to be handled under the existing bug if they are.
It is possible to use nft and good old iptables (sets !), simultaneously. Assuming careful coding 
you can benefit from ipsets speed advantage.
A mixture of iptables and nftables rules sounds like something that would definitely get me into a tangled mess 
For me personally I donāt think itās worth the hassle. Banip seems to work pretty well with my current blocklists (~80,000 elements) without causing any significant performance issues. Ideally Iād like to try using a couple of the larger lists as well, but they push the total number of elements up to 300,000+, which does cause problems with my other scripts.
As it stands itās not a massive issue in my setup - just an annoyance to be aware of and avoid.
Yes and no i.e. only via iptables-nft.
Considering that multiple sets existing in the same nft table seems to be significantly contributing to the gravity of the issue, perhaps using separate tables might be a workaround. If using geoip sets and rules along with other sets created by banip, you could test a mixed setup of banip + geoip-shell. geoip-shell creates its own table, so geoip-related nft sets would be naturally separated from non-geoip sets.
Unfortunately that doesnāt make any difference to the add element issue - the banip sets are already in a separate table to the sets that Iām modifying in my scripts, but it looks like add element gets all the sets from all the tables (running with --debug=netlink seems to confirm this).
2 Likes
That's odd because in my bug report from 2 years ago, I wrote that loading sets into other tables is unaffected, and I kinda recall testing this scenario.
Sorry, you are wrong. I used following .config:
CONFIG_PACKAGE_kmod-ipt-ipset=y
CONFIG_PACKAGE_ipset=y
CONFIG_PACKAGE_libipset=y
# CONFIG_PACKAGE_iptables-nft is not set
CONFIG_PACKAGE_iptables-zz-legacy=y
CONFIG_PACKAGE_tc-mod-iptables=y
and following ipt-cmd:
ipset restore -! < vpn_ips_saved_for_ipset
iptables -I PREROUTING -t raw -m set --match-set vpn_ips src -j DROP
for fast block of well-known-VPNs. Has nothing to do with nft (used for standard fw setup, only), right ?
iptables-nft would use same -m set kmod as iptables-legacy, but that is not reachable via nft rules.
The point was, that definitely iptables/ipsets CAN be used in parallel with nft rules. Which you partly objected. In more complicated combinations I guess, it is only a question of correct prio to use for the nft rules, not to conflict with iptable rules when using same hook.
You just have to shift nft priorities +- 1 tohave it somewhat predictable
You asked for a coredump, I have added a coredum for nftables-1.1.3 on https://github.com/jidhub/OnlyForCoreDumps
nft list ruleset
nft: src/mergesort.c:23: concat_expr_msort_value: Assertion `ilen > 0' failed.
Program received signal SIGABRT, Aborted.
Saved corefile core.1446543
Whole text transcript included in that .tar.gz file.