# Nft sets: Terrible performance **URL:** \ **Category:** For Developers\ **Created:** [August 26, 2024, 5:44am UTC](https://forum.openwrt.org/t/nft-sets-terrible-performance/208666 "2024-08-26T05:44:59Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![reinerotto](https://forum.openwrt.org/letter_avatar/reinerotto/32/5_5575768a8748004e209b776fc1b2916d.png) [@reinerotto](https://forum.openwrt.org/u/reinerotto)\ **Post date:** [August 26, 2024, 5:44am UTC](https://forum.openwrt.org/t/nft-sets-terrible-performance/208666/1 "2024-08-26T05:44:59Z")
add set ip filter fb\_ips { type ipv4\_addr ; auto-merge; flags interval; } time nft get element ip filter fb\_ips { 52.216.24.236 } \>/dev/null real 0m 0.15s user 0m 0.10s sys 0m 0.05s This is the typical timing to check for a facebook-ip in a set. Measurement done on a MT7620-based device. This looks quite slow to me. Any comments or timing examples from a more capable device ? ---
**Author:** ![anon63541380](https://forum.openwrt.org/letter_avatar/anon63541380/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon63541380](https://forum.openwrt.org/u/anon63541380)\ **Post date:** [August 26, 2024, 6:26am UTC](https://forum.openwrt.org/t/nft-sets-terrible-performance/208666/2 "2024-08-26T06:26:23Z")
Please add precise device id `ubus call system board` Given we get approx same init time it might be worth repeating thousand gets to get usable timings where load time plays no role. (firewalld on rawhide) ```plaintext time nft list ruleset | time nft -c -f - real 0m 0.03s user 0m 0.01s sys 0m 0.01s real 0m 0.13s user 0m 0.05s sys 0m 0.00s ``` ---
**Author:** ![reinerotto](https://forum.openwrt.org/letter_avatar/reinerotto/32/5_5575768a8748004e209b776fc1b2916d.png) [@reinerotto](https://forum.openwrt.org/u/reinerotto)\ **Post date:** [September 16, 2024, 12:31pm UTC](https://forum.openwrt.org/t/nft-sets-terrible-performance/208666/3 "2024-09-16T12:31:32Z")
ubus call system board { "kernel": "5.15.162", "hostname": "........", "system": "MediaTek MT7620A ver:2 eco:6", "model": "Zbtlink ZBT-WE826 (16M)", "board\_name": "zbtlink,zbt-we826-16m", "rootfs\_type": "squashfs", "release": { "distribution": "OpenWrt", "version": "23.05.4", "revision": "r24012-d8dd03c46f", "target": "ramips/mt7620", "description": "OpenWrt 23.05.4 r24012-d8dd03c46f" } } I do not consider thousands of gets to be interesting. Deviation is even more important, to consider the worst case. In the experiment mentionend above, the set was only about 2000 members large. Now, trying 10 times the size, results are terrible. For me, sets in nftable are not usable for higher amounts of members. Thus, trying to use ip-sets from iptables. iptables ONLY to be used for ip-sets, in raw table. Not to interfere with all the NFT stuff. UPDATE: This is from using ip-sets/iptables-raw, for a set of about 20k elements: time ipset test test\_ips 32.240.71.12 Warning: 32.240.71.12 is in set test\_ips. real 0m 0.01s user 0m 0.00s sys 0m 0.01s Populating this ipset using "ipset restore" takes about 5s. Initial load was about 3minutes. I could not create such a large set using NFT. My router rebooted, when trying to load this large set; however, success when loading only about 5000 elements. ---
**Author:** ![anon63541380](https://forum.openwrt.org/letter_avatar/anon63541380/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon63541380](https://forum.openwrt.org/u/anon63541380)\ **Post date:** [September 16, 2024, 2:33pm UTC](https://forum.openwrt.org/t/nft-sets-terrible-performance/208666/4 "2024-09-16T14:33:40Z")
Adding element to nftset pumps set to userland, then adjusts , then pumps back to kernel. It takes significantly more RAM than essential for the task. ---
**Author:** ![dibdot](https://forum.openwrt.org/user_avatar/forum.openwrt.org/dibdot/32/3100_2.png) [@dibdot](https://forum.openwrt.org/u/dibdot)\ **Post date:** [September 16, 2024, 6:15pm UTC](https://forum.openwrt.org/t/nft-sets-terrible-performance/208666/5 "2024-09-16T18:15:14Z")
> [@reinerotto](#): > > In the experiment mentionend above, the set was only about 2000 members large. Now, trying 10 times the size, results are terrible. For me, sets in nftable are not usable for higher amounts of members. What does that mean in detail? ---
**Author:** ![antonk](https://forum.openwrt.org/user_avatar/forum.openwrt.org/antonk/32/93698_2.png) [@antonk](https://forum.openwrt.org/u/antonk)\ **Post date:** [September 16, 2024, 6:49pm UTC](https://forum.openwrt.org/t/nft-sets-terrible-performance/208666/6 "2024-09-16T18:49:22Z")
See: > [Some thoughts about nftables performance](https://forum.openwrt.org/t/some-thoughts-about-nftables-performance/185952) ---
**Author:** ![antonk](https://forum.openwrt.org/user_avatar/forum.openwrt.org/antonk/32/93698_2.png) [@antonk](https://forum.openwrt.org/u/antonk)\ **Post date:** [September 16, 2024, 7:08pm UTC](https://forum.openwrt.org/t/nft-sets-terrible-performance/208666/7 "2024-09-16T19:08:07Z")
Bottom line, some operations are slow with current nft sets (and getting progressively slower as elements count goes up), but mostly there are workarounds. Memory consumption also gets progressively worse with larger sets when populating elements, to the extent where atomic sets replacement becomes not viable on memory-constrained devices. There are workarounds for this as well, although memory consumption is still significantly higher than with ipset, even with the workarounds. The netfilter people are aware of most of the bugs, although I'm not sure that there is a bug filed specifically for the `nft get element` command. ---
**Author:** ![reinerotto](https://forum.openwrt.org/letter_avatar/reinerotto/32/5_5575768a8748004e209b776fc1b2916d.png) [@reinerotto](https://forum.openwrt.org/u/reinerotto)\ **Post date:** [September 16, 2024, 7:15pm UTC](https://forum.openwrt.org/t/nft-sets-terrible-performance/208666/8 "2024-09-16T19:15:13Z")
Use ipset+iptables for large sets, if possible. Now I have it in parallel with nftables. But iptables(+ipset) ONLY for large set, used to block certain incoming packets, in special table "raw". (You might think about blocking DoH-servers ...) Seems to have no neg. impact, but need more time, to be shure. I have to note, that I do not use fw4 at all, but pure nft-rules + pure iptables rules. When using certain functionality (transparent proxy, captive portal etc.), fw3 or fw4 were/are a PITA.