Nope:
I observed this on the change to fw4. I have not tested the large sets since then.
I tried loading 300k+ elements on an OpenWrt VM with 256 MB RAM and it dies from OOM.
Increasing RAM to 1 GB makes it work and optimizes the set to ~30 elements including almost the entire IPv4 and IPv6 address spaces.
My little x86 box has 4GB of RAM and my set consumes about ~340MB of it. Obviously that's quite a lot, but I'm nowhere near hitting an OOM.
1 Like
Netfilter, iptables, nftables, gets extended performance testing. I doubt you try something nobody has ever tested before....
Can you share exactly what you do so others can try to reproduce and observe the issue you are describing?
2 Likes
Sure, here's my generated nft script to create the set:
1 Like
I just tested mine - it worked.
But took over 90 seconds to load (I stopped keeping track because I thought it failed - then the command prompt reappeared after some time). Mine only have 114k entries, and some are duplicates. As I recall, that's normal - as I previously tested on an x86_64:
root@OpenWrt:~# ls /tmp/block_list.txt -l
-rw-r--r-- 1 root root 1634582 Sep 25 15:18 /tmp/block_list.txt
config ipset
option name 'script_block'
option match 'src_net'
option loadfile '/tmp/block_list.txt'
config rule
option family 'ipv4'
option ipset 'script_block'
option target 'DROP'
list proto 'all'
option name 'Drop-script_block_In'
option src '*'
Today, I'm running a MT7621A device.
Are you running anything else that requires CPUs to process your traffic?
e.g.
- logging packets
- packet inspection
- mirroring packets
Nope:
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.663012] nft invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=0, oom_score_adj=0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.672685] CPU: 1 PID: 1128 Comm: nft Not tainted 5.15.127 #0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.678731] Stack : 8089b888 808123d8 00000840 80083bc8 00000000 00000000 00000000 00000000
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.687337] 00000000 00000000 00000000 00000000 00000000 00000001 827ef608 814855c0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.695942] 827ef6a0 00000000 00000000 827ef4b0 00000038 8039e4c4 ffffffea 00000000
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.704551] 827ef4bc 000001ac 8081aa78 ffffffff 80738d00 827ef5e8 81d14360 00000f3c
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.713151] 00000000 00040cc0 8089b888 808123d8 00000001 80410730 00000004 809d0004
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.721737] ...
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.724389] Call Trace:
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.724416] [<80083bc8>] 0x80083bc8
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.730766] [<8039e4c4>] 0x8039e4c4
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.734490] [<80410730>] 0x80410730
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.738199] [<80007908>] 0x80007908
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.741889] [<80007910>] 0x80007910
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.745578] [<803825bc>] 0x803825bc
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.749288] [<801417a8>] 0x801417a8
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.753000] [<80142000>] 0x80142000
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.756699] [<801429f0>] 0x801429f0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.760375] [<80194a40>] 0x80194a40
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.764118] [<801a4538>] 0x801a4538
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.767823] [<801a6cd4>] 0x801a6cd4
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.771522] [<8037ce0c>] 0x8037ce0c
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.775203] [<801a8aa8>] 0x801a8aa8
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.778927] [<83c83eb0>] 0x83c83eb0 [nf_tables@8cfaf769+0x2a3f0]
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.785189] [<83c92750>] 0x83c92750 [nf_tables@8cfaf769+0x2a3f0]
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.791466] [<801a8aa8>] 0x801a8aa8
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.795216] [<80032fbc>] 0x80032fbc
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.798941] [<80194108>] 0x80194108
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.802687] [<8059cb2c>] 0x8059cb2c
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.806475] [<8059cb2c>] 0x8059cb2c
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.810242] [<80194108>] 0x80194108
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.814069] [<83c92ca8>] 0x83c92ca8 [nf_tables@8cfaf769+0x2a3f0]
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.820299] [<8037ce00>] 0x8037ce00
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.823971] [<82ecef5c>] 0x82ecef5c [nfnetlink@4c670abd+0x1870]
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.830103] [<8018b510>] 0x8018b510
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.833768] [<80200020>] 0x80200020
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.837491] [<82ecf290>] 0x82ecf290 [nfnetlink@4c670abd+0x1870]
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.843593] [<80324cc8>] 0x80324cc8
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.847263] [<8059fac4>] 0x8059fac4
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.850935] [<8059ff70>] 0x8059ff70
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.854626] [<8059fc00>] 0x8059fc00
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.858292] [<804f1498>] 0x804f1498
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.861947] [<80327d6c>] 0x80327d6c
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.865633] [<804f33e4>] 0x804f33e4
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.869303] [<80184680>] 0x80184680
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.873018] [<804fa230>] 0x804fa230
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.876716] [<804f3560>] 0x804f3560
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.880389] [<80014550>] 0x80014550
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.884083]
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.886449] Mem-Info:
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.888948] active_anon:32759 inactive_anon:1125 isolated_anon:0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.888948] active_file:326 inactive_file:9 isolated_file:0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.888948] unevictable:0 dirty:0 writeback:0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.888948] slab_reclaimable:1487 slab_unreclaimable:63168
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.888948] mapped:1049 shmem:1075 pagetables:229 bounce:0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.888948] kernel_misc_reclaimable:0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.888948] free:4168 free_pcp:377 free_cma:0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.925579] Node 0 active_anon:4480kB inactive_anon:131056kB active_file:0kB inactive_file:1180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4272kB dirty:0kB writeback:0kB shmem:4300kB writeback_tmp:0kB kernel_stack:1328kB pagetables:916kB all_unreclaimable? no
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.950511] Normal free:16672kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:4480kB inactive_anon:131056kB active_file:0kB inactive_file:1616kB unevictable:0kB writepending:0kB present:458752kB managed:444440kB mlocked:0kB bounce:0kB free_pcp:1536kB local_pcp:580kB free_cma:0kB
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.977759] lowmem_reserve[]: 0 0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.981355] Normal: 942*4kB (UM) 340*8kB (UME) 134*16kB (UME) 57*32kB (UME) 22*64kB (UM) 12*128kB (UM) 3*256kB (M) 3*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 15704kB
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561354.996524] 1344 total pagecache pages
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561355.000468] 0 pages in swap cache
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561355.004037] Swap cache stats: add 0, delete 0, find 0/0
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561355.009449] Free swap = 0kB
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561355.012577] Total swap = 0kB
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561355.015651] 114688 pages RAM
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561355.018704] 0 pages HighMem/MovableOnly
Mon Sep 25 15:22:51 2023 kern.warn kernel: [1561355.022753] 3578 pages reserved
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.026083] Tasks state (memory values in pages):
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.030957] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.039818] [ 599] 81 599 429 48 20480 0 0 ubusd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.048282] [ 600] 0 600 265 7 20480 0 0 askfirst
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.056993] [ 635] 0 635 303 15 20480 0 0 urngd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.065488] [ 1032] 514 1032 371 37 16384 0 0 logd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.073840] [ 1091] 0 1091 982 200 20480 0 0 rpcd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.082145] [ 1351] 0 1351 319 14 20480 0 0 dropbear
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.090863] [ 1352] 0 1352 320 15 16384 0 0 dropbear
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.099654] [ 1456] 0 1456 728 14 20480 0 0 hostapd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.108290] [ 1457] 0 1457 728 14 20480 0 0 wpa_supplicant
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.117507] [ 1459] 101 1459 754 29 16384 0 0 wpa_supplicant
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.126732] [ 1461] 101 1461 797 47 20480 0 0 hostapd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.135359] [ 1521] 0 1521 572 86 16384 0 0 netifd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.143867] [ 1887] 0 1887 348 15 16384 0 0 crond
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.152238] [ 2033] 0 2033 739 76 16384 0 0 uhttpd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.160715] [ 2948] 0 2948 1693 1063 28672 0 0 softflowd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.169495] [ 2951] 0 2951 1089 725 24576 0 0 softflowd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.178262] [ 2982] 0 2982 728 28 16384 0 0 ntpd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.186591] [ 3000] 123 3000 345 14 20480 0 0 ntpd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.194913] [ 3433] 0 3433 753 39 20480 0 0 xxxx-ripd-2.4.1
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.204188] [ 21447] 0 21447 344 12 16384 0 0 udhcpc
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.212680] [ 21654] 0 21654 279 12 16384 0 0 igmpproxy
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.221400] [ 21997] 0 21997 444 111 20480 0 0 dynamic_dns_upd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.230695] [ 22000] 0 22000 444 111 20480 0 0 dynamic_dns_upd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.239993] [ 22001] 0 22001 437 104 20480 0 0 dynamic_dns_upd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.249298] [ 22002] 0 22002 437 103 20480 0 0 dynamic_dns_upd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.258615] [ 24393] 0 24393 305 16 20480 0 0 odhcp6c
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.267218] [ 25509] 0 25509 441 105 16384 0 0 dynamic_dns_upd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.276506] [ 26758] 0 26758 448 39 20480 0 0 odhcpd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.285018] [ 26910] 0 26910 958 308 20480 0 0 snmpd
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.293425] [ 10290] 0 10290 728 28 20480 0 0 dnsmasq
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.301943] [ 10292] 453 10292 1242 857 24576 0 0 dnsmasq
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.310518] [ 860] 0 860 324 16 16384 0 0 dropbear
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.319184] [ 861] 0 861 345 13 20480 0 0 ash
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.327440] [ 898] 0 898 348 12 20480 0 0 sleep
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.335871] [ 996] 0 996 348 13 16384 0 0 sleep
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.344273] [ 999] 0 999 348 12 20480 0 0 sleep
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.352652] [ 1109] 0 1109 348 12 16384 0 0 sleep
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.361005] [ 1120] 0 1120 344 8 16384 0 0 load_ipipfilter
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.370282] [ 1126] 0 1126 347 8 20480 0 0 fw4
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.378492] [ 1128] 0 1128 30559 29338 143360 0 0 nft
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.386691] [ 1206] 0 1206 348 13 16384 0 0 sleep
Mon Sep 25 15:22:51 2023 kern.info kernel: [1561355.395082] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=nft,pid=1128,uid=0
Mon Sep 25 15:22:51 2023 kern.err kernel: [1561355.407539] Out of memory: Killed process 1128 (nft) total-vm:122236kB, anon-rss:117348kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:140kB oom_score_adj:0
Nothing very significant. Just SQM and a set of rules to add DSCP tags to connections/packets based on ports, destination IP etc. CPU usage never gets over 25% even when downloading or uploading at 1Gb/s.
When nftables is behaving badly after adding the large set it doesn't seem to be caused by a shortage of CPU resource. CPU usage looks completely normal until I run a command that needs to use nft - for example, to add a single element to a very small and completely unrelated set. When I run a command like that it saturates a single core for ~10 seconds.
The linked code is very suboptimal.
You can greatly improve performance by using the option loadfile.
This should also automatically merge adjacent and overlapping elements, decreasing memory consumption.
1 Like
Thanks for that. I'm very much still learning nftables - I only upgraded from OpenWrt 21.02 last week. Is loadfile an nft option? I can't find any information on that anywhere. Note that it's not the creation of the set that's the problem though (the script runs in less than three seconds) - the problem is how nftables behaves once the set is populated.
I just tried specifying auto-merge when creating the set to tell nftables to automatically merge overlapping ranges, and it did seem to reduce memory usage slightly, but the performance problem was still there (maybe slightly less bad than before, presumably because there were fewer elements as a result).
No, it is specific to the firewall service.
Keep in mind that OpenWrt expects its users to operate high-level tools such as services and their UCI configs, instead of low-level commands.
To be clear, I increased the RAM to 1 GB, but your code still died from OOM, while the loadfile option works just fine, optimizing the size of the same set from ~200k to ~80k elements.
1 Like
Thanks for the info. I just tried this as well, and the performance problem with nftables does still seem to be present once the set is populated - it's just not as bad, presumably because the resulting set has fewer elements than before. After adding the set via UCI which ends up with 80K elements, as you said, my bandwidth seems ok but all nft commands are noticeably slow to complete (just not insanely slow like they were in the presence of the 200K set).
Are you able to reproduce anything like that? Or do nft commands complete in the normal time for you when the 80K set is populated?
I fully appreciate that the UCI method of adding the set produces a more optimized set with fewer entries (it seems to me that nftables' auto-merge option doesn't merge adjacent ranges, only overlapping ones), but I don't think the nftables performance problem I reported is really related to that.
The problem I'm hitting seems to correspond entirely to the number of entries in the set. UCI and the firewall service can optimize and reduce the number of elements in the set, but that doesn't fix the underlying problem with nftables. I'm sure if I added a set via UCI which ended up with 200K entries (even after optimization) I'd hit exactly the same performance problems as before.
Here are my results with the same ~80k set:
# List terse ruleset
> time nft -t list ruleset > /dev/null
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
# List full ruleset
> time nft list ruleset > /dev/null
real 0m 0.90s
user 0m 0.52s
sys 0m 0.38s
# Count set elements
> nft -j list sets \
| jsonfilter -e "$['nftables']
[*]['set']['elem'][*]" \
| wc -l
82463
I get similar results to you on my router after populating the 80K set. The performance in the presence of the 200K set is ten times worse:
# time nft list ruleset > /dev/null
real 0m 13.01s
user 0m 4.67s
sys 0m 8.11s
Then look at what happens when adding an element to a completely unrelated set:
# time nft add set inet fw4 test-set { type ipv4_addr \; comment \"Test set\" \; flags interval \; }
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
# time nft add element inet fw4 test-set { 192.168.1.1 }
real 0m 20.63s <--- !!!
user 0m 7.81s
sys 0m 12.66s
Anyway, I'm done for this evening - my family keep complaining that "the internet is going weird" 
1 Like
To tell the truth, I normally edit the UCI config and reload the service to apply the changes, so I don't need to run any nftables commands other than diagnostic ones.
IP set updates are typically scripted and scheduled, so it just requires to reload the service after updating the loadfile, which also allows to avoid direct interaction with nftables.
Thus, even if this issue happens, it doesn't affect me that much. 
I have a firewall script that runs on firewall reload/restart to setup stuff for DSCP tagging - about 20 sets of various types (MACs, ports, IPs) with fewer than 30 elements each, about 20 chains, and about 100 rules. It's not the most efficient thing in the world because it doesn't add the sets and rules atomically (not yet anyway, it's a work in progress) - it currently runs lots of individual nft commands.
That script usually completes in about three seconds, but it runs really slowly in the presence of the 80K set, and basically stalls forever in the presence of the 200K set.
I also have dnsmasq configured to add individual IP addresses to various sets based on hostname and I suspect the performance of those operations tanks as well, although I have no way of measuring that directly.
1 Like
Look at or install banIP and see how it handles this.
Note it uses country blocks from: IPdeny IP country CIDR blocks which might differ from the ones you are using.
3 Likes
Thanks for this. Somehow I've gone my whole OpenWrt career without hearing about this package! I'm not at home today but I'll definitely give it a go when I get a chance.
There are far fewer IP ranges in the IPdeny sets - e.g. only 65,855 ranges in the unaggregated US set vs 159,383 for the equivalent db-ip.com data. I have no idea which data is better in terms of accuracy but clearly the IPdeny sets will perform better, especially the aggregated sets.
In any case I still think there's an issue here worth investigating. Switching to a different approach might help my particular requirement right now, but it still seems that something starts to go very wrong inside nftables after populating sets with >100K elements. I would expect lots of memory usage, and maybe a performance dip while actually populating the table, but once that's done the performance of nftables shouldn't be affected, especially if the set isn't even being used. The presence of the set certainly shouldn't affect the performance of operations on unrelated sets.
I've been doing some more testing with interesting results.
Firstly, setting the set policy to "memory" instead of "performance" seems to make the nftables "auto-merge" setting work properly, and the CIDRs are merged in a similar way (maybe exactly the same way?) as they when the set is defined using the UCI firewall configuration file, resulting in a set of 80K elements. Ironically this results in much better performance than using the default "performance" setting...
Secondly, if leave the set policy as "performance", but split the 200K set into 10 sets of ~20K each, the performance is about the same as having a single 20K set. So the problem does seem to be related to the size of the largest set, not the total number of elements in all sets.
I'm currently trying BanIP with the much smaller geoip dataset from IPdeny (about 30K elements for the same country list). I've already hit issues with accuracy (an IP I know is definitely in the UK is listed as located in the US), but hopefully I can work around this and I won't have to pursue my home-rolled solution any further. Maybe I'll need a combination of banIP / home-rolled by generating a custom block list for BanIP from a more accurate location database.
Welcome to geoIP 
This is really at best a heuristic... that does not really map well onto how the internet actually operates. Granted for many things is clearly "better than nothing" but precise, robust or reliable it is not. But knowing that does not really help if the situation is bad enough to demand some action...