banIP 22.03 discussion and blocklist script offered

WIP

It never flushes, and giving it an empty blocklist is the same as flushing.
Needs diffutils, uses sort, awk, diff, and egrep, grep -E; doesn't have to use wget.

With the URL's in it currently, takes 30-36 seconds on ARM64 clocked to 1 Ghz.
For closer to 40k blocked ipv4 addresses, heavily processed.

Easy to change and work with, with basic mechanism functional and reasonably fast.

Many may need something like it to upgrade to 22.03 and use fw4.

Not more ambitous than hoping some parts go into banIP fw4 for emerging threats and rapidly changing blocklists, and may be the better application of this and also block output.

Admit that script does weird things for ipv4, not supporting CIDR /8 ranges etc, but up to /24 should work for calculating it into individual ip's. Something I have yet to test sufficiently with the awk program included as a function and used. The ipv4 blackhole set currently does not support adding /31 elements or anything like it and will error out if tried.

Whitelisting i suggest doing with grep -vf whitelist_file1 blocklist_ipv4 or something like it in apropriate disrespect of the inelegant script. This is however strangely fast for all the things done.

IPv6 part seem almost pointless and could be removed for a speed up.

So get on this lawn, for more get off my lawn. Not cleaning up, or just keeping the diff files give you a time machine of ip's added or remobed from the blackhole ipset.

3 Likes

Well, was hoping some interest.
As never flushing, and blank blocklist being the same as flushing.
To be sure about the state of the ipset managed by the script.

That quality of service improve, told it was less than 10 second runtime on Core i5 for voipbl.

And well, the mechanism, and using old and ancient tools with optimized algorithms for the tasks.
diff, sort, awk that may work fine on a boxes from the 90's - you're good with python if you beat it.

and well, having a diff between old and new. and fw4 and nft.

I am sure, that there are a lot of interest in an alternative to banIP for fw4.
But I think it is a little difficult to understand, what and how the script works. And as I understand it, I have to compile cidr and prips. So even though you might have put a lot of effort into this project, somebody needs to do some explanation and maybe compile, if the scripts should be testet by a broad usecase.

Hi!

There really isn't any interest, seems like the user base believe adblock is better.

And anyone managing networkrs, and blocking outgoing connections to known botnets or malware must be using other things for that specific task.

As you point out, prips and grepcidr isn't packaged for openwrt as opposed to ubuntu and other "network" appliance distributions.

Yeh, I'm puzzling a bit, as this is generally useful, even if prips and grepcidr is an inconvinience.
But the two programs are so simple they don't come with configure or autogen.sh.

Not awfullly difficult to understand what mine does, ensured you can look at lots of temporary files easily to check things.

Now getting slightly more complicated because i really really want this never flush feature, and the diff, and that it works properply and does sane things with netblocks, and between small netblocks and individual's ips.

Hard to do without grepcidr.

grepcidr -C 202.27.96.0/21 cidr_play 
202.27.100.0/22
202.27.96.0/21
202.27.96.0/23
202.27.98.0/24
202.27.99.0/24

That overlapping intervals as you were, which makes NFT reject them, the biggest is sufficient.

But get's a combination of irritating and oddly pleasureable procrastrination wonder how to do best do it.

grepcidr -D $line  cidr_play | sort -t "/" -k2 | head -n1
202.27.96.0/21

is guaranteed to output the biggest netblock for any line that you check.
as these programs lacks a few features, making unix pipe gymnastics necessary.
where rest of the lines need to stemped on and deleted over and over again with sed perhaps.
not sure if it is nft ipsets that are feature poor, or if these programs need work, or if i just haven't found the essentials.

In either case, you can run this on a faster router box i thought.
block outgoing for emerging threats on AP's, warn and email about it.

lots of regex and some gymnastics make these things inherently unreadable, and anyone serious would likely maintain own methods.

may not be much more, than matchin all lines found in cidr netblock file with the file and count line output, and loop from 2 lines up to most lines and all blocks are sanitized. Kept to a minimum, as i figure anything smaller than /24 can be single ip's perfectly fine.

think it should generally help average users with malware, even if not sexy cyber bla bla.
or well, never flushing and keeping track of updates and removals with diffs.
is something i see as minimal requirement for easily adaptation to a small set up.
comes with a level of awkward for that reason if you need more than block incoming.

I thought, have that in the monkey work, that if it can work as the "engine" within banip around quirks and nusances for those just wanting to block incoming to protect a service or two on a public ip, then even better.

Atm the features are limited to whitelisting, and another list for "alerts" if an ip is found in downloaded blocklists. And that is probably enough, and someone else with other interest would perhaps add lots of fw4 and wrap javascript and web stuff around it, feeding it a list of urls and whitelist.

But yeah, trying to keep it simple, that netblock currently has the highest hits among various public blocksts a bit more than 1MB in raw text. Sanitizing for biggest blocks, removing the smallest and "white listing" single ip's against ip blocks for a list without interval overlaps and no collisions with whats in the nft ipset from before with diff.

Biggest netblocks are well over 16 million ip's, and have collision with something as big as /12 for 1 million. And dealing with closer to 80k individual/small subnets. While honestly, just ignoring this and block individual ip's cover essential need or covers most of it for the majority.

Adblock doesn't do much towards, "internet immune system" for malware and crap.
Lowering the bak for running an AP at a breach front and informing over the hotspot of possible malware.

Doesn't really take much to be at "frontline" fighting botnets, and make that a common place networking topic in public networks.

I'm perplexed about scarcity and interest, as if monitoring traffic does anything useful for anyone other than those making bold statements with hand waving.

And that 22.03 was released without banIP and "gate way drugs" like it.
Running a firewall, but not so important to block contacting command and control for ransomware or botnets it seems.

1 Like

Scripts seem to have acheived basic functionality for the blockcidr stuff.

:/tmp/badhost# time sh /root/nft-badhost.sh 2> nft.log ; nft list set inet fw4 blackhole | wc -l
downloaded blocklists Sat Sep 24 23:28:30 CEST 2022
Start of nft ipset operations Sat Sep 24 23:30:24 CEST 2022
iteration of nft ipv4 delete
iteration of nft ipv4 add
wrt-badhost.sh completed Sat Sep 24 23:30:26 CEST 2022
total ipv4 elements in list: 87685 /tmp/badhost/wrk/blocklist_ipv4.JKPDeM
total ipv6 elements in list: 66 /tmp/badhost/wrk/blocklist_ipv6.jmhPiG
Amount of ipv4 entries before changes: 37230 /tmp/badhost/wrk/nft_ipv4_list.baFNMd
ipv4 added: 1
ipv4 removed: 2
ipv6 added: 0
ipv6 removed: 0
Done

real	2m4.104s
user	1m35.143s
sys	0m36.130s
18621

Last line being line count for nft blackhole ipset, so +6 and everything double.
So 37k entries to have those 85k addresses block after cidr address block interval overlap sanitation.

You need grepcidr and prips C programs, sed, awk, grep, sort and diffutils.

First run is 3 minutes, where it should be able to make it faster, definitely a few uncessary things left from earlier trial and error.

This is cool.

wrt-badhost.sh completed Sun Sep 25 00:45:16 CEST 2022
total ipv4 elements in list: 87938 /tmp/badhost/wrk/blocklist_ipv4.dnAICC
total ipv6 elements in list: 67 /tmp/badhost/wrk/blocklist_ipv6.fOgAbn
Amount of ipv4 entries before changes: 37230 /tmp/badhost/wrk/nft_ipv4_list.clCINh
ipv4 added: 265
ipv4 removed: 13
ipv6 added: 2
ipv6 removed: 1
Done

real	2m8.012s
user	1m37.709s
sys	0m36.153s

But troublesome, most lists now worked with in this work in progress and intermittent past time doesn't need to be downloaded all the time.

Due to a bug in nftables, for me, 224.0.0.0-255.255.255.255 gets deleted every run of the script, and 224.0.0.0/3 added.

This is shown from testing and what is about 2 minute runtime after download of blocklists at ARM64 clocked at 1Ghz on Cortex A72 cores, where these operations should run fast on any network equipment, especially if someone follows up this or I eventually find it important.

So just run the stupid script and hack away, where most of the runtime is grep regex.

Bits and bytes may break your bones, and logic certainly wrekchs your kernel.
And the bits in microprocessor registers are certanly fine, lest you be special about what your doing.

Alas, whther this 224.0.0.0/3 is a problem or not, it is above the paygrade of those that would do damage for profit.