banIP support thread

Community Builds, Projects & Packages
757

When 22.03 final comes out I'll look into putting together a work-around for nftables - but you're still going to have to manually do this each time you upgrade the build. The attended sys-upgrade will not work to install banip while the build is marked as broken.

759

Script to work with fw4/nftables. Now with ipv6!

#!/bin/sh

# IP blacklisting script for openwrt nftables

# bogons
# Emerging Threats lists 
# Blocklist.de 
URLS="https://www.team-cymru.org/Services/Bogons/bogon-bn-nonagg.txt http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt https://www.blocklist.de/downloads/export-ips_all.txt"

# create set to contain ip addresses
if ! nft list set inet fw4 blackhole6 > /dev/null 2> /dev/null; then
  nft add set inet fw4 blackhole { flags interval\; type ipv4_addr\; auto-merge\; }
  nft add set inet fw4 blackhole6 { flags interval\; type ipv6_addr\; auto-merge\; }
fi

# insert chain to drop where source address in blackhole set
if ! nft list chain inet fw4 input_wan > /dev/null 2> /dev/null | grep @blackhole6; then
  nft insert rule inet fw4 input_wan ip saddr @blackhole drop
  nft insert rule inet fw4 input_wan ip6 saddr @blackhole6 drop
fi

# temp filename
blocklist=$(mktemp)

# add new elements to the set
for url in $URLS; do
    # download the blocklist
    curl -L -s -k "$url" 2> /dev/null
    echo
done | grep -v ^# | grep -v ^$ > "${blocklist}"

# clear old ipv4
nft flush set inet fw4 blackhole

# create ipv4 block rules
grep -v : "${blocklist}" | while read line; do
  echo add element inet fw4 blackhole { $line }
done | nft -f -

# clear old ipv6
nft flush set inet fw4 blackhole6

# create ipv6 block rules
grep : "${blocklist}" | while read line; do
  echo add element inet fw4 blackhole6 { $line }
done | nft -f -

# cleanup
rm "${blocklist}"
2 Likes
760

Please be so kind and open another thread for your script - thanks!

2 Likes
761

There is tag out, and were more wondering if the hackish steps to get it work with fw4 could be done.
May also be good for working out bugs.

Like this banip nftables file you can find in an openwrt issue, and lines to change.
Various scripts and just adding ipsets to a block list is fine and all.
But a luxury with scripts with regexp that more have looked at.
Such things are plagued with, "just scripts, so not bother sharing".
banIP is closest to a pf-badhost quivalent.

762

Banip is designed around iptables (which is fw3). It injects chains using the iptables command instead of nftables and the formatting is different. You would need to uninstall fw4 and install fw3 in order for it to work on 22.03. Fw3 is still available as legacy software in the repositories at this point in time. I really don't want to make a "how to" for 22.03 in it's "release candidate" state - I have a feeling that as soon as I did the final version would come out and change everything anyways... not to mention - dibdot is working on a new version for fw4 (nftables) which we are all looking forward to. He mentioned it's still a long way off in the future on a previous post.

1 Like
763

Second time i link to it, but here you also find.
banIP.save.nft.gz

Where I assume he has this in /etc/nftables.d/ and slightly modify the commands to work with it in the scripts.

The scripts are also quite LuCI centric, and I have less interest in the web portion of things, but this and testing is generally what is required to get a patch accepted? Have been assuming things along these lines, and not backport the firewall to support a now deprecrated LuCI functionality so to speak.

That is currently marked as broken, where it doesn't break things in any significant way currently I assume. Building with support for packages marked broken, even if it is scripts we are talking about, enables BPF in the kernel for some reason that isn't good. And banIP should be a bit "core" as a firewall.

764

Sorry about this! Not intending to trash post.

But thanks to awk, it takes 24 seconds on arm64 with additional lists where,

nft list set inet fw4 blackhole | wc -l

17694

Bit silly to keep the while loop for ipv4 after that processing pipe.
Everything converted to dotted quad/32, comma seperated and sent to nft that slurps.
So mot things work and populating from scratch is fast. Miss all niceties, but that bit is fine.

2 Likes
765

Hi, can you add an allowed_list.txt int his script?
Thanks.

766

This is the last post in this thread about alternative scripts.
I am tinkering a bit with it, as learn why banIP is taking time.
Quirks with how big lists nft will accept, that most need to be awk.
comm is not available in busybox, and diff is just odd.

I don't really know, or am not comfortable with nft / nftables.
And I am an awful scripter, but don't have a problem with ugly for function over form.

@powtrix you can add to the pleasure and scripty yourself, and look at pf-badhost and banIP.
Also, just change gawk to awk, as that is available in buxybox and not all may have gawk.
I'm tinkering and puzzling with awk and other thing I don't know, and would prefer removing nft flush.
And take the blackhole ipset, compare it to a new iteration and populate add and del commands to update. What you are asking is more into nft land, but by all means, hope my trash post and atrocious script additions encourage others to play with this towards banIP for fw4.

Also get a bug with at least busybox awk with the awk program added as a shell function in the script.
79.110.62.84/31 misses .85/32 (in current blocklists worked with). Where this intermittently breaks parsing existing nft ipset in order to diff --old-line-format=$'delete %l\n' --new-line-format=$'add %l\n' --unchanged-line-format='' or similar to batch process within list length that nft accept.

Tried to find something fast and sane to work around what seems to be certain short comings to make it really robust and simple without awkwards scripts. Haven't tested the awk program with gawk and others, but may or may not have hit a bug with awk in busybox, so unexpected from both program and awk that it's been an avalance of trial and error and possibly getting banned from blocklist.de :stuck_out_tongue:

Maybe another nft issue, and not awk, but what I wanted ended up in a ditch.
And that ripping ip's in largely the same way from the nft set, diffing and doing adding and deleting fast without flushing. Most likely that bothersome ip-range syntax you find in nft list output, because there are actually two problems, and same problem with awk and gawk.

This is not UNIX friendly.

So another nft bug most likely. So my deppest sympathies for banIP development.

nft list set inet fw4 blackhole | grep 85.203.22

  	    85.203.22.76/31, 85.203.22.180,

nft delete element inet fw4 blackhole { 85.203.22.76/32 }

Error: interval not found in set
delete element inet fw4 blackhole { 85.203.22.76/32 }
^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory
delete element inet fw4 blackhole { 85.203.22.76/32 }
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This and the "interval" mannerism, ie. ip-ip, makes this awful and makes you flush.
I was batching in 1000 at a time, less for deletion to work this out.
Arbitrarily being hit by this and ip ranges/intervals, and nft as a tool is not intelligent enough for the operation listed, while grouping without being told and fed /32 excluusively.
So if 1000 ip operations fail, check and fix what rejected it or write c and patch nft?

FYI, less than two minutes for 36 000 ip's, and subsequent updates and chances less than 30 seconds. Starting with 1,4MB uncompressed text of raw input for various sources with overlaps.

For basic useable, ip's should be able to be removed from sets with nft from command line the same way they was entered. Having lots of ipsets doesn't remove the problem, I don't understand why biection and mapping through this grouping logic -- and two of them -- isn there.

767

I can't seem to find the banip and banip-luci packages on 22.03.0
Am I the only one?

EDIT: My bad, sorry for the dumb post

If anyone knows how to make it work on 22.03 please share the knowledge :blush:
EDIT2: or any workaround to block Russia and china using nftables while we wait for the ipban release :pray:

3 Likes
768

Really?

What compelled you to install the latest release in the first place?

What features does it contain that you must have over 21.02.3 that isn’t available to your use case?

Are those features really so compelling that staying with a supported 21.02.3 with working banIP cannot be countenanced until a working banIP on FW4 is realized?

2 Likes
769

LAN speed went from 100mb to 1gbps. Tried blocking Russia and china using the AdBlock package but I still can access rt.com so for now I think I will go back to 21.02.3 and wait.
Cheers.

770

Very off topic (or should be a topic of it own) but WHAT do you mean?

771

I was having the same issue as this user and fixed it by updating

773

Updated the script, right now it doesn't delete /tmp/badhost, and is sizeable with all the lists enabled.

Compressing and keeping for reboot, and adding white listing and doing other things and sharing should be easy with that.

774

@bldur How do I check what IPs have been added to the ruleset? Every nft command I've tried has spat the dummy other than "nft -s list ruleset". Might this be worth adding (as a debug switch perhaps?) to your script.

I'm trying to add the VoIPBL list (http://www.voipbl.org/update) as I'm running Asterisk on my router and it's getting hammered! This list works brilliantly as I've used it when Asterisk was on a VPS.

TIA

775

Not my script per say, it has one for loop for download of lists.

A few commands that do sort, diff or grep regex, and two nested while loops to delete and add ips.

nft list set inet fw4 blackhole | grep -E -o -- '((25[0-5]|(2[0-4]|1{0,1}[[:digit:]]){0,1}[[:digit:]])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[[:digit:]]){0,1}[[:digit:]])(/(3[0-2]|[1-2][[:digit:]]|[1-9]))?'

give's you a list of ip's (may miss ip ranges ip-ip), which can be gotten with:

echo "123.123.123.121-123.123.123.123 123.123.123.121 - 123.123.123.123 123.123.123.123 123.123.123.123" |  grep -E -o "(\b25[0-5]|\b2[0-4][0-9]|\b[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}-(\b25[0-5]|\b2[0-4][0-9]|\b[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}"

regex makes your eyes bleed, one of these is taken from:

and the other from pf-badhost. Regex is very very fast, and makes sure you dont send junk to the kernel and firewall that either crash or compiles or hits a security flaw. tried to put ?.-?. in the middle to match "ip - ip" and "ip-ip" but not that simple, at least wont match 192.168.0.1/24-192.168.0.2/24, and certainly get "ip-ip" if any, from nft ipset.

use "(\b25[0-5]|\b2[0-4][0-9]|\b[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3} to remove /32 after converting /24 to individual ip's. don't think it works on /8 or smaller than /24 if that small, and awk '$1=$1' RS= OFS=", " to change newline to ", " to place the ip list in { ip, ip, ip } to a max of 10 000 at a time until done.

grep -v gives you stuff that doesn't match, and -o only the matches.

Really, at the end of the day it is a bit simple with a fast "kernel" and these loops, and work as a proposal for how to replace the fw3 stuff from banIP.

And funny thing is, sort, diff, regex, diff and awk for the awkward data is faster than what you can expect from any library, for being very old, designed for the tasks involved and having academically best and optimal algorithms.

since what i pushed doesn't delete /tmp/badhost used for tmp files, you find there are diff_add_del_ipv4.XXXXXX file that is result of. diff -N --old-line-format=$'delete %l\n' --new-line-format=$'add %l\n' --unchanged-line-format='' file1 file2. Showin all the lines it wants to add or remove from previous/current nft list set fw4 blackhole ipset.

May not take long, but the ipset explodes.

/tmp/badhost# time nft list set inet fw4 blackhole | grep -E -o -- '((25[0-5]|(2[0-4]|1{0,1}[[:digit:]]){0,1}[[:digit:]]).){3,3}(25[0-5]|(2[0-4]|
1{0,1}[[:digit:]]){0,1}[[:digit:]])(/(3[0-2]|[1-2][[:digit:]]|[1-9]))?' | wc -l
47768

real 0m2.297s
user 0m2.413s
sys 0m0.529s

Be warned, this many IP's leaves behind close to 5MB of data in /tmp/badhost per run of the script if you change nothing and don't clean up. You could perhaps take the diff file and apply it to a tool for a time machine of banned ip's.

776

Hmm, something's not right. I get an empty set with this command running the script from your github a/c

table inet fw4 {
	set blackhole {
		type ipv4_addr
	}
}

The set is being created but not added to.

I'm not familiar with nftables so could you share the code to delete the blackhole sets so that I can start again?

TIA

777

it works here on 22.03, perhaps you are missing command, syntax or just error from the script?

need diff, likely egrep or gnu grep, awk from busy box is fine, assume the rest is there.

or maybe it's wget, you can change this to curl or uclient-fetch (will do next time i puzzle).
it really shouldn't need anything outside of buxybox to run, an inherent goal.
i know nothing about nft really, other than learning about unfriendly quirks of sets.
and the script is really really dumb, spent time on speed and simple.

the way things are done, diff utils is needed, impossible to replace without slowing things down severely. and right now, download and nft add is the slowest parts.

778

Thank you - diffutils package missing (I roll my own 22.03 and have never needed it before)