Hi,

I would like to create a shell script that will be called periodically (cron) to download and import lists of blacklisted IPs (e.g. those from iblocklist.com) into iptables' ipsets.

It should do some basic error handling (retry failed downloads) and sanity tests (e.g. check for empty blacklist)

Here's a quick sample:

if [[ "$(curl http://.../blacklist.txt.gz -z blacklist.txt.gz -o blacklist.txt.gz -s -L -w %{http_code})" == "200" ]];
  then
     echo "successful download of new version of blacklist.txt.gz";
     echo "start import blacklist.txt.gz into ipset";
     (for ip in $(zcat blacklist.txt.gz|sed -n '/^[0-9]/p'); do ipset -q add blacklist $ip -exist; done);
  else
     echo "file blacklist.txt.gz unchanged";
fi

Has anyone written such a script that he can share? (a bit of googling turned up some, but they seemed too simplistic).

#!/bin/bash

downloadList()
{
    while [ "$(curl http://.../blacklist.txt.gz -z blacklist.txt.gz -o blacklist.txt.gz -s -L -w %{http_code})" != "200" ]; do
        echo "Blocklist download failed! Trying again in 60 seconds..."
        sleep 60
    done
    echo "Blocklist download successful!"
}

applyList()
{
    local CHANGED=0
    for IP in $(zcat blacklist.txt.gz | sed -n '/^[0-9]/p'); do
        CHANGED=1
        ipset -q add blacklist $IP -exist
    done
    if [ "$CHANGED" -eq 1 ]; then
        echo "New IP addresses added to blacklist!"
    else
        echo "No new IP addresses added to blacklist!"
    fi
}

# kill any existing blocklist downloader PIDs
pkill -f <nameOfYourScriptFileHere>
# download the blocklist
downloadList
# apply the blocklist
applyList

exit 0

Just make sure you have pkill installed (need to add through menuconfig in the busybox settings if you can't find in packages.)

(Last edited by arfett on 27 Feb 2015, 23:12)

Thanks for the contribution, but I was looking for something more "intelligent" and with better error-handling (e.g. 404, empty file, or exceedingly big file, file locking against concurrent execution etc).

Btw in my original code-snippet, I skip importing the new blacklist-file, if the file hasn't changed since last download (i.e. if curl's http code was anything but 200).

PS: I put a timeout period of 3 days for each ipset entry, so older entries will gradually "expire" after a few days, if they have been deleted from the original blacklist. However, the ipset memory size (on OpenWrt) seems to increase over time as IPs get added/deleted, so I was thinking about wiping the ipset every week or so with ipset create / swap / destroy).

PPS: The script I have in mind will be something like this: http://www.tana.it/sw/spamhaus-drop/

(Last edited by kpv on 1 Mar 2015, 20:28)

404
Now might be a good time to learn more about shell scripting then. Just use mine as a base and add your own error checking for various conditions with if/then statements and a minor change to the while loop.

empty file
My script handles empty files just fine since the for loop would have nothing to do and CHANGED would still be 0 meaning it found nothing in the file.

exceedingly big file
Easy to add a check for this after downloading the file you could extract it and check the size.

file locking against concurrent execution
I'm not sure what you would need file locking for since I would assume this would be the only script modifying (downloading) the file and linux does a pretty good job of this by default.

You're not going to find this script ready made anywhere and unless you make it yourself or put in specific requirements no one will make it "intelligent" enough.

(Last edited by arfett on 1 Mar 2015, 22:25)

Well, considering OpenWrt's user-base, the hardware commonly used and the capabilities of the packet-filtering firewall in OpenWrt's base configuration (i.e. without installing some IDS like Snort on OpenWrt, or even attempting to do DPI), I would think that the filtering of ingress/egress traffic against ipsets containing a handful of well-known blacklists (e.g. SpamHaus DROP, EmergingThreats and a couple more from iblocklist.com) would be the most effective way to protect one's network.

There's imho very little variation between user needs (regardless if you're protecting your home network or a business, you'd still want to drop traffic from/to SpamHaus DROP list), which is why such a script would be of benefit to almost every user of OpenWrt and which is why I posted this publicly.

Yes I completely agree. I also think you did not read what I wrote at all otherwise you would see that I said two things:

1. You will not find this script ready made (someone needs to make it whether it is you or someone else)
2. If you don't specify what you're looking for then no one is going to make it the way you want (why don't you start outlining how you think this should work with specifics about the logic...)

I just don't see this as being a one-size-fits all script for everyones' needs although a foundation can be created which could be easily modified to meet different requirements.

(Last edited by arfett on 2 Mar 2015, 18:14)

The request for handling different errors is resolved with the addition of a simple infinite loop with a case switch:

downloadList()
{
    local RESULT
    while true; do
        RESULT="$(curl http://.../blacklist.txt.gz -z blacklist.txt.gz -o blacklist.txt.gz -s -L -w %{http_code})"
        case $RESULT in
            200)
                echo "Blocklist download successful!"
                break
                ;;
            404)
                echo "404 received! Trying again in 60 seconds..."
                sleep 60
                ;;
        esac
    done
}

Now to your other requests. What do you want to happen under what circumstances?

(Last edited by arfett on 2 Mar 2015, 22:38)

My initial idea was to adapt the script from http://www.tana.it/sw/spamhaus-drop/ to OpenWrt (btw I'm not sure which methods of locking against concurrent execution are supported by OpenWrt) and to amend that script by adding a couple more blacklists from iblocklist.com and ET, in addition to the two SpamHaus blacklists (DROP and EDROP). Some people might even want to add country-IP lists.

As I mentioned earlier, I think such filtering against frequently updated blacklists would be very useful to OpenWrt users, as long as they don't choose to run a full-blown IDS like Snort.