banIP support thread

Hi,

let me introduce my latest project called "banIP" - a package to block incoming & outgoing ip adresses/subnets via ipset. Screenshots will follow in the second post.

Features:

  • a shell script which uses ipset and iptables to ban a large number of IP addresses published in IP blacklists
  • support blocking by ASN numbers
  • support blocking by iso country codes
  • support local white & blacklist (IPv4, IPv6 & CIDR notation)
  • auto-add unsuccessful ssh login attempts to local blacklist
  • auto-add the uplink subnet to local whitelist
  • per source configuration of SRC (incoming) and DST (outgoing)
  • supports IPv4 & IPv6 (the latter one is completely untested!)
  • 'backup mode' to re-use blocklist backups during startup, get fresh lists via reload or restart action
  • Strong LuCI support:
    ** easy interface to change all aspects of your iptables/ipset configuration on the fly
    ** integrated IPSet-Lookup
    ** integrated RIPE-Lookup
    ** Log-Viewer & online configuration of white- & blacklist

latest snapshot version: banIP 0.3.1 plus luci companion package

Link to the latest banIP documentation

Feel free to test, ask questions or make suggestions.

Have fun!
Dirk


Changelog

---
release 0.3.1

  • the WAN auto detection now supports multiple interfaces, too
  • no longer filter out possible LAN devices
  • add a new DoH (DNS over HTTPS) blocklist source with public DoH DNS server addresses, to effectively block client side DoH communication, e.g. via Firefox or Chrome
    ---
    release 0.3.0
  • new 'ca-bundle' dependency as all https connections are now validated by default
  • automatically select the download utility: 'aria2', 'curl', 'uclient-fetch' with libustream-* or wget are supported
  • track & ban failed LuCI login attempts as well
  • add a small log/banIP background monitor to block SSH/LuCI brute force attacks in realtime (disabled by default)
  • add a config version check (please update your default config!)
  • made the automatic wan detection more stable
  • fix the IPv6 logfile parser
  • fix the service status message
  • update readme
  • update LuCI frontend (separate PR)
    ---
    release 0.2.1
  • fix for #9954
    ---
    release 0.2.0
  • remove 'http-only' mode, all sources are now fetched from https sites
  • the backup mode is now mandatory ('/tmp' is the default backup
    directory), always create and re-use backups if available. To force a re-download take the 'reload' action.
  • support 'sshd' in addition to 'dropbear' for logfile parsing to detect break-in events
  • always update the black-/whitelist with logfile parsing results in 'refresh' mode (no new downloads)
  • rework the return code handling
  • tweak procd trigger
  • various small fixes
  • (s)hellsheck cosmetics
    ---
    release 0.1.5
  • add extra options to control auto-addons to blacklist & whitelist ('ban_autoblacklist' & 'ban_autowhitelist',
    both enabled by default). If disabled auto-addons are only stored temporary in the black/whitelist ipset but not in the list itself, fixes #9631
  • remove old, no longer needed procd workaround
  • remove 'zeus' source from default config (discontinued)
    ---
    release 0.1.4
  • refine 'refresh' mode, add normal processing/download as fallback
  • remove needless reload trigger
  • fix various ipset warnings
  • fix timer in 'refresh' mode
  • adapt ssbl regex to new source list format
    ---
    release 0.1.3
  • change iptables whitelist target from 'ACCEPT' to 'RETURN' to stop traversing the banIP chain and resume at the next chain
  • cosmetics
    ---
    release 0.1.2
  • add more IPv4 & IPv6 consistency checks
  • various cleanups
    ---
    release 0.1.1
  • remove needless sort step to reduce system load
  • change maxqueue default in backend and LuCI frontend to '4' to reduce (default) system load
  • cosmetics
    ---
    release 0.1.0
  • add automatic blocklist backup & restore, they will be used in case of download errors or during startup in backup mode
  • add a 'backup mode' to re-use blocklist backups during startup, get fresh lists via reload or restart action
  • procd interface trigger now supports multiple WAN interfaces
  • change URL for abuse.ch/feodo list source in default config
  • small fixes
  • update readme
    ---
    release 0.0.7
  • determine L3 and L2 network devices to support pppoe interfaces correctly
    ---
    release 0.0.6
  • support multiple WAN interfaces in iptables rules, set 'ban_iface' option accordingly (as space separated list) or use the LuCI frontend
  • add new "refresh" mode while triggered by fw changes (no download)
  • add required ip dependency
  • fix wrong 'settype' definition for firehol1 in config
    ---
    initial github release
  • see commit message
15 Likes

Screenshots from LuCI interface:

  • Overview page with Runtime Infomation (will be updated dynamically via XHR poll)
    banIP_01

  • Currently supported IP blacklists
    banIP_02

  • Extra options (you can change firewall aspects via "Addional Field" select box
    banIP_03

  • IPSet-Lookup for all active IPsets
    banIP_04

  • RIPE-Lookup with many pre-configured reports - output is raw json
    banIP_05

  • Advanced section overview
    banIP_06

1 Like

Can you precise that "current"? Will this be true even in 2 years time from now?
I'm coming from the OpenWrt wiki, where many "current" or "currently" are heavily outdated, years after they have been added and never got removed again...

Notably the LuCI frontend part requires the latest changes in master, which are not in 17.x or 18.x branch.

I've been using firehol's ipset lists for some time, finally someone made a frontend for that. Unfortunately there's no IPv6 here. D:

I'll add your packages to my next build. Thank you.

@dibdot finally a project like sub2rbl but with a LuCI frontend for OpenWRT. Quick question, how is the ram usage with ipsets?
Also, tiny nitpick, it is ransomware, not ransomeware.
Thank you!

On the netfilter mailing list I found this calculation formula: https://www.spinics.net/lists/netfilter/msg56265.html

banIP always tries to start with a very small hashsize. Anyway, the kernel rounds this up to the first (lowest) valid hashsize during ipset creation. Compared to dns level blocking the RAM usage is very low.

1 Like

@dibdot thanks for this, looks really useful.

I installed and all seems to work except blocking by iso country codes, anything I can do to tweak/test?

Also, I seem my subnet has been auto-added to white list, should it be there? Or does that disable banip on my network?

Please provide debug logs - thanks!

That's the intended behaviour to keep your uplink/subnet always accessible.

IPv6 testing/feedback would be fine! :wink:

1 Like

Did some more testing; "auto-add unsuccessful ssh login attempts to local blacklist" this feature seems to be non-functional as well for me.

Regarding "except blocking by iso country codes" in situations where multiple countries are added, the most recently added country, seems to be the only one blocked.

Wed Aug  1 22:20:25 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: country, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 8711/0/8711, time(s): 12
Wed Aug  1 22:20:25 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: country_6, mode: create, settype: net, setipv: inet6, ruletype: src+dst, count(sum/ip/cidr): 1771/0/1771, time(s): 12
Wed Aug  1 22:20:25 2018 user.info banIP-[0.0.1]: 18 IPSets with overall 165102 IPs/Prefixes loaded successfully (Linksys WRT1900ACS, Cantenna_22-06-18 v.1.15- Lede SNAPSHOT r6865+1-419238f)
Wed Aug  1 22:20:25 2018 user.debug banIP-[0.0.1]: f_jsnup ::: status: enabled, setcnt: 18, cnt: 165102
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_jsnup ::: status: running, setcnt: 0, cnt: 0
Wed Aug  1 22:20:46 2018 user.info banIP-[0.0.1]: start banIP processing (start)
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: fetch_util: /usr/bin/curl (built-in), iface: lan, dev: br-lan, mem_total: 511, mem_free: 336, max_queue: 32
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: -, mode: initial, chain: banIP, ruleset: input_wan_rule forwarding_wan_rule input_lan_rule forwarding_lan_rule, ruleset_6: input_wan_rule forwarding_wan_rule input_lan_rule forwarding_lan_rule
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: whitelist, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: whitelist_6, src_on: 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: whitelist_6, mode: flush
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: blacklist, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: blacklist_6, src_on: 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: blacklist_6, mode: flush
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: tor, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: threat, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: debl, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: debl_6, src_on: 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: debl_6, mode: flush
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: blacklist, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 0/0/0, time(s): 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: myip, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: myip_6, src_on: 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: myip_6, mode: flush
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: bogon, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: bogon_6, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: yoyo, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: zeus, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: sslbl, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: ransomeware, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: feodo, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: dshield, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: proxy, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: iblocklist, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: drop, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: drop_6, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: drop_6, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: edrop, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: firehol1, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: whitelist, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 1/0/1, time(s): 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: firehol1, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: firehol2, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: firehol2, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: firehol3, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: firehol3, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: firehol4, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: firehol4, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: country, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: country_6, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: asn, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: asn, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: asn_6, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: asn_6, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: iblocklist, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 0/0/0, time(s): 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: edrop, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 112/0/112, time(s): 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: dshield, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 20/0/20, time(s): 0
Wed Aug  1 22:20:48 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: threat, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 2412/1567/845, time(s): 2
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: drop, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 825/0/825, time(s): 3
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: ransomeware, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 304/304/0, time(s): 3
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: zeus, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 108/108/0, time(s): 3
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: feodo, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 1464/1464/0, time(s): 3
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: proxy, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 2749/2749/0, time(s): 3
Wed Aug  1 22:20:51 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: yoyo, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 11843/11843/0, time(s): 5
Wed Aug  1 22:20:51 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: sslbl, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 99/99/0, time(s): 4
Wed Aug  1 22:20:51 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: tor, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 920/920/0, time(s): 5
Wed Aug  1 22:20:52 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: bogon, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 3239/0/3239, time(s): 6
Wed Aug  1 22:20:52 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: myip, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 3692/3692/0, time(s): 6
Wed Aug  1 22:20:52 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: debl, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 27287/27287/0, time(s): 6
Wed Aug  1 22:20:56 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: country_6, mode: create, settype: net, setipv: inet6, ruletype: src+dst, count(sum/ip/cidr): 1771/0/1771, time(s): 9
Wed Aug  1 22:20:58 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: country, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 8711/0/8711, time(s): 11
Wed Aug  1 22:20:59 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: bogon_6, mode: create, settype: net, setipv: inet6, ruletype: src+dst, count(sum/ip/cidr): 99545/0/99545, time(s): 13
Wed Aug  1 22:20:59 2018 user.info banIP-[0.0.1]: 18 IPSets with overall 165102 IPs/Prefixes loaded successfully (Linksys WRT1900ACS, Cantenna_22-06-18 v.1.15- Lede SNAPSHOT r6865+1-419238f)
Wed Aug  1 22:20:59 2018 user.debug banIP-[0.0.1]: f_jsnup ::: status: enabled, setcnt: 18, cnt: 165102
1 Like

Thanks for testing! :wink:
What's your testcase? Currently only lines like below will be auto-added (whenever you've reached the max-attempts):

Exit before auth (user 'root', 3 fails): Max auth tries reached - user 'root' from 10.168.1.103:53372

Regarding iso codes: which country codes did you use for testing?

Thanks again!

confirmed & fixed with the next update (same bug applied to dynamic ASN IPsets, too) - thanks!

1 Like

(sorry for late reply, was asleep)

I use ssh key+pass dropbear.

Country codes DE, RU, CN

But I see you made progress anyways :wink: Looking forward to you next update.

Could you please send me a logfile/logread excerpt from the intentionally failed login via PM?

Thanks!

@cantenna I've reproduced & fixed that issue, too. I need no further logs ... thanks.

banIP v 0.0.2 is now in my google drive folder (see first post), with the following changes:

  • fix auto-add function of failed ssh logins
  • fix dynamic ASN & Country IPSet creation where multiple sources are selected
  • fix "ransomware" typo
  • updated LuCI components - should work with 18.06 release & latest snapshots

Please remove the old version before you update and reset the LuCI cache (rm -rf /tmp/luci-*)

Happy testing! :wink:

3 Likes

Literally just working on getting that log to you now, came here to double check what you were asking for again. Awesome that you got it done anyways. Will give the update a go, thanks again:)

Thanks for the update. Country block does seem to work but for some reason, occasionally after a reboot, country ipset fails to be known to banip i.e.) Country is missing from IPSet-Lookup. EDIT: increasing/adding trigger delay of 10 seconds seems to have helped

Haven't tested ssh block yet, will do next.

Is it possible to o stuff via console? I know it's early days.

Check the debug log regarding failed downloads ... usually you can fine tune this with a reduced number of parallel processes (you've raised this to 32 if I remember right :wink: ) plus a higher trigger delay (default: 2).

/etc/init.d/banip start

to refresh your IPSets ... or did you mean something different!?

1 Like

That's great. Wasn't sure if restart would be same as refresh, thanks again, really useful package for LEDE