banIP support thread

Hi,

let me introduce my latest project called "banIP" - a package to block incoming & outgoing ip adresses/subnets via ipset. Screenshots will follow in the second post.

Features:

  • a shell script which uses ipset and iptables to ban a large number of IP addresses published in IP blacklists
  • support blocking by ASN numbers
  • support blocking by iso country codes
  • support local white & blacklist (IPv4, IPv6 & CIDR notation)
  • auto-add unsuccessful ssh login attempts to local blacklist
  • auto-add the uplink subnet to local whitelist
  • black- and whitelist supporting domain names as well, they will be resolved in a detached background process and added to the IPsets
  • supports a 'whitelist only' mode, this option allows to restrict Internet access from/to a small number of secure websites/IPs
  • per source configuration of SRC (incoming) and DST (outgoing)
  • supports IPv4 & IPv6
  • re-use blocklist backups during startup, get fresh lists via reload or restart action
  • small monitor to block suspicious login attempts (supports ssh, LuCI und NGINX)
  • report engine to get detailed IPset statistics
  • MAC whitelisting to exclude certain clients from blocking
  • supports email notifications, e.g. with report data before reloading your IPSets
  • strong LuCI support: easy interface to change all aspects of your banIP configuration on the fly

stable OpenWrt version 21.02.x: banIP 0.7.10 plus luci companion package

latest snapshot version: banIP 0.7.10 plus luci companion package

Link to the latest banIP documentation

Feel free to test, ask questions or make suggestions.

Have fun!
Dirk


Changelog

---
release 0.7.10

  • Updated firehol ipset URLs
  • optimize dns resolve function
  • cosmetics
  • switch to unencrypted http downloads for ipdeny.com due to persistent certificate issues
  • compact json generator code (tested with report files > 2MB)
  • various code cleanups and optimizations
    ---
    release 0.7.9
  • add switch 'ban_fetchinsecure' to allow insecure downloads without certificate check (disabled by default)
  • better explain 'ban_fetchparm' in readme
    ---
    release 0.7.8
  • fix pid file processing of the background monitor plus child processes (bug reported in the forum)
  • made the enabled/disabled switch of the background monitor functional
    ---
    release 0.7.7-2
  • fix whitelist housekeeping if you switch between normal- and 'whitelist only' mode
  • add a "whitelist only" mode, this option allows to restrict Internet access from/to a small number of secure websites/IPs, and block access from/to the rest of the Internet.
    ---
    release 0.7.6
  • rework the central iptables function to significantly reduce the code complexity and the overall number of iptables calls
  • check early and only once in the chain for ctstate NEW and return otherwise (thanks @ldir-EDB0)
  • made the whitelist ordering within the chain more flexible
    ---
    release 0.7.5-4
  • fix another IPv4/IPv6 related iptables chain creation problem
  • fix counter during ipset creation
  • fix regex for debug counters
  • fix ipset housekeeping for local sources
    ---
    release 0.7.5-3
  • fix iptables/chain creation in setups without IPv6 support
  • refine the new dns resolving process
  • add a caching mechanism for the resolved IPs, the detached name lookup takes place only during 'restart' or 'reload' action, 'start' and 'refresh' actions are using an auto-generated backup instead
  • update the readme
  • black- and whitelist now supporting domain names as well - the corresponding IPs (IPv4 & IPv6) will be resolved in a detached background process and added to the IPsets
    ---
    release 0.7.3
  • fix search string/pipe preparation for the background service
  • fix IPSet maxelem limitation, made it more flexible
  • fix potential error during resume action
  • add Cisco Talos IP blacklist
  • update readme
    ---
    release 0.7.2
  • add scanning for suspicious nginx events
  • add a log counter to track the number of the failed requests or login repetitions of the same ip in the log before banning, defaults are: ssh (3), luci (3), nginx (5)
  • optimize the background service handling
  • add 'greensnow' as a new source
  • update readme and LuCI frontend regarding the new log count options
    ---
    release 0.7.1
  • add 'ban_extrasources' to handle banIP-unrelated sets for reporting and queries
  • add set timeouts for local sources (maclist, whitelist, blacklist)
  • expose 'ban_extrasources', 'ban_localsources' and the new set timeouts to LuCI
    ---
    release 0.7.0
  • major rewrite
  • add support for multiple chains
  • add mac whitelisting
  • add support for multiple ssh daemons in parallel
  • add an ipset report engine
  • add mail notifications
  • add suspend/resume functions
  • add a cron wrapper to set an ipset related auto-timer for automatic blocklist updates
  • add a list wrapper to add/remove blocklist sources
  • add 19.x and Turris OS 5.x compatibility code
  • sources stored in an external compressed json file (/etc/banip/banip.sources.gz)
  • change Country/ASN download sources (faster/more reliable)
  • fix DHCPv6/icmpv6 issues
Ancient Releases (Unsupported!)

OpenWrt version 19.07: banIP 0.3.11 plus luci companion package

27 Likes

Screenshots from LuCI interface:

  • Overview page with Runtime Infomation, config tabs and list selections

  • Reporting page with detailed IPSet information, direct whitelisting and/or bgview IP information (ext. link)

3 Likes

Can you precise that "current"? Will this be true even in 2 years time from now?
I'm coming from the OpenWrt wiki, where many "current" or "currently" are heavily outdated, years after they have been added and never got removed again...

1 Like

Notably the LuCI frontend part requires the latest changes in master, which are not in 17.x or 18.x branch.

1 Like

I've been using firehol's ipset lists for some time, finally someone made a frontend for that. Unfortunately there's no IPv6 here. D:

I'll add your packages to my next build. Thank you.

1 Like

@dibdot finally a project like sub2rbl but with a LuCI frontend for OpenWRT. Quick question, how is the ram usage with ipsets?
Also, tiny nitpick, it is ransomware, not ransomeware.
Thank you!

1 Like

On the netfilter mailing list I found this calculation formula: https://www.spinics.net/lists/netfilter/msg56265.html

banIP always tries to start with a very small hashsize. Anyway, the kernel rounds this up to the first (lowest) valid hashsize during ipset creation. Compared to dns level blocking the RAM usage is very low.

3 Likes

@dibdot thanks for this, looks really useful.

I installed and all seems to work except blocking by iso country codes, anything I can do to tweak/test?

Also, I seem my subnet has been auto-added to white list, should it be there? Or does that disable banip on my network?

1 Like

Please provide debug logs - thanks!

That's the intended behaviour to keep your uplink/subnet always accessible.

IPv6 testing/feedback would be fine! :wink:

2 Likes

Did some more testing; "auto-add unsuccessful ssh login attempts to local blacklist" this feature seems to be non-functional as well for me.

Regarding "except blocking by iso country codes" in situations where multiple countries are added, the most recently added country, seems to be the only one blocked.

Wed Aug  1 22:20:25 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: country, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 8711/0/8711, time(s): 12
Wed Aug  1 22:20:25 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: country_6, mode: create, settype: net, setipv: inet6, ruletype: src+dst, count(sum/ip/cidr): 1771/0/1771, time(s): 12
Wed Aug  1 22:20:25 2018 user.info banIP-[0.0.1]: 18 IPSets with overall 165102 IPs/Prefixes loaded successfully (Linksys WRT1900ACS, Cantenna_22-06-18 v.1.15- Lede SNAPSHOT r6865+1-419238f)
Wed Aug  1 22:20:25 2018 user.debug banIP-[0.0.1]: f_jsnup ::: status: enabled, setcnt: 18, cnt: 165102
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_jsnup ::: status: running, setcnt: 0, cnt: 0
Wed Aug  1 22:20:46 2018 user.info banIP-[0.0.1]: start banIP processing (start)
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: fetch_util: /usr/bin/curl (built-in), iface: lan, dev: br-lan, mem_total: 511, mem_free: 336, max_queue: 32
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: -, mode: initial, chain: banIP, ruleset: input_wan_rule forwarding_wan_rule input_lan_rule forwarding_lan_rule, ruleset_6: input_wan_rule forwarding_wan_rule input_lan_rule forwarding_lan_rule
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: whitelist, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: whitelist_6, src_on: 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: whitelist_6, mode: flush
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: blacklist, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: blacklist_6, src_on: 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: blacklist_6, mode: flush
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: tor, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: threat, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: debl, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: debl_6, src_on: 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: debl_6, mode: flush
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: blacklist, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 0/0/0, time(s): 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: myip, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: myip_6, src_on: 0
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: myip_6, mode: flush
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: bogon, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: bogon_6, src_on: 1
Wed Aug  1 22:20:46 2018 user.debug banIP-[0.0.1]: f_main  ::: name: yoyo, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: zeus, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: sslbl, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: ransomeware, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: feodo, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: dshield, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: proxy, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: iblocklist, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: drop, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: drop_6, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: drop_6, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: edrop, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: firehol1, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: whitelist, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 1/0/1, time(s): 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: firehol1, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: firehol2, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: firehol2, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: firehol3, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: firehol3, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: firehol4, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: firehol4, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: country, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: country_6, src_on: 1
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: asn, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: asn, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_main  ::: name: asn_6, src_on: 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: asn_6, mode: flush
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: iblocklist, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 0/0/0, time(s): 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: edrop, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 112/0/112, time(s): 0
Wed Aug  1 22:20:47 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: dshield, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 20/0/20, time(s): 0
Wed Aug  1 22:20:48 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: threat, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 2412/1567/845, time(s): 2
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: drop, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 825/0/825, time(s): 3
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: ransomeware, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 304/304/0, time(s): 3
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: zeus, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 108/108/0, time(s): 3
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: feodo, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 1464/1464/0, time(s): 3
Wed Aug  1 22:20:50 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: proxy, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 2749/2749/0, time(s): 3
Wed Aug  1 22:20:51 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: yoyo, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 11843/11843/0, time(s): 5
Wed Aug  1 22:20:51 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: sslbl, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 99/99/0, time(s): 4
Wed Aug  1 22:20:51 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: tor, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 920/920/0, time(s): 5
Wed Aug  1 22:20:52 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: bogon, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 3239/0/3239, time(s): 6
Wed Aug  1 22:20:52 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: myip, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 3692/3692/0, time(s): 6
Wed Aug  1 22:20:52 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: debl, mode: create, settype: ip, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 27287/27287/0, time(s): 6
Wed Aug  1 22:20:56 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: country_6, mode: create, settype: net, setipv: inet6, ruletype: src+dst, count(sum/ip/cidr): 1771/0/1771, time(s): 9
Wed Aug  1 22:20:58 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: country, mode: create, settype: net, setipv: inet, ruletype: src+dst, count(sum/ip/cidr): 8711/0/8711, time(s): 11
Wed Aug  1 22:20:59 2018 user.debug banIP-[0.0.1]: f_ipset ::: name: bogon_6, mode: create, settype: net, setipv: inet6, ruletype: src+dst, count(sum/ip/cidr): 99545/0/99545, time(s): 13
Wed Aug  1 22:20:59 2018 user.info banIP-[0.0.1]: 18 IPSets with overall 165102 IPs/Prefixes loaded successfully (Linksys WRT1900ACS, Cantenna_22-06-18 v.1.15- Lede SNAPSHOT r6865+1-419238f)
Wed Aug  1 22:20:59 2018 user.debug banIP-[0.0.1]: f_jsnup ::: status: enabled, setcnt: 18, cnt: 165102
2 Likes

Thanks for testing! :wink:
What's your testcase? Currently only lines like below will be auto-added (whenever you've reached the max-attempts):

Exit before auth (user 'root', 3 fails): Max auth tries reached - user 'root' from 10.168.1.103:53372

Regarding iso codes: which country codes did you use for testing?

Thanks again!

1 Like

confirmed & fixed with the next update (same bug applied to dynamic ASN IPsets, too) - thanks!

2 Likes

(sorry for late reply, was asleep)

I use ssh key+pass dropbear.

Country codes DE, RU, CN

But I see you made progress anyways :wink: Looking forward to you next update.

1 Like

Could you please send me a logfile/logread excerpt from the intentionally failed login via PM?

Thanks!

1 Like

@cantenna I've reproduced & fixed that issue, too. I need no further logs ... thanks.

1 Like

banIP v 0.0.2 is now in my google drive folder (see first post), with the following changes:

  • fix auto-add function of failed ssh logins
  • fix dynamic ASN & Country IPSet creation where multiple sources are selected
  • fix "ransomware" typo
  • updated LuCI components - should work with 18.06 release & latest snapshots

Please remove the old version before you update and reset the LuCI cache (rm -rf /tmp/luci-*)

Happy testing! :wink:

4 Likes

Literally just working on getting that log to you now, came here to double check what you were asking for again. Awesome that you got it done anyways. Will give the update a go, thanks again:)

1 Like

Thanks for the update. Country block does seem to work but for some reason, occasionally after a reboot, country ipset fails to be known to banip i.e.) Country is missing from IPSet-Lookup. EDIT: increasing/adding trigger delay of 10 seconds seems to have helped

Haven't tested ssh block yet, will do next.

Is it possible to o stuff via console? I know it's early days.

1 Like

Check the debug log regarding failed downloads ... usually you can fine tune this with a reduced number of parallel processes (you've raised this to 32 if I remember right :wink: ) plus a higher trigger delay (default: 2).

/etc/init.d/banip start

to refresh your IPSets ... or did you mean something different!?

1 Like

That's great. Wasn't sure if restart would be same as refresh, thanks again, really useful package for LEDE