[Solved] 22.03.0: workaround for running CrowdSec bouncer

pigeon35 · 2022-12-15T17:02:17.536Z

OpenWRT doesn't default to output syslog to file, so how does the crowdsec package scan for threats beyond importing IP bans from rest of world?

Default setup has this:

cscli metrics

INFO[15-12-2022 04:55:05 PM] Local Api Metrics:                           
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | GET    |    7 |
| /v1/decisions/stream | GET    |  625 |
| /v1/watchers/login   | POST   |   10 |
+----------------------+--------+------+
INFO[15-12-2022 04:55:05 PM] Local Api Machines Metrics:                  
+--------------------------------------------------+------------+--------+------+
|                     MACHINE                      |   ROUTE    | METHOD | HITS |
+--------------------------------------------------+------------+--------+------+
| d26ffdf46b234e47a83324358a9aa308saAtnf4uLr27aJ3P | /v1/alerts | GET    |    7 |
+--------------------------------------------------+------------+--------+------+
INFO[15-12-2022 04:55:05 PM] Local Api Bouncers Metrics:                  
+------------------------------------+----------------------+--------+------+
|              BOUNCER               |        ROUTE         | METHOD | HITS |
+------------------------------------+----------------------+--------+------+
| crowdsec-firewall-bouncer-QUBnPT2q | /v1/decisions/stream | GET    |  625 |
+------------------------------------+----------------------+--------+------+

So I changed OpenWRT config to output syslog to a file. Now i get:

cscli metrics
INFO[15-12-2022 04:55:05 PM] Acquisition Metrics:                         
+----------------------+------------+--------------+----------------+------------------------+
|        SOURCE        | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+----------------------+------------+--------------+----------------+------------------------+
| file:/var/log/syslog |       1251 | -            |           1251 | -                      |
+----------------------+------------+--------------+----------------+------------------------+
INFO[15-12-2022 04:55:05 PM] Parser Metrics:                              
+---------------------------------+------+--------+----------+
|             PARSERS             | HITS | PARSED | UNPARSED |
+---------------------------------+------+--------+----------+
| child-crowdsecurity/syslog-logs | 2502 | -      |     2502 |
| crowdsecurity/syslog-logs       | 1251 | -      |     1251 |
+---------------------------------+------+--------+----------+
INFO[15-12-2022 04:55:05 PM] Local Api Metrics:                           
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | GET    |    7 |
| /v1/decisions/stream | GET    |  625 |
| /v1/watchers/login   | POST   |   10 |
+----------------------+--------+------+
INFO[15-12-2022 04:55:05 PM] Local Api Machines Metrics:                  
+--------------------------------------------------+------------+--------+------+
|                     MACHINE                      |   ROUTE    | METHOD | HITS |
+--------------------------------------------------+------------+--------+------+
| d26ffdf46b234e47a83324358a9aa308saAtnf4uLr27aJ3P | /v1/alerts | GET    |    7 |
+--------------------------------------------------+------------+--------+------+
INFO[15-12-2022 04:55:05 PM] Local Api Bouncers Metrics:                  
+------------------------------------+----------------------+--------+------+
|              BOUNCER               |        ROUTE         | METHOD | HITS |
+------------------------------------+----------------------+--------+------+
| crowdsec-firewall-bouncer-QUBnPT2q | /v1/decisions/stream | GET    |  625 |
+------------------------------------+----------------------+--------+------+

Which makes "more sense" in that i can see syslog being read.
In terms of alerts, there are none (only the community blocklist)
Zero "Lines Parsed" from syslog:


root@OpenWrt:/etc/crowdsec# cscli alerts list
+----+-----------------------------------+------------------------+---------+----+-----------+-------------------------------+
| ID |               VALUE               |         REASON         | COUNTRY | AS | DECISIONS |          CREATED AT           |
+----+-----------------------------------+------------------------+---------+----+-----------+-------------------------------+
|  8 | crowdsecurity/community-blocklist | update : +32/-0 IPs    |         |    | ban:32    | 2022-12-15 15:27:20 +0000 UTC |
|  7 | crowdsecurity/community-blocklist | update : +15000/-0 IPs |         |    | ban:14981 | 2022-12-15 12:30:35 +0000 UTC |
|  6 | crowdsecurity/community-blocklist | update : +15000/-0 IPs |         |    | ban:53    | 2022-12-15 10:25:22 +0000 UTC |
|  5 | crowdsecurity/community-blocklist | update : +15000/-0 IPs |         |    | ban:348   | 2022-12-14 21:29:08 +0000 UTC |
|  4 | crowdsecurity/community-blocklist | update : +15000/-0 IPs |         |    | ban:61    | 2022-12-14 19:29:08 +0000 UTC |
|  3 | crowdsecurity/community-blocklist | update : +15000/-0 IPs |         |    | ban:58    | 2022-12-14 17:29:08 +0000 UTC |
|  2 | crowdsecurity/community-blocklist | update : +15000/-0 IPs |         |    | ban:6     | 2022-12-14 15:27:19 +0000 UTC |
|  1 | crowdsecurity/community-blocklist | update : +15000/-0 IPs |         |    | ban:12071 | 2022-12-14 13:24:21 +0000 UTC |
+----+-----------------------------------+------------------------+---------+----+-----------+-------------------------------+

ne20002 · 2022-12-15T17:35:40.611Z

As the title of this thread implies: this is a workaround to get the bouncer running on OpenWrt.

With my WRT3200ACM I don't have more space to install the crowdsec cscli package. It is also not needed for the bouncer. The bouncer just blocks the ips it retrieves from the local Crowdsec API.

As long as someone does not run some services that need protection on the OpenWrt device itself, the bouncer only is sufficiant. The usual setup is to have a local API running somewhere in the internal network plus a (few) agent(s) runnign on the services to protect. Those send decissions to the local API from where the bouncer retrieves this list.

Also: the router shall be protected by firewall rules to not accept connections from outside to the router itself. In this case, there is no need to protect the router with the Crowdsec bouncer. That's why the rules I posted are hooking into forward chain, protected the services inside the network by blocking the malicious connection attemps on the router already.

pigeon35 · 2022-12-15T18:49:07.634Z

I'm hoping to run Exim+Dovecot directly from the router, have setup the DDNS and within 24 hours can see attempts on port 22

What would I need to look at changing please? Change the hook to input?

ne20002 · 2022-12-15T20:12:01.120Z

Yes, if you run your services on the Openwrt device you need to protect the input hook.

I've set up my own mail server with postfix and dovecot on a device in dmz. Setting up and securing a mail server is some work to do.

pigeon35 · 2022-12-15T22:19:01.847Z

Interesting, was this using the openWRT packages? I ask because Postfix didn't look to be updated compared to Exim packages.

I will admit, had no idea setting up email server would be so tricky. So far I've got about 2 pages of A4 notes and when it started to dawn on me that mistakes will lead to big problems, thats when I started looking at trying to add Crowdsec.

Speaking of Crowdsec. I managed to use the 22.03.2 OpenWRT SDK and compile Crowdsec 1.4.3 (x86_64).
For what its worth, it does work with the cs-firewall-bouncer 0.0.21 that we've installed from openWRT 21.02.3 onto 22.03.2

But I could not repeat the same trick in terms of SDK compiling of cs-firewall-bouncer 0.0.24-rc1. It doesnt show up in make menuconfig and simply editing the .config and Makefiles doesnt get an ipk.
Going to park that one for a bit and come back to it given 1.4.3 is working with 0.0.21 anyway.

EDIT:

Just to confirm, I changed /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
from set-only true to false

nftables:
  ipv4:
    enabled: true
    set-only: false
    table: crowdsec
    chain: crowdsec-chain
  ipv6:
    enabled: true
    set-only: false
    table: crowdsec6
    chain: crowdsec6-chain

EDIT2:

On reboot, unfortunately the nftables are not populated with the IP bans. And cat /var/log/crowdsec-firewall-bouncer.log reports error "auth with api key failed" which apparently is due to bouncer starting before crowdsec (this happened on 1.3.0 too) . So I added "sleep 2" to /etc/firewall.cs (see below). A bit bodgy......but works

Errors
time="15-12-2022 22:37:00" level=info msg="backend type : nftables"
time="15-12-2022 22:37:00" level=info msg="nftables initiated"
time="15-12-2022 22:37:00" level=info msg="Processing new and deleted decisions . . ."
time="15-12-2022 22:37:00" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp [::1]:8080: connect: connection refused"
time="15-12-2022 22:37:00" level=fatal msg="Get \"http://localhost:8080/v1/decisions/stream?startup=true\": dial tcp [::1]:8080: connect: connection refused"
time="15-12-2022 22:37:02" level=info msg="backend type : nftables"
time="15-12-2022 22:37:02" level=info msg="nftables initiated"
time="15-12-2022 22:37:02" level=info msg="Processing new and deleted decisions . . ."
time="15-12-2022 22:37:02" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp [::1]:8080: connect: connection refused"
time="15-12-2022 22:37:02" level=fatal msg="Get \"http://localhost:8080/v1/decisions/stream?startup=true\": dial tcp [::1]:8080: connect: connection refused"

Weirdly on re-boot, there are repeats of the same rules. I did read earlier in this thread someone else had same issue, solved by the modified /etc/firewall.cs with rule deletions; more bodges

Duplicate rules showing:

nft list table crowdsec
table ip crowdsec {
	set crowdsec-blacklists {
		type ipv4_addr
		flags timeout
	}

	chain crowdsec-chain {
		type filter hook input priority filter; policy accept;
		ip saddr @crowdsec-blacklists drop
		ip saddr @crowdsec-blacklists drop
		ip saddr @crowdsec-blacklists drop
		ip saddr @crowdsec-blacklists drop
	}
}

My /etc/firewall.cs now looks like this

#!/bin/sh
#/etc/init.d/crowdsec enabled && /etc/init.d/crowdsec restart
#/etc/init.d/crowdsec-firewall-bouncer enabled && /etc/init.d/crowdsec-firewall-bouncer restart
#nft delete chain ip crowdsec crowdsec-chain
#nft delete chain ip6 crowdsec6 crowdsec6-chain
#nft add chain ip crowdsec crowdsec-chain '{ type filter hook forward priority 4; policy accept; }'
#nft add rule ip crowdsec crowdsec-chain iifname { usb0, eth4 } ct state new ip saddr @crowdsec-blacklists log prefix \"crowdsec: \" counter drop
#nft add chain ip6 crowdsec6 crowdsec6-chain '{ type filter hook forward priority 4; policy accept; }'
#nft add rule ip6 crowdsec6 crowdsec6-chain iifname { usb0, eth4 } ct state new ip6 saddr @crowdsec6-blacklists log prefix \"crowdsec: \" counter drop
#service crowdsec reload
sleep 2
service crowdsec-firewall-bouncer reload
sleep 2
nft delete rule crowdsec crowdsec-chain handle 6
nft delete rule crowdsec crowdsec-chain handle 5
nft delete rule crowdsec crowdsec-chain handle 4
nft delete rule ip6 crowdsec6 crowdsec6-chain handle 6
nft delete rule ip6 crowdsec6 crowdsec6-chain handle 5
nft delete rule ip6 crowdsec6 crowdsec6-chain handle 4
exit 0

So now on bootup

cat /var/log/crowdsec-firewall-bouncer.log shows

time="15-12-2022 23:08:25" level=info msg="backend type : nftables"
time="15-12-2022 23:08:25" level=info msg="nftables initiated"
time="15-12-2022 23:08:25" level=info msg="Processing new and deleted decisions . . ."
time="15-12-2022 23:09:38" level=info msg="'1352' decisions deleted"
time="15-12-2022 23:09:39" level=info msg="'26261' decisions added"

And nft list table crowdsec has data + no duplicate rule.

			     223.197.145.33 timeout 2d17h15m55s140ms expires 2d17h1s10ms, 223.197.151.55 timeout 4d18h15m55s130ms expires 4d18h1s200ms,
			     223.197.153.135 timeout 4d10h22m9s130ms expires 4d10h6m15s140ms, 223.197.175.91 timeout 5d12h15m55s120ms expires 5d12h1s490ms,
			     223.197.186.7 timeout 5d3h15m55s130ms expires 5d3h1s280ms, 223.197.188.206 timeout 5d12h15m55s130ms expires 5d12h1s410ms,
			     223.197.208.186 timeout 4d19h22m9s130ms expires 4d19h6m15s230ms, 223.197.220.67 timeout 6d10h22m9s120ms expires 6d10h6m15s710ms,
			     223.240.83.206 timeout 5d12h15m55s130ms expires 5d12h1s460ms, 223.240.96.1 timeout 5d12h15m55s130ms expires 5d12h1s470ms,
			     223.241.222.151 timeout 13h22m9s140ms expires 13h6m14s870ms, 223.245.0.5 timeout 5d12h15m55s120ms expires 5d12h1s460ms }
	}

	chain crowdsec-chain {
		type filter hook input priority filter; policy accept;
		ip saddr @crowdsec-blacklists drop
	}
}
root@OpenWrt:~#

lopez67876 · 2022-12-17T14:38:01.621Z

I have found crowdsec 1.4.3 for armv7 and is working good on my Linksys WRT1900ACS

image1526×847 25.1 KB

I just replaced my binary files crowdsec and crowdsec-cli with the ones in here: crowdsec-1.4.3-r1.apk.

I could not work out the same for cs-firewall-bouncer-0.0.24-r3.apk so I stay on "crowdsec-firewall-bouncer_0.0.21-3_arm_cortex-a9_vfpv3-d16".

norbertjoni · 2023-01-24T09:04:43.069Z

I am lazy I know, but there is any plugin workaround for crowdsec?

ne20002 · 2023-01-25T16:57:36.027Z

I'm currently working on a new package for the Crowdsec-firewall-bouncer with uci configuration file and will afterwards try to set up a luci package for configuration as well.
No date yet.

norbertjoni · 2023-01-27T15:17:26.125Z

There is a trick to installing cloudflare bouncer to OpenWRT? I can't found in the repo.

docs.crowdsec.net

Cloudflare Bouncer | CrowdSec

📚 Documentation

ne20002 · 2023-01-27T16:43:13.648Z

I'm making progess so this is a call for testers.

What I have achieved so far: I have a crowdesc-firewall-bouncer package that is working with OpenWrt 22.03. It uses a uci config file which at the moment looks like this:

config bouncer
        option api_url 'http://cs-lapi:8080/'
        option api_key '*******************'
        option update_frequency '10s'
        option deny_action 'drop'
        option log_prefix 'crowdsec: '
        option ipv4 '1'
        option filter_input '1'
        option filter_forward '1'
        option input_chain_name 'crowdsec-chain'
        option forward_chain_name 'forward'
        option input6_chain_name 'crowdsec6-chain'
        option forward6_chain_name 'forward'
        list interface 'eth1'
        list interface 'wg1'
        option enabled '1'
        option deny_log '0'
        option ipv6 '1'

I also have a first Luci package available:

grafik1077×761 65.6 KB

I'm currently working on a AVM Fritzbox 4040 so the target is ipq40xx/generic.

lopez67876 · 2023-01-27T19:21:32.097Z

It looks nice

ne20002 · 2023-01-30T08:28:10.809Z

I have finished work so far and an updated package works on my router. At the moment I tweak the old package which makes developing and testing much easier.
I will start to update the Makefile and then create a pull request. Let's see how this works ...

lopez67876 · 2023-01-30T08:48:37.025Z

Thanks for your great work
Looking forwad for testing it on my wrt1900acsv2.

norbertjoni · 2023-01-30T17:34:58.708Z

ne20002:

target is ipq40xx/generic.

Sorry for my ignorance, but this will work with my rpi4?

ne20002 · 2023-01-31T07:10:07.599Z

I have opened the pull requests. Unfortunately it fails build checks with docker setup. I need to see where I can get help for this.
But when build and when/if accepted to OpenWrt it will be available for all tagets including Raspberry Pi.

ne20002 · 2023-02-04T09:50:22.410Z

My pull request for the bouncer has been merged. The PR for the LuCi app is still open.

The bouncer package shall appear shortly. It will install the bouncer without enabling it and thus no firewall rules will be applied.

The default config file is /etc/config/crowdsec:

config bouncer
	option enabled '0'
	option ipv4 '1'
	option ipv6 '1'
	option api_url 'http://localhost:8080/'
	option api_key ''
	option update_frequency '10s'
	option deny_action 'drop'
	option deny_log '0'
	option log_prefix 'crowdsec: '
	option log_level 'info'
	option filter_input '1'
	option filter_forward '1'
	list interface 'eth1'

You need to update the api_url and api_key and depending on your setup the interface list option as well. Afterwards you can restart the Crowdsec bouncer in System -> Startup.
As soon as the LuCi app is available, the configuration and start/restart (be enabling/disabling) can be done from the UI.

lopez67876 · 2023-02-04T11:24:58.011Z

Great job
Thank you for sharing your work.

image1826×405 88.2 KB

ne20002 · 2023-02-06T09:08:54.225Z

I hope it is working ... (my first contribution with building from go sources).

I hope I get the LuCI app also approved and merged (and get both packages cherry picked to 22.03. branch as well).
Also to mention, the documentation needs to be updated for the bouncer. Maybe @gandalf can help here.

After updating the bcp38 filter and the CrowdSec bouncer I will now work on a new filter based on dnsmasq resolving (like the FritzBox has it and I like it). I need to do my annual tax declaration first so this might take some time.

lopez67876 · 2023-02-06T10:43:28.314Z

Great
This afternoon I'll start the testing to see if I can get it up and running (I am an expert in copy/paste and little else ).
Thanks again and we'll be on the lookout for that filter

ne20002 · 2023-02-11T10:18:46.468Z

So, did it work?