Crowdsec: initial packages v1.2.0 for OpenWrt

Héllo all,

Here is a PR for CrowdSec v1.1.1 and CrowdSec-FireWall-Bouncer v0.0.13

I still get an issue with ARM compile and buildroot…
See https://github.com/openwrt/packages/issues/16193

Is there anyone who can help to fix this, please ?

UPDATE : now fixed and working !

How can I enable (force) a CONFIG value from a Package Makefile ?
Is it only possible to ?
I need the toolchain (binutils) to being built with EXTRA as :

CONFIG_EXTRA_BINUTILS_CONFIG_OPTIONS="--enable-gold --enable-plugins"

UPDATE : now unnecessary and fixed

Edit : made a PR to enable gold linker in binutils toolchain -> https://github.com/openwrt/openwrt/pull/4412

UPDATE : now unnecessary and fixed

packages available now for testing also for mipsel_24kc ( like Device: Xiaomi R3P (mt7621)... )
-> https://github.com/erdoukki/crowdsec-openwrt/tree/master/package/custom

UPDATE : now more up to date...

Now CrowdSec v1.2.0 and CrowdSec-FireWall-Bouncer v0.0.15

You can find the packages for testing at the Artifacts section from the build check of the PR section https://github.com/openwrt/packages/actions/runs/1265111620

HowTo (short / quick / dirty / temporary) :
Download package for your architecture and install with :

opkg install crowdsec*.ipk

verify :

root@ULTRA-5G:~/custom/1.2.0# service crowdsec status
running
root@ULTRA-5G:~/custom/1.2.0# service crowdsec-firewall-bouncer status
running

check :

root@ULTRA-5G:~/custom/1.2.0# cscli metrics
INFO[20-09-2021 07:58:05 AM] Local Api Metrics:                           
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
+----------------------+------+---+
| /v1/decisions/stream | GET  | 1 |
+======================+======+===+
| /v1/watchers/login   | POST | 1 |
+----------------------+------+---+
+----------------------+--------+------+
INFO[20-09-2021 07:58:05 AM] Local Api Bouncers Metrics:                  
+------------------------------+----------------------+--------+------+
|           BOUNCER            |        ROUTE         | METHOD | HITS |
+------------------------------+----------------------+--------+------+
| cs-firewall-bouncer-1AZW62x2 | /v1/decisions/stream | GET    |    1 |
+------------------------------+----------------------+--------+------+

test :

root@LPM:~# cscli alerts list
+-----+------------------------------+-----------------------------------+---------+--------------------------------+-----------+--------------------------------+
| ID  |            VALUE             |              REASON               | COUNTRY |               AS               | DECISIONS |           CREATED AT           |
+-----+------------------------------+-----------------------------------+---------+--------------------------------+-----------+--------------------------------+
| 871 | Ip:143.198.56.171            | crowdsecurity/http-probing        | US      |  MCI Communications Services,  | ban:1     | 2021-10-02 07:27:23.57835364   |
|     |                              |                                   |         | Inc. d/b/a Verizon Business    |           | +0200 +0200                    |
| 870 | crowdsec/community-blocklist | update : +494/-0 IPs              |         |                                | ban:494   | 2021-10-02 07:18:38 +0200      |
|     |                              |                                   |         |                                |           | +0200                          |
| 869 | crowdsec/community-blocklist | update : +496/-0 IPs              |         |                                | ban:4     | 2021-10-02 05:18:38 +0200      |
|     |                              |                                   |         |                                |           | +0200                          |
...

And feel free to report here about OpenWrt packages, or ask directly at CrowdSec community https://discourse.crowdsec.net/ for specific usage and recommendation !

crowdsec: initial package v1.2.0 -> https://github.com/openwrt/packages/pull/16244
crowdsec-firewall-bouncer: initial package v0.0.15 -> https://github.com/openwrt/packages/pull/16844

Merged yesterday in master !

1 Like

The PR was merged this morning and packages will be available in few days for 21.02 !

:partying_face:

1 Like

installing owns console is this expected behavior?

console-hangs
dca632 /usbstick 54°# opkg install crowdsec
Installing crowdsec (1.2.0-1) to root...
Downloading http://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a72/packages/crowdsec_1.2.0-1_aarch64_cortex-a72.ipk
Configuring crowdsec.
WARN[14-10-2021 07:16:13 PM] can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field) 
INFO[14-10-2021 07:16:13 PM] push and pull to Central API disabled        
INFO[14-10-2021 07:16:13 PM] Machine 'MACHINE' successfully added to the local API 
INFO[14-10-2021 07:16:13 PM] API credentials dumped to '/etc/crowdsec/local_api_credentials.yaml' 
WARN[14-10-2021 07:16:14 PM] can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field) 
INFO[14-10-2021 07:16:14 PM] push and pull to Central API disabled        
INFO[14-10-2021 07:16:16 PM] Successfully registered to Central API (CAPI) 
INFO[14-10-2021 07:16:16 PM] Central API credentials dumped to '/etc/crowdsec/online_api_credentials.yaml' 
WARN[14-10-2021 07:16:16 PM] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 
INFO[14-10-2021 07:16:17 PM] Wrote new 147807 bytes index to /etc/crowdsec/hub/.index.json 
INFO[14-10-2021 07:16:18 PM] crowdsecurity/syslog-logs : OK               
INFO[14-10-2021 07:16:18 PM] /etc/crowdsec/parsers/s00-raw doesn't exist, create 
INFO[14-10-2021 07:16:18 PM] Enabled parsers : crowdsecurity/syslog-logs  
INFO[14-10-2021 07:16:18 PM] crowdsecurity/geoip-enrich : OK              
INFO[14-10-2021 07:16:18 PM] downloading data 'https://crowdsec-statics-assets.s3-eu-west-1.amazonaws.com/GeoLite2-City.mmdb' in '/srv/crowdsec/data/GeoLite2-City.mmdb' 




^CInterrupted. Writing out status database.

seems to be from your uci-defaults/99_crowdsec last line;

cscli hub update && cscli collections install crowdsecurity/linux && cscli parsers install crowdsecurity/whitelists && cscli hub upgrade

edit: console finally exited after 2mins... how big is the download... (is interactive/uci-defaults the best way to handle this... firstboot internet connectivity might cause some issues here)

60M... slow servers maybe...

dca632 /usbstick 55°# du -chs /srv/crowdsec/data/*
5.7M	/srv/crowdsec/data/GeoLite2-ASN.mmdb
53.2M	/srv/crowdsec/data/GeoLite2-City.mmdb
136.0K	/srv/crowdsec/data/crowdsec.db
59.0M	total

Thanks for the feedback…

Yes,
It had some pre-install command ntegrated.
I must write a full OpenWrt documentation, but you may look at the default crowdsec documentations since I wrote OpenWrt specific…

The /srv/crowdsec must exist in a persistent storage.

I look at this ASAP…

Some initial commands may have been moved while PR reviews !
It need to be ready after installation, but I think there is no problem, since the Crowdsec package is not integrate in a firmware image for now.

1 Like
################# grep INSTALL_DIR net/crowdsec/Makefile 
	$(INSTALL_DIR) $(1)/etc/crowdsec
	$(INSTALL_DIR) $(1)/etc/crowdsec/scenarios
	$(INSTALL_DIR) $(1)/etc/crowdsec/postoverflows
	$(INSTALL_DIR) $(1)/etc/crowdsec/collections
	$(INSTALL_DIR) $(1)/etc/crowdsec/patterns
	$(INSTALL_DIR) $(1)/etc/crowdsec/hub
	$(INSTALL_DIR) $(1)/srv/crowdsec/data/
	$(INSTALL_DIR) $(1)/etc/init.d
	$(INSTALL_DIR) $(1)/etc/config
	$(INSTALL_DIR) $(1)/etc/uci-defaults
dca632 /usbstick 53°# grep plugins /etc/crowdsec/config.yaml
  plugin_dir: /usr/local/lib/crowdsec/plugins/

dca632 /usbstick 54°# ls -lah /usr/local/lib/crowdsec/plugins/
ls: /usr/local/lib/crowdsec/plugins/: No such file or directory

afaik, we don't have /usr/local in OpenWrt

it may not be a default package... but it's not unreasonable to assume that people will attempt to include it by default when they create their own images...

the question(s) is(are);

  • how does the daemon behave when/if these commands fail on firstboot?
  • does the daemon / init script inform the user?
  • does the daemon / init script re-attempt?
[   41.389597]  welcomeback localversion="3.5.75-7" > localversion="3.5.95-5"
[   42.033515] 99_crowdsec-ucidefault start
[   42.366744] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
[   42.936193] 99_crowdsec-ucidefault finished
[   45.786811] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[   45.795668] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   45.796957] br-lan: port 1(eth0) entered blocking state
[   45.809008] br-lan: port 1(eth0) entered disabled state
[   45.814410] device eth0 entered promiscuous mode
[   45.820181] br-lan: port 1(eth0) entered blocking state
[   45.825423] br-lan: port 1(eth0) entered forwarding state
[   46.791390] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   47.115424] netlink: 'iw': attribute type 302 has an invalid length.
...(eth1 is WAN)
[   49.544213] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   49.550994] r8152 2-2:1.0 eth1: carrier on
[root@dca632 /usbstick/_CROWDSEC 52°]# ps w | grep crowd | grep -v grep
[root@dca632 /usbstick/_CROWDSEC 53°]# logread | grep crowd
[root@dca632 /usbstick/_CROWDSEC 53°]# cat /var/log/crowdsec.log
time="15-10-2021 16:50:19" level=info msg="Crowdsec v1.2.0-openwrt-openwrt"
time="15-10-2021 16:50:19" level=info msg="Loading prometheus collectors"
time="15-10-2021 16:50:19" level=fatal msg="crowdsec init: Failed to load hub index : unable to read index file: open /etc/crowdsec/hub/.index.json: no such file or directory"

Have you let the full install process be done ?

If you have interrupt the install, the initial commands will not have processed correctly !

Yes, it is why I have said I will look at your first report...

I may propose a crowdsec-data package to integrate the first install data download.

Or move this download further...

Or simply remove the first initial installation I have integrate in the package to simplify the usage for the user, and let him do all by itself reading the documentations...

1 Like

It simply do not work since you remade the needed commands.
But some checks like in upgrade may bot work.

The log may, but the Init script don't.

No. not for now.

1 Like

negative (your commands were all present at firstboot but failed because they run before any network interfaces are available)

correct

?

Yes, and it is not used, or I have missing something...

I will patch to move this folder.

Update : To be fixed, it is a new feature from the v1.2.0 and I have missed this part !

1 Like

I do not understand how you have installed the package ?
If you let it fully install and then reboot, is it okay ?

Because you have interrupt the install, the commands did not complete...

What the firstboot do here ?
The script is also apply at installation, and then uneeded at firstboot.
It will only be used at firstboot if you integrate the package in a custom firmware, what is it not supported for now, as I have already said.