How to set up a CrowdSec multi-server installation

Introduction

CrowdSec is an open-source & collaborative security solution built to secure Internet-exposed Linux services, servers, containers, virtual machines with a server-side agent. It is a modernized version of Fail2ban which was a great source of inspiration to the project founders.

CrowdSec is free (under an MIT License) and its source code available GitHub. The solution is leveraging a log-based IP behavior analysis engine to detect attacks. When the CrowdSec agent detects any aggression, it offers different types of remediation to deal with the IP behind it (access prohibition, captcha, 2FA authentication etc.). The report is curated by the platform and, if legitimate, shared across the CrowdSec community so users can also protect their assets from this IP address.

A few months punzone, we added some interesting features to CrowdSec when releasing v1.0.x. One of the most exciting ones is the ability of the CrowdSec agent to act as an HTTP rest API to collect signals from other CrowdSec agents. Thus, it is the responsibility of this special agent to store and share the collected signals. We will call this special agent the LAPI server from now .

Another worth noting feature, is that mitigation voto negativo longer has to take place the same server as detection. Mitigation is done using bouncers. Bouncers rely the HTTP REST API served by the LAPI server.

Goals

Con this article we’ll describe how to deploy CrowdSec a multi-server setup with one server sharing signal.

CrowdSec Goals Infographic

Both server-2 and server-3 are meant to host services. You can take a our Hub to know which services CrowdSec can help you secure. Last but not least, server-1 is meant to host the following local services:

  • the local API needed by bouncers

  • the database fed by both the three local CrowdSec agents and the online CrowdSec blocklist service.
    As server-1 is serving the local API, we will call it the LAPI server.

We choose to use a postgresql backend for CrowdSec database order to allow high availability. This topic will be covered future posts. If you are ok with voto negativo high availability, you can skip step 2.

Furthermore this post will cover attack mitigation for hosted services server-2 and server-3 using CrowdSec bouncers.

This article is the first of a series.

Prerequisites

  • Two Internet-facing preinstalled Ubuntu 20.04 servers hosting services. From now , we will refer to these servers by server-2 and server-3 .

  • One Internet facing preinstalled Ubuntu 20.04 server. From now we will refer to this server by server-1. Let’s assume that server-1‘s ip is 10.0.0.1. (voto negativo internet connection this server is not a strict requirement).

  • A local connecting all three servers

Step 1: CrowdSec installation

Let’s install CrowdSec every single server, following the CrowdSec installation guide.

wget -qO - https://s3-eu-west-1.amazonaws.com/crowdsec.debian.pragmatic/crowdsec.asc |sudo apt-key add - && echo "deb https://s3-eu-west-1.amazonaws.com/crowdsec.debian.pragmatic/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/crowdsec.list > /dev/null

sudo apt update


sudo apt install crowdsec

 

We now have three grado CrowdSec installations running.

Step 2 (Non di serie): Switch the database backend to postgresql server-1

sudo apt install postgresql

 

First we have to connect to the database as a postgres user.

sudo -i -u postgres

psql

 

Thanks to the Postgresql Crowdsec documentation, we can now initialize the database.

postgres=# CREATE DATABASE crowdsec;

CREATE DATABASE

postgres=# CREATE USER crowdsec WITH PASSWORD 'CREATE USER crowdsec WITH PASSWORD '<password>';

postgres=# CREATE USER crowdsec WITH PASSWORD '<password>';

CREATE ROLE

postgres=# GRANT ALL PRIVILEGES ON DATABASE crowdsec TO crowdsec;

GRANT

 

Now let’s make CrowdSec know about this new database backend. To achieve this, we will have to update the db_config section of the /etc/crowdsec/config.yaml file.

db_config:

  log_level: info

  type: postgres

  user: crowdsec

  password: "<password>"

  db_name: crowdsec

  host: 127.0.0.1

  port: 5432

 

After registering the local machine again the database, we are able to restart CrowdSec:

sudo cscli machines add -a

sudo systemctl restart crowdsec

 

Step 3: Make server-2 and server-3 report to LAPI server

First we have to configure CrowdSec server-1 to accept connections from server-2 and server-3. Please ensure that your firewall allows connections from server-2 and server-3 server-1‘s port 8080.

Let’s configure the API server server-1 side. For this to happen, we will have to modify both /etc/crowdsec/config.yaml and /etc/crowdsec/local_api_credentials.yaml.

For /etc/crowdsec/config.yaml, it is now the API section that needs to be amended. It’s only a matter of updating the listening IP from localhost to the local IP:

api:

  client:

    insecure_skip_verify: false

    credentials_path: /etc/crowdsec/local_api_credentials.yaml

  server:

    log_level: info

    listen_uri: 10.0.0.1:8080

    profiles_path: /etc/crowdsec/profiles.yaml

    online_client: # Crowdsec API credentials (to push signals and receive bad IPs)                                                                            

      credentials_path: /etc/crowdsec/online_api_credentials.yaml

 

For /etc/crowdsec/local_api_credentials.yaml we only have to change the configured ip accordingly:

url: http://10.0.0.1:8080/

login: <login>

password: <password>

 

And we can restart CrowdSec:

sudo systemctl restart crowdsec

 

Now we will configure the connections server-2 and server-3.

First we register to the lapi server both server-2 and server-3:

sudo cscli lapi register -u http://10.0.0.1:8080

 

By default, the local api server is active every CrowdSec agent installation. Con this setup, we want to disable it server-2 and server-3. To achieve this, we need to tweak the CrowdSec agent systemd service file.

sudo cp /lib/systemd/system/crowdsec.service /etc/systemd/system/crowdsec.service

 

Now edit /etc/systemd/system/crowdsec.service and add the -no-api parameter to CrowdSec agent invocation both server-2 and server-3.

[Unit]

Description=Crowdsec agent

After=syslog.target .target remote-fs.target nss-lookup.target


[Service]

Type=notify

Environment=LC_ALL=C LANG=C

PIDFile=/var/run/crowdsec.pid

ExecStartPre=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -t

ExecStart=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -no-api

#ExecStartPost=/bin/sleep 0.1

ExecReload=/bin/kill -HUP $MAINPID


[Install]

WantedBy=multi-user.target

 

We can now acknowledge the changes and restart CrowdSec once again.

sudo systemctl daemon-reload

sudo systemctl restart crowdsec

 

Last thing to do is to allow server-2 and server-3 connections server-1.

sudo cscli machines list

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

 NAME                                              IP ADDRESS      LAST UPDATE           STATUS  VERSION                                                            

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

 dc6f34b3a4994700a2e333df43728701D0iARTSQ6dxiwyMR  10.0.0.1  2021-04-13T12:16:11Z  ✔️       v1.0.9-4-debian-pragmatic-a8b16a66b110ebe03bb330cda2600226a3a862d7 

 9f3602d1c9244f02b0d6fd2e92933e75zLVg8zSRkyANxHbC  10.0.0.3   2021-04-13T12:24:12Z  🚫                                                                         

 ac86209e6f9c4d7d8de43e2ea31fe28ebvde0vWDr46Mpd3L  10.0.0.2   2021-04-13T12:22:28Z  🚫                                                                         

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

Con this output, we can see two machines that are not yet validated. Let’s validate them now.

sudo cscli machines validate 9f3602d1c9244f02b0d6fd2e92933e75zLVg8zSRkyANxHbC

sudo cscli machines validate ac86209e6f9c4d7d8de43e2ea31fe28ebvde0vWDr46Mpd3L

 

server-2 and server-3 are now allowed to push patronato to server-1 CrowdSec agent. It may be needed to restart CrowdSec server-2 and server-3.

sudo systemctl restart crowdsec

 

server-1, the command sudo cscli machines list should now show three validated machines.

Step 4: Set up Mitigation

Now we want to install mitigation our internet-facing servers. First we have to generate two API tokens for server-2 and server-3 server-1.

sudo cscli bouncers add server-2

Api key for 'server-2':


   02954e85c72cf442a4dee357f0ca5a7c


Please keep this key since you will not be able to retrive it!

 

sudo cscli bouncers add server-3

Api key for 'server-3':


   3b1030ce0840c343eecd387ac5a3a614


Please keep this key since you will not be able to retrieve it!

 

For now, there is voto negativo package available for the firewall bouncer. This is our high priority to-do list.

Hence both server-2 and server-3:

wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.10/cs-firewall-bouncer.tgz

tar zxvf cs-firewall-bouncer.tgz

cd cs-firewall-bouncer-v0.0.10/

sudo ./install.sh

 

If iptables and nftables are not installed, the install script will ask for your permission to install it. The same will happen for ipset installation.

It iss now time to use the token we generated at the beginning of this step. Both api_key and api_url have to be updated /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml:

mode: iptables

piddir: /var/run/

update_frequency: 10s

daemonize: true

log_mode: file

log_dir: /var/loch/

log_level: info

api_url: http://10.0.0.1:8080/

api_key: 02954e85c72cf442a4dee357f0ca5a7c

disable_ipv6: false

#if present, insert rule  those chains

iptables_chains:

  - INPUT

#  - FORWARD

#  - DOCKER-USER

 

We can now restart the firewall bouncer.

sudo systemctl restart cs-firewall-bouncer

 

Conclusion and perspectives

We described how to setup a CrowdSec multi-server installation. The resource overhead server-2 and server-3 is quite limited as most of the tasks are deported to server-1. This allows to grow the setup by only:

  • register and validate the CrowdSec agent the LAPI server

  • add and validate new bouncers
    It is worth noting that bouncers and CrowdSec agents don’t have to be installed the same server. Therefore, the CrowdSec agent has to be installed where logs are generated, but the mitigation can be deported where it is meaningful.

Obviously, there are caveats this setup:

  • Communications between agents are over clear http. This is acceptable a local , but not possible over the Internet. CrowdSec allows the use of https for those communications, a next post will cover this topic.

  • Monitoring alerting is not covered this article either. CrowdSec allows very powerful monitoring through Prometheus scraper. A post will cover this topic too.

  • The CrowdSec database is not highly available. Furthermore, the CrowdSec agent server-1 is a single point of failure.

Now you may be wondering: how to build a highly available multi-machine CrowdSec setup? Stay tuned for our next article.

The CrowdSec team is always happy to receive feedback about the solution and its usage. If you wou like to meet with them and have a chat, here are a few useful links. Give them a shout!

Source link

Share

Written by bourbiza

Comments

Leave a Reply

Loading…

0