Using CrowdSec

I will not go through the complete installation process of CrowdSec. Rather, I will cover the steps of a bare minimum installation that will allow you to understand what you need for the section “Learning From CrowdSec.” The objective of this installation is to get to a point to see the community data that is collected by a central server replicated to a local database.

The directory contains a variety of files including the CrowdSec command line tools crowdsec and cscli along with a folder called data that you will look at in the next section in more detail. The database with extension .mmdb is the database that you will look at in detail in the “GeoIP Database” section.

crowdsec.db

CrowdSec stores data inside a SQLite database called crowdsec.db. The database contains a number of tables, shown in Figure 14-1.

A screenshot of the crowd's database includes the main potion, tables, alerts, bouncers, decisions, events, machines, meta, SQLite slash master, SQLite slash sequence, and server objects.

Notice the last log message that says added 8761 entries, which means that it has added 8761 entries into your database. If you are not getting this message, rerun the crowdsec command.

Looking into the decisions table, you will the populated data, as shown in Figure 14-2

A screenshot of the decision table show categories according to I P address, data until when a particular I P is banned, and scenarios when an I P address is detected.

You have learned briefly how to set up CrowdSec and you have seen the data it uses. In the next section, you will look at parts of CrowdSec that are interesting. You will look at how certain things are implemented inside CrowdSec and then look at a simpler code sample of how to do it.

System Signal Handling

As a system, CrowdSec provides an extensive list of features that are broken down into several different modules. The reason for features to be broken down into modules is to make it easy for development, maintenance, and testing. When building a system, one of the key things to remember is to make sure all the different modules can be gracefully terminated and all resources such as memory, network connections, and disk space are released. To make sure that different parts of the system shut down properly, you need some sort of coordinated communication to understand when modules need to prepare for the shutdown process.

Imagine a scenario where you are designing an application and it is terminated by the operating system because of some resource constraint. The application must be aware of this and have the capability to shut down all the different modules independently before shutting itself down permanently. You will look at an example on how this is done using the code sample in the chapter14/signalhandler folder.

The application successfully shuts down gracefully because the Ctrl+C key combo is detected. Before going through the code, Figure 14-3 shows the app design. Use Figure 14-3 as guidance when you walk through the sample code.

A flowchart of the Crowdsec system starts from main.go then register signal channel after that classified into a loop in 100 and 1000 times function then wait for function.

The code listens to all these signals to ensure that if any of them are detected, it will do its job to shut itself down properly.

The signalChan variable is a channel that accepts os.Signal and it is passed as parameter when calling signal.Notify(). The goroutine takes care of handling the signal received from the library in a for{} loop (step 2). Receiving a signal (step 6) means that there is an interruption, so the code must take the necessary steps to start the shutdown process (step 7).

The loop100Times function runs inside a for{} loop where it checks the channel condition inside the select{} statement. To make it easy to understand, basically the for{ select {} } block of code translate to the following:

The same logic is used inside the loop1000Times function, so it works exactly the same. Both functions will stop processing and will print the counter value to the terminal once the stop channel is closed. The application has achieved the state of shutting down itself gracefully by informing the different parts of the code that it is shutting down.

Handling Service Dependencies

Complex applications like CrowdSec have multiple services that run at the same time or at scheduled times. In order for services to run properly, there needs to be service coordination that takes care of the dependencies between services.

Figure 14-4 shows how the service coordination is done inside CrowdSec using the channel.

A flowchart of the service coordinator starts from the main.go then startRunSvc after that classified into two categories named serve, serve Prometheus, serveAPIServer, and wait for the signal.

In Figure 14-4, the apiReady channel is the central part of the service coordination when CrowdSec starts up. The diagram shows that the apiServer.Run function sends a signal to the apiReady channel, which allows the other service, servePrometheus, to run the server listening on port 6060.

GeoIP Database

CrowdSec uses a GeoIP database that contains geographical information of an IP address. This database is downloaded as part of setting up the test environment discussed in the “Using CrowdSec” section.

In this section, you will look into this database and learn how to read the data from the database. One of the use cases for this database is the ability to build a security tool for your infrastructure to label each incoming IP, which is useful to monitor and understand the incoming traffic to your infrastructure. The GeoIP database comes from the following website: https://dev.maxmind.com/geoip/geolite2-free-geolocation-data?lang=en#databases. Have a read through the website to get an understanding of the licensing

The code reads the database to get all IP addresses in the 2.0.0.0 IP range and prints all the IP addresses found in that range along with other country- and continent-related information. Let’s go through the code and understand how it uses the database.

The data is stored in a single file, which is efficiently packed together, so in order to read the database, you must to use another library. Use the github.com/oschwald/maxminddb-golang library. The documentation of the library can be found at https://pkg.go.dev/github.com/oschwald/maxminddb-golang.

networks.Next() loops through the records found and reads all geographical information from the database by calling the networks.Network(..) function, which populates the rec variable.

Once the JSON has been unmarshalled back to the r variable, the code prints out the information into the console.