Prometheus is an open source monitoring system inspired by Borgmon, Google's production monitoring system. Since it was introduced in 2012, the project has built up an active community and has already been used by many companies. It is an example of a TSDB, a time-series monitoring database. This means that time is a deliberate X-axis in all of your measurements. Every new entry is an insert and not an update of a data-row.

This is best visualized in a table:

The two main advantages of this approach are as follows:

• Scalability: These databases are tuned to ingest data and do so very efficiently.
• Usability: They provide all kinds of tools and functions, for example, data retention and continuous queries.

Whereas most monitoring systems focus on measuring external system behaviors through health checks, Prometheus emphasizes measuring internal system behaviors by requesting metrics from the software itself. With Prometheus, you can set up monitoring dynamically, making it a valuable tool for gaining insight into distributed setups, such as software on a Kubernetes cluster.

The Prometheus project provides client libraries that make it easier to export metrics from software. These libraries enable you to integrate monitoring as part of your software. It opens a port to metrics inside your application, so to speak.

Another method is to run separate smaller programs that gather data that will be scraped by the Prometheus server. Some officially endorsed exporters are available in the Prometheus group on GitHub: https://github.com/prometheus.

GitLab is packaged with both methods. When enabled, it can expose metrics from within GitLab itself, but the omnibus package can also deliver extra exporter instances that are able to monitor a number of components. The monitoring architecture is as follows (as you can see, the functionality runs on the GitLab application server):

You can see several exporters in the image; let's go through them:

• The (GitLab) Metrics Exporter: GitLab has incorporated the client functions into itself and can expose metrics if the setting is enabled. You can find this feature in the Admin | Settings | Metrics and profiling | Metrics Prometheus, as shown in the following screenshot:

This works by allowing you to view the exporters' output on http://{your gitlab url}}/-/metrics?token=something. An example is given on the Health Check page via Monitoring | Health check, as shown in the following screenshot:

Make sure that you append the token to the request otherwise you won't see any data. An example output shown when visiting the URL is as follows:

• The Redis Exporter: This is an external program you can find at https://github.com/oliver006/redis_exporter/blob/master/README.md#whats-exported. It is a Go binary that exposes metrics about Redis, the in-memory database and cache that GitLab uses for storing background job queues, and session state and UI caching.

When used together with Grafana, dashboards can be quickly set up, as shown in the following screenshot:

• The Postgres Exporter: Again, this is a Go binary and external project available at: https://github.com/wrouesnel/postgres_exporter.

• The GitLab Monitor: This exporter is a bit different in that it is built-in with Ruby and exposes metrics with the Sinatra web server gem. The project page is https://gitlab.com/gitlab-org/gitlab-monitor.

When you query the GitLab Monitor, you can provide different parameters, which represent the kind of metric you want to scrape; they include the following:

1. 1. • Database: Provides information about tables, rows, and CI build
      • Git: Provides information about Git pulls
      • Process: Provides information about CPU, process count, sidekiq stats, and more

• The Node Exporter: This is perhaps one of the best-known exporters for Prometheus. It consists of a lot of basic metrics for an application node. Again, this exporter is written in Go and is available at https://github.com/prometheus/node_exporter.
• To enable the built-in Prometheus server on the GitLab application server, edit the /etc/gitlab/gitlab.rb file. Search for prometheus['enable'], uncomment it, and set it to true. There are several other options, but just enabling it will already deliver you a working instance.

The part of the gitlab.rb file you should change is as follows:

################################################################################
 ## Prometheus
 ##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/
 ###############################################################################
 prometheus['enable'] = true
 # prometheus['monitor_kubernetes'] = true
 # prometheus['username'] = 'gitlab-prometheus'
 # prometheus['uid'] = nil
 # prometheus['gid'] = nil

After changing gitlab.rb, you should run a reconfiguration to activate the changes. You will see messages explaining that certain exporters and the Prometheus server have been started.

You can navigate to the Prometheus console by going to the address configured in prometheus['listen_address']. You will be presented with a query interface. If you click the drop-down list, you should see a list of metrics that can be queried, as shown in the following screenshot:

Once you have chosen a metric, it will display all values it has recorded in the database (including data that was actively scraped from Prometheus exporters). In this case, the chose view mode was probably Console. You can also view the data as a graph by clicking on Graph, as shown in the following screenshot: