If you are not using the omnibus package to manage GitLab or insist on using an external Prometheus server, the picture will be a bit different. You should be aware that the default security model that Prometheus uses is rather simple; it assumes that anyone can view the stored time series data, and the server provides no authentication, authorization, or encryption. If you need these features, you should prepare a reverse proxy in front of the Prometheus server to help. More information about this can be found at https://prometheus.io/docs/operating/security/.
The monitoring architecture for this situation is shown in the following diagram (as you can see, some functionality is running on a separate server):
Now we have to make sure we have an external Prometheus host configured that is ready to scrape the data. As we've previously mentioned, Prometheus is a single Go binary. To specify which configuration file to load, use the --config.file flag. This configuration file has to have the YAML format. How a single Prometheus server monitors the GitLab Metrics Exporter embedded in GitLab is shown in the following prometheus.yml example:
- job_name: 'git-metrics'
params:
token: [ gitlab_health_check_access_token ]
metrics_path: /-/metrics
scrape_interval: 5s
scheme: https
tls_config:
insecure_skip_verify: true
file_sd_configs:
- files:
- /etc/prometheus/sd/gitlab_metrics_exporter_sd.yml
The gitlab_metrics_exporter_sd.yml file contains the following code:
- targets: ['gitlab.joustie.nl']
labels:
app: gitlab
If you put both files in /tmp, or elsewhere, and run Prometheus as a Docker container (as seen in the following example), you should have an external Prometheus ready for action. Of course, you can also install it from source and run it on a dedicated server somewhere else if preferred.
docker run -it --name my-prometheus
-v /tmp:/etc/prometheus
--publish 9090:9090
prom/prometheus
You now know how Prometheus runs on the GitLab application server itself, as well as on a separate server.