The recommended way to integrate a custom service discovery is by relying on the file-based service discovery, file_sd. The way this integration should be implemented is to have a process (local or remote, scheduled or permanently running) query a data source (catalogue/API/database/configuration management database (CMDB)) and then write a JSON- or YAML-formatted file with all the targets and their respective labels on a path that's accessible by Prometheus. The file is then read by Prometheus either automatically through disk watches or on a schedule, which in turn allows you to dynamically update the targets that are available for scraping.

The following diagram illustrates the aforementioned workflow:

This type of approach is generic enough to comply with most, if not all, required use cases, making it possible to build a custom service discovery mechanism in a straightforward manner.

Now that we know how this type of integration should work, let's dive right in and start building our own.