Besides the Store API that's exposed by the Thanos sidecar we covered previously, this component is also is able to gather TSDB blocks from disk and ship them to an object storage bucket. This allows you to decrease the retention of the Prometheus server by keeping historical data in a durable medium. In order to use the block upload feature in the sidecar, the --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration flags need to be set to the same value (two hours to match the default behavior), so that Prometheus local compaction is disabled.

The configuration in use is available on the prometheus instance and can be inspected by executing the following instruction:

vagrant@prometheus:~$ systemctl cat thanos-sidecar
...
ExecStart=/usr/bin/thanos sidecar 
           --log.level debug 
           --prometheus.url "http://localhost:9090" 
           --tsdb.path "/var/lib/prometheus/data" 
           --objstore.config-file "/etc/thanos/storage.yml"
...

As we can see, the --objstore.config-file flag loads all the required configuration from the file in order to ship the TSDB blocks to an object storage bucket, such as the bucket name (in our case, thanos), the storage endpoint, and access credentials. The following are the contents of that file:

vagrant@prometheus:~$ sudo cat /etc/thanos/storage.yml
type: S3
config:
  bucket: 'thanos'
  endpoint: 'storage:9000'
  insecure: true
  signature_version2: true
  access_key: 'strongACCESSkey'
  secret_key: 'strongSECRETkey'

In our Prometheus test instance, a new TSDB block will be generated every 10 minutes, and the Thanos sidecar will take care of shipping it to the object storage endpoint. We can review the available blocks in our bucket by using MinIO's web interface, available at http://192.168.42.11:9000/minio/thanos/. After logging in with the access_key and secret_key shown in the previous code snippet, you'll be greeted with something resembling the following screenshot:

We should now have some historical data available for testing. We'll need a way to query this data. That's where the Thanos store gateway comes into play.