Chapter 14, Integrating Long-Term Storage with Prometheus
The main advantages of basing the remote write feature on the WAL are: it makes streaming of metrics possible, has a much smaller memory footprint, and it’s more resilient to crashes.
You can request Prometheus to produce a snapshot of the TSDB by using the /api/v1/admin/tsdb/snapshot API endpoint (only available when the --web.enable-admin-api flag is enabled), and then back up the snapshot.
You can delete time series from the TSDB by using the /api/v1/admin/tsdb/delete_seriesAPI endpoint and then using the/api/v1/admin/tsdb/clean_tombstonesto make Prometheus clean up the deleted series (these endpoints will only be available when the--web.enable-admin-apiflag is enabled).
Object storage usually provides 99.999999999% durability and 99.99% availability service-level agreements, and it’s quite cheap in comparison to block storage.
Yes. For example, keeping the raw data is useful for zooming into short time ranges in the past.
Thanos store provides an API gateway between Thanos Querier and object storage.
Data in object storage can be inspected using the thanos bucketsub-command, which also allows verifying, repairing, listing and inspecting storage buckets.