Following the same logic from the previous section, the --storage.tsdb.path flag should be set to configure the base path to the data storage location. This defaults to data/ on the current working directory, and so it is advisable to point this to a more appropriate path—possibly to a different drive/volume, where data can be safely persisted and I/O contention can be mitigated. To note that NFS (AWS EFS included) is not supported, as it doesn't support the POSIX locking primitives needed for safe database files management. Placing the Prometheus data storage directory in a network share is also ill-advised as transient network failures would impact the monitoring system's ability to keep functioning - just when you'd need it the most.

The Prometheus local storage can only be written to by a single Prometheus instance at a time. To make sure this is the case, it uses a lock file in the data directory. On startup, it tries to lock this file using OS-specific system calls, and will refuse to start if the file is already locked by another process.

There can be an edge case to this behavior; when using persistent volumes to store the data directory, there is a chance that, when relaunching Prometheus as another container instance using the same volume, the previous instance might not have unlocked the database. This problem would make a setup of this kind susceptible to race conditions. Luckily, there is the --storage.tsdb.no-lockfile flag, which can be used in exactly this type of situation. Be warned though that, in general (and namely, in most Prometheus deployments), it is a bad idea to disable the lock file, as doing so makes unintended data corruption easier.