Ceph monitors use leveldb to store all of the required monitor data for your cluster. This includes things such as the monitor map, OSD map, and PG map, which OSDs and clients pull from the monitors to be able to locate objects in the RADOS cluster. One particular feature that you should be aware of is that during a period where the health of the cluster doesn't equal HEALTH_OK, the monitors do not discard any of the older cluster maps from its database. If the cluster is in a degraded state for an extended period of time and/or the cluster has a large number of OSDs, the monitor database can grow very large.

In normal operating conditions, the monitors are very lightweight on resource consumption; because of this, it's quite common for smaller disk sizes to be used for the monitors. In the scenario where a degraded condition continues for an extended period, it's possible for the disk holding the monitor database to fill up, which, if it occurs across all your monitor nodes, will take down the entire cluster.

To guard against this behavior, it may be worth deploying your monitor nodes using LVM so that, if the disks need to be expanded, it can be done a lot more easily. When you get into this situation, adding disk space is the only solution, until you can get the rest of your cluster into a HEALTH_OK state.

If your cluster is in a HEALTH_OK state, but the monitor database is still large, you can compact it by running the following command:

sudo ceph tell mon.{id} compact

However, this will only work if your cluster is in a HEALTH_OK state; the cluster will not discard old cluster maps, which can be compacted, until it's in a HEALTH_OK state.