By monitoring performance counters from both the operating system and Ceph, you are arming yourself with a wealth of knowledge to gain a better understanding of how your Ceph cluster is performing. If storage permits, it's worth trying to capture as many of these metrics as possible; you never know when the metrics will come in handy. It's quite often the case when diagnosing a problem that a metric that was previously thought to have no connection to the issue suddenly sheds light on the actual cause. The traditional approach of only monitoring key metrics is very limiting in this regard.

Most monitoring agents that run on Linux will allow you to capture a large array of metrics, from resource consumption to filesystem usage. It's worth spending time analyzing what metric you can collect and configuring them appropriately. Some of these monitoring agents will also have plugins for Ceph, which can pull out all of the performance counters from Ceph's various components, such as osd and mon nodes.