The latest versions of Ceph contain an auto-tuning functionality for BlueStore OSDs. The auto-tuning works by analyzing the cache utilization of the OSDs and adjusting the caching thresholds for the OSD, RocksDB, and data caches, depending on the current hit rate. It also limits the sum of these caches to try to limit the total OSD memory usage to the limit set by the osd_memory_target variable, which is set to 4 GB by default.

Obviously, if you have less RAM in the Ceph node and therefore it unable to provide 4 GB for each OSD, this figure would need to be reduced to avoid the node running out of memory. However, if the Ceph node has sufficient memory, it would be recommended to increase the osd_memory_target variable to allow Ceph to make as much use of the installed memory as possible. Once enough RAM has been assigned to the OSD and RocksDB, any additional RAM will be used as a data cache and will help to service the top-percentile read IOs much more effectively. The current auto-tuning algorithm is fairly slow and takes a while to ramp up, so at least 24-48 hours should be given to see the full effect of a change to the osd_memory_target variable.