As mentioned at the start of the chapter, the tiering functionality should be thought of as tiering and not a cache. The reason behind this statement is that the act of promotions has a detrimental effect to cluster performance when compared with most caching solutions, which do normally not degrade performance if enabled on noncacheable workloads. The performance impact of promotions are caused by two main reasons. First, the promotion happens in the I/O path, the entire object to be promoted needs to be read from the base tier and then written into the top tier before the I/O is returned to the client.
Second, this promotion action will likely also cause a flush and an eviction, which cause even more reads and writes to both tiers. If both tiers are using 3x replication, this starts to cause a large amount of write amplification for even just a single promotion. In the worse case scenario, a single 4 KB access that causes a promotion could cause 8 MB of read I/O and 24 MB of write I/O across the two tiers. This increased I/O will cause an increase in latency; for this reason, promotions should be considered expensive, and tuning should be done to minimize them.
With that in mind, Ceph tiering should only be used where the hot or active part of the data will fit into the top tier. Workloads that are uniformly random will likely see no benefit and in many cases may actually cause a performance degradation, either due to no suitable objects being available to promote, or too many promotions occurring.
Most workloads that involve providing storage for generic virtual machines tend to be good candidates as normally only a small percentage of VM tends to be accessed.
Online transaction processing (OLTP) databases, will normally show improvements when used with either caching or tiering as their hot set of data is relatively small and data patterns are reasonably consistent. However, reporting or batch processing database are generally not a good fit as they can quite often require a large range of the data to be accessed without any prior warm up period.
RADOS Block Devices (RBD) workloads that involve random access with no specific pattern or workloads that involve large read or write streaming should be avoided and will likely suffer from the addition of a cache tier.