As has been covered earlier, promotions are very expensive operations in tiering and care should be taken to make sure that they only happen when necessary. A large part of this is done by carefully tuning the HitSet and recency settings. However, in order to limit the impact of promotions, there is an additional throttle that restricts the number of promotions to a certain speed. This limit can either be specified as number of bytes or objects per second via two OSD configuration options:

    osd_tier_promote_max_bytes_sec

    osd_tier_promote_max_objects_sec

The default limits are 4 MBps or five objects a second. While these figures may sound low, especially when compared with the performance of the latest SSDs, their primary goal is to minimize the impact of promotions on latency. Careful tuning should be done to find a good balance on your cluster. It should be noted that this value is configured per OSD, and so the total promotion speed will be a sum across all OSDs.

Finally, the following configuration options allow tuning of the selection process for flushing objects:

    hit_set_grade_search_last_n

This controls how may HitSets are queried in order to determine object temperature, with the the temperature of an object reflects how often it is accessed. A cold object is rarely accessed, with a hot object being accessed far more frequently being candidates for eviction. Setting this to a similar figure as the recency settings is recommended. We have the following code:

    hit_set_grade_decay_rate

This works in combination with the hit_set_grade_search_last_n setting and decays the HitSet results the older they become. Objects that have been accessed more frequently than others have a hotter rating and will make sure that objects that are more frequently accessed are not incorrectly flushed. It should be noted that the min_flush and evict_age settings may override the temperature of an object when it comes to being flushed or evicted:

    cache_min_flush_age

    cache_min_evict_age

The cache_min_evict_age and cache_min_flush_age settings simply define how long an object must have not been modified for before it is allowed to be flushed or evicted. These can be used to stop objects that are only just not enough to be promoted, from continually being stuck in a cycle of moving between tiers. Setting them between 10 and 30 minutes is probably a good approach although care needs to be taken that the top tier does not fill up, in the case where there are no eligible objects to be flushed or evicted.