The next tuning options that should be looked at are the ones that define the HitSets and the required recency to trigger a promotion:

    hitset_count

    hitset_period

The hitset_count setting controls how many HitSets can exist before the oldest one starts getting trimmed. The hitset_period setting controls how often a HitSet should be created. If you are testing tiering in a laboratory environment, it should be noted that I/O to the PG needs to be occurring in order for a HitSet to be created; on an idle cluster, no HitSets will be created or trimmed.

Having the correct number and controlling how often HitSets are created is a key to being able to reliably control when objects get promoted. Remember that HitSets only contain data about whether an object has been accessed or not; they don't contain a count of the number of times an object was accessed. If hitset_period is too large, then even relatively low-accessed objects will appear in the majority of the HitSets. For example, with a hitset_period of 2 minutes, an RBD object containing the disk block where a log file is updated once a minute would be in all the same HitSets as an object getting access 100 times a second.

Conversely, if the period is too low, then even hot objects may fail to appear in enough HitSets to make them candidates for promotion and your top tier will likely not be fully used. By finding the correct HitSet period, you should be able to capture the right view of your I/O that a suitable sized proportion of hot objects are candidates for promotion:

    min_read_recency_for_promote

    min_write_recency_for_promote

These two settings define how many of the last recent HitSets an object must appear in to be promoted. Due to the effect of probability, the relationship between semi-hot objects and recency setting is not linear. Once the recency settings are set past about 3 or 4, the number of eligible objects for promotion drops off in a logarithmic fashion. It should be noted that while promotion decisions can be made on reads or writes separately, they both reference the same HitSet data, which has no way of determining an access from being either a read or a write. As a handy feature, if you set the recency higher than the hitset_count setting, then it will never promote. This can be used for example to make sure that a write I/O will never cause an object to be promoted, by setting the write recency higher than the hitset_count setting.