Once you have configured a RADOS pool to be an overlay of another RADOS pool, Cephs tiering functionality works on the basic principal that if an object does not exist in the top-level tier, then it must exist in the base tier. All object requests from clients are sent to the top tier; if the OSD does not have the requested object, then depending on the tiering mode, it may either proxy the read or write request down to the base tier or force a promotion. The base tier then proxies the request back through the top tier to the client. It's important to note that the tiering functionality is transparent to clients, and there is no specific client configuration needed.
There are three main actions in tiering that move objects between tiers. Promotions copy objects from the base tier up to the top tier. If tiering is configured in writeback mode, the flushing action is used to update the contents of the base tier object from the top tier. Finally, when the top tier pool reaches capacity, objects are evicted by the eviction action.
In order to be able to make decisions on what objects to move between the two tiers, Ceph uses HitSets to track accesses to objects. A HitSet is a collection of all object access requests and is consulted to determine if an object has had either a read or write request since that HitSet was created. The HitSets use a bloom filter to statistically track object accesses rather than storing every access to every object, which would generate large overheads. The bloom filter only stores binary states; an object can only be marked as accessed or not; and there is no concept of storing the number of accesses to an object in a single HitSet. If an object appears in a number of the most recent HitSets and is in the base pool, then it will be promoted.
Likewise, objects that no longer appear in recent HitSets will become candidates for flushing or eviction if the top tier comes under pressure. The number of HitSets and how often a new one gets created can be configured, along with the required number of recent HitSets a write or read I/O must appear in, in order for a promotion to take place. The size of the top-level tier can also be configured and is disconnected from the available capacity of the RADOS pool it sits on.
There are a number of configuration and tuning options that define how Ceph reacts to the generated HitSets and the thresholds at which promotions, flushes, and evictions occur. These will be covered in more detail later in the chapter.