This technology can be found in more expensive (or highertier) hardware- or software-based storage arrays. As the name suggests, the idea is that we do not need to store data that is already written to the storage; only data that is different is stored.

Deduplication usually works in blocks. Each block (containing the actual data) has its hash calculated. If the hash already exists, then the data is not written but is instead pointed to the location of the data that is already stored.

There are two types of deduplication:

• In-line: Data is deduplicated as it is being written to other storage. This commonly involves a cache. The data is written to the persistent cache, the hash is calculated and compared to the stored blocks, and then the data is written to the storage itself (if it is not already stored).
• Off-line: The data is written to the storage as-is, and at a defined interval, the deduplication process is invoked to deduplicate the data.