First, let's understand what a shard is. An index contains documents of one or more types. Shards help in distributing an index over the cluster. Shards help in dividing the documents of a single index over multiple nodes. There is a limit to the amount of data that can be stored on a single node, and that limit is dictated by the storage, memory, and processing capacities of that node. Shards help by splitting the data of a single index over the cluster, and hence allowing the storage, memory, and processing capacities of the cluster to be utilized.
The process of dividing the data among shards is called sharding. Sharding is inherent in Elasticsearch and is a way of scaling and parallelizing, as follows:
- It helps in utilizing storage across different nodes of the cluster
- It helps in utilizing the processing power of different nodes of the cluster
By default, every index is configured to have five shards in Elasticsearch. At the time of creating the index, you can specify the number of shards from which the data will be divided for your index. Once an index is created, the number of shards cannot be modified.
The following diagram illustrates how five shards of one index may be distributed on a three-node cluster:
The shards are named P1 to P5 in the preceding diagram. Each shard contains roughly one fifth of the total data stored in the index. When a query is made against this index, Elasticsearch takes care of going through all the shards and consolidating the result.
Now, imagine that one of the nodes (Node 1) goes down. With Node 1, we also lose the share of data, which was stored in shards P1 and P2:
Distributed systems such as Elasticsearch are expected to run in spite of hardware failure. This issue is addressed by replica shards or replicas. Each shard in an index can be configured to have zero or more replica shards. Replica shards are extra copies of the original or primary shard and provide a high availability of data.
For example, with one replica of each shard, we will have one extra copy of each replica. In the following diagram, we have five primary shards, with one replica of each shard:
Primary shards are depicted in green and replica shards are depicted in yellow. With the replicas in place, if Node 1 goes down, we still have all the shards available in Node 2 and Node 3. Replica shards may be promoted to primary shards when the corresponding primary shard fails.
Apart from providing high availability and failover, replica shards also enable the querying workload to be executed over replicas. Read operations such as search, query, and aggregations can be executed on replicas as well. Elasticsearch transparently distributes the execution of queries across nodes of the cluster where the required shards or replicas are located.
To summarize, nodes get together to form a cluster. Clusters provide a physical layer of services on which multiple indexes can be created. An index may contain one or more types, with each type containing millions or billions of documents. Indexes are split into shards, which are partitions of underlying data within an index. Shards are distributed across the nodes of a cluster. Replicas are copies of primary shards and provide high availability and failover.