Managing the repository is the most import issue in Elasticsearch backup management. Due to its native distributed architecture, the snapshot and the restore are designed in a cluster style.
During a snapshot, the shards are copied to the defined repository. If this repository is local to the nodes, the backup data is spread across all the nodes. For this reason, it's necessary to have shared repository storage if you have a multimode cluster.
A common approach is to use a Network File System (NFS), as it's very easy to set up and it's a very fast solution (also, standard Windows Samba shares can be used.)
We have a network with the following nodes:
- Host server:
192.168.1.30(where we will store the backup data) - Elasticsearch master node 1:
192.168.1.40 - Elasticsearch data node 1:
192.168.1.50 - Elasticsearch data node 2:
192.168.1.51
You need an up-and-running Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe in Chapter 2, Downloading and Setup.
The following instructions are for standard Debian or Ubuntu distributions: they can be easily changed for other Linux distribution.
To create an NFS shared repository, we need to execute the following steps on the NFS server:
- We need to install the NFS server (generally the
nfs-kernel-serverpackage) on the host server. On the host server192.168.1.30we will execute:sudo apt-get update sudo apt-get install nfs-kernel-server
- Once the package is installed, we can create a directory to be shared among all the clients. Let's create a directory:
sudo mkdir /mnt/shared-directory - Give the access permission of this directory to user
nobodyand groupnogroup. These are a special reserved user/group in Linux operating systems that do not need any special permission to run things:sudo chown -R nobody:nogroup /mnt/shared-directory - Then, we need to configure the NFS exports, where we can specify that this directory will be shared with certain machines. Edit the
/etc/exportsfile (sudo nano /etc/exports) and add the following line containing the directory to be shared and space-separated client IP lists:/mnt/shared-directory 192.168.1.40(rw,sync,no_subtree_check) 192.168.1.50(rw,sync,no_subtree_check) 192.168.1.51(rw,sync,no_subtree_check)
- To refresh the NFS table that holds the export of the share, the following command must be executed:
sudo exportfs -a - Finally, we can start the NFS service by running the following command:
sudo service nfs-kernel-server start
After the NFS server is up-and-running, we need to configure the clients. We'll repeat the following steps on every Elasticsearch node:
- We need to install the NFS client on our Elasticsearch node:
sudo apt-get update sudo apt-get install nfs-common
- Now, create a directory on the client machine and we'll try to mount the remote shared directory:
sudo mkdir /mnt/nfs sudo mount 192.168.1.30:/mnt/shared-directory /mnt/nfs
- If everything is fine, we can add the mount directory in our node
/etc/fstabfile, so that it will be mounted at the next boot:sudo nano /etc/fstab - And add following lines into this file:
192.168.1.30:/mnt/shared-directory /mnt/nfs/ nfs auto,noatime,nolock,bg,nfsvers=4,sec=krb5p,intr,tcp,actimeo =1800 0 0
- We update our Elasticsearch node configuration (
config/elasticsearch.yml)path.repoin this way:path.repo: /mnt/nfs/ - After having restarted all Elasticsearch nodes, we can create our share repository on the cluster via a single standard repository creation call:
curl -XPUT 'http://192.168.1.40:9200/_snapshot/my_repository' - d '{ "type": "fs", "settings": { "location": "/ mnt/nfs/my_repository", "compress": true } }'
NFS is a distributed filesystem protocol that is very common in the Unix world, and which allows you to mount remote directories on your server. The mounted directories look like the local directory of the server, and therefore, by using NFS, multiple servers can write to same directory.
This is very handy if you need to do a shared backup: all the nodes will write/read from the same shared directory.