Backup and restore of an IBM Cloud Private clusterA backup and restore strategy plays an important role in any IBM Cloud Private cluster, as it defines how the whole environment, it’s configuration and it’s applications are restored to a working state in the event of a disaster.
This chapter describes the various options available to Cluster Administrators when thinking about backing up an IBM Cloud Private cluster, ranging from each individual core component to the infrastructure itself. It will discuss the various alternatives and considerations to a backup strategy, and provide several approaches to backing up and restoring the configuration to the same or a different instance of IBM Cloud Private 3.1.2.
This chapter has the following sections:
3.1 The purpose of backing up a cluster
A properly defined resiliency strategy is a key part in keeping your applications running continuously, and most of the time a backup plan is one of the core features for any form of resiliency effort to provide peace of mind, knowing that your infrastructure and application data is able to be recovered in the event of some failure or disaster that disrupts day to day operations.
For IBM Cloud Private, there are three common options for defining a backup strategy
•Backup the infrastructure
•Backup the core components
•Backup using a preferred third party tool
Due to the vast range of tools available, backing up using third party software is not covered in detail in this book. Instead, this chapter will focus on how to effectively backup and restore the infrastructure and the core components.
In any case, regular backups should be taken so that you’re able to restore to a recent point in time. This point in time entirely depends on your own policies, but backups are recommended to be take immediately after installation and prior to any major changes or upgrades, where the risk of a cluster failure is higher than the normal day to day business operations.
Taking a full infrastructure backup can be quite storage intensive, so this chapter will provide a variety of methods to backup the core IBM Cloud Private components in a flexible manner, using both manual and automated methods to ensure the most efficient process is put in place on a regular basis without worrying about the manual effort involved.
The main purpose of a backup plan is to actually restore it, so in addition to backups, the relevant processes should be in place to test that the backups are actually good backups. This helps to avoid nasty surprises in the event something does go wrong and to provide peace of mind knowing that you can recover from failures.
3.2 Backup versus high availability, disaster recovery, and continuous availability
Whilst backup and restore is the main subject for this chapter, there are other alternatives that should be considered when designing the cluster architecture, and in all case the architecture should be designed in such a way that gracefully caters for failure. This is applicable on both the infrastructure level and the application level, although the latter is less of a concern as the IBM Cloud Private platform takes care of keeping applications highly available. Before designing the processes to backup a cluster, consider the use of High Availability (HA), Continuous Availability (CA) and Disaster Recovery (DR) methodologies
HA is the ability to withstand failures, by eliminating any single points of failure from the whole system. This typically entails having two or more instances of any component so that the system is able to continue providing services by switching to instance 2, when instance 1 fails. CA is the ability to withstand both unplanned and planned downtime for any given component, providing greater flexibility for maintenance periods, or general component failures. DR on the other hand, is the capability to recover from a failure.
A common example used in DR discussions is the failure of an entire datacenter, and will typically involve all the hardware in one geographic location simultaneously shut down. A good DR strategy will be able to recover from this by recreating the same infrastructure (compute, networking and storage) in another geographical location within a specific Recovery Time Objective (RTO).
These methods can be combined with the backup strategy to ensure you are backing up the platform effectively to allow you to recover from any given failure. When doing the system architecture it is important to consider the backup strategy and also if there is a need for high availability (HA) or disaster recovery (DR). The following specifications must be taken into account: Recovery Point Objective (RPO) and Recovery Time Objective (RTO).
The backup strategy will grant that the system is restored to a given time when the backup was taken and in a case of the disaster, the data created between the time that the backup was taken and the moment of the failure will be lost in most of cases. This should be considered when creating the backup policy
The high availability environment will grant that the servers still responsive including when one or more nodes fail. So other servers will remain online and perform the tasks. When thinking about HA it is frequently referred as the local system redundancy and each of HA environments should be able to handle the full load. The main difference between the HA and backup is that if data gets corrupted in one of the nodes, it will be propagated to the other nodes, with that both nodes will having the failure, and will need the backup to resume to its normal operation.
The disaster recovery (DR) environment is a copy of the production (or running) environment and in case of failure this environment will take over the requests and all operation, when the main system is ready to be put back online the data should be replicated from the DR side to the main system.
3.3 Backup options
When considering the backup strategy, it is recommended to verify the most suitable types of backup to meet the requirements. This section will briefly discuss each of the components within the IBM Cloud Private infrastructure and platform, and if/why each should be backed up.
3.3.1 Infrastructure backups
Infrastructure backups are the most common method to ensure a full copy of an IBM Cloud Private node exists in the event of a failure. Suitable tools for infrastructure backups are usually available with the hypervisor hosting IBM Cloud Private.
At the time of writing, there are 6 different types of IBM Cloud Private nodes; boot, master, proxy, management, Vulnerability Advisor and worker nodes. Each node plays a specific role in the cluster, and each will have a different impact on the backup and restore strategy.
•Boot nodes: A boot (or bootstrap) node is used for running installation, configuration, node scaling, and cluster updates. Only one boot node is required for any cluster, and a single boot node can cater for multiple installations of IBM Cloud Private. This node stores the cluster configuration data and is important to at least backup the filesystem so that the original configuration and cluster certificates are able to be reused if necessary. In small clusters, boot nodes are typically combined with the master nodes.
•Master nodes: A master node provides management services and controls the worker nodes in a cluster. Master nodes host processes that are responsible for resource allocation, state maintenance, scheduling and monitoring. Because a high availability (HA) environment contains multiple master nodes, if the leading master node fails, failover logic automatically promotes a different node to the master role. Hosts that can act as the master are called master candidates. The master nodes hosts the most important core components, such as etcd (if etcd is not externalized) and MongoDB and should almost certainly be the main focus for platform level backups.
•Proxy nodes: A proxy node is a node that transmits external request to the services created inside your cluster. These are not considered important to backup, as they are generally easily replaced in any cluster.
•Management nodes: A management node hosts management services such as Istio, monitoring, metering, and logging. The importance of backing up this node entirely depends on how valuable you consider the data stored on it. For example, the logging data (by default) will retain the platform log data for one day, so it may not always make sense to keep infrastructure backups of the data if it will be deleted again shortly after restoration.
•Vulnerability Advisor nodes: A Vulnerability Advisor (VA) node is an optional node that is used for running the Vulnerability Advisor services. The importance of backing up this node entirely depends on how valuable you consider the data stored on it. VA data is a representation of the current cluster, so if a cluster fails and becomes unrecoverable, recovering the data to another cluster is not meaningful for reporting, but may be desired if you require all data to be retained.
•Worker nodes: A worker node is a node that provides a containerized environment for running tasks. It’s generally not recommended to backup these nodes as they are easily replaced, but if applications running on the worker nodes depend on a file path on the host file system then an appropriate backup method (or a different storage method) should be considered.
•etcd nodes: An etcd node is an optional node that contains the etcd distributed key store components for the IBM Cloud Private cluster. The etcd components are hosted by the master nodes by default, but in large clusters with more than 100 worker nodes, it’s recommended to separate etcd to reduce the volume of network traffic and resource requirements for heavy etcd usage.
Application storage considerations
As a Cluster Administrator, you need to think about the storage methods and how it fits in with the backup and restore strategy. There are a range of storage solutions for IBM Cloud Private that provide different storage capabilities and the most suitable solution(s) should be used to ensure you’re able to backup the data from the PersistentVolume mounted to the application and restore it as necessary. This chapter does not provide a specific method to backing up and restoring PersistentVolume data and it’s recommended to agree with the application owners on how to ensure the application is resilient to failures.
3.3.2 Platform backups
There are several components of the IBM Cloud Private platform that require persisted data to be backed up and retained in order restore the same working configuration to a fresh installation, either on the same infrastructure (local) or different infrastructure entirely (remote). In all cases, the backup data should be stored externally to the cluster outside the failure range (for example a different datacenter in a multiple datacenter environment), to ensure the data is globally available in the event of a disaster.
Platform components to back up
The following are components requiring a backup.
Cluster installation data
The cluster directory used for the IBM Cloud Private installation has several directories that contain core data generated during the cluster installation. It’s important to back up this data as it will need to be reused in a situation where a cluster is considered unrecoverable, requiring a fresh installation of IBM Cloud Private on the same infrastructure. The following data should be backed up to a storage medium situated outside of the cluster.
etcd
etcd is a distributed key-value store that maintains all of the configuration data for an IBM Cloud Private cluster. Without a fully functioning and healthy etcd, the cluster will be inoperable. etcd is a core component of IBM Cloud Private that should be backed up at regular intervals so that a cluster can be restored to the most recent state in the event of a failure.
MongoDB
MongoDB is a database used in IBM Cloud Private to store data for the metering service, Helm repository, Helm API server, LDAP configuration and team/user/role information. This is a core component of IBM Cloud Private that should be backed up at regular intervals so that a cluster can be restored to the most recent state in the event of a failure.
MariaDB
MariaDB is a database used to store OpenIDConnect (OIDC) data used for the authentication to IBM Cloud Private. By default IBM Cloud Private refreshes the user access tokens every 12 hours, so the data in MariaDB is transient and therefore an optional component to back up. It’s worth noting that whilst the content of the MariaDB database is not essential, it is worth backing up at least once to store the database structure, in the event that it becomes corrupt and the authentication modules in IBM Cloud Private cannot authenticate users.
Private image registry
The private image registry is a Docker image registry used to store all the images for the IBM Cloud Private platform. It contains all the system images and user images segregated by repositories. It is important that the image registry data is backed up to ensure that both the platform and user deployed workloads can reliably pull the same images regardless of whether the workloads are running on the same or a restored cluster. In a Highly Available configuration, the image repository is mounted on shared storage, so does not require a backup unless the cluster administrator wishes to do so.
Helm repository
The Helm repository stores all the Helm charts uploaded to the IBM Cloud Private catalog. This should be backed up to ensure the same Helm charts are available for deployment from the catalog.
Elasticsearch logging data
In IBM Cloud Private, an Elasticsearch cluster consists of Elasticsearch, Logstash and Kibana and is used to collect and process all the log data generated from all containers running on the platform. By default, the collected log data is stored for 24 hours, after which it is removed from the system and no longer available for querying. Whilst the Elasticsearch cluster itself is a core component and fundamental to cluster administrators and users that wish to view log data for their workloads, the log data stored is not essential to maintaining a functioning cluster, therefore it is optional to back up the logging data.
The Vulnerability Advisor and Mutation Advisor management services use Elasticsearch to store data, therefore in a cluster configuration where the Vulnerability Advisor and Mutation Advisor management services are enabled (and the cluster administrator deems the data valuable) then it is advised to back up the Elasticsearch log data.
Monitoring data
In IBM Cloud Private, the monitoring stack includes Alert Manager, Prometheus, and Grafana. These components provide the capabilities to expose metrics about the whole cluster and the applications running within it, trigger alerts based on data analysis and provide a graphical view of the collected metrics to the users. The monitoring stack is deployed, by default, with no persistent storage and therefore does not require backing up. After installation, if persistent storage is added to each component then backing up the Persistent Volume data is necessary to ensure any changes made can be restored.
Vulnerability Advisor
Vulnerability Advisor (VA) is an optional management service that actively scans the images used in an IBM Cloud Private cluster and identifies security risks in these images. The reporting data for VA is stored in Kafka and Elasticsearch. Therefore, to enable a time-based analysis of the cluster image security, the VA components should be backed up.
This means that the logging data also needs to be backed up to ensure VA data is available upon restoring VA functionality. The scans and reports are generated based on the current environment, so restoring this data to a new cluster means that the restored data is no longer meaningful, and depending on the requirements for VA reporting, it may not be necessary to restore the backup data at all.
Mutation Advisor
Mutation Advisor (MA) is an optional management service (installed with Vulnerability Advisor) that actively scans running containers for any changes in system files, configuration files, content files, or OS process within the container. Time-based reports are generated and stored in the same way as Vulnerability Advisor, and therefore the same conditions apply.
Application persistent volume data
Persistent volume data mounted by user deployed workloads can be hosted on a variety of storage solutions, and cluster administrators should ensure that the right tools/methods are used to backup the relevant storage platform accordingly. In a situation where a cluster becomes unrecoverable, and persistent volumes need to be recreated on a different cluster, it is the Cluster Administrators (or application owners, whichever is applicable) responsibility to know the data restoration process for the implemented storage solution(s).
3.4 Backup and restore strategy
The IBM Cloud Private backup and restore process described in this section is based on a best-effort basis. There are many different combinations of cluster configurations and user customizations that may impact how the whole infrastructure or individual platform components should be backed up, as well as different user preferences on exactly what data is considered ‘meaningful’ to restore.
Figure 3-1 shows each of the components to backup as mentioned earlier and their respective host.
Figure 3-1 Components to backup
The general strategy for successful IBM Cloud Private backup cycles is shown in Figure 3-2. It’s a typical iterative approach during operations where periodic backups are taken at either regular intervals (for example daily) or before maintenance.
Figure 3-2 General strategy for IBM Cloud Private backup cycles
The backup process needs to be carefully planned to minimise disruption to end users. The following sections will explain the use of several options for backing up and restoring both infrastructure and platform components to allow cluster administrators to apply the most efficient process for their environment.
3.4.1 Infrastructure backup process
It is recommended to take an infrastructure level snapshot immediately after a successful installation of IBM Cloud Private, where there are no user deployed workloads running. This ensures that a clean cluster installation is always available in the event of failure and to reduce the time required to restore a working cluster.
Stopping an IBM Cloud Private node
To gracefully stop an IBM Cloud Private node the recommended approach is to stop kubelet and docker on the node first, then shut down the node. This ensures a clean shut down for Docker. For worker nodes containing user deployments that should be kept available, it may be necessary to drain the node first, using kubectl drain <node>. It’s important to stop the Kubelet process prior to stopping Docker, because Kubelet will continue to run, attempting to access the containers that were previously running on the host:
1. First, stop Kubelet:
sudo systemctl stop kubelet
2. With Kubelet stopped, stop Docker:
sudo systemctl stop docker
3. Stopping Docker may take some time. When Docker is stopped (dockerd should no longer be running), shut down the node:
shutdown now
More information about taking nodes offline can be found in the IBM Knowledge Center:
Hot backups of an IBM Cloud Private cluster
Depending on the virtualization platform hosting the cluster nodes, it is possible to take a running (hot) backup of each IBM Cloud Private node and later use it for restoration. As hot backups can be taken while nodes are running, it is recommended to backup the nodes simultaneously, if possible, so that the cluster can be restored in a unified state in the event of failure. Hot backups are generally seen as a short term point-in-time snapshot and should not be used as the sole method of backing up cluster nodes.
Cold backups of an IBM Cloud Private cluster
When taking a cold backup of an IBM Cloud Private topology, shutting down the whole cluster at the same time is not required, but it’s important to consider the order in which nodes are shut down. The master nodes contain core processes such as etcd, the Kubernetes api-server, controller manager and scheduler that will attempt to recover pods that have ‘failed’ due to the host becoming unavailable.
This recovery effort is normal behavior for a running system, but is not considered an organized state that provides a stable backup of the cluster. Therefore, stopping the master nodes first will provide the most stable backup to recover from.
The most stable approach, when taking cold backups of an IBM Cloud Private cluster, is to shut down all master nodes (and etcd nodes, if they are separate from the masters) at the same time, take a backup, then start up the node again. This ensures that all the core components are backed up in exactly the same state. However, the drawback to this approach is that the cluster is temporarily offline while backups are taken, and therefore is not suitable for production installation beyond the point of initial installation.
The recommended time to use this approach is immediately after installation with a stable cluster, so that it’s possible to quickly revert back to a fresh installation if required. Generally, backups should be taken of all nodes after installation but proxy and worker nodes are easily replaced and only need backups if the workload data resides on the node itself. Management and Vulnerability Advisor nodes are also easily replaced, but many of the management services write data directly to the host filesystem, so at a minimum the management node filesystems should have a backup.
In a HA installation, it is possible that the cluster can be kept in a running state by taking node backups one by one. This ensures that the cluster is still available for end users, and also provides a stable state for etcd. It’s crucial that etcd is kept in a stable state and still able to serve requests, as without a healthy etcd cluster, neither Kubernetes or IBM Cloud Private will function correctly. In topologies where etcd is separated from the master nodes, backups must also be in a similar fashion to ensure etcd is always kept in a stable state.
However, there is a caveat to this process; if too much time has passed in between taking a cold backup of each node, the risk of the etcd database becoming too inconsistent is increased. Furthermore, in a very active cluster, there may have been too many write operations performed on a particular etcd instance, which means etcd will not be able to properly recover.
The cluster administrator needs to consider the use of an active-active style architecture, as discussed earlier, if they require as close to zero-downtime as possible for running applications that rely on the cluster (or management) services on IBM Cloud Private (such as the api-server, or Istio). In most cases, workloads running on worker nodes (using the IBM Cloud Private proxy nodes for access to the applications) are not affected by master node downtime unless there is an event on the application host node that requires attention from the core IBM Cloud Private components whilst the master nodes are offline.
Based on the 1.3, “IBM Cloud Private architecture” on page 10, the following nodes contain core components that persist data and should backed up regularly
•etcd/master
•management (optional but recommended)
•Vulnerability Advisor (optional but recommended)
The other nodes not mentioned in the list above (proxy and worker nodes) are generally replaceable without consequence to the health of the cluster. Information on replacing various IBM Cloud Private nodes can be found in the IBM Knowledge Center documentation:
3.4.2 Infrastructure restore process
Care should be taken when restoring IBM Cloud Private nodes. It’s important to ensure that the backup process is followed carefully to maximise the chance of a successful cluster restore and attempting to restore a bad backup will almost certainly result in an unstable cluster. The etcd component carries the greatest weight to keep Kubernetes running and should always be validated immediately after restoration to validate the cluster integrity.
There are several other crucial components that also require quorum, such as MongoDB and MariaDB, but as they are Kubernetes resources it is always possible to recover it (assuming the host filesystem is intact. If etcd is permanently failed, so is the whole cluster, so the main focus in this section is retaining a healthy etcd cluster. The key thing to understand when restoring infrastructure backups is that etcd will rebuild it’s cluster state using the latest available data on the current “leader”, and the elected leader will have the highest raft index.
When the etcd/master nodes are brought back online, the etcd instances will continue to operate where they left off at the point in time where the backup was taken, requesting new elections and updating the current raft term. If any other etcd members making requests to the cluster (becoming a ‘follower’) have a different raft index that the leader does not recognise (for example a different copy of the data) and an inconsistent raft term then it will no longer be a member of the cluster.
Consider the following example: In a HA cluster with 3 masters, a cold backup is taken sequentially starting from master 1, then 2, then 3. Each master is powered off (using the recommended approach in the previous sections), backed up, then powered on again. During this time period, master 2 and 3 are still running, and the cluster is still fully operational for end users. When restoring, the same order is used so master 1 is restored first, then 2, then 3.
The issue here is that each backup for the master nodes is taken at a completely different time, with master 3 being the last. This means that master 3 will have different data when restored, and when brought online last it will attempt to communicate with the existing cluster. Therefore, it’s essential to ensure that in the event of a cluster failure, and all 3 masters need to be restored from a backup, that the order of restoration is the reverse of the order of backup. This process applies to restoring both hot and cold backups.
Verifying a restored backup
Restoring a cluster entirely depends on the hosting virtualization platform. The examples in this section uses VMWare to host the IBM Cloud Private installation, and therefore uses the Virtual Machine (VM) snapshot capabilities provided with the platform. Note that a VMWare snapshot is not a complete backup mechanism and is used here to demonstrate the practical application of the backup methods described in this chapter.
Using the previous example of a HA cluster, master nodes 1, 2, and 3 were backed up using VMWare snapshots while the machines were offline (cold snapshot) in sequence starting from master 1. Nginx workloads were deployed to simulate user deployments so that the cluster state in etcd was changed. To simulate a disaster, masters 1 and 2 were destroyed. At this point etcd is now running as a single node cluster, in read-only mode and cannot accept any write requests from Kubernetes.
The CoreOS documentation (found at https://coreos.com/etcd/docs/latest/op-guide/failures.html) provides a good explanation about different etcd failure scenarios and what it can or can’t tolerate. In this situation, quorum is lost and as you cannot currently replace IBM Cloud Private master nodes using different IP addresses, you cannot recover. Therefore, the snapshots must be used to deploy masters 1 and 2, and restore all 3 masters to a previously working state.
Restore all 3 masters to a previous snapshot, power on the nodes and allow some time for all the containers to start. To ensure the etcd cluster has started successfully and the cluster is healthy, etcd provides a useful command line utility to query the cluster status, which can be downloaded to your local machine or ran from the etcd container itself. To get the cluster health status, complete the following steps from the IBM Cloud Private boot node (or whatever system has cluster access with kubectl):
1. Run kubectl -n kube-system get pods | grep k8s-etcd to retrieve the etc pod name (the format is k8s-etcd-<node-ip>):
[root@icp-ha-boot ~]# kubectl -n kube-system get pods -o name| grep k8s-etcd
k8s-etcd-172.24.19.201
k8s-etcd-172.24.19.202
k8s-etcd-172.24.19.203
2. Use any one of the pod names returned to execute an etcdctl cluster-health command, using one of the master node IP addresses in the endpoint parameter:
[root@icp-ha-boot ~]# kubectl -n kube-system exec k8s-etcd-172.24.19.201 -- sh -c "export ETCDCTL_API=2; etcdctl --cert-file /etc/cfc/conf/etcd/client.pem --key-file /etc/cfc/conf/etcd/client-key.pem --ca-file /etc/cfc/conf/etcd/ca.pem --endpoints https://172.24.19.201:4001 cluster-health"
member 8a2d3ec6df19666f is healthy: got healthy result from https://172.24.19.201:4001
member ae708d12aa012fdc is healthy: got healthy result from https://172.24.19.202:4001
member dd2afb46d331fdd2 is healthy: got healthy result from https://172.24.19.203:4001
cluster is healthy
This outputs the cluster state for all etcd endpoints, and will either state cluster is healthy, or cluster is unhealthy as a final status. If the output is cluster is healthy, then restoring etcd was successful. If the output is cluster is unhealthy, then further investigation is needed.
It’s also important to check the other core components, such as MongoDB and MariaDB to ensure they are running correctly. See the troubleshooting section for more information about verifying a successful IBM Cloud Private cluster state.
Further testing
This section provides some real-world testing of hot and cold backups where a cluster is under heavier stress from continuous workload deployment. The testing in this section was conducted in a VMWare lab environment using VMWare snapshot capabilities and does not represent a real customer deployment, it is only intended for simulation purposes.
To test the backup process resiliency for master nodes, the following test plan was used, as shown in Figure 3-3.
Figure 3-3 VMWare snapshot simulation action list
This test plan took “hot” and “cold” snapshots of nodes at different times, In items 1 to 18, there was a minimal workload in the cluster, but this does not simulate an active cluster. At item 19, heavy workload was applied to the cluster, by deploying a random number of replicas of Nginx containers to different namespaces every 10 seconds.
Prior to taking backups, the etcd cluster was queried to retrieve information about the current database size, current raft term and current raft index. Below shows the output of an etcdctl endpoint status command.
[root@icp-ha-boot ~]# docker run --entrypoint=etcdctl -e ETCDCTL_API=3 -v /etc/cfc/conf/etcd:/certs -v /var/lib/etcd:/var/lib/etcd -v /tmp:/data mycluster.icp:8500/ibmcom/etcd:3.2.24 --cert=/certs/client.pem --key=/certs/client-key.pem --cacert=/certs/ca.pem --endpoints https://172.24.19.201:4001,https://172.24.19.202:4001,https://172.24.19.203:4001 -w table endpoint status
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://172.24.19.201:4001 | 8a2d3ec6df19666f | 3.2.24 | 30 MB | false | 1617 | 1879006 |
| https://172.24.19.202:4001 | ae708d12aa012fdc | 3.2.24 | 31 MB | true | 1617 | 1879006 |
| https://172.24.19.203:4001 | dd2afb46d331fdd2 | 3.2.24 | 31 MB | false | 1617 | 1879006 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
After 30 seconds, the same query was run to analyze the changes in data.
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://172.24.19.201:4001 | 8a2d3ec6df19666f | 3.2.24 | 30 MB | false | 1617 | 1879291 |
| https://172.24.19.202:4001 | ae708d12aa012fdc | 3.2.24 | 31 MB | true | 1617 | 1879292 |
| https://172.24.19.203:4001 | dd2afb46d331fdd2 | 3.2.24 | 31 MB | false | 1617 | 1879293 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
The leader and raft term is still the same, so there is no issues around frequent elections, but the raft index has changed, which represents changes in the data.
Master 1 was taken offline, snapshot, then left offline for some time to simulate maintenance. After 10 minutes, master 1was brought back online, and the process repeated for master 2 and 3. Once all masters were available and running, the etcd status now shows the following.
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://172.24.19.201:4001 | 8a2d3ec6df19666f | 3.2.24 | 43 MB | true | 1619 | 1931389 |
| https://172.24.19.202:4001 | ae708d12aa012fdc | 3.2.24 | 43 MB | false | 1619 | 1931391 |
| https://172.24.19.203:4001 | dd2afb46d331fdd2 | 3.2.24 | 43 MB | false | 1619 | 1931395 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
This maintenance simulation process took place over 47 minutes, providing a realistic test for restoring the snapshots to a point in time before maintenance took place on the masters.
Prior to restoring the nodes, the endpoint status resembles the following.
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://172.24.19.201:4001 | 8a2d3ec6df19666f | 3.2.24 | 46 MB | true | 1619 | 1945597 |
| https://172.24.19.202:4001 | ae708d12aa012fdc | 3.2.24 | 46 MB | false | 1619 | 1945597 |
| https://172.24.19.203:4001 | dd2afb46d331fdd2 | 3.2.24 | 46 MB | false | 1619 | 1945597 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
To verify that restoring the masters in the original sequential order yields an inconsistent cluster, the masters were brought online starting from master 1. The endpoint status shows the following.
Failed to get the status of endpoint https://172.24.19.203:4001 (context deadline exceeded)
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://172.24.19.201:4001 | 8a2d3ec6df19666f | 3.2.24 | 44 MB | false | 1638 | 1927603 |
| https://172.24.19.202:4001 | ae708d12aa012fdc | 3.2.24 | 43 MB | true | 1638 | 1927603 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
This test was repeated 5 times, of which every test produced consistent results; the third master could not join the cluster.
Upon restoring the nodes in reverse order (from master 3 to 1), the endpoint status resembles the following.
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://172.24.19.201:4001 | 8a2d3ec6df19666f | 3.2.24 | 43 MB | false | 1661 | 1928697 |
| https://172.24.19.202:4001 | ae708d12aa012fdc | 3.2.24 | 43 MB | false | 1661 | 1928697 |
| https://172.24.19.203:4001 | dd2afb46d331fdd2 | 3.2.24 | 43 MB | true | 1661 | 1928697 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
This test was repeated 5 times, of which every test produced consistent results; the etcd cluster recovered successfully.
Other combinations were also tested. The below tables show the results per varying combinations of restoration order for both hot and cold snapshots.
Figure 3-4 Tables showing simple test results for hot and cold restores
The results above reiterate the need to restore cold backups in reverse order. Restoring backups in any other order yielded unpredictable results, and in some cases, an unstable cluster. Hot backups were much more effective at restoring the original cluster state a lot quicker (less time to start up services as memory was preserved). Hot snapshots were taken within a much closer time frame, which means etcd had a very high chance of recovering from minor inconsistencies between instances.
In most cases, hot snapshots are not used as the sole backup method and should be used appropriately as documented by the platform provider. However, based on the above results they can be reliable to quickly recover from fatal changes to the cluster.
3.4.3 Platform backup process
As described in section 3.2, the recommended approach is to regularly backup the individual components of an IBM Cloud Private cluster. More frequent backups mean that a cluster administrator always has a recent backup of all the core components to successfully restore a cluster in the event of a failure, without reverting to a previous infrastructure backup. This section will describe the steps required to back up the various components to external storage using the tools available in IBM Cloud Private.
It’s advised that the core components are backed up in the following order:
1. etcd
2. MongoDB
3. MariaDB (Optional)
This is not a strict backup order, just an advised order. MongoDB persists Team and User data, and etcd persists the relevant Role Base Access Control (RBAC) Kubernetes resources associated to those Teams and users. If you were to back up etcd first, add a new Team with Users, then back up MongoDB, you would end up with out of sync data as etcd would not contain the data to restore the RBAC Kubernetes resources for the new Team.
Alternatively, if you backup MongoDB first, add a new Team and Users, then back up etcd, data would still be out of sync as MongoDB would not contain the Team and User data upon restore. Based on this, the backup order is entirely the choice of the Cluster Administrator, but following the approach in the infrastructure backup sections this chapter takes a backup of etcd first and everything else after. Ideally, both etcd and MongoDB should be backed up within a close time frame, at a period of time where cluster activity is low.
The non-core components can be backed up in any order. When restoring the Vulnerability Advisor, it’s recommended to restore full functionality to the Elasticsearch stack first, as Vulnerability Advisor and Mutation Advisor both rely on a healthy Elasticsearch to store and retrieve data.
The recommended approach is to regularly back up these core components by using a CronJob. This creates an instance of a Kubernetes job on a time-based schedule, allowing users to execute jobs at any time using the Unix crontab format. In IBM Cloud Private, CronJobs can be used to run etcd and MongoDB backups periodically, using a PersistentVolume so the data can be stored on an external storage provider. This has the major benefit of keeping the latest backups, with no cluster down time and no heavy storage implications, which would normally be the case for full infrastructure backups.
For all of the components that require a PersistentVolume (PV) and PersistentVolumeClaim (PVC), the same PV for an NFS server is used. If the backups should be segregated on different volumes, create additional PVs and PVCs for each component.
Creating the core-backup PersistentVolume and PersistentVolumeClaim
Create a new PersistentVolume (PV) and PersistentVolumeClaim (PVC) to use to store the backups. Use Example 3-1 as a template for a new NFS PV, and Example 3-2 as a template for an NFS PVC.
Example 3-1 YAML definition for core-backup PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: core-backup
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 200Gi
nfs:
path: /backup
server: 198.0.0.1
persistentVolumeReclaimPolicy: Retain
Example 3-2 YAML definition for core-backup PV
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: core-backup
namespace: kube-system
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 200Gi
storageClassName: ""
The use of accessModes: ReadWriteMany here allows multiple containers to write to the volume simultaneously. Some storage types do not provide this capability and in such scenarios, one PV and PVC should be created per component. The storage capacity of 200Gi is used as an example and the real-world value depends on the size of the backups, and the duration the backup data is kept (which typically depends on internal customer requirements/policies). As this is reused among all components that require a PV and PVC, the directory structure is the following:
backup
├── etcd
├── logging
├── mariadb
└── mongodb
The volumes will be mounted to the appropriate directories on the NFS server.
etcd and MongoDB CronJob
This CronJob creates a job every day at 23:30pm to deploy two containers: one to snapshot etcd and one to export the current MongoDB database content. Each container has the same shared storage mounted (NFS in this example).
The easiest way to determine how much storage etcd and MongoDB require is to manually run a backup of etcd and MongoDB using kubectl and check the backup size. From this, the storage capacity required for the PVs can be calculated from the CronJob schedule. “Manual etcd and MongoDB backups” on page 89 provides more information on manual backups.
Complete the following steps to run a manual backup:
1. Copy the CronJob YAML definition in Example 3-3 on page 86 to your local machine and save to a file named core-backup_cronjob.yaml. For security and isolation from the worker nodes, the pods created in this job are scheduled to a master node.
Example 3-3 core-backup_cronjob.yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: core-backup
namespace: kube-system
spec:
schedule: "30 23 * * *"
jobTemplate:
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- ppc64le
- s390x
- key: master
operator: In
values:
- "true"
containers:
- name: etcddump
image: ibmcom/etcd:3.2.24
command: ["/bin/sh","-c","cat /certs/etcd-cert > client.pem; cat /certs/etcd-key > client-key.pem; cat /certs/etcd-ca > ca.pem; etcdctl --cert=client.pem --key=client-key.pem --cacert=ca.pem --endpoints https://$(ENDPOINT):4001 snapshot save /backup/etcd/etcd.$(date +%Y-%m-%d_%H:%M:%S).db"]
env:
- name: ENDPOINT
value: "#ETCD-ENDPOINT"
- name: ETCDCTL_API
value: "3"
volumeMounts:
- mountPath: "/backup"
name: backup
- mountPath: "/certs"
name: etcd-certs
- name: mongodbdump
image: ibmcom/icp-mongodb:4.0.5
command: ["/bin/bash","-c","cat /certs/tls.crt certs/tls.key > mongo.pem; export PRIMARY=$(mongo --host rs0/mongodb:27017 --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /ca/tls.crt --sslPEMKeyFile mongo.pem --eval="db.isMaster()['"'primary'"']" | grep ^icp-mongodb-); mongodump --host $PRIMARY --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /ca/tls.crt --sslPEMKeyFile mongo.pem --oplog --verbose --gzip --archive=/backup/mongodb/mongodb-backup-$(date +%Y-%m-%d_%H:%M:%S).gz"]
env:
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: icp-mongodb-admin
volumeMounts:
- mountPath: "/backup"
name: backup
- mountPath: "/ca"
name: cluster-ca
- mountPath: "/certs"
name: mongodb-certs
tolerations:
- effect: NoSchedule
key: dedicated
operator: Exists
volumes:
- name: backup
persistentVolumeClaim:
claimName: core-backup
- name: cluster-ca
secret:
secretName: cluster-ca-cert
- name: mongodb-certs
secret:
secretName: icp-mongodb-client-cert
- name: etcd-certs
secret:
secretName: etcd-secret
restartPolicy: Never
This file can also be downloaded from the following location:
2. The ETCD-ENDPOINT environment variable for the etcd container should be changed to reflect an actual etcd IP address. Modify the containers section in Example 3-4 and replace #ETCD-ENDPOINT with an appropriate etcd IP address.
Example 3-4 etcd container definition
containers:
- name: etcddump
image: ibmcom/etcd:3.2.24
command: ["/bin/sh","-c","cat /certs/etcd-cert > client.pem; cat /certs/etcd-key > client-key.pem; cat /certs/etcd-ca > ca.pem; etcdctl --cert=client.pem --key=client-key.pem --cacert=ca.pem --endpoints https://$(ENDPOINT):4001 snapshot save /data/etcd.$(date +%Y-%m-%d_%H:%M:%S).db"]
env:
- name: ENDPOINT
value: "#ETCD-ENDPOINT"
3. Alternatively, sed can be used to easily do this, by executing sed -i -e s/#ETCD-ENDPOINT/<etcd-endpoint>/g core-backup_cronjob.yaml replacing <etcd-endpoint> with your own.
CronJob schedules use the Unix crontab format, so adjust the spec.schedule value as required. For more information and guidance in generating a valid crontab schedule, see https://crontab.guru/. As an example, to run a job every 10 minutes, the schedule value would be */10 * * * *, for every hour 0 */1 * * *, and every Friday at 9pm 0 21 * * FRI.
4. The ibmcom/etcd:3.2.24 image is the default etcd image for IBM Cloud Private 3.1.2. If the environment does not have internet access, replace ibmcom with the private image registry local to the cluster, for example, mycluster.icp:8500/ibmcom/etcd:3.2.24.
5. Create the CronJob using kubectl create -f core-backup_cronjob.yaml. The CronJob should be visible in the Workloads → Jobs → CronJobs tab in IBM Cloud Private (Figure 3-5).
Figure 3-5 Running CronJob
6. All generated jobs (at the defined schedule) should be visible in the Workloads → Jobs → BatchJobs tab (Figure 3-6).
Figure 3-6 Running jobs
7. If a job is completed successfully, it will show a 1 in the Successful column. If the job was not successful, troubleshoot it using the guidance provided in the troubleshooting chapter. Verify that the backups exist on the storage provider (external NFS server in this example) as demonstrated in Example 3-5.
Example 3-5 Tree output of NFS backup directory
[root@icp-nfs backup]# tree --du -h
.
├── [87M] etcd
│ ├── [ 29M] etcd.2019-02-26_02:50:08.db
│ ├── [ 29M] etcd.2019-02-26_03:00:02.db
│ └── [ 29M] etcd.2019-02-26_03:10:05.db
└── [2.8M] mongodb
├── [966K] mongodb-backup-2019-02-26_02:50:09.gz
├── [967K] mongodb-backup-2019-02-26_03:00:03.gz
└── [967K] mongodb-backup-2019-02-26_03:10:06.gz
89M used in 2 directories, 6 files
This output also provides a baseline to calculate the necessary storage requirements. For example, if etcd data should be kept for 30 days and each backup is about ~30 Mb, the total needed for etcd backups is 900 Mb. Similarly for MongoDB, ~30 Mb is required for 30 days worth of backup data. Naturally, the backup size is likely to grow as the cluster is used over time, so ensure the backup sizes are monitored periodically and adjust the PV size as required.
Manual etcd and MongoDB backups
Manual backups of etcd and MongoDB can be taken in a variety of ways. This section will cover how to do this using Docker, kubectl, and Kubernetes jobs.
Backup etcd using Docker
Perform the following steps on an IBM Cloud Private master node to backup etcd using Docker. If etcd is running on dedicated nodes, perform the steps on that node.
1. Find the k8s-etcd container (Linux grep and awk tools are used to simplify the output):
[root@icp-ha-master-1 ~]# docker ps | grep k8s_etcd_k8s-etcd | awk 'FNR==1{print $1}'
af39b9392d85
2. Execute the snapshot command:
[root@icp-ha-master-1 ~]# docker run --entrypoint=etcdctl -e ETCDCTL_API=3 -v /tmp/backup:/backup -v /etc/cfc/conf/etcd:/certs -v /var/lib/etcd:/var/lib/etcd mycluster.icp:8500/ibmcom/etcd:3.2.24 --cert /certs/client.pem --key /certs/client-key.pem --cacert /certs/ca.pem --endpoints https://172.24.19.201:4001 snapshot save /backup/snapshot.db
Snapshot saved at /backup/snapshot.db
3. Verify the backup exists:
[root@icp-ha-master-1 ~]# ls -sh /tmp/backup/
total 30M
30M snapshot.db
4. Copy the snapshot to a backup storage location outside of the cluster.
Backup etcd using kubectl
Perform the following steps on any machine that has kubectl configured to access to the cluster:
1. Get the etcd pod name:
[root@icp-ha-boot ~]# kubectl -n kube-system get pods | grep "k8s-etcd" | awk 'NR==1{print $1}'
k8s-etcd-172.24.19.201
2. Run the snapshot command:
[root@icp-ha-boot ~]# kubectl -n kube-system exec k8s-etcd-172.24.19.201 -- sh -c "export ETCDCTL_API=3; etcdctl --cert /etc/cfc/conf/etcd/client.pem --key /etc/cfc/conf/etcd/client-key.pem --cacert /etc/cfc/conf/etcd/ca.pem --endpoints https://172.24.19.201:4001 snapshot save snapshot.db"
Snapshot saved at snapshot.db
3. Copy the snapshot.db file from the etcd container to the local machine:
[root@icp-ha-boot ~]# kubectl cp kube-system/k8s-etcd-172.24.19.201:snapshot.db .
4. Verify the backup exists:
[root@icp-ha-boot ~]# ls -sh .
total 30M
30M snapshot.db
Backup etcd using a Kubernetes job
etcd can be snapshot to a PersistentVolume using a Kubernetes job. For security and isolation from the worker nodes, the pods created in this job are scheduled to a master node. Perform the following steps on any machine that has kubectl configured to access to the cluster:
1. Create a PersistentVolume (PV) and PersistentVolumeClaim (PVC), similar to the examples in Example 3-1 on page 84 and Example 3-2 on page 85 to store the etcd snapshot. This job definition uses the same PV and PVC created in those examples.
2. Create the etcd-backup_job.yaml file shown in Example 3-6. Alternatively, download the file from the following location:
Example 3-6 The etcd-backup_job.yaml file
apiVersion: batch/v1
kind: Job
metadata:
name: etcddump
namespace: kube-system
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- ppc64le
- s390x
- key: master
operator: In
values:
- "true"
containers:
- name: etcddump
image: ibmcom/etcd:3.2.24
command: ["/bin/sh","-c","cat /certs/etcd-cert > client.pem; cat /certs/etcd-key > client-key.pem; cat /certs/etcd-ca > ca.pem; etcdctl --cert=client.pem --key=client-key.pem --cacert=ca.pem --endpoints https://$(ENDPOINT):4001 snapshot save /backup/etcd/etcddump.$(date +%Y-%m-%d_%H:%M:%S).db"]
env:
- name: ENDPOINT
value: "#ETCD-ENDPOINT"
- name: ETCDCTL_API
value: "3"
volumeMounts:
- mountPath: "/backup"
name: backup
- mountPath: "/certs"
name: etcd-certs
tolerations:
- effect: NoSchedule
key: dedicated
operator: Exists
volumes:
- name: backup
persistentVolumeClaim:
claimName: core-backup
- name: etcd-certs
secret:
secretName: etcd-secret
restartPolicy: Never
backoffLimit: 1
The ETCD-ENDPOINT environment variable for the etcd container should be changed to reflect an actual etcd IP address. Modify the containers section in Example 3-7 and replace #ETCD-ENDPOINT with an appropriate etcd IP address.
Example 3-7 etcd container definition
containers:
- name: etcddump
image: ibmcom/etcd:3.2.24
command: ["/bin/sh","-c","cat /certs/etcd-cert > client.pem; cat /certs/etcd-key > client-key.pem; cat /certs/etcd-ca > ca.pem; etcdctl --cert=client.pem --key=client-key.pem --cacert=ca.pem --endpoints https://$(ENDPOINT):4001 snapshot save /data/etcd.$(date +%Y-%m-%d_%H:%M:%S).db"]
env:
- name: ENDPOINT
value: "#ETCD-ENDPOINT"
Alternatively, sed can be used to easily do this, by executing sed -i -e s/#ETCD-ENDPOINT/<etcd-endpoint>/g core-backup_cronjob.yaml replacing <etcd-endpoint> with your own.
The ibmcom/etcd:3.2.24 image is the default etcd image for IBM Cloud Private 3.1.2. If the environment does not have internet access, replace ibmcom with the private image registry local to the cluster, for example, mycluster.icp:8500/ibmcom/etcd:3.2.24.
3. Create the job:
[root@icp-ha-boot ~]# kubectl create -f etcd-backup_job.yaml
job.batch "etcddump" created
4. Verify that the job was successful:
[root@icp-ha-boot ~]# kubectl -n kube-system get jobs etcddump
NAME DESIRED SUCCESSFUL AGE
etcddump 1 1 1m
5. Verify that the etcd snapshot exists:
[root@icp-nfs etcd]# ls -sh etcddump*
30M etcddump.2019-02-26_04:49:06.db
Backup MongoDB using kubectl
Perform the following steps on any machine that has kubectl configured to access to the cluster:
1. Get the MongoDB pod name:
[root@icp-ha-boot ~]# kubectl -n kube-system get pods -l app=icp-mongodb -o=jsonpath='{.items[0].metadata.name}'
icp-mongodb-0
2. Retrieve the primary replica:
[root@icp-ha-boot ~]# kubectl -n kube-system exec icp-mongodb-0 -n kube-system -- sh -c 'mongo --host rs0/mongodb:27017 --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /data/configdb/tls.crt --sslPEMKeyFile /work-dir/mongo.pem --eval="db.isMaster()["primary"]" | grep ^icp-mongodb-'
icp-mongodb-1.icp-mongodb.kube-system.svc.cluster.local:27017
[root@icp-ha-boot ~]# kubectl -n kube-system exec icp-mongodb-0 -- sh -c 'mkdir -p /work-dir/backup/; mongodump --host icp-mongodb-1.icp-mongodb.kube-system.svc.cluster.local:27017 --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /data/configdb/tls.crt --sslPEMKeyFile /work-dir/mongo.pem --oplog --quiet --gzip --archive="/work-dir/backup/backup.gz"'
4. Copy the backup.gz file from the icp-mongodb-0 container to the local machine:
[root@icp-ha-boot ~]# kubectl cp kube-system/icp-mongodb-0:/work-dir/backup/backup.gz .
5. Verify that the backup exists:
[root@icp-ha-boot ~]# ls -sh .
total 972K
972K backup.gz
Backup MongoDB using a Kubernetes job
MongoDB can be backed up to a PersistentVolume using a Kubernetes job. For security and isolation from the worker nodes, the pods created in this job are scheduled to a master node. Perform the following steps on any machine that has kubectl configured to access to the cluster:
1. Create a PersistentVolume (PV) and PersistentVolumeClaim (PV), similar to the examples in Example 3-1 and Example 3-2 to store the backup file. This job definition uses the same PV and PVC created in those examples.
Example 3-8 mongodb-backup_job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: mongodbdump
namespace: kube-system
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- ppc64le
- s390x
- key: master
operator: In
values:
- "true"
containers:
- name: mongodbdump
image: ibmcom/icp-mongodb:4.0.5
command: ["/bin/bash","-c","cat /certs/tls.crt certs/tls.key > mongo.pem; export PRIMARY=$(mongo --host rs0/mongodb:27017 --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /ca/tls.crt --sslPEMKeyFile mongo.pem --eval="db.isMaster()['"'primary'"']" | grep ^icp-mongodb-); mongodump --host $PRIMARY --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /ca/tls.crt --sslPEMKeyFile mongo.pem --oplog --verbose --gzip --archive=/backup/mongodb/mongodbdump-$(date +%Y-%m-%d_%H:%M:%S).gz"]
env:
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: icp-mongodb-admin
volumeMounts:
- mountPath: "/backup"
name: backup
- mountPath: "/ca"
name: cluster-ca
- mountPath: "/certs"
name: mongodb-certs
tolerations:
- effect: NoSchedule
key: dedicated
operator: Exists
volumes:
- name: backup
persistentVolumeClaim:
claimName: core-backup
- name: cluster-ca
secret:
secretName: cluster-ca-cert
- name: mongodb-certs
secret:
secretName: icp-mongodb-client-cert
restartPolicy: Never
backoffLimit: 1
Alternatively, download the file from the following link:
The ibmcom/icp-mongodb:4.0.5 image is the default MongoDB image for IBM Cloud Private 3.1.2. If the environment does not have internet access, replace ibmcom with the private image registry local to the cluster, for example: mycluster.icp:8500/ibmcom/icp-mongodb:4.0.5.
3. Create the job:
[root@icp-ha-boot ~]# kubectl create -f mongodb-backup_job.yaml
job.batch "mongodbdump" created
4. Verify that the job was successful:
[root@icp-ha-boot ~]# kubectl get jobs mongodbdump
NAME DESIRED SUCCESSFUL AGE
mongodbdump 1 1 3m
5. Verify that the etcd snapshot exists:
[root@lf-icp-nfs mongodb]# ls -sh mongodbdump*
976K mongodbdump-2019-02-26_05:58:22.gz
Backing up MariaDB
MariaDB in IBM Cloud Private is only used for storing the transient OpenIDConnect data. The default refresh period for user tokens is 12 hours. If this data is lost between restores, the data can simply be regenerated by logging in to the platform. If there is a use case to retain the MariaDB data, you can back up the database using kubectl or a Kubernetes job.
Backing up MariaDB using kubectl
Perform the following steps on any machine that has kubectl configured to access to the cluster:
1. Get the MariaDB pod name:
[root@icp-ha-boot ~]# kubectl -n kube-system get pods -l release=mariadb -o=jsonpath='{.items[0].metadata.name}'
mariadb-0
2. Run the mysql command:
[root@icp-ha-boot ~]# kubectl -n kube-system exec -it mariadb-0 -c mariadb -- sh -c 'mysqldump --host=$MARIADB_SERVICE_HOST --user=root --password=$MYSQL_ROOT_PASSWORD --all-databases > /backup.sql'
3. Copy the backup file from the mariadb-0 container to the local machine:
[root@icp-ha-boot ~]# kubectl cp kube-system/mariadb-0:backup.sql -c mariadb .
4. Move the backup.sql to a location outside of the cluster.
Backing up MariaDB using a Kubernetes job
The mariadb-backup_job.yaml and mariadb-backup_cronjob.yaml can be retrieved here:
Run the job in the same way as the etcd and MongoDB jobs.
Backing up the private image registry
The private image registry stores all system and user deployed images in the /var/lib/registry directory on the master nodes. When backing up this data, it is sufficient to create an archive of the data using the OS tools available, such as tar. The loss of this data means losing all users images which will result in failed deployments when the cluster is restored. You can copy the whole contents of /var/lib/registry and move to a location outside of the cluster.
When later restoring a cluster, the system images will already exist after installation, so it does not always make sense to back up the whole /var/lib/registry directory unless it is easier to do so. It’s recommended to also store the application images in a location external to the cluster, so that each individual image can be pushed to the image repository if needed.
To back up an individual image, you can use commands such as docker save -o <backup-name>.tar <repo>/<namespace>/<image-name>:<tag>
For example, to back up a user deployed python image:
docker save -o python-image.tar mycluster.icp:8500/development/python:v1.0.0
Backing up the Helm repository
All Helm charts pushed to the IBM Cloud Private Helm repository are stored in MongoDB, and the deployable chart data can be located on the master nodes. Due to this method it’s not mandatory to back up the chart archives from the master nodes, but it’s recommended to do so in the event of a MongoDB failure, resulting in data loss.
Only one helm-repo pod is running at one time and is scheduled to a master node, using a LocalVolume PersistentVolume. This means that the most up to date chart information from MongoDB is stored on the master node hosting the helm-repo pod, so chart backups should be retrieved from that node.
Backing up the contents of the /var/lib/icp/helmrepo directory on the master node hosting the helm-repo pod is sufficient. Example 3-9 shows how to retrieve the host node name or IP address using kubectl.
Example 3-9 Retrieve helm-repo host IP
[root@icp-ha-boot cluster]# kubectl -n kube-system get pods -l app=helm-repo -o=custom-columns=NAME:.metadata.name,NODE:.spec.nodeName
NAME NODE
helm-repo-866cd5f9bd-c7cxt 172.24.19.203
Use your preferred methods to retrieve the contents of /var/lib/icp/helmrepo from the master node and store the data outside of the cluster.
Backing up the Elasticsearch cluster
As the Elasticsearch, Logstash and Kibana (ELK) stack provides users with the ability to retrieve historical log data it is an important component to backup so the same data can be restored in the event of cluster failure. The current logging release does not provide the capability to backup the logging data without suspending the logging service (only partially for installations with multiple management node).
General considerations
By default, IBM Cloud Private configures the ELK stack to retain log data using the logstash index for one day, and in this configuration, it is worth considering whether or not the platform log data is actually meaningful enough to keep. In most cases, the platform log data is transient, especially if the default curation is kept and logs are removed after 24 hours.
Backing up log data is only really valuable if the default 24 hours is extended to a longer duration and where the platform ELK stack is the primary source of application log data. The Logging and Monitoring Chapter provides information on the various use cases for platform and application logging and the alternative approaches to consider, such as deploying a dedicated ELK for applications.
Backing up the logging filesystem
This method requires downtime of management nodes to be able to accurately backup the logging data. This is because Elasticsearch is constantly reading/writing data to the filesystem at /var/lib/icp/logging on the management nodes especially when the cluster is active. Attempting to copy the logging data from the filesystem whilst the Elasticsearch cluster is online has a high chance of creating a faulty backup and some shards will be corrupted upon restoration, resulting in data loss.
The recommended way to backup the logging file system is to ensure Elasticsearch is not running on the management node by stopping all containers running on it. Use the method described in “Stopping an IBM Cloud Private node” on page 78 to stop kubelet and docker, then the /var/lib/icp/logging/elk-data/nodes/0/indices directory can be safely copied. During this time, no log data generated by the platform or application will be persisted.
In environments with more than one management node, multiple Elasticsearch data pods are deployed (one on each management node), so it is possible to keep the logging services running by repeating the above approach on each management node one by one. During this time, Elasticsearch will persist the data only to the available data pods, which means that there will be an imbalance of spread of replicas per shard on each management node.
Elasticsearch will attempt to correct this as data pods are brought back online so higher CPU utilization and disk I/O is normal in this situation. More information about how replica shards are stored and promoted when data pods are taken offline can be found here:
Backing up the monitoring data
The monitoring stack comprises of the Alert Manager, Prometheus and Grafana. By default, IBM Cloud Private does not deploy these components with a PersistentVolume (PV) or PersistentVolumeClaim (PVC), so any data stored by these components will be deleted if the container gets restarted. The platform does, however, store any user-configured AlertRules and MonitoringDashboard resources within Kubernetes itself, so these will persist during container restarts.
Alerts and Dashboards
If you have created custom alert rules or Grafana dashboards and modified them in Prometheus or Grafana directly, you should export the YAML definitions to a file and store these resources outside of the cluster, as part of your backup routine.
To export an AlertRule to a file, use the command kubectl -n <namespace> get alertrule <name> > <name>.yaml replacing <namespace> with the hosting namespace and <name> with the AlertRule name.
To export a Dashboard to a file, use the command kubectl -n <namespace> get monitoringdashboard <name> > <name>.yaml replacing <namespace> with the hosting namespace and <name> with the Dashboard name.
Adding persistence to the monitoring stack
The recommended approach to persisting data in the monitoring stack is by adding a PV and PVC to the deployments using helm upgrade. The following steps will create a new PV and PVC and mount them to the Alert Manager, Prometheus and Grafana deployments.
|
Important: These steps will force the containers to be restarted in order to mount the new volume, so any existing data stored in the containers will no longer be available. AlertRules and Dashboards are stored as Kubernetes resources and can be exported, but current monitoring data in Prometheus or additional configurations from modifying ConfigMaps directly may be lost. Be sure to export your additional configurations prior to executing these steps.
|
1. If you do not have dynamic storage provisioning, create a separate PV and PVC for the Alert Manager (1 Gb), Prometheus (10 Gb) and Grafana (10 Gb). Examples of PVs and PVCs can be found throughout this chapter and the Kubernetes documentation at https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistent-volumes.
2. Retrieve the current monitoring Helm chart and the release values, replacing mycluster.icp with a value suitable for your environment:
wget --no-check-certificate https://mycluster.icp:8443/mgmt-repo/requiredAssets/ibm-icpmonitoring-1.4.0.tgz
3. Create a new YAML file called monitoring-override.yaml containing the PV definition in Example 3-10.
Example 3-10 monitoring-override.yaml
prometheus:
persistentVolume:
enabled: true
useDynamicProvisioning: false
size: 10Gi
storageClass: ""
existingClaimName: "monitoring-prometheus"
selector:
label: ""
value: ""
alertmanager:
persistentVolume:
enabled: true
useDynamicProvisioning: false
size: 1Gi
storageClass: ""
existingClaimName: "monitoring-alertmanager"
selector:
label: ""
value: ""
grafana:
persistentVolume:
enabled: true
useDynamicProvisioning: false
size: 1Gi
storageClass: ""
existingClaimName: "monitoring-grafana"
selector:
label: ""
value: ""
Replace the values for useDynamicProvisioning and storageClass to suit your environment.
4. Use helm upgrade to apply the changes to the monitoring release:
helm upgrade monitoring ibm-icpmonitoring-1.4.0.tgz -f monitoring-override.yaml --reuse-values --recreate-pods --tls
The data stored by the monitoring stack is portable and can be reused in different instances, so use the storage backup tools available to backup the PV data. If the monitoring data may be restored on a different cluster, you need to ensure that the configuration is the same (for example the Grafana ConfigMap is the same) for the data to be reused in a new deployment of Alert Manager, Prometheus or Grafana.
Using your preferred methods, copy the PV data to a location outside of the IBM Cloud Private cluster.
Backing up Vulnerability Advisor
As discussed in the section “Vulnerability Advisor” on page 76, consider whether it is worth backing up Vulnerability Advisor (VA) and Mutation Advisor (MA) at all. Imported data in to a new cluster will mean that the restored VA and MA data is no longer relevant to the current environment and would not offer any benefit other than analysis of the previous environment. If backing up the VA and MA data is required then this section is relevant.
VA and MA store all its data in two places; on the hosts filesystem and in Elasticsearch. The data stored by VA and MA is not configured to a specific host so data can be transported between installations of IBM Cloud Private. The VA and MA filesystems that require backup are as follows:
•/var/lib/icp/va/minio
•/var/lib/icp/va/zookeeper
•/var/lib/icp/va/kafka
In high availability (HA) deployments, the /var/lib/icp/va/minio is on shared storage so back this up using the preferred methods.
Copy this data to a location outside of the IBM Cloud Private cluster.
Backing up the cluster installation data
Depending on how an IBM Cloud Private cluster is restored, it may be necessary to copy the cluster installation data. If IBM Cloud Private needs to be reinstalled on the existing nodes with the same IP addresses, copy the following data in the <installation-directory>/cluster directory on the boot node.
•cfc-certs (directory)
•config.yaml (optional)
•hosts (optional)
•ssh_key (optional)
The most important data to be backed up is the cfc-certs directory, as it contains the certificates and keys generated for the existing cluster that are used for the whole platform. The config.yaml, hosts and ssh_key files can be easily replaced.
Copy this data to a location outside of the IBM Cloud Private cluster.
Backing up PersistentVolume data
Each storage provider used in IBM Cloud Private will typically have it’s own method for backing up data or replicating it to another system. For example, GlusterFS can snapshot volumes, Ceph can export images, vSphere volumes can make use of VMWare native backup functions and hostpath/LocalVolumes benefit from any standard Linux file copy methods.
The platform components mostly store data on the host node, simplifying the backup procedures by allowing the use of common tools to backup the data. For applications that rely on PersistentVolumes, it is the cluster administrators responsibility to ensure that they are familiar with the storage technologies they offer to application developers.
The use of a storage provider running as containers on IBM Cloud Private, such as GlusterFS and Ceph, can complicate the backup process as these technologies typically require the write operations to be suspended while the volume/block is backed up using the native snapshot tools. If the cluster fails and you can no longer interact with the storage provider, there is a higher risk of data loss unless the application can handle such scenarios gracefully.
The recommended way to reduce the risk of application data loss is to use storage providers such as GlusterFS or Ceph on Virtual Machines external to the cluster. This way the dependency on the IBM Cloud Private platform itself is removed and the use of dynamic volume provisioning is still enabled. Regardless of the storage provider, ensure backups of the application data are taken regularly.
3.4.4 Platform restore process
No backup is good without first testing it can be restored, and this process should be thoroughly tested to ensure that each backup can be restored multiple times for any given environment. The information in this section assumes that a new IBM Cloud Private cluster needs to be installed on new or existing infrastructure and that the guidelines in “Platform backup process” on page 84 has been followed to for each of the necessary components.
In the example commands provided, the same backup files created in the “Platform backup process” section will be used. It’s important to take a full infrastructure backup of the new environment before attempting to restore the cluster. It’s also recommended to use the steps in “Backup etcd using kubectl” on page 89 to create an etcd backup of the current environment, in case the restore process fails and you end up in a situation where some nodes are restored and others are not. There are several steps that involve replacing core data files and there is a high margin for error.
The restore process will address the following use cases:
•Restore an IBM Cloud Private configuration and data to an existing cluster
•Restore an IBM Cloud Private configuration and data to a remote cluster
The local cluster restore steps in this section were performed in a lab environment with LDAP configured and light workloads, tested on the following cluster configurations:
•Single master, management, proxy, Vulnerability Advisor and 3 x worker nodes
•3 x masters, 2 x management, 2 x proxies, 3 x Vulnerability Advisor and 3 x worker nodes
•3 x etcd, 3 x masters, 2 x management, 2 x proxies, 3 x Vulnerability Advisor and 3 x worker nodes
The remote cluster restore steps in this section were performed in a lab environment with LDAP configured and light workloads, tested on the following cluster configurations: Single master, management, proxy, Vulnerability Advisor, and 3 x worker nodes.
The core components should be restored in the following order:
1. MongoDB
2. Private image registry
3. etcd
4. Persistent Volume data
The other components can be restored in any order, although it is recommended that the logging data is restored before the Vulnerability Advisor. The Persistent Volume Data is restored last so that when a cluster is restored, the backed up data is not overwritten with new cluster data, which would later corrupt the restored applications.
Local cluster restore prerequisites
If IBM Cloud Private is reinstalled on the existing infrastructure, the content in the cfc-certs directory must be copied to the <installation_directory>/cluster directory before installation. This ensures the same certificates from the previous installation are used in the new installation and that any restored components are able to interact with the cluster.
Remote cluster restore prerequisites
If IBM Cloud Private is reinstalled on new infrastructure (for example different nodes with different IP addresses) then you need to ensure that the cluster configuration is as close as possible to the old configuration, in particular the cluster name. The cluster name is used for the image repository (for example, mycluster.icp:8500), so all deployments restored from etcd will use the old cluster image repository and if this is different between clusters, you’ll need to manually edit each deployed resource to use a different image.
Before restoring the cluster, you’ll need to extract all of the platform ConfigMaps and Secrets immediately post-installation, as these will contain the certificates and keys specific to this cluster. When etcd is restored, it will restore the same resources that existed in the backed up installation, which means the new installation will not function correctly if these are not replaced with the current ones. To export all of the current ConfigMaps and Secrets to your local machine, run the script in Example 3-11.
Example 3-11 The save-yamls.sh script
#!/bin/bash
namespaces=(kube-system kube-public services istio-system ibmcom cert-manager)
echo "Saving all ConfigMaps, Secrets, Deployments and Daemonsets..."
for ns in ${namespaces[@]}
do
mkdir $ns.configmap $ns.secret $ns.deployment.extensions $ns.daemonset.extensions
# save configmap, secret, deployment, daemonset
for n in $(kubectl get -o=name configmap,secret,ds,deployment -n $ns)
do
kubectl get -o yaml -n $ns $n > $ns.$n.yaml
done
done
kubectl get APIService -n kube-system -o yaml v1beta1.servicecatalog.k8s.io > kube-system.servicecatalog.yaml
echo "Done."
It’s important to note that unlike the local cluster installation, you do not need to use the same certificates from the backed up cfc-certs directory.
Restoring MongoDB
The recommended way to restore MongoDB from a backup is by using the mongorestore utility. This section will cover two methods; using kubectl and using a Kubernetes job. The use of each method entirely depends on the method used to backup MongoDB. For example, if you used kubectl to backup the database, it makes sense to use kubectl to restore it. If a CronJob or Job was used, then the core-backup PersistentVolume created in the backup steps can be used to remove the manual effort of copying the file to the local machine.
It’s worth checking the current database statistics as a reference point, to know whether the mongorestore command has successfully repopulated the database.
1. Use kubectl exec to create a shell session in the icp-mongodb-0 pod.
[root@icp-ha-boot cluster]# kubectl -n kube-system exec -it icp-mongodb-0 -c icp-mongodb sh
$
2. Log in to the primary replica:
$ mongo --host rs0/mongodb:27017 --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /data/configdb/tls.crt --sslPEMKeyFile /work-dir/mongo.pem
$ mongo --host rs0/mongodb:27017 --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /data/configdb/tls.crt --sslPEMKeyFile /work-dir/mongo.pem
This will output some connectivity information and eventually show the rs0:PRIMARY> prompt.
3. Use the show dbs command to view the current list of databases and each databases current size.
rs0:PRIMARY> show dbs
HELM_API_DB 0.000GB
admin 0.000GB
config 0.000GB
helm-repo_DB 0.001GB
key_protect_metadata 0.000GB
key_protect_secrets 0.000GB
local 0.018GB
metering_DB 0.001GB
platform-db 0.000GB
Restoring using kubectl
Perform the following steps to restore MongDB. Ensure the backup exists on the local machine, and that you can access the cluster using kubectl.
1. Find the primary MongoDB replica. During testing this process, some observations were made where the database did not restore correctly when explicitly using the --host rs0/mongodb:27017 replicaset instead of the hostname of the primary replica. This step, shown in Example 3-12, ensures that the correct primary instance is used.
Example 3-12 Find the primary MongoDB replica
[root@icp-ha-boot cluster]# kubectl exec icp-mongodb-0 -n kube-system -- sh -c 'mongo --host rs0/mongodb:27017 --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /data/configdb/tls.crt --sslPEMKeyFile /work-dir/mongo.pem --eval="db.isMaster()["primary"]" | grep ^icp-mongodb-'
icp-mongodb-0.icp-mongodb.kube-system.svc.cluster.local:27017
2. Copy the backup file to the pod from the previous step.
[root@icp-ha-boot ~]# kubectl cp backup.gz "kube-system/icp-mongodb-0:/work-dir/backup.gz" -c icp-mongodb
3. Verify the file was copied successfully.
[root@icp-ha-boot ~]# kubectl -n kube-system exec icp-mongodb-0 -c icp-mongodb ls /work-dir
backup.gz
credentials.txt
log.txt
mongo.pem
openssl.cnf
peer-finder
4. Execute the restore.
[root@icp-ha-boot ~]# kubectl exec icp-mongodb-0 -n kube-system -- sh -c 'mongorestore --host icp-mongodb-0.icp-mongodb.kube-system.svc.cluster.local:27017 --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /data/configdb/tls.crt --sslPEMKeyFile mongo.pem --oplogReplay --gzip --archive=/work-dir/backup.gz'
2019-03-05T13:07:57.309+0000 preparing collections to restore from
2019-03-05T13:07:57.373+0000 reading metadata for metering_DB.usage from /work-dir/backup/metering_DB/usage.metadata.json
2019-03-05T13:07:57.373+0000 restoring metering_DB.usage from /work-dir/backup/metering_DB/usage.bson
2019-03-05T13:07:57.406+0000 reading metadata for helm-repo_DB.mgmtrepo-assets from /work-dir/backup/helm-repo_DB/mgmtrepo-assets.metadata.json
2019-03-05T13:07:57.406+0000 restoring helm-repo_DB.mgmtrepo-assets from /work-dir/backup/helm-repo_DB/mgmtrepo-assets.bson
...
2019-03-05T13:08:41.235+0000 done
5. Repeat step 1 on page 101 to verify that the mongorestore has been successful and added data to the database. The output might look similar to the following:
rs0:PRIMARY> show dbs
HELM_API_DB 0.000GB
admin 0.000GB
config 0.000GB
helm-repo_DB 0.003GB
key_protect_metadata 0.000GB
key_protect_secrets 0.000GB
local 2.174GB
metering_DB 0.177GB
platform-db 0.001GB
Restoring using a Kubernetes job
Execute the following steps to restore the MongoDB using a job.
1. Create a PersistentVolume (PV) and PersistentVolumeClaim (PVC), similar to the examples in Example 3-1 and Example 3-2 on page 85. This job definition uses the same PV and PVC created in those examples.
Example 3-13 YAML definition for mongodb-restore_job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: mongodbrestore
namespace: kube-system
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- ppc64le
- s390x
- key: master
operator: In
values:
- "true"
containers:
- name: mongodbrestore
image: ibmcom/icp-mongodb:4.0.5
command: ["/bin/bash","-c","cat /certs/tls.crt certs/tls.key > mongo.pem; export PRIMARY=$(mongo --host rs0/mongodb:27017 --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /ca/tls.crt --sslPEMKeyFile mongo.pem --eval="db.isMaster()['"'primary'"']" | grep ^icp-mongodb-); mongorestore --host $PRIMARY --username admin --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /ca/tls.crt --sslPEMKeyFile mongo.pem --oplogReplay --gzip --archive=/backup/mongodb/$BACKUP_NAME"]
env:
- name: BACKUP_NAME
value: "#BACKUP"
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: icp-mongodb-admin
volumeMounts:
- mountPath: "/backup"
name: backup
- mountPath: "/ca"
name: cluster-ca
- mountPath: "/certs"
name: mongodb-certs
tolerations:
- effect: NoSchedule
key: dedicated
operator: Exists
volumes:
- name: backup
persistentVolumeClaim:
claimName: core-backup
- name: cluster-ca
secret:
secretName: cluster-ca-cert
- name: mongodb-certs
secret:
secretName: icp-mongodb-client-cert
restartPolicy: Never
backoffLimit: 1
This file can also be downloaded from the following link:
The BACKUP_NAME environment variable should be changed to reflect the name of an actual MongoDB backup file that exists on the PV. If the MongoDB backup job or CronJob in this chapter was used to create the backup, the file name format is mongodb-backup-<year-month-day_hour:minutes:seconds>.gz. Alternatively, sed can be used to easily do this, by executing sed -i -e s/#BACKUP/<name.gz>/g mongodb-restore_job.yaml replacing <name.gz> with your own.
The ibmcom/icp-mongodb:4.0.5 image is the default MongoDB image for IBM Cloud Private 3.1.2. If the environment does not have internet access, replace ibmcom with the private image registry local to the cluster, for example, mycluster.icp:8500/ibmcom/icp-mongodb:4.0.5.
Create the job using kubectl create -f mongodb-restore_job.yaml. The job should be visible in the Workloads → Jobs → BatchJobs tab in IBM Cloud Private. If the job did not complete successfully, troubleshoot the failure using the guidance provided in the troubleshooting chapter. If the job ran successfully, use the same method of verification using show dbs as described earlier. At this point, MongoDB should be restored.
Restoring the private image registry
Copy the contents of the private image registry backup to the /var/lib/registry directory on the master node. In high availability environments, the data should be restored to the shared filesystem and all master nodes should be able to access this directory. The system images will already exist after installing IBM Cloud Private, so it may be preferable to restore the user images from other image backups by pushing these images to the image repository before restoring etcd.
After filesystem data is restored, use kubectl to delete the image manager pods running on all master nodes:
[root@icp-ha-boot ~]# kubectl delete pods -l app=image-manager
pod "image-manager-0" deleted
pod "image-manager-1" deleted
pod "image-manager-2" deleted
When the image-manager pods are running again, the restored images should now be available. If you backed up images using docker save, you can reload them using dockerload. For example, use the following command:
docker load -i backup-image.tar
PersistentVolumes
Due to the large number of storage providers available in IBM Cloud Private, restoring the data to the PersistentVolumes is not covered in this chapter. When restoring etcd, the original PersistentVolume and PersistentVolumeClaim names will be restored so at this point, the original data should be restored as per the storage providers recommendations.
The restored volumes will use the same configuration from the backed up cluster, so restoring this data is a good idea to do before restoring etcd, so that there is minimal disruption to the applications that start up and expecting data to be present. For hostPath or LocalVolume type volumes and any external storage volumes where the data points on the host/remote storage have not moved, the data restoration should be straight forward.
If you’re restoring a cluster that previously used a containerized storage provider such as GlusterFS or Ceph, you will need to ensure that any LVM groups/volumes and disk paths used in the storage configuration are identical to the backed up cluster. Restoring a cluster with containerized storage providers was not tested in this chapter, and may yield unpredicted results.
Restoring etcd on local cluster
Restoring etcd is not a trivial task and there are several steps required, including modifying core files. Ensure the following pre-requisites are met before restoring etcd:
•You have backed up the cluster at the infrastructure level
•You have access to the master and management nodes
•The etcd backup file exists on your local machine
Most steps are run on the etcd/master nodes but the management nodes also require some actions.
Restoring etcd on single and High-Availability installations
The restore procedure for etcd on both single and multi-master installations (where etcd is running on the master nodes) use the same set of steps. Ensure that any steps directed at master nodes are repeated on all master nodes. It’s also important to perform each step on all specified nodes before moving on to the next step, to ensure all nodes are at the same stage. This helps with reversing the process if a step should fail. Perform the following steps:
1. On all master nodes, create a backup directory and move the etcd pods to it. This will stop the etcd container, so verify that it is stopped:
a. Create the backup directories:
mkdir -p /etc/cfc/podbackup
mkdir -p /var/lib/etcdbackup
mkdir -p /var/lib/etcdwalbackup
b. Move the pod data to the backup directory:
mv /etc/cfc/pods/* /etc/cfc/podbackup/
c. Move the current etcd data to the backup directory:
mv /var/lib/etcd/* /var/lib/etcdbackup/
mv /var/lib/etcd-wal/* /var/lib/etcdwalbackup/
d. Verify the etcd pod has stopped.
docker ps | grep etcd
2. On the master and management nodes, stop kubelet and restart docker to stop all running pods:
systemctl stop kubelet
systemctl restart docker
3. Copy the etcd snapshot file to all masters.
4. Copy the script in Example 3-14 and run it on all masters. This script will retrieve the current environment information from the /etc/cfc/podbackup/etcd.json file on the master nodes, then construct and execute a docker command to generate the new etcd data from the snapshot provided.
When running the script ensure the etcd snapshot file name is passed as a parameter and it is located in the current directory (where the script is run), as the docker container will mount the current directory and use the snapshot file.
Example 3-14 The etcd-restore.sh script
#!/bin/bash
[ -z "$1" ] && { echo "Please provide the etcd snapshot file name"; exit 1; }
data_dir="/var/lib/etcd"
restore_dir="/var/lib/etcd/restored"
# Get etcd docker image details
[ -z $etcd_image ] && etcd_image=$(grep '"image"' /etc/cfc/podbackup/etcd.json | sed -e 's/^s*//' -e 's/",//g' -e 's/"//g' -e 's/--//g' -e 's/image://g')
# Set volume mounts
volume_mounts="-v $(pwd):/data -v /etc/cfc/conf/etcd:/certs -v /var/lib/etcd:/var/lib/etcd"
self=$(grep 'advertise-client-urls=' /etc/cfc/podbackup/etcd.json | sed -e 's/^s*//' -e 's/",//g' -e 's/"//g')
# Get etcd cluster settings
node_name=$(grep 'name=' /etc/cfc/podbackup/etcd.json | sed -e 's/^s*//' -e 's/",//g' -e 's/"//g')
initial_advertise_peer_urls=$(grep 'initial-advertise-peer-urls=' /etc/cfc/podbackup/etcd.json | sed -e 's/^s*//' -e 's/",//g' -e 's/"//g')
initial_cluster=$(grep 'initial-cluster=' /etc/cfc/podbackup/etcd.json | sed -e 's/^s*//' -e 's/",//g' -e 's/"//g')
initial_cluster_token=$(grep 'initial-cluster-token=' /etc/cfc/podbackup/etcd.json | sed -e 's/^s*//' -e 's/",//g' -e 's/"//g')
# Delete the etcd data
rm -rf /var/lib/etcd
# Run the restore on the node
docker run --entrypoint=etcdctl -e ETCDCTL_API=3 ${volume_mounts} ${etcd_image} --cert /certs/client.pem --key /certs/client-key.pem --cacert /certs/ca.pem --endpoints ${self} snapshot restore /data/$1 --data-dir=$restore_dir $node_name $initial_advertise_peer_urls $initial_cluster_token $initial_cluster
# Print result
[ $? -eq 0 ] && echo "etcd restore successful" || echo "etcd restore failed"
Alternatively, download and run the etcd-restore.sh script from the following link:
If successful, the output should be etcd restore successful.
|
Tip: There are other ways to read the correct variable values from the etcd.json file by using tools such as jq, but the method used here is designed to be as generic as possible. If this script fails with an error similar to Unable to find image, the etcd.json file may not have been parsed correctly, potentially due to double hyphens or similar characters in the cluster name. Set the etcd_image environment variable manually by getting the docker image name using docker images | grep etcd and then setting etcd_image using export etcd_image=<docker-image>.
|
[root@icp-master ~]# ./etcd-restore.sh snapshot.db
etcd restore successful
If etcd failed to restore, review the error message and take appropriate action. A common error is providing the snapshot name but failing to copy/move it to the current director, so etcd will report that the snapshot file was not found.
5. Move the data in /var/lib/etcd/restored to /var/lib/etcd and remove the empty /var/lib/etcd/restored directory:
mv /var/lib/etcd/restored/* /var/lib/etcd/
rmdir /var/lib/etcd/restored
6. The current kubelet pods need to be deleted so that the cache will be consistent between the kubelet and etcd data. Do this on the master and management nodes:
mount | grep kubelet | awk '{ system("umount "$3)}'
rm -rf /var/lib/kubelet/pods
7. Start kubelet on the master and management nodes:
systemctl start kubelet
8. Enable the etcd pods on all master nodes:
mv /etc/cfc/podbackup/* /etc/cfc/pods/
9. Allow around 30 seconds for etcd to rebuild its cluster state and verify it is healthy, replacing 172.24.19.201 with the IP address of any master node. See Example 3-15.
Example 3-15 Verify that etcd is healthy,
[root@icp-master ~]# docker run --entrypoint=etcdctl -v /etc/cfc/conf/etcd:/certs -v /var/lib/etcd:/var/lib/etcd ibmcom/etcd:3.2.24 --cert-file=/certs/client.pem --key-file=/certs/client-key.pem --ca-file=/certs/ca.pem --endpoints https://172.24.19.201:4001 cluster-health
member 8a2d3ec6df19666f is healthy: got healthy result from https://172.24.19.201:4001
cluster is healthy
Note that the ibmcom/etcd:3.2.24 image may need to be replaced with the private image registry name for the environment. For example, mycluster.icp:8500/ibmcom/etcd:3.2.24.
10. Clear your web browser cache.
11. The IBM Cloud Private platform may take some time to start up all the pods on the master node and become available for use. Once the dashboard is accessible. verify that the cluster has started recreating the previous state by redeploying workloads from the backed up cluster. You may experience a few pod failures for some time until the system becomes stable again. Logging may be particularly slow as it deals with the sudden influx of log data from newly recreated pods all at once.
Restoring etcd on dedicated etcd installations
To restore etcd on an installation with dedicated etcd nodes, you can use the same steps for single and multi-master installations, but using the dedicated etcd nodes in place of the masters. All etcd data resides on the dedicated etcd nodes so the only steps that apply to the masters in this case is the same steps that apply to the management nodes.
Failed etcd restores
If the etcd snapshot was not restored, it may be possible to recover the initial state of the cluster. A common error on RHEL systems is that systemd becomes unresponsive when Docker is stopped and started. A usual symptom of this is when the docker restart command takes a long time to run, and simple commands that depend on systemd (such as reboot) just hang. In this situation, only a reboot from the hypervisor will fix a hung systemd.
If the etcd snapshot restore was successful on some nodes and failed on others, you will need to either address the issue on the failed nodes and try again, or revert the successful nodes to the original state by restoring the etcd snapshot taken from the new installation.
To revert successful nodes back to the previous state, try the following steps. If these commands do not restore a healthy state, you may need to reinstall the cluster.
On all master nodes (or dedicated etcd nodes if they are separate)
1. Stop Kubelet:
systemctl stop kubelet
2. Restart Docker:
systemctl restart docker
3. Remove existing etcd data:
rm -rf /var/lib/etcd
4. Recreate Etcd directory:
mkdir -p /var/lib/etcd
5. Copy data from backup:
cp -r /var/lib/etcdbackup/* /var/lib/etcd/
cp -r /var/lib/etcdwalbackup/* /var/lib/etcd-wal/
6. Recreate pods directory:
mkdir -p /etc/cfc/pods
7. Restore pod data:
mv /etc/cfc/podbackup/* /etc/cfc/pods/
8. Remove kubernetes pods:
rm -rf /var/lib/kubelet/pods
9. Start kubelet:
systemctl start kubelet
10. Check the Etcd health:
docker run --entrypoint=etcdctl -v /etc/cfc/conf/etcd:/certs -v /var/lib/etcd:/var/lib/etcd ibmcom/etcd:3.2.24 --cert-file=/certs/client.pem --key-file=/certs/client-key.pem --ca-file=/certs/ca.pem --endpoints https://172.24.19.201:4001 cluster-health
member 8a2d3ec6df19666f is healthy: got healthy result from https://172.24.19.201:4001
cluster is healthy
Replace 172.24.19.201 with any etcd/master node IP address.
10. On all management nodes (and master nodes too if etcd is running on dedicated nodes)
a. Move the pod files back to the original directory:
mv /etc/cfc/podbackup/* /etc/cfc/pods/
b. Start kubelet:
systemctl start kubelet
Allow some time for all pods to start up. The cluster should be returned to a normal working state.
Restoring etcd on remote cluster
To restore etcd in a remote cluster, repeat steps 1 to 9 in the section “Restoring etcd on local cluster” on page 106. Some further steps need to be performed to clean up the old nodes and data from the restored etcd snapshot and apply the working configuration. Perform the following steps:
1. After restoring etcd, all the cluster nodes from the initial installation are not present in the new Kubernetes cluster, so for each node kubelet and docker should be restarted. You can check which nodes currently exist using kubectl get nodes. Restart kubelet and docker on all cluster nodes using systemctl stop kubelet && systemctl restart docker && systemctl start kubelet. After this, the new cluster nodes should be visible using kubectl get nodes.
2. Wait for about 90 seconds after restoring etcd for the restored nodes to change from Ready to NotReady state, then remove the NotReady nodes. Be sure to only remove the nodes that contain the old cluster IP addresses. If nodes with the current cluster IP addresses are in NotReady state, you should investigate whether the node is online first.
a. Get the current list of nodes in the NotReady state.
kubectl get nodes | grep NotReady | awk '{print $1}'
b. Remove the old nodes.
kubectl delete node $(kubectl get nodes | grep NotReady | awk '{print $1}')
3. Replace the ConfigMap and Secrets restored by etcd with the data exported in the section “Restoring MongoDB” on page 101. Run the script in Example 3-16.
Example 3-16 The restore-yamls.sh script
#!/bin/bash
echo "Restoring..."
# re-genereate default secret
kubectl get secret -n kube-system -o wide | grep "-token-" | awk '{system("kubectl delete secret "$1 " -n kube-system")}'
kubectl get secret -n services -o wide | grep "-token-" | awk '{system("kubectl delete secret "$1 " -n services")}'
kubectl get secret -n istio-system -o wide | grep "-token-"| awk '{system("kubectl delete secret "$1 " -n istio-system")}'
kubectl get secret -n kube-public -o wide | grep "-token-"| awk '{system("kubectl delete secret "$1 " -n kube-public")}'
kubectl get secret -n ibmcom -o wide | grep "-token-"| awk '{system("kubectl delete secret "$1 " -n ibmcom")}'
kubectl get secret -n cert-manager -o wide | grep "-token-"| awk '{system("kubectl delete secret "$1 " -n cert-manager")}'
namespaces=(kube-system services istio-system kube-public ibmcom cert-manager)
for ns in ${namespaces[@]}
do
#secret
for s in $(ls $ns.secret/ | grep -v "-token-")
do
kubectl delete -f $ns.secret/$s && kubectl create -f $ns.secret/$s
done
#configmap
for s in $(ls $ns.configmap/)
do
kubectl delete -f $ns.configmap/$s && kubectl create -f $ns.configmap/$s
done
done
kubectl --force --grace-period=0 delete pv $(kubectl get pv | grep -e "image-manager-" -e "icp-mongodb-" -e "mariadb-" -e "logging-datanode-" -e "kafka-" -e "zookeeper-" -e "minio-" | awk '{print $1}')
kubectl patch pv $(kubectl get pv | grep Terminating | awk '{print $1}') -p '{"metadata":{"finalizers":null}}'
kubectl delete pvc -n kube-system $(kubectl get pvc -n kube-system | grep -e "image-manager-image-manager-" -e "mongodbdir-icp-mongodb-" -e "mysqldata-mariadb-" -e "data-logging-elk-data-" -e "-vulnerability-advisor-" -e "datadir-vulnerability-advisor-kafka-" -e "datadir-vulnerability-advisor-zookeeper-" -e "datadir-vulnerability-advisor-minio-" | awk '{print $1}')
echo "Done."
Alternatively download and run the script from the following link:
If Vulnerability Advisor is not installed, ignore the errors.
4. Several files created during the initial cluster installation need to be reapplied:
a. Change the directory to <installation_directory>/cluster/cfc-components/:
cd <installation_directory>/cluster/cfc-components/
b. Apply the bootstrap-secret.yaml file:
kubectl apply -f bootstrap-secret.yaml
c. Apply the tiller.yaml file:
kubectl apply -f tiller.yaml
d. Apply the image-manager.yaml file:
kubectl apply -f image-manager/image-manager.yaml
e. Apply the whole storage directory:
kubectl apply -f storage/
5. Restore the resources that use IP addresses in the YAML configuration. Run the script in Example 3-17.
Example 3-17 The restore-ip-yamls.sh script
#!/bin/bash
# Set files
files=(
servicecatalog.yaml
daemonset.extensions/service-catalog-apiserver.yaml
daemonset.extensions/auth-idp.yaml
daemonset.extensions/calico-node.yaml
daemonset.extensions/nginx-ingress-controller.yaml
daemonset.extensions/icp-management-ingress.yaml
deployment.extensions/helm-api.yaml
deployment.extensions/helm-repo.yaml
deployment.extensions/metering-ui.yaml
deployment.extensions/metering-dm.yaml
deployment.extensions/monitoring-prometheus-alertmanager.yaml
deployment.extensions/monitoring-prometheus.yaml
)
# Delete and recreate resources
for f in ${files[@]}
do
kubectl delete -f kube-system.$f && kubectl create -f kube-system.$f
done
Alternatively, download and run the script from the following link:
6. Restart kubelet and docker on all nodes again. When restarted, the pod should start scheduling across all the cluster nodes.
7. After all nodes are online, all kube-system pods need to be deleted so they inherit the Secrets and ConfigMaps restored earlier.
kubectl -n kube-system delete pods $(kubectl -n kube-system get pods | awk '{print $1}')
Check that all the pods start being deleted, using kubectl -n kube-system get pods. Some pods may be stuck in Terminating state, which can prevent the new pods from starting up properly.
To delete all stuck pods, using the following code:
kubectl delete pod --force --grace-period=0 -n kube-system $(kubectl get pods -n kube-system | grep Terminating | awk '{print $1}')
You may also need to delete all pods running in the istio-system and cert-manager namespaces, so use the same approach as above.
kubectl delete pod -n istio-system $(kubectl get pods -n istio-system | awk '{print $1}')
kubectl delete pod -n cert-manager $(kubectl get pods -n cert-manager | awk '{print $1}')
Allow some time for all pods to start up, which may take 10 - 40 minutes, depending on the size of the cluster. The logging pods might take even longer than this to cater for the influx of logs generated from the restore. If individual pods are still in error state, review the errors and attempt to resolve the problems. Issues will vary with each environment, so there is no common resolution. If the restore was successful, you should be able to access and operate the cluster again.
Restoring MariaDB
As discussed in “Backing up MariaDB” on page 94, it’s not usually necessary to restore MariaDB data, however for completeness, the information is provided here. The recommended way to restore MariaDB from a backup is by using the mysql utility. This section will cover two methods; using kubectl and using a Kubernetes job. The use of each method entirely depends on the method used to backup MariaDB.
For example, if you used kubectl to back up the database, it makes sense to use kubectl to restore it. If a job was used, then the core-backup PersistentVolume created in the backup steps can be used to remove the manual effort of copying the file to the local machine.
Restore MariaDB using kubectl
Perform the following steps to restore MariaDB. Ensure that the backup exists on the local machine, and that you can access the cluster using kubectl.
1. Get the MariaDB pod name:
kubectl -n kube-system get pods -l release=mariadb -o=jsonpath='{.items[0].metadata.name}'
mariadb-0
2. Copy the backup file to the mariadb-0 container:
kubectl cp backup.sql kube-system/mariadb-0:/backup.sql
3. Execute the mysql command:
kubectl -n kube-system exec -it mariadb-0 -c mariadb -- sh -c 'mysql --host=$MARIADB_SERVICE_HOST --user=root --password=$MYSQL_ROOT_PASSWORD < /backup.sql'
Restore MariaDB using a Kubernetes job
If MariaDB was backed up using a job, recreate the PersistentVolume and PersistentVolumeClaim definition that was used for the backup job. Download the mariadb-restore_job.yaml from https://github.com/IBMRedbooks/SG248440-IBM-Cloud-Private-System-Administrator-s-Guide/tree/master/Ch3-Backup-and-Restore/Restore/mariadb-restore_job.yaml, replacing the volumeClaimName with your own, and create it using kubectl create -f mariadb-restore_job.yaml.
Restoring the logging data
You can restore the logging data by copying the backed up data back to the management node filesystem.
Use the preferred tools to restore the backed up data to /var/lib/icp/logging/elk-data/nodes/0/indices on each management node. The Elasticsearch data node(s) need to be restarted to reload the new indices. You can do this by using kubectl -n kube-system delete pods logging-elk-data-0, and allowing some time for the pod to start up again. Once it is started, all the backed up indices should be restored and available in Elasticsearch.
Restoring the monitoring data
Restore the monitoring data by recreating the PersistentVolume (PV) and PersistentVolumeClaim used during the backup steps. Assuming that the appropriate restore methods for the storage technology used for this PV, the monitoring pods should continue from the backed up data.
As AlertRules and Dashboards are stored in CustomResourceDefinitions, these should be restored when etcd is restored. Alternatively, you can recreate the resources from the YAML definitions used to create the AlertRule or Dashboard, or from the exported YAML definitions as described in “Backing up the monitoring data” on page 96. See the Logging and Monitoring chapter for more information about creating monitoring resources.
Restoring Vulnerability Advisor data
The complete restoration of Vulnerability Advisor (VA) and Mutation Advisor (MA) in this step depends on whether or not the logging data (specifically the indices related to VA and MA) was restored successfully, as VA stores data in Elasticsearch. To restore the VA and MA data, extract the backed up filesystem data to /var/lib/icp/va on the VA node(s) using the preferred OS tools.
Restoring Helm charts
All Helm chart data is stored in MongoDB and should be recreated on the master nodes when the helm-repo pod is restored. However, in the event this does not happen, perform the following steps:
1. Make sure that helm-repo is running and that the /var/lib/icp/helmrepo directory exists all master nodes.
2. Move the charts that you backed up to the /var/lib/icp/helmrepo directory on all master nodes.
3. Delete the helm-repo deployment pod, and then the index.yaml repopulates with the restored charts.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.