Chapter 10. Backup Strategies
The objective of any backup strategy is first to ensure nothing is lost, and second to verify that it can be restored. The reality is that, while this is achievable, with virtualization, we need two distinct paradigms to make this a reality. As we discussed in our design session, a XenServer host has a privileged VM that looks and feels like Linux but has been completely customized. While applying standard Linux backup strategies to it isn’t a bad idea, they only go part of the way to supporting the “ensure nothing is lost” criterion. This section will cover those missing pieces.
We’ll also touch on what it means to backup a guest VM. While XenServer provides the ability to perform snapshots on running VMs, there are many third-party solutions to manage those snapshots. Unfortunately, many users have discovered that simply having a backup of VMs in snapshot form isn’t sufficient. This is because a VM disk by itself lacks the metadata to define the network, memory, and CPU configuration. As a result, any successful strategy needs to encompass both infrastructure and VM backup.
Frequency of Backup
If you are running a XenServer environment that is actively creating and re-creating VMs, then you should pay particular attention to the metadata topics and create an updated metadata backup after each object is configured. In the event a restore is needed and the configuration is stale, errors will be flagged and additional work will be required to restore the pool configuration.
Backup dom0
dom0 is a critical component of all XenServer deployments and should be backed up.
Problem
You wish to back up the core configuration for a single XenServer host, even if pooled.
Solution
XenServer provides a simple command to perform a host backup. This backup should be performed after install and after any reconfiguration or upgrade.
Discussion
In order to back up the core configuration for a XenServer host, a single xe command is used (Example 10-1).
Example 10-1. Backup host
# xe host-backup file-name=[filename] -h [hostname] -u root -pw [password]
The resultant backup file can then be saved for future use. Note that this is how host-level configuration data is preserved during upgrades, which means that this backup can then be restored onto any fresh installation to re-create the host.
To restore the backup, issue the command in Example 10-2, then reboot using the installation media, and select the “restore from backup” option to complete the restoration.
Example 10-2. Restore host backup
# xe host-restore file-name=[filename] -h [hostname] -u root -pw [password]
XAPI Database Still Needs to Be Backed Up!
Performing a host backup doesn’t back up the XAPI database because it’s possible the host is a pool member. To complete a single host backup, you will need to also back up the XAPI database.
Pool and XAPI Database Backup
The XAPI database contains all information related to the XenServer toolstack.
Problem
You wish to back up the XAPI database and associated pool configuration.
Solution
XenServer provides a simple command to perform a backup of the XAPI database. This backup should be performed after install and after any reconfiguration or upgrade.
Discussion
The XAPI database contains all information related to the current pool configuration. This includes storage, network, VM, GPU, user, and host information. Having a current backup of the pool metadata will greatly simplify the process of restoring a pool after a major hardware issue, such as a storage controller failure. To create a backup of the pool metadata, issue the command in Example 10-3.
Example 10-3. Backup pool metadata to NFS mount
# xe pool-dump-database file-name=[NFS backup]
In the event the metadata needs to be restored, issue the command in Example 10-4 on the host.
Example 10-4. Restore pool metadata
# xe pool-database-restore file-name=[backup file]
Performing a Dry Run
The pool-database-restore command has an optional parameter called dry-run. If set to “true,” the command will verify the integrity of the backup and issue any warnings that might prevent a successful backup.
VM Backup
Guest virtual machines come in essentially two configurations: stateless VM or stateful VM. Stateless VMs have no meaningful state information in them and often lack a backup model. Instead, the provisioning solution simply re-creates them as required. On the other hand, a stateful VM contains some material information, be it configuration information or data. Stateful VMs are often part of a backup strategy.
Problem
You wish to back up an individual VM and its configuration without downtime.
Solution
While XenServer supports a VM export command that will perform a full backup of a VM, this command requires the VM be powered down. In order to perform a snapshot of a running VM, you will need to take a snapshot of that VM, export any VM metadata, and then export the snapshot.
Discussion
Creating a VM backup in a crash-consistent manner is easily accomplished using snapshots. Crash-consistent VMs are ones that automatically recover from a hard shutdown, which could occur from a power outage.
Start by locating the VM’s UUID, which will be used throughout this procedure:
# xe vm-list name-label=[VM name] --minimal
Then create a snapshot for the VM, which also returns a UUID:
# xe vm-snapshot uuid=[uuid] new-name-label=[snapshot name]
Next convert the snapshot into a VM:
# xe template-param-set is-a-template=false uuid=[snapshot uuid]
Then export the VM with all associated metadata:
# xe vm-export vm=[snapshot uuid]filename=filename.xva
Then clean up by removing the snapshot and requesting storage garbage collection to be run:
# xe vm-uninstall uuid=[snapshot uuid] force=true # xe sr-scan uuid=[SR uuid]
The exported VM can then be restored onto any XenServer host with a compatible XenServer version to the original host.
Account for Snapshot Chains
When using snapshots as a part of a backup strategy, it’s important to understand disk chains. Snapshots collected by virtual appliance-based or external solutions can pose a risk to successful backups. The primary reason is that, after 28–30 snapshots, a mathematical limitation of VHD files with excessive snapshots is encountered. Even with a base disk still intact, it becomes difficult, time-consuming, and storage intensive to coalesce those snapshots back into a base VM disk. Furthermore, if the base disk is lost, there is no alpha point with which to base snapshots off of to revert to a previous moment in time of a Guest VM’s life.
Metadata Backup and Restore
Throughout this book, we have placed heavy emphasis in avoiding single points of failure and disaster recovery. In considering both of these major topics for a single-server or pool-configured deployment, this section discusses how to back up and restore VM metadata in the event of a host failure.
As each VM is created, information is stored within XAPI’s database. This information, known as metadata, describes each VM, its physical resource requirements, and its virtual resources requirements. This data includes, but is not limited to:
- VM name and description
- Boot order and OS information
- Virtual network information
- Virtual disk definitions and locations
In the event of a complete host or pool failure, there are scenarios where recovery of VMs can be an arduous task or worse, impossible to restore without metadata backups.
Problem
You wish to back up all relationships between a virtual machine and the virtual infrastructure.
Discussion
Metadata backups can be created at will in xsconsole, XenCenter, or from the command line within dom0. While we will focus on the command line, the end result from either source is a backup of all current VM metadata stored in a virtual disk (VDI) container. The reason for this format is so that, like a VM’s VDI, it can be moved, exported, or accessed by pool members.
Avoid Single Points of Failure
Metadata backups should always be stored on a shared storage repository. Local storage should never be used because this backup now becomes a single point of failure in the event of disk corruption.
The xe-backup-metadata utility can be executed manually—or automated—from the command line. Example 10-5 shows you the command to obtain all backup options, while Example 10-6 shows the command to back up to a specified SR.
Example 10-5. Obtain all backup options
# xe-backup-metadata -h
Example 10-6. Back up metadata to specified SR
# xe-backup-metadata -c -i -u [SR UUID for backup]
Restoring metadata requires that the XenServer host be operational and that the SR on which the backup is present has been added. Once those requirements have been met, the metadata back up can be restored, as shown in Example 10-7.
Example 10-7. Restore all metadata on the specified SR
# xe-restore-metadata -u [UUID of Shared Storage] -m sr
Portable Storage Repositories
The most common storage repository used in XenServer is one that is fixed to either a host or a pool for the lifespan of the underlying storage. Portable storage repositories have sufficient information contained within them to allow all VMs contained in the repository to be attached to a new host or pool.
Problem
A full backup of all VMs on a storage repository is required.
Solution
A portable storage repository is an SR that has all necessary VM information to re-create those VMs on any XenServer pool. Portable SRs are particularly valuable during hardware upgrades where new pools are being created, but VM migration options like storage migration aren’t an option.
In order to create a portable SR, the underlying virtual disk images must all reside on the same SR. A special backup disk is then created by XenSever that contains all metadata required to restore the VMs using that SR.
Discussion
The process of creating a portable SR is best done from within the XenServer command-line console, xsconsole, and must be done on the pool master. Example 10-8 shows the simple command to start xsconsole.
Example 10-8. Start xsconsole on the pool master
# xsconsole
Then select the option for “Backup, Restore and Update.” We’ll first start by creating a new backup for the desired storage by selecting “Backup Virtual Machine Metadata.” Once complete, a portable SR has been created, but as new VMs are created and old ones retired, this data will become stale. To resolve this, define a schedule under “Schedule Virtual Machine Metadata.” Create a schedule that matches the frequency VM metadata changes within that SR.
Restoring a portable SR in a new pool is easily accomplished by adding the portable SR to the new pool, starting xsconsole on the pool master, selecting “Backup, Restore and Update” and then choosing the portable SR from the list of storage options.