In any project, the backup is one of the most important ways to avoid losing data. In this chapter, we will learn the backup strategies to be used in your application.
Backup is the process of saving code or data in a different place to where the code or data is usually stored. This process can be done using different strategies, but all of them have the same goal--not to lose data in order for it to be accessed in the future.
A backup can be done for two reasons. The first one is the loss of data due to a hack attack, corrupted data, or any mistakes executing queries on the production server. This backup will help restore the lost or corrupted data.
The second reason is policy. The law says that it is required to store user data for years. Sometimes, this functionality is done by a system, but a backup is another way to store this data.
To sum up, backups allow us to be calm. We ensure that we are doing things properly and in case any disasters happen, we have a solution to fix them fast and without (significant) data loss.
If we are using some repositories, such as Git, in our application, this can be our backup place for the files. The assets or any other files uploaded by users should be backed up too.
A good practice to see if we are backing up all the necessary files is reviewing the .gitignore file and ensuring that we have backed up all the files and folders included in that folder.
Also, the most important and precious thing to back up is the database. This should be backed up more frequently.
The backups can be full, incremental, or differential. We will look at the difference between them and how they work. Applications usually combine different backup types: a full backup with incremental or differential backups.
The full backup is the basic one; it consists of generating a full copy of the current application. This option is used by large applications periodically, and small applications can use it daily.
The advantages are as follows:
- The full application is backed up in a single file
- It always generates a full copy
The disadvantages are as follows:
- The time to generate it can be very long
- The backup will need a lot of disk space
Please note that it's generally good practice to include date/time in the backup file name so you can know when was created only looking to the file name.
The incremental backup copies the data that has changed since the last backup. The datetime should be included in this kind of backup in order to be checked by the backup tools the next time a new backup is generated.
The advantages are as follows:
- It is faster than a full backup
- It takes up less disk space
The disadvantages are as follows:
- The entire application is not stored in a single generated backup
There is another type, called differential. This is like the incremental backup (copying all the data that has changed since the last backup); it is executed the first time and then it will continue copying all the modified data from the last full backup.
So, it will generate more data than the incremental one but less than the full one after the first time. This type is an intermediate one between full and incremental. It needs more space and time than incremental and less than full.
It is possible to find many backup tools. The most common tool in large projects is Bacula. There are other similar ones for small projects like a custom script that will be run frequently.
Bacula is a backup management tool. This application manages and automates the backup tasks and it is perfect for large applications. This tool is a little complex to set up, but once you have it ready, it does not need any other changes and it will work without any problems.
Three different parts exist on Bacula, and every single part needs to be installed in a different package:
- Director: This manages all the backup processes
- Storage: This is where the backup is stored
- File: This is the client machine where we have our application running
In our application based on microservices, we will have many files (one for each microservice) and optionally, many storage locations (in order to have different locations for the backups) and directors.
This tool works with daemons. Each part has its own daemon and each daemon works following its own config files. The config files are set up in the installation process, and it is only necessary to change some little things, such as the remote IP address, certificates, or the plan to automate backups.
The security of Bacula is really amazing--every part (director, storage, and file) has its own key and depending on the connection, it is encrypted using it. Also, Bacula allows TLS connections for more security.
Bacula allows you to do full, incremental, or differential backups, and they can be automated on the director's part.
XtraBackup is an open source tool to do hot database backups on your application without blocking the database. This is possibly the most important feature of this application.
This tool allows MySQL databases, such as MariaDB and Percona, to perform streaming and compression and do incremental backups.
The advantages are as follows:
- Fast backups and restores
- Uninterrupted transaction processing during the backups
- Saves disk space and network bandwidth
- Automatic backup verification
The fastest way to create backups is to use a custom script in production. This is a script that when it runs, it creates a backup by doing a mysqldump (if we are using a MySQL database), compressing the desired files, and putting them in the desired location (ideally, remotely on a different machine).
These scripts should be executed by a cronjob that can be set up to run them every day or week.
As part of the backup strategies, it is a good practice to have techniques to validate the data stored on the backups. If you have backups with errors, it is like not having any backups at all.
In order to check that our backups are valid, are not corrupted, and they work as expected, it is necessary to frequently do a mock restore to avoid failure if we need to restore them in the future.
No one wants to have to restore a backup, but it is necessary to be ready in case any of your microservices are broken or corrupted and we have to react fast.
The first step is knowing where the most recent backup of your application is in order to restore it as soon as possible.
If the problem is related to the database, we have to put the application under maintenance, restore the database backup, check that it works properly, and then, make the application live again.
If the problem is related to something like assets or files, it is possible to restore without putting the application under maintenance.
Keep calm and get your backup.