It is a good feeling to have a backup when things go wrong. When setting up a monitoring system, it is a good idea to spend some time to figure out how backups could be made so that the good feeling is not replaced by a bad feeling. With Zabbix, there are components and data to be considered:
- Zabbix binaries: Such as the server and proxy agent: They're probably not worth backing up. Hopefully, they're easily available from packages or by recompiling.
- Zabbix frontend files: Hopefully, they're easily available as well. If any changes have been made, they're presumably stored as a patch in a version control system.
- Zabbix configuration files: Hopefully, these are stored in a version control system or a system configuration tool.
- Zabbix server database: This contains all the monitoring-related configuration data, such as hosts and items, and it also holds all the collected values. Now that is worth backing up!
Several different databases could be used for the Zabbix backend. We won't spend much time on database-specific information, besides a brief look at a simple possible way to create backups with the most widely used backend—MySQL—or one of its forks. A very simple way to back up a database with MySQL, compressing it on the way, would be this:
$ mysqldump zabbix --add-drop-table --add-locks --extended-insert --single-transaction --quick -u zabbix -p | bzip2 > zabbix_database_backup.db.bz2
Here, we are allowing the backup to drop existing tables in the target database and telling it to lock each table when restoring, which is supposed to offer better restore performance. We're also using extended insert, which uses one insert for many values instead of one per value—a much smaller backup and much faster restore. Performing the backup in a single transaction should ensure a consistent state across all the tables being backed up. And finally, the --quick option should instruct MySQL to dump large tables partially instead of buffering all of their contents in memory.
We also used bzip2 to compress the data before writing it to the disk. You can choose other compression software such as gzip or xz or change the compression level, depending on what you need more—disk space savings or a less-taxed CPU during the backup and restore. Memory usage can also be quite high with some compression utilities. The great thing is you can run this backup process without stopping the MySQL server (actually, it has to run) and even the Zabbix server.
Now, you can let your usual backup software grab this created file and store it on a disk array, tape, or some other, more exotic media.
Restoring such a backup is simple as well. We pass the saved statements to the MySQL client, uncompressing them first, if necessary:
$ bzcat zabbix_database_backup.db.bz2 | mysql zabbix -u zabbix -p
Of course, backups are useful only if it is possible to restore them. As required by any backup policy, the ability to restore from backups should be tested. This includes restoring the database dump, but it is also suggested to compare the schema of the restored database and the default schema as well as running a copy of Zabbix Server on a test system. Make sure to disallow any outgoing network connections by the test server, though; otherwise, it might overload the network or send false alerts.
While we can dump a whole database in a single file, it is not always the best solution. There might be cases when restoring only the configuration data would be useful:
- When testing a Zabbix upgrade on a less powerful system than the Zabbix server.
- When attempting to recover from a disastrous event, it would be useful to restore configuration only and resume monitoring as quickly as possible. If needed, history and trend data can be restored later in small portions to avoid overloading the database.
Usually, data tables, such as the ones holding history, trend, and event information, will be much bigger than the configuration tables. Restoring the data tables would take much longer or even be impossible on a test system. We could split all the tables into configuration and data ones, but it is likely even more simple to back each table up separately and deal with the desired tables when restoring. An example command to do so is as follows:
$ for table in $(mysql -N -e "show tables;" zabbix); do mysqldump --add-locks --extended-insert --single-transaction --quick zabbix $table | bzip2 > zabbix_database_backup_$table.bz2; done
Note that in this case, we are not performing the backup for the whole database in a single transaction, and changes to the configuration could lead to inconsistencies across the tables. It is a good idea to schedule such a backup at a time when configuration changes would be unlikely.
If the consistency of the configuration tables is a likely problem, we could instead back up the configuration tables in a single transaction, and the tables that hold collected and recorded information separately:
$ mysqldump --add-locks --extended-insert --single-transaction zabbix --ignore-table=zabbix.history --ignore-table=zabbix.history_uint --ignore-table=zabbix.history_text --ignore-table=zabbix.history_str --ignore-table=zabbix.history_log --ignore-table=zabbix.trends --ignore-table=zabbix.trends_uint --ignore-table=zabbix.events --ignore-table=zabbix.alerts --ignore-table=zabbix.auditlog --ignore-table=zabbix.auditlog_details --ignore-table=zabbix.acknowledges | bzip2 > zabbix_database_backup_config_tables.bz2 $ mysqldump --add-locks --extended-insert --single-transaction zabbix history history_uint history_text history_str history_log trends trends_uint events alerts auditlog auditlog_details acknowledges | bzip2 > zabbix_database_backup_data_tables.bz2
Note that the configuration and data table distinction is a bit fuzzy in Zabbix, and several configuration tables still hold runtime information.