Things to Think About

We have talked about disaster recovery and business continuity and the importance of securing your data, but there is one other thing we want to talk about. What we really need to think about is how quickly we can recover from a data loss or hardware failure. You have hosts, and you have data. In the event of hardware loss, can you recover your hosts and data quickly?

If you have a database, your database will probably be the single most important data and server you will have. A database server is usually built lovingly, with disk arrays created and the operating system installed and tuned. In this scenario, provided you have recent data, rebuilding this host can take a long time.

Of course, there are a couple of ways around this time recovery time.

• The best is having primary/secondary databases. Your secondary database can be promoted to primary, providing time for you to rebuild your host.

• You can make use of virtualization technology. Your host can be rebuilt quickly from a template or image, or it can be moved off failing hardware without interruption. Combined with configuration management, like Puppet or Ansible, you will be able to recover your systems quickly.

• If you have bare-metal hosts, then you need to back up the disk architecture as well as the operating system and the data. Something like Relax-and-Recover might be a good idea, or something like Mass or Cobbler would help rebuild your servers quickly from a known build.

The worst situation we can have is relying on a running list of how we might have built the host 6 months ago while never having tried to build a host like it since. Actually, there probably is one worse situation: not having the running list.

For other types of hosts, like web servers, you might not need to back them up. You should take careful consideration of things like logs, but if you can rebuild your hosts quicker than restoring them, that would be your best option. If the rebuilding process is automated via configuration management, then you are in a great position as an administrator. This is the notion of treating servers as cattle and not pets and leaves your backup regimes much simpler.

Setting Up S3 Buckets

To back up our data, we are going to use Amazon’s S3 object store. For this we need to create an S3 bucket. An S3 bucket is used to store S3 objects . S3 objects are made up of a unique key, data, and metadata and are stored in a flat structure. While the keys can have prefixes (such as nightly/backupfile.zip, where nightly is the prefix and nightly/backupfile.zip is the key), they are not nested hierarchically but instead all live in the one bucket.

• We are going to set up an S3 bucket.

• We are to create a life-cycle version policy.

• Set the access policy to allow our backup user to put files in S3, list them, and get them.

When you have logged into the AWS web console for your account, you will need to make your way to the S3 service tab like in Figure 14-2.

Figure 14-2. Going to S3

From there we are going to create a bucket called our-backups. You are free to choose an appropriate name. Click the Create Bucket button to proceed.

As shown in Figure 14-3, we have created an S3 bucket called our-backups. We are next going to make sure that we have a life-cycle versioning policy on this bucket so that we don’t have to pay too much for any old data. What this policy will do is to move to lower-cost storage after 35 days and delete any objects just older than one year. The reason is that if we haven’t used the data after a month, the likelihood we will need it beyond that is small and so we will move it to a lower-cost tier of storage. We allow 5 days (30+5) after the month for a little buffer like we do at the expiration of 1 year. This does mean that retrieval will be slower, but the chance of us actually needing it is much lower after only a few weeks. There may be times when this is not the case, and we should revise as we went along.

Figure 14-3. Creating an S3 bucket

We can apply these life-cycle policies to different paths in our bucket or to the whole bucket. We are going to apply these to a prefix nightly, as these are the nightly backups for our hosts . After you have created the bucket, on the right pane of the console you will see a Lifecycle option. Click it and you will see the screen shown in Figure 14-4.

Figure 14-4. Adding a life-cycle rule

Click “Add rule” to start adding our rule.

In Figure 14-5 we are creating a life-cycle policy that is concerned only with objects that have the “nightly” prefix in their keys. Now we click Configure Rule.

Figure 14-5. Setting life-cycle policy target

In Figure 14-6 you can see there are three nobs we can tweak. Remember we talked about the S3 tiers earlier? We back up originally to the Standard tier. The next below that is Standard Infrequently Accessed (Standard-IA) and then below that is the Glacier storage . The Standard tier costs the most, so we want to transition off that as soon as possible, and we have chosen 35 days. This gives us the standard month plus a couple of days for holidays or people being off work as a buffer. We will not bother moving to Glacier storage because if we need it, we will need it relatively quicker than Glacier will take retrieve. Standard-IA also has a minimum object size of 128Kb, which means objects smaller than that will not be transitioned and will be deleted after the year is up. Also, once we have transitioned the files into Standard-IA, they need to remain there for 30 days before we can transition them to another tier like Glacier.

Figure 14-6. Configuring life-cycle rules

We permanently delete the files just after a year with again a buffer in case we discover we need to retrieve them just after a year is up. Any partly uploaded files will be cleaned up, and multipart uploads will end after 7 days. A multipart upload is when you upload files in smaller chunks. This will clean up and end any of those jobs. You can learn more about that process here:

• https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config

We can click the Review button when we are done.

We have the opportunity in Figure 14-7 to give our policy a name, and we have called it nightly-lifecycle. You can have more than one policy on a bucket. You use this page to review the settings; we are just going to click the Create and Activate Rule button.

Figure 14-7. Reviewing the life cycle

AWS User Policies

We want to secure the S3 bucket we have created to allow only those AWS users with the right credentials to access and upload to it. We are going to add an inline policy to the AWS user we asked you to create earlier.

Now we can add the policy to allow the proper access to the S3 bucket from our AWS user. To do that, we need to use the IAM service in AWS.

In Figure 14-8 we can see the policy we are going to attach to our AWS backup user. Lines 3–11 allow us to list all our S3 buckets. It is made up of three parts: the effect, the action, and resource. The effect can be Allow or Deny. The action is the API action being performed (in this case ListAllMyBuckets). The resource is the AWS resource (the Amazon resource name) you are applying this to; arn:aws:s3:::* is our S3.

Figure 14-8. Attaching an inline policy to our user

The second part of the policy statement shows the other API actions we allow. They manage our objects and allow us to put (PutObject), get (GetObject), and list (ListObject). In lines 22–25 we see that we are limiting these actions to the backup bucket ARN, arn:aws:s3:::our-backup, only.

For further information on this, please read the following:

• http://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html

Testing S3 Bucket Access

We are going to test our access to our backup S3 bucket. To do that, we are going to download and configure the awscli program that allows us to interact with AWS resources from our command line.

The first thing we will do is download the awscli package. This can be done using your native package manager or by downloading via PIP, the Python package manager.

$ sudo aptitude install –y awscli

If you use PIP, try this:

$ sudo pip install awscli

Once that has been installed, we can begin to use the aws command to interact with AWS resources. As the local Linux user that will run our backups, we need to configure our AWS credentials to access the AWS resources. We need to run the following command:

$ aws configure
AWS Access Key ID [None]: ALIAIJBASN4NOMJ4FLTR
AWS Secret Access Key [None]: +GDHpm+FLPj311tu6YJ29c9luMlQTHwntgy7vgs2
Default region name [None]: ap-southeast-2
Default output format [None]: json

Here we have used the aws configure command to add our AWS access key ID and our access secret. We have also specified our default region, which is the one closest to us, and the output format of JSON; these can be overridden at any time and are not really important.

The aws command takes several subcommands that deal with different resources. We are asking it to list an S3 resource. If we were to be working on EC2 resources, we would use aws ec2 <command>. You can get a list of AWS resources by issuing the following:

$ aws help

Now that we have our AWS credentials set up, we are going to test that we can list our S3 bucket.

$ aws s3 ls our-backup/

If there are no errors, that shows we can list our S3 bucket. We can try to copy, or put, a local file up into our S3 bucket. To do that, we issue the following:

$ aws s3 cp afile s3://our-backup/
upload: ./afile to s3://our-backup/afile
$ aws s3 ls our-backup/
2016-10-16 05:02:47      32768 afile

There you can see we have uploaded a small file to our S3 bucket. Let’s now see if we can copy the file back down to our host and then delete that file from S3.

$ aws s3 cp s3://our-backup/afile afile2
download: s3://our-backup/afile to ./afile2
$ aws s3 rm s3://our-backup/afile
delete: s3://our-backup/afile

These are all the actions that our backup program will perform, and we can see that our credentials allow us to perform these actions. Let’s now move on to creating our nightly backup with Duply.

Installing and Configuring Duply

Duply, as we have said, is the convenient way to manage our Duplicity backups. Duplicity can run multiple methods to securely back up files to remote systems . With AWS, it is going to copy our backup in file chunks of about 25Mb. It does this because it is easier to manage several small files than one rather large file.

There are two ways you can secure your data with Duplicity, one with GPG symmetric keys and the other using GPG asymmetric keys. We are going to use symmetric keys, but if you have GPG asymmetric keys available, please use those. We also store our data encrypted at rest with S3, and we could further encrypt our files using server-side encryption.

Installing Duply will install Duplicity as a requirement. It may not, however, install python-boto, and you may have to install this also. However, by installing awscli previously, we should have already installed python-boto.

$ sudo aptitude install –y duply

For CentOS, you will need to use YUM, of course.

Now that Duply is installed, we can investigate how to use the command. Table 14-3 shows the command usage to get help information (--help or help do not work).

When we interact with our backups , we will need to use the duply command like so:

$ duply <profile> command

To configure Duplicity via Duply, we need to create a backup profile. This is going to store our configuration details for Duplicity to use.

$ duply nightly create

# although called exclude, this file is actually a globbing file list

Congratulations. You just created the profile 'nightly'.
The initial config file has been created as
'/home/exbackup/.duply/nightly/conf'.
You should now adjust this config file to your needs.

IMPORTANT:
  Copy the _whole_ profile folder after the first backup to a safe place.
  It contains everything needed to restore your backups. You will need
  it if you have to restore the backup from another system (e.g. after a
  system crash). Keep access to these files restricted as they contain
  _all_ informations (gpg data, ftp data) to access and modify your backups.

  Repeat this step after _all_ configuration changes. Some configuration
  options are crucial for restoration.

This command creates a directory (∼/.duply) in the home directory of the user issuing the command. In that directory, a profile (∼/.duply/nightly) directory is created, and in that directory, a ∼/.duply/nightly/conf file is created. Created is an excludes file (∼/.duply/nightly/exclude) that can contain the directories we want to include or exclude from our backup profile.

In this nightly profile directory we can also include pre- and post-scripts that can be run before or after our backups. These should be included in .duply/nightly/pre or .duply/nightly/post files.

Also, the output in the end of the create command tells us to make a safe copy of the whole profile directory after the first backup as we may need it when we restore our systems.

GPG_PW='mygeniusbigbikeisnice'

<scheme://><uid>:<password>@<other.host>/<some_dir>

TARGET='s3://s3-ap-southeast-2.amazonaws.com/our-backup/nightly'

TARGET_USER='ALIAIJBASN4NOMJ4FLTR'
TARGET_PASS='+GDHpm+FLPj311tu6YJ29c9luMlQTHwntgy7vgs2'

TARGET_USER=`awk '/aws_access_key_id/ {print $3}' ∼.aws/credentials`
TARGET_PASS=`awk '/aws_secret_access_key/ {print $3}' ∼.aws/credentials`

TARGET_USER=`echo $AWS_ACCESS_KEY_ID`
TARGET_PASS=`echo $AWS_SECRET_ACCESS_KEY`

SOURCE='/'

SOURCE='/home/jsmith'

MAX_AGE=1M

- /path/to/exclude
+ /path/dir/of/directory/to/include
+ /path/*/**.py

- /data/
- /dev/
- /lost+found/
- /media/
- /mnt/
- /proc/
- /run/
- /srv/
- /sys/
- /tmp/
- /var/tmp

$ sudo visudo
# Cmnd alias specification
Cmnd_Alias BACKUPS = /usr/bin/duply

%backup ALL=(ALL) NOPASSWD:  BACKUPS

$ sudo usermod -aG backup exbackup

$ sudo duply nightly backup
Start duply v1.11, time is 2016-10-15 23:57:21.
Using profile '/home/exbackup/.duply/nightly'.
Using installed duplicity version 0.7.06, python 2.7.12, gpg: unsafe (Home: ∼/.gnupg), awk 'GNU Awk 4.1.3, API: 1.1 (GNU MPFR 3.1.4, GNU MP 6.1.0)', grep 'grep (GNU grep) 2.25', bash '4.3.46(1)-release (x86_64-pc-linux-gnu)'.
Signing disabled. Not GPG_KEY entries in config.
Checking TEMP_DIR '/tmp' is a folder and writable (OK)
Test - Encryption with passphrase (OK)
Test - Decryption with passphrase (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.12989.1476575841_*'(OK)

--- Start running command PRE at 23:57:21.238 ---
Skipping n/a script '/home/exbackup/.duply/nightly/pre'.
--- Finished state OK at 23:57:21.252 - Runtime 00:00:00.013 ---

--- Start running command BKP at 23:57:21.262 ---
Reading globbing filelist /home/exbackup/.duply/nightly/exclude
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Error accessing possibly locked file /var/lib/lxcfs
--------------[ Backup Statistics ]--------------
StartTime 1476588974.26 (Sun Oct 16 03:36:14 2016)
EndTime 1476591675.63 (Sun Oct 16 04:21:15 2016)
ElapsedTime 2701.37 (45 minutes 1.37 seconds)
SourceFiles 140466
SourceFileSize 2375250457 (2.21 GB)
NewFiles 140466
NewFileSize 2375250457 (2.21 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 140466
RawDeltaSize 2281512996 (2.12 GB)
TotalDestinationSizeChange 198548201 (189 MB)
Errors 1
-------------------------------------------------

--- Finished state OK at 04:39:42.342 - Runtime 01:03:29.436 ---

--- Start running command POST at 04:39:42.354 ---
Skipping n/a script '/home/exbackup/.duply/nightly/post'.
--- Finished state OK at 04:39:42.375 - Runtime 00:00:00.020 ---

$ aws s3 ls our-backup/nightly/
2016-10-16 03:35:34          0
2016-10-16 04:25:55   36877167 duplicity-full-signatures.20161015T235721Z.sigtar.gpg
2016-10-16 04:39:28     634738 duplicity-full.20161015T235721Z.manifest.gpg
2016-10-16 03:35:37   26244347 duplicity-full.20161015T235721Z.vol1.difftar.gpg
2016-10-16 03:35:37   26194091 duplicity-full.20161015T235721Z.vol10.difftar.gpg
2016-10-16 03:35:37   26196749 duplicity-full.20161015T235721Z.vol11.difftar.gpg
2016-10-16 03:35:38   26227650 duplicity-full.20161015T235721Z.vol12.difftar.gpg
...

$ sudo duply nightly list
Start duply v1.11, time is 2016-10-16 10:41:48.
Using profile '/home/exbackup/.duply/nightly'.
Using installed duplicity version 0.7.06, python 2.7.12, gpg: unsafe (Home: ∼/.gnupg), awk 'GNU Awk 4.1.3, API: 1.1 (GNU MPFR 3.1.4, GNU MP 6.1.0)', grep 'grep (GNU grep) 2.25', bash '4.3.46(1)-release (x86_64-pc-linux-gnu)'.
Signing disabled. Not GPG_KEY entries in config.
Checking TEMP_DIR '/tmp' is a folder and writable (OK)
Test - Encryption with passphrase (OK)
Test - Decryption with passphrase (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.18655.1476614508_*'(OK)

--- Start running command LIST at 10:41:49.064 ---
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 16 10:01:58 2016
Thu Oct 13 07:21:56 2016 .
Thu Sep 29 23:57:15 2016 bin
Fri Jun 24 15:44:14 2016 bin/bash
Tue Jan 19 23:11:42 2016 bin/btrfs
Tue Jan 19 23:11:41 2016 bin/btrfs-calc-size
...<snip>...
Thu May 26 23:31:30 2016 sbin/wipefs
Tue Feb 16 13:19:00 2016 sbin/xfs_repair
Fri Feb 19 15:21:11 2016 sbin/xtables-multi
Thu May 26 23:31:30 2016 sbin/zramctl
Thu Sep  1 17:37:32 2016 snap
Tue Oct 11 14:07:06 2016 vmlinuz
Thu Sep 29 12:36:50 2016 vmlinuz.old
--- Finished state OK at 10:41:58.667 - Runtime 00:00:09.603 ---

$ sudo duply nightly fetch bin/bash /tmp/restores/bash
Start duply v1.11, time is 2016-10-16 11:41:15.
Using profile '/home/exbackup/.duply/nightly'.
Using installed duplicity version 0.7.06, python 2.7.12, gpg: unsafe (Home: ∼/.gnupg), awk 'GNU Awk 4.1.3, API: 1.1 (GNU MPFR 3.1.4, GNU MP 6.1.0)', grep 'grep (GNU grep) 2.25', bash '4.3.46(1)-release (x86_64-pc-linux-gnu)'.
Signing disabled. Not GPG_KEY entries in config.
Checking TEMP_DIR '/tmp' is a folder and writable (OK)
Test - Encryption with passphrase (OK)
Test - Decryption with passphrase (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.20766.1476618075_*'(OK)

--- Start running command FETCH at 11:41:15.242 ---
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20161015T235721Z.sigtar.gpg to local cache.
Copying duplicity-full-signatures.20161016T100158Z.sigtar.gpg to local cache.
Copying duplicity-full.20161015T235721Z.manifest.gpg to local cache.
Copying duplicity-full.20161016T100158Z.manifest.gpg to local cache.
Last full backup date: Sun Oct 16 10:01:58 2016
--- Finished state OK at 12:04:32.407 - Runtime 00:23:17.164 ---

$ crontab –e
0 2 * * * env HOME=/home/exbackup sudo duply nightly backup

Getting the Software

The Bareos download page ( https://www.bareos.org/en/download.html ) provides links to RPM and DEB packages of the latest releases for you to download. Distributions can be several versions behind the latest release, and only Ubuntu is currently shipping with Bareos. CentOS still has Bacula available but not Bareos. You can get the packages and tarballs here: http://download.bareos.org/bareos/ .

For CentOS we will install the following YUM repo contents into the file /etc/yum.repos.d/bareos.repo, as shown here:

[bareos_bareos-15.2]
name=Backup Archiving Recovery Open Sourced 15.2 (CentOS_7)
type=rpm-md
baseurl=http://download.bareos.org/bareos/release/15.2/CentOS_7/
gpgcheck=1
gpgkey=http://download.bareos.org/bareos/release/15.2/CentOS_7/repodata/repomd.xml.key
enabled=1

We are then able to install the following packages:

$ sudo yum install bareos-client bareos-director bareos-storage bareos-storage-glusterfs bareos-bconsole

On Ubuntu, you have to use the following:

$ sudo aptitude install bareos-filedaemon bareos-bconsole bareos-director bareos-storage

On Ubuntu, you will be asked to give a username and password for your MariaDB service via the dbconfig-common helper. At this stage we are not ready to provide that information, so we are going to choose <No>. If you have your details or are setting this up on your localhost, you can select yes and go through the steps for setting up the database. We are going to go through the database configuration shortly.

The installation process installs all the required packages and any dependencies and creates the default configuration files in the configuration directory /etc/bareos. The bareos user is also created at this point. To get Bareos up and running, you will have to edit the following files in the configuration directory to access the Bareos management console: /etc/bareos/bareos-dir.conf, /etc/bareos/bareos-sd.conf, /etc/bareos/bareos-fd.conf, and /etc/bareos/bconsole.conf.

Configuring the Database

We talked about MariaDB servers in Chapter 11. We are now going to create a database on that host to house our catalog. Along the way, we’ll configure the MariaDB host to be backed up. First, we need to create the correct accounts and the database on our MariaDB host.

We will assume that the headoffice.example.com host has the MariaDB database server installed and that we are running our backups from backup.example.com. We can, of course, house this database on the one host, headoffice.example.com, if we didn’t have an extra host available to be the separate backup host. In that case, you should use localhost in the following examples where we have used backup.example.com. The other alternatives are to store a separate MariaDB database server (or PostgreSQL, if you want) on your backup host or to use a SQLite database on the backup host.

On the headoffice.example.com MariaDB host, we will create a new database for Bareos and grant access to the user bareos from our backup.example.com host. We will then make sure that our backup.example.com host can connect to port 3306 on headoffice.example.com by making a change to the firewall. While we do that, we will also make sure that our Bareos File daemon is installed on headoffice.example.com and that we have firewall access for TCP port 9103 on that host from backup.example.com.

We need to install the following packages on the host that will house the Bareos database; they contain the helper scripts that help manage the database:

$ sudo aptitude install –y bareo-database-common bareos-database-mysql

Of course, on CentOS, you would use the yum command.

On headoffice.example.com we will do the following:

$ sudo mysql –u root –p
Enter Password:
...
mysql> CREATE DATABASE 'bareos';
Query OK, 1 row affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON bareos.* to [email protected]
IDENTIFIED BY 'somepassword';
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql> q;
Bye

In the previous code, we have created a database called bareos. On that database, we have given the user bareos signing in from backup.example.com all privileges if they use the somepassword password. If you were setting this up for a localhost, you would change [email protected] to bareos@localhost. Finally, we use FLUSH PRIVILEGES, which reloads the grants table with all the recent changes.

On CentOS and Ubuntu hosts, there are scripts to create, drop, update, and back up your Bareos databases. You will find them in /usr/lib/bareos/scripts/.

To create the database tables for the Bareos database, we will run the following on the backup.example.com host:

$ sudo /usr/lib/bareos/scripts/create_bareos_database
Creating mysql database
Creating of bareos database succeeded.
$ sudo /usr/lib/bareos/scripts/grant_bareos_privileges
Granting mysql tables
Privileges for user bareos granted ON database bareos.

Running these scripts supplied by Bareos will create the necessary tables for our catalog and appropriate grants for the Bareos user.

We confirm that the database was created by running the following command from backup.example.com:

$ sudo –u bareos mysql –h headoffice.example.com –u bareos –p –D bareos
–e 'show tables;'

This command connects to the MariaDB database running on the headoffice.example.com host using the Bareos username and password. -p prompts for a password, and -psomepassword is how you would submit the password on the command line (there is no space between –p and the password). Getting the output will do two things: confirm that our username and password are working correctly and that the scripts actually did create some tables.

Other things to be aware of are that the /data/backups/bareos/FileStorage directory must exist, and it must also be writable by the Bareos user, as should the /var/log/bareos.log file.

Configuring Bareos

We’ll now examine the configuration files that need to be defined for your Bareos server. As mentioned earlier, there are three files for dealing with the Bareos server itself and one for the management console.

What we are going to do in these exercises is the following:

• In the bareos-dir.conf file:

  • Configure the director with the correct name

  • Configure the default Linux backup job definition

  • Configure a backup job for the headoffice server

  • Configure the set of files to back up

  • Configure the schedule of the backup

  • Configure the client to back up

  • Add the Storage daemon configuration for the Director

  • Configure the pools

  • Configure the console

  • Configure the messages

• In the bareos-sd.conf file:

  • Configure the storage daemon

• In the bareos-fd.conf file:

  • Configure the client daemon

Before we configure anything in Bareos , we should explain the relationship these files have with each other. For the Bareos daemons to communicate with each other, they require the exchange of a set of passwords. Each Storage daemon (you can have more than one) and each File daemon (you will generally have more than one) must contain the name of the Bareos Director and the password it uses to verify itself. The password is set in two places for each daemon that wants to communicate with the Bareos Director.

Figure 14-9 shows the relationship between the configuration files and their common definitions.

Figure 14-9. Bareos common configuration file definitions

As you can see in Figure 14-9, the Bareos /etc/bareos/bareos-dir.conf file for the Director will contain information for the Storage daemon, the Client daemon, and the bconsole program. You can also see that definitions of configuration objects, such as Storage, Pools, FileSets, and so forth, require the name of the object being configured and the definitions to be enclosed in curly brackets: {<definition = value>}. The definitions must be one per line.

You can also see in Figure 14-9 that in the Storage file, the Name and Media Type entries must match the Device Type and Media Type entries in the Device object definition in the Director file. The Director Name and Password in the Director object definition in the Storage file must match the storage password in the Storage section of the Director file.

When we define a client in our Director file , the password assigned to the client must be present in the File daemon configuration for that Director name. Also in Figure 13-2, you see that the bconsole program configuration file requires the name of the Director and the password for that Director before it will establish a connection.

Let’s start with the main file, the Director, at /etc/bareos/bareos-dir.conf.

Director {                            # define myself
  Name = backup-dir
  QueryFile = "/usr/lib/bareos/scripts/query.sql"
  Maximum Concurrent Jobs = 10
  Password = "uogx2tNL9dRUwGfNvY/b+uQrU8osZn+JOM7t8iIrpszN"     # Console password
  Messages = Daemon
  Auditing = yes

  # Enable the Heartbeat if you experience connection losses
  # (eg. because of your router or firewall configuration).
  # Additionally the Heartbeat can be enabled in bareos-sd and bareos-fd.
  #
  # Heartbeat Interval = 1m

  # remove comment in next line to load dynamic backends from specified directory
  # Backend Directory = /usr/lib64/bareos/backends

  # remove comment from "Plugin Directory" to load plugins from specified directory.
  # if "Plugin Names" is defined, only the specified plugins will be loaded,
  # otherwise all director plugins (*-dir.so) from the "Plugin Directory".
  #
  # Plugin Directory = /usr/lib64/bareos/plugins
  # Plugin Names = ""
}

JobDefs {
  Name = "DefaultLinux"
  Type = Backup
  Level = Incremental
  Client = bareos-fd
  FileSet = "Linux”
  Schedule = "WeeklyCycle"
  Storage = File
  Messages = Standard
  Pool = Incremental
  Priority = 10
  Write Bootstrap = "/var/lib/bareos/%c.bsr"
  Full Backup Pool = Full
  Incremental Backup Pool = Incremental
}

Job {
  Name = "BackupCatalog"
  JobDefs = "DefaultJob"
  Level = Full
  FileSet="Catalog"
  Schedule = "WeeklyCycleAfterBackup"

  # This creates an ASCII copy of the catalog
  # Arguments to make_catalog_backup.pl are:
  #  make_catalog_backup.pl <catalog-name>
  RunBeforeJob = "/usr/lib/bareos/scripts/make_catalog_backup.pl MyCatalog"

  # This deletes the copy of the catalog
  RunAfterJob  = "/usr/lib/bareos/scripts/delete_catalog_backup"

  # This sends the bootstrap via mail for disaster recovery.
  # Should be sent to another system, please change recipient accordingly
  Write Bootstrap = "|/usr/bin/bsmtp -h localhost -f "(Bareos) " -s "Bootstrap for Job %j" root@localhost" # (#01)
  Priority = 11                   # run after main backup
}

Job {
  Name = "RestoreFiles"
  Type = Restore
  Client=au-mel-centos-1-fd
  FileSet = "Linux All"
  Storage = File
  Pool = Incremental
  Messages = Standard
  Where = /tmp/bareos-restores
}

Job {
  Name = headoffice.example.com
  Client = headoffice-fd
  Enabled = yes
  JobDefs = "DefaultLinux"
}

FileSet {
  Name = "Linux"
  Include {
    Options {
      Compression = GZIP
      Signature = SHA1
     }
     File = "/etc"
     File = "/var/lib" File = "/data" File = "/home"
     }
  Exclude {
    File = "/proc"
    File = "/sys"
    File = "/dev"
    File = “/run”
    File = “/tmp”
    File = "/data/backups/FileStorage"
  }
}

File = "C:/"
File = "D:/"

FileSet {
  Name = "Windows All Drives"
  Enable VSS = yes
  Include {
    Options {
      Signature = MD5
      Drive Type = fixed
      IgnoreCase = yes
      WildFile = "[A-Z]:/pagefile.sys"
      WildDir = "[A-Z]:/RECYCLER"
      WildDir = "[A-Z]:/$RECYCLE.BIN"
      WildDir = "[A-Z]:/System Volume Information"
      WildDir = "*/Cache/*"
      WildDir = "*/cache/*"
      Exclude = yes
      }
  @/etc/bareos/windows.list
  }
}

FileSet {
  Name = "Catalog"
  Include {
    Options {
      signature = MD5
    }
    File = "/var/lib/bareos/bareos.sql" # database dump
    File = "/etc/bareos"                # configuration
  }
}

Schedule {
  Name = "WeeklyCycle"
  Run = Full 1st 3rd sun at 23:05
  Run = Differential 2nd 4th 5th sun at 23:05
  Run = Incremental mon-sat at 23:05
}
Schedule {
  Name = "WeeklyCycleAfterBackup"
  Run = Full sun-sat at 23:10
}

Client {
  Name = headoffice-fd
  Address = headoffice.example.com
  Password = "6rZQQqVsOJeeTPefrG2AslT5ODxtPWO1hNsd7Re1u6J/"
}

Storage {
  Name = File
# Do not use "localhost" here
  Address = backup.example.com
  Password = "4ncvx7V+Mw4NOMDMyuYjCHsNg1nKgcdh8nl0szWpi6t4"
  Device = FileStorage
  Media Type = File
}

Catalog {
  Name = MyCatalog
  # Uncomment the following lines if you want the dbi driver
  # dbdriver = "dbi:postgresql"; dbaddress = 127.0.0.1; dbport =
  #dbdriver = "postgresql"
  dbdriver = "mysql"
  dbname = "bareos"
  dbuser = "bareos"
  dbpassword = "somepassword"
  dbaddress = headoffice.example.com
}

Pool {
  Name = Full
  Pool Type = Backup
  Recycle = yes
  Recycle Oldest Volume = Yes
  AutoPrune = yes
  Volume Retention = 365 days
  Maximum Volume Bytes = 50G
  Maximum Volumes = 100
  Label Format = "Full-${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}"
}

Pool {
  Name = Differential
  Pool Type = Backup
  Recycle = yes
  Recycle Oldest Volume = Yes
  AutoPrune = yes
  Volume Retention = 90 days
  Maximum Volume Bytes = 10G
  Maximum Volumes = 100
  Label Format = "Differential-${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}"
}

Pool {
  Name = Incremental
  Pool Type = Backup
  Recycle = yes
  Recycle Oldest Volume = Yes
  AutoPrune = yes
  Volume Retention = 30 days
  Maximum Volume Bytes = 1G
  Maximum Volumes = 100
  Label Format = "Incremental-${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}"
}

Console {
  Name = example-mon
  Password = "tcqT4bG9NLl11YoZ5GXFLrcb8o9mSuM2U4rxl7mgH8eD"
  CommandACL = status, .status
  JobACL = *all*
}

Messages {
  Name = Standard
  mailcommand = "/usr/bin/bsmtp -h localhost -f "(Bareos) <%r>" -s "Bareos: %t %e of %c %l" %r"
  operatorcommand = "/usr/bin/bsmtp -h localhost -f "(Bareos) <%r>" -s "Bareos: Intervention needed for %j" %r"
  mail = [email protected] = all, !skipped, !audit # (#02)
  operator = [email protected] = mount     # (#03)
  console = all, !skipped, !saved, !audit
  append = "/var/log/bareos/bareos.log" = all, !skipped, !audit
  catalog = all, !audit
}

#
# Message delivery for daemon messages (no job).
#
Messages {
  Name = Daemon
  mailcommand = "/usr/bin/bsmtp -h localhost -f "(Bareos) <%r>" -s "Bareos daemon message" %r"
  mail = [email protected] = all, !skipped, !audit # (#02)
  console = all, !skipped, !saved, !audit
  append = "/var/log/bareos/bareos.log" = all, !skipped, !audit
  append = "/var/log/bareos/bareos-audit.log" = audit
}

Storage {                             # definition of myself
  Name = example-sd
  Maximum Concurrent Jobs = 20

  # remove comment from "Plugin Directory" to load plugins from specified directory.
  # if "Plugin Names" is defined, only the specified plugins will be loaded,
  # otherwise all storage plugins (*-sd.so) from the "Plugin Directory".
  #
  # Plugin Directory = /usr/lib64/bareos/plugins
  # Plugin Names = ""
}

#
# List Directors who are permitted to contact Storage daemon
#
Director {
  Name = example-dir
  Password = "4ncvx7V+Mw4NOMDMyuYjCHsNg1nKgcdh8nl0szWpi6t4"
}

Director {
  Name = example-mon
  Password = "/H9QQD9YJOcd9jTnX1w/aFMMghxFuHmPYuJv882Sk54F"
  Monitor = yes
}
Messages {
  Name = Standard
  director = example-dir = all
}

Device {
  Name = FileStorage
  Media Type = File
  Archive Device = /data/backups/FileStorage
  LabelMedia = yes;
  Random Access = Yes;
  AutomaticMount = yes;
  RemovableMedia = no;
  AlwaysOpen = no;
}

Director {
FileDaemon {                          # definition of myself
  Name = headoffice-fd
  WorkingDirectory = /var/lib/bareos
  Maximum Concurrent Jobs = 20

  # remove comment from "Plugin Directory" to load plugins from specified directory.
  # if "Plugin Names" is defined, only the specified plugins will be loaded,
  # otherwise all storage plugins (*-fd.so) from the "Plugin Directory".
  #
  # Plugin Directory = /usr/lib64/bareos/plugins
  # Plugin Names = ""

  # if compatible is set to yes, we are compatible with bacula
  # if set to no, new bareos features are enabled which is the default
  # compatible = yes
}

#
# List Directors who are permitted to contact this File daemon
#
Director {
  Name = example-dir
  Password = "6rZQQqVsOJeeTPefrG2AslT5ODxtPWO1hNsd7Re1u6J/"
}

#
# Restricted Director, used by tray-monitor to get the
#   status of the file daemon
#
Director {
  Name = example-mon
  Password = "WOhEm9JC+uBvpNBJta1RuA7NJSmUgQrz3QPRgPYhRzyZ"
  Monitor = yes
}

# Send all messages except skipped files back to Director
Messages {
  Name = Standard
  director = example-dir = all, !skipped, !restored
}

$ sudo /usr/sbin/bareos-dir -t -c /etc/bareos/bareos-dir.conf

$ sudo /usr/sbin/bareos-fd -t -c /etc/bareos/bareos-fd.conf

$ sudo /usr/sbin/bareos-sd -t -c /etc/bareos/bareos-sd.conf

$ sudo systemctl enable bareos-dir bareos-sd bareos-fd
$ sudo systemctl start bareos-dir bareos-sd bareos-fd
$ sudo systemctl status bareos-dir bareos-sd bareos-fd

Director {
  Name = example-dir
  DIRport = 9101
  Address = backup.example.com
  Password = "uogx2tNL9dRUwGfNvY/b+uQrU8osZn+JOM7t8iIrpszN"
}

Managing Bareos with bconsole

After you configure Bareos, you can start the services by using the systemctl command, like we just did previously. We can now start the Bareos console program using the bconsole command like so:

$ sudo /usr/sbin/bconsole

First, we can list the current status of the Director by issuing the status all command. Issuing this command gives a detailed list of what is happening on the Bareos Director, including the attached Storage daemons and File daemons.

To find out what the Director itself is doing, we use the stat dir command. This command gives an overview of the pending jobs, the current running jobs, and the jobs that have been completed. We access the console by issuing the following command:

$ sudo /usr/sbin/bconsole
Connecting to Director backup.example.com:9101
1000 OK: example-dir Version: 15.2.2 (16 November 2015)
Enter a period to cancel a command.
* stat dir
example-dir Version: 15.2.2 (16 November 2015) x86_64-redhat-linux-gnu redhat CentOS Linux release 7.0.1406 (Core)
Daemon started 10-Oct-16 00:56. Jobs: run=0, running=0 mode=0 db=mysql
 Heap: heap=274,432 smbytes=86,156 max_bytes=86,156 bufs=321 max_bufs=322

Scheduled Jobs:
Level          Type     Pri  Scheduled          Name                      Volume
=================================================================================
Incremental    Backup    10  10-Oct-16 23:05    headoffice.example.com     *unknown*
Full                 Backup    11  10-Oct-16 23:10    BackupCatalog                   *unknown*

Running Jobs:
Console connected at 10-Oct-16 00:57
No Jobs running.
====
No Terminated Jobs.
====
*

As you can see, we have no jobs completed and no jobs currently running. We have two jobs scheduled to run on October 11 at 23:05, one that backs up headoffice.example.com and one that backs up the catalog (BackupCatalog). Take some time to explore bconsolea little more. For example, type help and then press Enter to see a list of commands available to you on the command line. Table 14-4 lists the most useful commands.

You can run a job or a backup in two ways. One is to use the run command and press Enter. You are then presented with an easy-to-understand menu-driven way to submit your backup. The other is to place the commands directly on the command line like this:

* run job=headoffice.example.com level=full priority=7 yes

When we run the * stat dir command, we can see that our job is running.

Running Jobs:
Console connected at 10-Oct-16 01:21
 JobId Level   Name                       Status
============================================================
     1 Full    headoffice.example.com.2016-10-10_01.21.34_04 is running
====

From that status you can see that we have a job ID of 1 and that we are running a full backup, since we do not have a full backup yet. Bareos will automatic start a full backup if it does not exist yet. A new volume has been created, and the backup is restoring to it. If you type mess at the command line, you will receive any messages that are available about your job.

Now if we wanted to stop this job, we would use the cancel command with JobId 1 as an argument. You can see in the previous Running Jobs output the JobId to use.

*cancel jobid=1

The cancel command can be run without arguments and will automatically select the JobId for you (if only one job is running; otherwise, it presents a list to choose from). We have run the job again, this time without canceling it, and it has completed. When we do a stat dir in the Terminated Jobs section, we find the following:

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name
==================================================================
 2      Full     8,908      140.1 M  OK      10-Oct-16 01:21 headoffice.example.com

The preceding output shows that we have a full save of the Linux FileSet for the host headoffice.example.com. You can see the number of files it holds and the data volume. Let’s try to restore the /etc/hosts file using the restore command.

* restore
To select the JobIds, you have the following choices:
     1: List last 20 Jobs run
     2: List Jobs where a given File is saved
     3: Enter list of comma separated JobIds to select

The list of options to use here actually goes to 12, but we will use option 2 to find out the JobId (which we already know from the Terminated Jobs output to be 4).

Select item: (1-12): 2
Automatically selected Client: headoffice-fd
Enter Filename (no path):hosts
+-------+------------+---------------------+---------+-----------+----------+-------------+
| JobId | Name       | StartTime           | JobType | JobStatus | JobFiles | JobBytes    |
+-------+------------+---------------------+---------+-----------+----------+-------------+
| 2     | /etc/hosts | 2016-10-10 01:21:36 | B       | T         | 8908     | 140144767   |
+-------+------------+---------------------+---------+-----------+----------+-------------+

Now that we know our JobId, we would then use option 3 to restore our hosts file, as follows:

Select item: (1-12): 3
Enter JobId(s), comma separated, to restore: 4
You have selected the following JobId: 4

Building directory tree for JobId 4 ... ++++++++++++++++++++++++++++++++++++
1 Job, 1,174 files inserted into the tree. cwd is: /
$ cd /etc
cwd is: /etc/
$ mark hosts
1 file marked.
$ done
Bootstrap records written to /var/lib/bareos/example-dir.restore.1.bsr

The job will require the following
   Volume(s)                 Storage(s)                SD Device(s)
===========================================================================

    Full-0001                 File                      FileStorage

Volumes marked with "*" are online.

1 file selected to be restored.

Run Restore job
JobName:         RestoreFiles
Bootstrap:       /var/lib/bareos/example-dir.restore.1.bsr
Where:           /tmp/bareos-restores
Replace:         Always
FileSet:         Linux All
Backup Client:   headoffice-fd
Restore Client:  headoffice-fd
Format:          Native
Storage:         File
When:            2016-10-10 01:32:35
Catalog:         MyCatalog
Priority:        10
Plugin Options:  *None*
OK to run? (yes/mod/no): yes
Job queued. JobId=4

When selecting the files you want to restore, you can use the ls and cd commands to list and change directories as you search for the files .

cwd is: /
$ cd /etc
cwd is: /etc/
$ mark hosts
1 file marked.
$ done

Here we have used the mark command to select the file we want to restore after using the cd command to navigate through the backed-up file system. To select everything recursively in a directory, you can use the mark * command. When you are finished selecting the files you want to restore, use the done command.

You are given a list of the media volumes required to perform this restore. You will have to load them if they are not already available to your backup host. Then you are given the rundown of the restore job you are going to run. Make a note of Where: /tmp/bareos-restores, as this is where your restored file will be found.

Looking at stat dir and the Terminated Jobs section, a complete restore will look like this:

JobId  Level    Files      Bytes    Status    Finished          Name
==========================================================================
    4      1      403        OK     10-Oct-16 01:32          RestoreFiles

Now we can check that the file has been restored to the /tmp/bareos-restores directory.

$ $ sudo ls -l /tmp/bareos-restores/etc/
total 4
-rw-r--r--. 1 root root 403 Oct 10 00:53 hosts

We could always use the restore command to restore to a different host and a different destination by changing the Client and Where options before we confirm that we want to run the restore job.

Using GlusterFS for Backup Storage

We are going to quickly go through how to set up your Bareos server to use the GlusterFS storage as our backup device. As you can remember, GlusterFS provides us with reliable and expandable storage options on commodity hardware and can be a great option for safely keeping backups on disk. In this section, we are going to do the following:

• Create a new backup volume in GlusterFS and update our GlusterFS configuration

• Add a new record to our DNS server

• Mount volume and setup directories

• Configure Bareos storage

Bareos can talk directly to the Gluster daemon via the libgfapi provided by the bareos-storage-glusterfs package. This means we don’t FUSE mount the Gluster volume but allow Bareos direct access to the daemon.

$ sudo mkfs.xfs -f -L brick3_r1 /dev/sdf && 
   sudo mkdir -p /data/brick3_r1 && 
   sudo bash -c 'echo LABEL=brick3_r1 /data/brick3_r1 xfs defaults 1 2 >> /etc/fstab' && 
   sudo mount –a && 
   sudo mkdir /data/brick3_r1/backups

$ sudo gluster volume create backups replica 3 
  au-mel-dfs-1:/data/brick3_r1/backups 
  au-mel-dfs-2:/data/brick3_r1/backups 
  au-mel-dfs-3:/data/brick3_r1/backups

$ sudo gluster volume start backups

volume management
    type mgmt/glusterd
    option working-directory /var/lib/glusterd
...<snip>...
    option rpc-auth-allow-insecure on
end-volume

$ gluster volume set backups server.allow-insecure on

$ sudo nsupdate -k /etc/bind/ddns_update.key
> server localhost
> update add storage.example.com 360 A 192.168.0.240
> update add storage.example.com 360 A 192.168.0.241
> update add storage.example.com 360 A 192.168.0.242
> send
> quit

$ dig @localhost storage.example.com
....
;; ANSWER SECTION:
storage.example.com.    360    IN    A    192.168.0.242
storage.example.com.    360    IN    A    192.168.0.240
storage.example.com.    360    IN    A    192.168.0.241

$ sudo mount –t glusterfs storage.example.com:/backups /mnt

$ id bareos
uid=899(bareos) gid=986(bareos) groups=986(bareos),6(disk),30(tape)

$ sudo mkdir /mnt/FileStore && 
  sudo chown 899:986 /mnt/FileStore && 
  sudo chmod 2770 /mnt/FileStore

$ sudo umount /mnt

/etc/bareos/bareos-sd.d/device-gluster.conf
Device {
  Name = GlusterStorage
  Archive Device = "Gluster Device"
  Device Options = "uri=gluster://storage.example.com/backups/FileStore"
  Device Type = gfapi
  Media Type = GlusterFile
  Label Media = yes
  Random Access = yes
  Automatic Mount = yes
  Removable Media = no
  Always Open = no
}

/etc/bareos/bareos-dir.conf
Storage {
  Name = Gluster
  Address = backup.example.com                # N.B. Use a fully qualified name here
  Password = "4ncvx7V+Mw4NOMDMyuYjCHsNg1nKgcdh8nl0szWpi6t4"
  Device = GlusterStorage
  Media Type = GlusterFile
  Maximum Concurrent Jobs = 10
}

JobDefs {
  Name = "DefaultLinux"
  Type = Backup
...<snip> ...
  Storage = Gluster
...<snip>...
}

/etc/bareos/bareos-sd.conf
...<snip>...
@/etc/bareos/bareos-sd.d/device-gluster.conf
...<snip>...

*status Storage=Gluster
... <snip>...
====

Device status:

Device "GlusterStorage" (Gluster Device) is not open.
==
...<snip>...

Backing Up Databases with Bareos Plug-Ins

We now want to show you a more advanced scenario. We have a MariaDB database running on our headoffice.example.com host. Currently it holds our Bareos catalog database, but it could also contain the WordPress database that we created in Chapter 11 or any other databases we use.

As we have mentioned, you can install Bareos plug-ins on your Storage and File daemons to do special tasks. We are going to install a Python plug-in on the database server so the File daemon can back up the MariaDB database for us. This plug-in will make use of the mysqldump and mysql commands to make a database backup, consisting of a file for each database found.

The plug-in we are going to use can be found here:

• https://github.com/bareos/bareos-contrib/tree/master/fd-plugins/mysql-python

We are going to install it in /usr/lib64/bareos/plugins and then make a configuration change to update our bareos-fd.conf file. To do that, we going to copy two files from the GitHub repository with the curl command.

# curl -s https://raw.githubusercontent.com/bareos/bareos-contrib/master/fd-plugins/mysql-python/BareosFdMySQLclass.py 
  -o /usr/lib64/bareos/plugins/BareosFdMySQLclass.py
# curl -s https://raw.githubusercontent.com/bareos/bareos-contrib/master/fd-plugins/mysql-python/bareos-fd-mysql.py 
  -o /usr/lib64/bareos/plugins/bareos-fd-mysql.py

These two files have been copied via curl into the plugins directory as the root user. Now we have to configure the /etc/bareos/bareos-fd.conf file on the headoffice.example.com server where the database lives.

FileDaemon {                          # definition of myself
  Name = headoffice-fd
  Maximum Concurrent Jobs = 20

  # remove comment from "Plugin Directory" to load plugins from specified directory.
  # if "Plugin Names" is defined, only the specified plugins will be loaded,
  # otherwise all storage plugins (*-fd.so) from the "Plugin Directory".
  #
  Plugin Directory = /usr/lib64/bareos/plugins
  Plugin Names = "bareos-fd-mysql.py"
...
}

You will need to restart the File daemon on the headoffice.example.com server. Now we are going to change the Director configuration and create a new FileSet for the database server.

FileSet {
  Name = "DBLinux"
  Include {
    Options {
      Compression = GZIP
      Signature = SHA1
     }
     File = "/etc"
     File = "/var/lib" File = "/data" File = "/home"
     Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-mysql"
     }
  Exclude {
    File = "/proc"
    File = "/sys"
    File = "/dev"
    File = "/run"
    File = "/tmp"
    File = "/data/backups"
  }
}

We are going to create a new JobDefs for headoffice.example.com also.

JobDefs {
  Name = "DBLinux"
  Type = Backup
  Level = Incremental
  Client = headoffice-fd
  FileSet = "DBLinux"
  Schedule = "WeeklyCycle"
  Storage = Gluster
  Messages = Standard
  Pool = Incremental
  Priority = 10
  Write Bootstrap = "/var/lib/bareos/%c.bsr"
  Full Backup Pool = Full
  Incremental Backup Pool = Incremental
}

We have created a new JobDefs and added the DBLinux FileSet for the database server. We can change other defaults for the job as we require, like schedules or storage. Lastly we need to change the job for the database server.

Job {
  Name = headoffice.example.com
  Client = headoffice-fd
  Enabled = yes
  JobDefs = "DBLinux"
}

We need to reload or restart the Director service before this becomes active. Let’s try to run our backup for headoffice.example.com.

*  run job=headoffice.example.com client=headoffice-fd
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
Run Backup job
JobName:  headoffice.example.com
Level:    Incremental
Client:   headoffice-fd
Format:   Native
FileSet:  DBLinux
Pool:     Incremental (From Job IncPool override)
Storage:  Gluster (From Job resource)
When:     2016-10-11 22:29:37
Priority: 10
OK to run? (yes/mod/no): yes
Job queued. JobId=15

Here you can see our job being scheduled and run with the DBLinux FileSet. As this job runs, we can follow it in /var/log/bareos/bareos.log, and we can see the databases being backed up here:

11-Oct 22:29 example-sd JobId 15: Ready to append to end of Volume "Full-0004" size=145926917
11-Oct 22:29 headoffice-fd JobId 15:      /var/lib/nfs/rpc_pipefs is a different filesystem. Will not descend from /var/lib into it.
11-Oct 22:29 headoffice-fd JobId 15: Starting backup of /_mysqlbackups_/test.sql
11-Oct 22:29 headoffice-fd JobId 15: Starting backup of /_mysqlbackups_/mysql.sql
11-Oct 22:29 headoffice-fd JobId 15: Starting backup of /_mysqlbackups_/bareos.sql
11-Oct 22:29 example-sd JobId 15: Elapsed time=00:00:08, Transfer rate=18.22 M Bytes/second

When we issue the restore command in bconsole and select the jobid for the last restore, we can see the backups of our database files.

cwd is: /
$ cd _mysqlbackups_/
cwd is: /_mysqlbackups_/
$ ls
bareos.sql
mysql.sql
test.sql
$ mark bareos.sql
1 file marked.
$ done

The contents of the bareos.sql file when restored contain the full backup of our bareos database.

Bareos also provides two instructions in the Jobresource that we can take advantage of when running complex jobs: Client Run Before Job and Client Run After Job. As their names imply, these instructions can be used to run scripts on the target host. Client Run Before Job will execute the command or script on the target host and require a successful completion before it continues. If it does not succeed, the job will terminate and produce an error message. Client Run After Job also requires a successful completion of the script or command on the target host; otherwise, it stops further processing for that job and submits an error message. If we did not want to use the Python plug-ins, we could write our own scripts. To use these, we could change the headoffice.example.com Job resource we defined earlier. We would change it as follows:

Job {
  Name = headoffice.example.com
  Client = headoffice-fd
  Enabled = yes
  JobDefs = "DefaultLinux"
  Client Run Before Job = "/usr/local/bin/mysql_backup start"
  Client Run After Job = "/usr/local/bin/mysql_backup stop"
}

The script mysql_backup must exist on our target host in the /usr/local/bin directory.

You can now see how you can manage complex backups on the Bareos backup server. You can use Bareos to back up a complete file system, or you can target data directories only. If you are going to target data directories only, be sure to look at how you are going to rebuild your hosts in the event of a problem with the hosts.

Next, we’re going look at using the Web-UI console to manage our Bareos configurations.

Introducing the Bareos Web-UI

Using a text-based console is a quick and easy way to set up and manage your Bareos services. Some people prefer a GUI, however, so Bareos created a nice, clean web interface called Web-UI for managing the Bareos Director.

We set up a PHP web service in Chapter 11 and now would be a good time to review that if you have forgotten. The Web-UI can be installed on any web host; it does not require a Bareos Director to be installed. You install the Web-UI via the bareos-webui package, which is available from the online repository. It will install the required dependencies for you.

The bareos-webui package will install the web files into /usr/share/bareos-webui and install an Apache configuration file in /etc/httpd/conf.d/bareos-webui.conf.

We can restart the Apache service, and then we will create a user to access the web interface. We will create a file /etc/bareos/bareos-dir.d/admin.conf if it is not already created by your installation process.

Console {
  Name = "admin"
  Password = "secret"
  Profile = "webui"
}

Next we will need to create the file /etc/bareos/bareos-dir.d/webui-consoles.conf and enter the following contents for the file:

Profile {
  Name = webui
  CommandACL = status, messages, show, version, run, rerun, cancel, .api, .bvfs_*, list, llist, use, restore, .jobs, .filesets, .clients
  Job ACL = *all*
  Schedule ACL = *all*
  Catalog ACL = *all*
  Pool ACL = *all*
  Storage ACL = *all*
  Client ACL = *all*
  FileSet ACL = *all*
  Where ACL = *all*
}

This profile relates to the commands that the admin user can access using the webui profile. If you wanted to restrict access to certain profiles, you would edit this file by listing the resources the profile has access to (see http://doc.bareos.org/master/html/bareos-manual-main-reference.html#x1-1510009.12 for more details).

We now need the Bareos Director to read these files in when it starts. To do that, we need to add the following to the bottom of our bareos-dir.conf file:

@/etc/bareos/bareos-dir.d/webui-consoles.conf
@/etc/bareos/bareos-dir.d/webui-profiles.conf

Again, we are using the @ symbol to tell the Bareos Director to read in these files. We now have to restart or reload the Director.

Once all this is done, we should be able to restart the Apache server and log in to the Bareos Web-UI. Let’s log in, as shown in Figure 14-10.

Figure 14-10. Log Web-UI page

We use the password secrets and the username admin, which we specified in webui-consoles.conf earlier. In Figure 14-11 we see the main dashboard.

Figure 14-11. Dashboard page of Web-UI

We will not go through every page; as you can see, there are many parts to explore that maps to parts of the Bareos configuration. We do want to show you the Restore page, as you can use this to find and restore your backups like in Figure 14-12.

Figure 14-12. Restore page on Web-UI

Here you can see that you can explore and select your restore options via the Web-UI , and this will become useful for you if you need to restore any files.