One of the most valuable assets on a server is its configuration. This is second only to the data the server stores. Often, when we implement a new technology on a server, we'll spend a great deal of time editing configuration files all over the server to make it work as best as we can. This can include any number of things from Apache virtual host files, DHCP server configuration, DNS zone files, and more. If a server were to encounter a disaster from which the only recourse is to completely rebuild it, the last thing we'll want to do is re-engineer all of this configuration from scratch. This is where Git comes in.

Git is a development tool, used by software engineers everywhere for the purpose of version control for source code. In a typical development environment, an application being developed by a team of engineers can be managed by Git, each contributing to a repository that hosts the source code for their software. One of the things that makes Git so useful is how you're able to go back to previous versions of a file in an instant, as it keeps a history of all changes made to the files within the repository.

Git isn't just useful for software engineers, though. It's also a really useful tool we can leverage for keeping track of configuration files on our servers. For our use case, we can use it to record changes to configuration files and push them to a central server for backup. When we make changes to configuration, we'll push the change back to our Git server. If for some reason we need to restore the configuration after a server fails, we can simply download our configuration files from Git back onto our new server. Another useful aspect of this approach is that if an administrator implements a change to a configuration file that breaks a service, we can simply revert back to a known working commit and we'll be immediately back up and running.

Configuration management on servers is so important, in fact, I highly recommend that every Linux administrator takes advantage of version control for this purpose. Although it may seem a bit tricky at first, it's actually really easy to get going once you practice with it. Once you've implemented Git for keeping track of all your server's configuration files, you'll wonder how you ever lived without it. We covered Git briefly in Chapter 14, Automating Server Configuration with Ansible, where I walked you through creating a repository on GitHub to host Ansible configuration. However, GitHub may or may not be a good place for your company's configuration, because not only do most companies have policies against sharing internal configuration, you may have sensitive information that you wouldn't want others to see. Thankfully, you don't really need GitHub in order to use Git; you can use a local server for Git on your network.

I'll walk you through what you'll need to do in order to implement this approach. To get started, you'll want to install the git package:

sudo apt install git

In regard to your Git server, you don't necessarily have to dedicate a server just for this purpose, you can use an existing server. The only important aspect here is that you have a central server onto which you can store your Git repositories. All your other servers will need to be able to reach it via your network. On whatever machine you've designated as your Git server, install the git package on it as well. Believe it or not, that's all there is to it. Since Git uses OpenSSH by default, we only need to make sure the git package is installed on the server as well as our clients. We'll need a directory on that server to house our Git repositories, and the users on your servers that utilize Git will need to be able to modify that directory.

Now, think of a configuration directory that's important to you, that you want to place into version control. A good example is the /etc/apache2 directory on a web server. That's what I'll use in my examples in this section. But you're certainly not limited to that. Any configuration directory you would rather not lose is a good candidate. If you choose to use a different configuration path, change the paths I give you in my examples to that path.

On the server, create a directory to host your repositories. I'll use /git in my examples:

sudo mkdir /git

Next, you'll want to modify this directory to be owned by the administrative user you use on your Ubuntu servers. Typically, this is the user that was created during the installation of the distribution. You can use any user you want actually, just make sure this user is allowed to use OpenSSH to access your Git server. Change the ownership of the /git directory so it is owned by this user. My user on my Git server is jay, so in my case I would change the ownership with the following command:

sudo chown jay:jay /git

Next, we'll create our Git repository within the /git directory. For Apache, I'll create a bare repository for it within the /git directory. A bare repository is basically a skeleton of a Git repository that doesn't contain any useful data, just some default configuration to allow it to act as a Git folder. To create the bare repository, cd into the /git directory and execute:

git init --bare apache2

You should see the following output:

Initialized empty Git repository in /git/apache2/

That's all we need to do on the server for now for the purposes of our Apache repository. On your client (the server that houses the configuration you want to place under version control), we'll copy this bare repository by cloning it. To set that up, create a /git directory on your Apache server (or whatever kind of server you're backing up) just as we did before. Then, cd into that directory and clone your repository with the following command:

git clone 192.168.1.101:/git/apache2

For that command, replace the IP address with either the IP address of your Git server, or its hostname if you've created a DNS entry for it. You should see the following output, warning us that we've cloned an empty repository:

warning: You appear to have cloned an empty repository

This is fine, we haven't actually added anything to our repository yet. If you were to cd into the directory we just cloned and list its storage, you'd see it as an empty directory. If you use ls -a to view hidden directories as well, you'll see a .git directory inside. Inside the .git directory, we'll have configuration items for Git that allow this repository to function properly. For example, the config file in the .git directory contains information on where the remote server is located. We won't be manipulating this directory, I just wanted to give you a quick overview on what its purpose is. Note that if you delete the .git directory in your cloned repository, that basically removes version control from the directory and makes it a normal directory.

Anyway, let's continue. We should first make a backup of our current /etc/apache2 directory on our web server, in case we make a mistake while converting it to being version controlled:

sudo cp -rp /etc/apache2 /etc/apache2.bak

Then, we can move all the contents of /etc/apache2 into our repository:

sudo mv /etc/apache2/* /git/apache2

The /etc/apache2 directory is now empty. Be careful not to restart Apache at this point; it won't see its configuration files and will fail. Remove the (now empty) /etc/apache2 directory:

sudo rm /etc/apache2

Now, let's make sure that Apache's files are owned by root. The problem though is if we use the chown command as we normally would to change ownership, we'll also change the .git directory to be owned by root as well. We don't want that, because the user responsible for pushing changes should be the owner of the .git folder. The following command will change the ownership of the files to root, but won't touch hidden directories such as .git:

sudo find /git/apache2 -name '.?*' -prune -o -exec chown root:root {} +

When you list the contents of your repository directory now, you should see that all files are owned by root, except for the .git directory, which should be owned by your administrative user account.

Next, create a symbolic link to your Git repository so the apache2 daemon can find it:

sudo ln -s /git/apache2 /etc/apache2

At this point, you should see a symbolic link for Apache, located at /etc/apache2. If you list the contents of /etc while grepping for apache2, you should see it as a symbolic link:

ls -l /etc | grep apache2

The directory listing will look similar to the following:

lrwxrwxrwx 1 root root 37 2016-06-25 20:59 apache2 -> /git/apache2

If you reload Apache, nothing should change and it should find the same configuration files as it did before, since its directory in /etc maps to /git/apache2, which includes the same files it did before:

sudo systemctl reload apache2

If you see no errors, you should be all set. Otherwise, make sure you created the symbolic link properly.

Next, we get to the main attraction. We've copied Apache's files into our repository, but we didn't actually push those changes back to our Git server yet. To set that up, we'll need to associate the files within our /git/apache2 directory into version control. The reason for this is because the files simply being in the git repository folder isn't enough for Git to care about them. We have to tell Git to pay attention to individual files. We can add every file within our Git repository for Apache by entering the following command from within that directory:

git add

This basically tells Git to add everything in the directory to version control. You can actually do the following to add an individual file:

git add <filename>

In this case, we want to add everything, so we used a period in place of a directory name to add the entire current directory.

If you run the git status command from within your Git repository, you should see output indicating that Git has new files that haven't been committed yet. A Git commit simply finalizes the changes locally. Basically, it packages up your current changes to prepare them for being copied to the server. To create a commit of all the files we've added so far, cd into your /git/apache2 directory and run the following to stage a new commit:

git commit -a -m "My first commit."

With this command, the -a option tells Git that you want to include anything that's changed in your repository. The -m option allows you to attach a message to the commit, which is actually required. If you don't use the -m option, it will open your default text editor and allow you to add a comment from there.

Finally, we can push our changes back to the Git server:

git push origin master

By default, the git suite of commands utilizes OpenSSH, so our git push command should create an SSH connection back to our server and push the files there. You won't be able to inspect the contents of the Git directory on your Git server, because it won't contain the same file structure as your original directory. Whenever you pull a Git repository though, the resulting directory structure will be just as you left it.

From this point forward, if you need to restore a repository onto another server, all you should need to do is perform a Git clone. To clone the repository into your current working directory, execute the following:

git clone 192.168.1.101:/git/apache2

Now, each time you make changes to your configuration files, you can perform a git commit and then push the changes up to the server to keep the content safe:

git commit -a -m "Updated config files."
git push origin master

Now we know how to create a repository, push changes to a server, and pull the changes back down. Finally, we'll need to know how to revert changes should our configuration get changed with non-working files. First, we'll need to locate a known working commit. My favorite method is using the tig command. The tig package must be installed for this to work, but it's a great utility to have:

sudo apt install tig

The tig command (which is just git backwards) gives us a semi-graphical interface to browse through our Git commits. To use it, simply execute the tig command from within a Git repository. In the following example screenshot, I've executed tig from within a repository for a Git repository on one of my servers:

While using tig, you'll see a list of Git commits, along with their dates and comments that were entered with each. To inspect one, press the up and down arrows to change your selection, then press Enter on the one you want to view. You'll see a new window, which will show you the commit hash (which is a long string of alphanumeric characters), as well as an overview of which lines were added or removed from the files within the commit. To revert one, you'll first need to find the commit you want to revert to and get its commit hash. The tig command is great for finding this information. In most cases, the commit you'll want to revert to is the one before the change took place. In my example screenshot, I fixed some syntax issues on 3/14/2018. If I want to restore that file, I should revert to the commit before that, on 3/13/2018. I can get the commit hash by highlighting that entry and pressing Enter. It's at the top of the window. Then, I can exit tig by pressing q, and then revert to that commit:

git checkout 356dd6153f187c1918f6e2398aa6d8c20fd26032

And just like that, the entire directory tree for the repository instantly changes to exactly what it was before the bad commit took place. I can then restart or reload the daemon for this repository, and it will be back to normal. At this point, you'd want to test the application to make sure that the issue is completely fixed. After some time has passed and you're finished testing, you can make the change permanent. First, we switch back to the most recent commit:

git checkout master

Then, we permanently switch master back to the commit that was found to be working properly:

git revert --no-commit 356dd6153f187c1918f6e2398aa6d8c20fd26032

Then, we can commit our reverted Git repository and push it back to the server:

git commit -a -m "The previous commit broke the application. Reverting."
git push origin master

As you can see, Git is a very useful ally to utilize when managing configuration files on your servers. This benefits disaster recovery, because if a bad change is made that breaks a daemon, you can easily revert the change. If the server were to fail, you can recreate your configuration almost instantly by just cloning the repository again. There's certainly a lot more to Git than what we've gone over in this section, so feel free to pick up a book about it if you wish to take your knowledge to the next level. But in regard to managing your configuration with Git, all you'll need to know is how to place files into version control, update them, and clone them to new servers. Some services you run on a server may not be a good candidate for Git, however. For example, managing an entire MariaDB database via Git would be a nightmare, since there is too much overhead with such a use case and database entries will likely change too rapidly for Git to keep up. Use your best judgment. If you have some configuration files that are only manipulated every once in a while, they'll be a perfect candidate for Git.