Powering On

Let’s look in a bit more detail at what happens when you boot your host. You may have noticed that when you turn on your host, it is common to hear a few beeps and whirrs and see some blinking lights on your front panel and keyboard. This is the first step of your host starting up, and this process, on a host running BIOS, is controlled by a small chip on the motherboard called the BIOS.

The Unified Extensible Firmware Interface

More recent motherboards will support UEFI. When you boot your host with UEFI you will notice very little difference compared to BIOS; the lights will flash and things will whirr. It is expected to be a little faster than BIOS. However, behind the scenes things are quite different.

UEFI was created to overcome some of the very real limitations of BIOS. BIOS runs on a 16-bit processor; UEFI can run on modern processors. BIOS was limited in RAM; UEFI can run in as much RAM as required. BIOS reads from a small section of a hard drive (the master boot record, or MBR) to load the operating system. UEFI instead reads from a special FAT32 partition created at installation time and is not limited to 512 bytes. Also UEFI requires that the partition table on the disk is GPT (GUID Partition Table), and this means that we can boot off disks over 2TiB in size. It is said that UEFI is more like a modern operating system than the old BIOS; it can even run nicer graphical interfaces with mouse and keyboard support.

One of the ways that malware can be loaded on a system is via the boot loader. UEFI offers “secure boot” that is designed to protect against this by verifying signed boot loaders and hardware via public/private keys before loading them. Most modern 64-bit distros should support secure boot, but this is still relatively new and you may have problems with some hardware under secure boot. In this case you can file a bug report with the distribution and disable secure boot via the UEFI configuration screen.

Finally, it is currently not mandatory that you use UEFI. BIOS is still available in most motherboards as “legacy” and has been widely tested and supported. It is, however, mandatory under the following circumstances; you are running dual boot with a system already running UEFI and you want to access disks larger than 2TiB. The purpose of both BIOS and UEFI is to get your hardware ready and find the boot loader to load your operating system.

More on Disks

In previous chapters we have talked about partitioning disks and said that a disk is like a cake, in that you can carve up slices, but how does that work? When you slice up your disk you create table on the disk called the partition table . The partition table is normally found at the beginning of the disk and tells the computer how the disk has been laid out. That is, how much of the disk has been allocated to the '/boot' partition, how much is expected to be logical volume management (LVM), and so on.

Above we have talked about MBR and GPT and these are two types of partition tables. A partition table holds information about the block addresses (in chunks of 512 bytes or 4KB of disk). The MBR can only hold 2TiBs worth of addresses while GPT can hold 8-9ZiB.

Figure 6-1. Your hard disk

A disk will either have a MBR at the start (in the first 512 bytes) or GPT . If it has a MBR the partition table is written after the first 446 bytes of the disk. The size of the partition table is 64 bytes and is followed by the MBR boot signature (2bytes). The remaining space up until the first partition is often called the MBR gap and varies in size as most partitioners align the start of partitions to the first 1MB.

GPT on the other hand has an address space of 16,384 bytes and can hold up to 128 partitions. GPT makes a copy of itself at the end of the disk for recovery in case of corruption. The remaining disk, excluding the partition tables, can be divided as you see fit. See the following for more detail:

• https://en.wikipedia.org/wiki/Master_boot_record

• https://en.wikipedia.org/wiki/GUID_Partition_Table

Boot Loaders

The Boot Loader with UEFI

If you use an UEFI system the boot process is a little different. The UEFI can read the GPT partition table and can find and execute the boot loader code in the /EFI/ directory in the EFI system partition. Unlike BIOS, UEFI has its own partition that the boot loader and modules can be installed into.

In Figure 6-2 we can see that after our installation we have three partitions created on drive /dev/sda. The first one is our EFI System Partition and it is a FAT32 file system of 537MB in size. It has the boot and esp flags (both meaning it is a boot partition). The other thing to note is that the partition table is GPT.

Figure 6-2. EFI partition on Ubuntu

So when UEFI is ready to run the boot loader it reads this partition looking for a boot loader. In this case on our Ubuntu host will find /EFI/ubuntu/grubx64.efi. This initial part of the boot loader will find the '/boot' partition that contains the GRUB2 software and load core.efi from and bring up the GRUB2 menu.

For more information on UEFI and the UEFI Shell (an interactive shell that you can use to manage your UEFI) see the following:

• https://help.ubuntu.com/community/UEFI

• https://fedoraproject.org/wiki/Unified_Extensible_Firmware_Interface

• https://www.happyassassin.net/2014/01/25/uefi-boot-how-does-that-actually-work-then/

• https://software.intel.com/en-us/articles/uefi-shell

Without any intervention GRUB2 will boot the default kernel after a brief countdown. Pressing any key will stop the countdown, show you a more detailed menu of available options, and give you the opportunity to edit your boot configuration. Later in the section “Using the GRUB2 Menu,” we will explain these choices and how to manipulate them.

After picking the kernel you wish to boot into (or waiting until the default is loaded after the timeout), GRUB2 will now find the kernel binary (vmlinuz-<release-number>) and then load a special file called initrd.img into memory. This file contains the drivers your kernel needs to load to make use of the hardware of your host.

Starting the Operating System

After loading initrd.img, GRUB2 completes and hands over control to the kernel, which continues the boot process by initiating your hardware, including your hard disks. Your operating system is started, and a special program called systemd, upstart or init is called that starts your services and brings your host to life. We’ll take a look at all three of these system initializers and how to manage services later in this chapter, but first, more on the boot loader.

Using the GRUB2 Menu

When your host boots, it will boot into the default kernel (or operating system) or you can override it and display the GRUB2 menu. Once the menu is displayed, you will be presented with a list of boot options, and you can use the up and down arrows on your keyboard to choose the kernel you wish to boot. You can also edit the GRUB menu and change parameters, commands, and arguments before proceeding to boot into your chosen kernel.

For example, we could choose to boot into what is called single-user mode, or maintenance mode. This special mode, used when something is broken on your host, restricts access to the host to the root user: usually at the system console. This mode, which functions much like the Microsoft Windows Recovery Console, allows you to work with resources like disks and files without worrying about conflicts or other users manipulating the host. Let’s look at booting into single-user mode now.

First, again, this is slightly different between CentOS and Ubuntu. Both show a similar GRUB2 boot menu that allows you to select which kernel you would like to run with your arrow keys. CentOS presents the two menu entries that we talked about earlier, one being the kernel and the other described as a rescue kernel. On Ubuntu you will see a sub-menu entry called Advanced Options. You can use your arrow keys to navigate to see the recovery mode kernel.

We are going to use the normal kernel of an Ubuntu host in this exercise. Press the e key to edit the highlighted kernel. You will see the configuration of that kernel, such as its location and parameters, displayed as in Figure 6-3.

Figure 6-3. Booting to single-user mode , or maintenance mode , using the GRUB2 menu

You’ll see the details here as you saw in the grub.cfg menuentry. The line we are interested in this particular example is the linux /vmlinuz-4.4.0-15-generic.efi.signed… line. Use the arrow keys to navigate to the end of that line. To boot into single-user mode, add the word ‘single’ (without quotes) to the end of this line. In Figure 6-3, you can see that we have already added the word single to the end of the line. At the bottom of Figure 6-3 you can see that we can now enter Ctrl-x or F10 to boot with this setting or Ctrl-c to get a command line or ESC to abandon these changes and go back to the main menu.

We can choose Ctrl-x and boot into single-user mode. We are first asked for the password to decrypt our hard drive. Then, depending on your distribution, you will either be given a maintenance password prompt that expects the root user (CentOS only) or you will be able to hit enter and be given a shell prompt (see Figure 6-4).

Figure 6-4. Maintenance mode

Now that we have the shell prompt we can go about repairing any of the problems we might have on our system. These changes to the kernel line are not permanent though, and the next time the host is booted, it will boot normally. Using the GRUB2 menu, you can manipulate almost all of the boot runtime configuration settings available.

With a note of caution, we are providing the above as an example and it clearly has some security implications associated with it—this provides very powerful access to your hosts if you have access to a console at boot time. We help to address that issue next. There are also other methods for recovery and in Chapter 9 we will go over this further. In the meantime the following are good sources of information:

• https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/sec-Terminal_Menu_Editing_During_Boot.html

• https://wiki.ubuntu.com/RecoveryMode

Configuring GRUB2

The GRUB2 boot loader is highly configurable, and its configuration is contained in the grub.conf configuration file. GRUB2 has made the grub.cfg more modular than its simpler predecessor. The chance that you will need to make many changes to the GRUB2 configuration is small but possible.

The GRUB2 files are located in different places on different distributions. On Red Hat-based hosts like CentOS, they can be found at /boot/grub2/grub.cfg (and the file is usually linked symbolically to /etc/grub.conf). On Ubuntu and Debian hosts, the files can be found at /boot/grub/grub.cfg.

The grub.cfg file in GRUB2 is made up of a series of configuration files. On both Ubuntu and CentOS hosts you will find the configuration files in the directory /etc/grub.d. In that directory you will see a number ordered list of files. In Listing 6-1 you can see the files found on the CentOS 7 host.

Listing 6-1. ls –la /etc/grub.d

total 72
-rwxr-xr-x. 1 root root  8702  Nov 25 02:49  00_header
-rwxr-xr-x. 1 root root   992   May   4  2015  00_tuned
-rwxr-xr-x. 1 root root   230   Nov 25 02:49  01_users
-rwxr-xr-x. 1 root root 10232 Nov 25 02:49  10_linux
-rwxr-xr-x. 1 root root 10275 Nov 25 02:49  20_linux_xen
-rwxr-xr-x. 1 root root  2559  Nov 25 02:49  20_ppc_terminfo
-rwxr-xr-x. 1 root root 11169 Nov 25 02:49  30_os-prober
-rwxr-xr-x. 1 root root   214   Nov 25 02:49  40_custom
-rwxr-xr-x. 1 root root   216   Nov 25 02:49  41_custom
-rw-r--r--.   1 root root   483   Nov 25 02:49  README

Listing 6-1 shows the files that make up a grub.cfg file. These files are read in via the grub2-mkconfig command from 00_header to 41_custom in this case. You will not need to know much about these files normally as you rarely need to touch them on a regular basis. The files you might be interested in are 01_users and 40_custom files. The 01_users file loads in the boot loader password if it set. The 40_custom file is where you may wish to add custom GRUB2 configurations.

The GRUB2 configuration file is made via the grub2-mkconfig command. It takes one argument, the –o or --output option of where you would like the file to be created. You can create your own grub.cfg file now. It will probe your system and run through your configuration files and output a file that you can view. Of course, you don’t have to override your current grub.cfg with this file, so we will change the output to be a file in our local directory.

Listing 6-2. Creating a grub.cfg

$ sudo grub2-mkconfig --output mygrub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-327.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-65e567f18fe84907a6f1d8519e921c97
Found initrd image: /boot/initramfs-0-rescue-65e567f18fe84907a6f1d8519e921c97.img
done

In Listing 6-2 you can see that we have used the sudo command to escalate our privileges to perform the grub2-mkconfig command . This command runs through the files in the /etc/grub.d directory and executes them in numbered order. The output of the command to the console shows that is has found two kernels (vmlinuz) with one being a “rescue” kernel. The grub2-mkconfig command has probed our system and found them via /etc/grub.d/30_os-prober script.

Taking a look at the mygrub.cfg file that has been produced (use $ less mygrub.cfg) you will see the output of all those files in /etc/grub.d. We are going to concentrate on the part that starts your operating system, and that starts with the following line:

### BEGIN /etc/grub.d/10_linux ###

Below that BEGIN line is the menuentry that configures our menu item.

Listing 6-3. menuentry in the grub.cfg

menuentry 'CentOS Linux (3.10.0-327.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-327.el7.x86_64-advanced-f60589f6-4d39-4c5b-8ac6-b1252f28f323' {
  ....
}

That already seems like a lot to digest and we have reduced Listing 6-3 to just the first line. Beginning with menuentry, this tells grub that we have a new menu item for it to list. ‘CentOS Linux (...) 7 (Core)’ gives us the name as it will appear in the menu list. Other information is provided by the --class parameter are used to display certain themes (like splash screens and so on).

The --unrestricted option is used to allow any user to run that entry at boot time via the console menu. This can be changed with the --users option where you can list users that may be able to run that entry (note that anyone listed as a superuser will still have access). Of course, this means that any entry with --unrestricted removed will not boot automatically and will require intervention. The GRUB2 documentation says that this is not really the best form of security as it will require access to the console on reboots but might be considered in kiosk type of installations. We will talk more on this and superusers in the section “Securing Your Boot Loader.”

In Listing 6-3 we have left out the code between the curly braces, { }. Let’s go through that code now in Listing 6-4.

Listing 6-4. Grub Boot Menu Code

menuentry ... {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  85932bb0-c5fe-431f-b129-93c567e4f76f
        else
          search --no-floppy --fs-uuid --set=root 85932bb0-c5fe-431f-b129-93c567e4f76f
        fi
        linux16 /vmlinuz-3.10.0-327.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet
        initrd16 /initramfs-3.10.0-327.el7.x86_64.img
}

Inside the curly braces of the menuentry code the first few lines of code install various modules. For example, the first line, load_video calls a function within the grub.cfg script that inserts the various video modules for the splash screen. The modules on a CentOS 7 host can be found in /boot/grub/i386-pc/ and are loaded via the insmod command you see in the subsequent lines. The three insmod lines you see here are loading compression and file system modules.

The set root='hd0,msdos1' is how we set the root device for GRUB2. GRUB2 uses this setting to help find files that GRUB2 needs. What it means is the root device is on a hard drive (hd) and is the first hard drive (hd0). The 'msdos' is the filesystem of the root device and the final ‘1’ is the partition number on hd0. If you were booting off a USB drive you might expect it to look like 'usb0,msdos1' instead.

Next in Listing 6-4 we see an if…fi conditional statement that has a search command inside it. The search command is different depending if ‘$feature_platform_search_hint’ is set to y. What is happening here is that the search command is looking for a device by a UUID (universal unique identifier). This is used to again set the root device for GRUB2 so it can find the Linux kernel we are going to load.

In Listing 6-4 we are at last loading our Linux kernel. The first part of the line linux16 says we are going to load the kernel using a 16-bit mode (16-bit mode is used to get around certain problems with video modes). The next part is the Linux kernel itself, /vmlinuz-3.10.0-327.el7.x86_64. You will find this file in the /boot directory. Next we pass in options to the kernel. We pass root=/dev/mapper/centos-root to provide the path to the ‘/’ partition. Initially we load the kernel as ‘ro’ or read only. The crashkernel=auto option is a setting for kdump kernel indicating the amount of memory reserved for it. The next two options tell the kernel that there are two LVM logical volumes it needs to load: centos/root and centos/swap. Finally, we can see that we load the Red Hat Graphical Boot (rhgb), a nicer boot experience than text, and we set the quiet option to supress noisy output during boot time.

Finally, the last line we are concerned with in Listing 6-4 is loading the kernel initrd. The initrd is a temporary root file system that is used by the boot process to first load the needed executables and drivers and then mount the system’s “true” root (‘/’) file system. It is only used at boot and is unmounted once the true file system has been mounted.

We have shown you only one GRUB2 menuentry and only from a CentOS host. The menuentry will be slightly different for an Ubuntu host. Normally you see at least two menu entries per kernel you have installed, with one entry being a “rescue” kernel for system recovery. It is a copy of the kernel in case one is corrupted on CentOS with no special kernel options. Ubuntu is different as it passes the recovery and nomodeset kernel options. We will talk more about these shortly.

Securing Your Boot Loader

Having such a versatile and configurable boot loader can have its drawbacks. One of those we have mentioned is security. An unsecured boot loader can be altered at boot time by any person with malicious intent. Many small offices will have their servers on or under a couple of desks rather than in a locked computer room, and these servers are therefore very susceptible to such attacks. We strongly suggest you consider setting a password on your boot loader and store it in a very safe place along with your other passwords.

Luckily GRUB2 provides the ability to set a password to the boot loader so that any changes to the preconfigured boot process require the user to enter a password. CentOS hosts have tools that make this easy for us and we will explore that distribution in this exercise. First we issue the following command:

$ sudo grub2-setpassword

This will ask you for a password and confirmation and will only create a password for the root user. What this command actually does is create a file /boot/grub2/user.cfg. The contents of that file looks as follows:

GRUB2_PASSWORD=grub.pbkdf2.sha512.10000.00E574C[...]DDF512CC090A9B[...]CCEA3B

Now that is file is created, when your host is rebooted this code in the /boot/grub2/grub.cfg file will now be executed:

Listing 6-5. Adding Root User to grub.cfg

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n ${GRUB2_PASSWORD} ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

What that code is saying in Listing 6-5 is, if the user.cfg file exists, then we will “source” the contents. If there is a $GRUB2_PASSWORD variable declared, then we will add rootto the superusers list and this locks out other users to the GRUB2 command line. Then, finally, password_pbkdf2 is a GRUB2 command that associates the user with a password. We now need to run the grub2-mkconfig –o /boot/grub2/grub.cfg to set the superuser root password.

When your host next boots, you will notice that if you interrupt the boot process at the GRUB2 stage, you will be able to select one of the kernels with the --unrestricted menuentry option. However, if you want to edit any of the GRUB2 configuration details, by pressing e, you are required to enter a password. You will then be able to edit the GRUB2 configuration as normal.

To remove the password, remove the user.cfg file. Remember the GRUB2 documentation does suggest that this is not recommended except in situations where your hosts are publicly exposed. We suggest you use your best judgment considering your circumstances.

Remember that Ubuntu doesn’t provide these tools to you out of the box, but you can follow a similar process with that distribution. For an Ubuntu specific guide see https://help.ubuntu.com/community/Grub2/Passwords .

Understanding Systemd

As we have already mentioned, Systemd is the newest system initializer that is used across many modern distributions. Systemd is the replacement for both SysV Init and Upstart. It has been criticized as being against the Unix philosophy (do one thing, do it well), but it offers many advantages over its predecessors in the following ways (to a greater or lesser degree):

• It is event driven. This means it can respond to system events (such as new hardware being plugged in, traffic starting on a network port)

• Concurrent and parallel boot processing

• It can respawn processes

• Event logging

• Tracks processes via the kernel’s CGroups

For those already familiar with SysV, Systemd doesn’t have runlevels like SysV but instead has “targets.” Targets are used to group service dependencies together. There are some common targets in systemd like multiuser, reboot, and rescue. So much as in runlevels, we can have different targets that bring us up to certain and discrete states. For example, if we want the system to be in the state where everyone can log in and use services we would use the target multiuser. In that state, we would like sshd, logging, and networking to be available. These are considered “wants” of the multiuser target.

Remember in Figure 6-3 where we showed you how to enter “maintenance” mode from the GRUB2 menu? We typed the word “single” at the end of the line where we launched the Linux kernel. Well, with Systemd we can also do the same thing by declaring the target we wish to launch into.

Listing 6-6. Using Systemd Targets to Boot to Rescue Mode

linux16 /vmlinuz-3.10.0-327.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet systemd.unit=rescue.target

From looking at Listing 6-6 you might be able to recognize that we are on a CentOS server; we can do the same on an Ubuntu server. You can see that we have issued the systemd.unit=rescue.target on the end of the line. We are telling the kernel that when it runs systemd we should got to the rescue target.

Systemd uses a set of “unit” files to manage the services on your system. When systemd starts it looks for the required unit files by traversing the following load path:

Because we used the system.unit=rescue.target parameter, when the system process starts it will look for the rescue.target file in these directories in order of precedence, the /etc/systemd/system, then the /run/systemd/system directories and then finally in the /usr/lib/systemd/system directory. The /etc/systemd/system and /run/systemd/system directories are called “drop-in” directories and systemd will search for unit and *.conf files there. The target file will look like the following rescue.target.

Listing 6-7. The rescue.target File

[Unit]
Description=Rescue Mode
Documentation=man:systemd.special(7)
Requires=sysinit.target rescue.service
After=sysinit.target rescue.service
AllowIsolate=yes

[Install]
Alias=kbrequest.target

In Listing 6-7 we have removed the comments beginning with the ‘#’ which are ignored by the system process. The options are case sensitive. In breaking this file down you can see that it has two sections, denoted by the square brackets([…]). Systemd unit files require at least the [Unit] and [Install] sections. You have the option to include a [Service] section, and we will discuss that shortly.

The [Unit] section has a Description and a Documentation setting, and they are pretty self-explanatory. In Listing 6-7 we can see that the rescue target Requires the sysinit.target and rescue.service. Requires does not provide ordering but tells systemd that these services should be executed too. If one of the required targets or services fails, this unit will be deactivated too.

Ordering support of targets or services is provided by adding options like the After. The rescue.unit should now be started after sysinit.target and rescue.service. It is a common pattern to see the same services and targets in the Required option also listed in the After option.

The AllowIsolate option is a Boolean read by the systemctl isolate command. The isolate argument to the systemctl command is similar in nature to a SysV runlevel. You can use it to put your system in a particular “state,” so to start your graphical environment you may choose to run systemctl isolate graphical.target. For this reason, AllowIsolate is set to “false/no” by default because not every target file provides a stable system state.

The [Install] section is used by the systemctl tool when the unit is enabled or disabled. The systemctl tool is the utility tool used to manage systemd. The Alias option here points to the kbrequest.target which is a special systemd unit that is started whenever the Alt+ArrowUp is pressed on the console. Take a look at Figure 6-7 to see how that works.

Figure 6-7. Using systemctl to enable the Alt-ArrowUp alias

In Figure 6-7 you can see that when we issue the $ sudo systemctl enable rescue.target, the Alias option of the Install section means that we create a symlink from kbrequest.target to the rescue.target. Now, on the console, if we hit the Alt+ArrowUp keys we get the following as shown in Figure 6-8.

Figure 6-8. Entering rescue console from pressing the Alt+ArrowUp keys

Let’s take a quick look at a service file that configures how we manage the rsyslog daemon. Rsyslog is the logging service on Linux and we look at it in depth in Chapter 18. In Listing 6-8 we have displayed the contents of our rsyslog.service file.

Listing 6-8. The rsyslog.service systemd File

[Unit]
Description=System Logging Service
;Requires=syslog.socket

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/rsyslog
ExecStart=/usr/sbin/rsyslogd -n $SYSLOGD_OPTIONS
Restart=on-failure
UMask=0066
StandardOutput=null

[Install]
WantedBy=multi-user.target
;Alias=syslog.service

In this file we have the [Unit] and [Install] sections as we have seen before. It is worth noting that in these sections you can see a semicolon on the Requires and Alias options. If lines start with a #or a;then they are considered comments and are ignored. In Listing 6-8 we see that there is a [Service] section and we use that to describe the service we are configuring to run on our system.

All [Service] sections require a Type option. It can be set to “simple,” “forking,” “oneshot,” “dbus,” “notify,” or “idle.” The default setting is “simple.” With a type of simple you expect that the process run by the ExecStart option is to be the main process (i.e., it doesn’t fork a child process). Notify is similar to simple but it will send a notification message once it has finished starting up. Systemd will then start other units once it reads this message. Since this is a logging service, it makes sense to make sure the service is up and running before starting other units dependent on recording log events.

EnvironmentFileis where we can load environmental parameters. There are many optional parameters that can be set for rsyslogd and you can add them into the /etc/sysconfig/rsyslog file. Options found in man rsyslogd can be added to the file.

Next we have the ExecStart option . This is the command that is executed, along with any arguments. This should be the main process of the daemon, unless you are going to fork the process (with Type=fork).

The Restart option controls how systemd monitors the service and reacts to any issues. The options are “no,” “on-success,” “on-failure,” “on-abnormal,” “on-watchdog,” “on-abort,” or “always.” The rsyslog service is set to restart on failure and the default is “no” if not set. Setting on-failure means that systemd will attempt to restart the service when the process exits with a non-zero exit code, it is terminated by a signal, an operation times out, or a watch-dog timeout is triggered.

UMask sets the permissions of any files created by this process. If we remember what we learned back in Chapter 4, the umask is an octet and is used to set permissions on files and directories. When applied it means we give files created by this process permissions of 0711, or rwx-x-x. This means that the owner of the process is given read, write, and execute permission on the files, the group and everyone else will have execute only.

The last option in the [Service] section is StandardOutput and defines where the file descriptor 1 or standard output of the process should be connected to. The values are “inherit,” “null,” “tty,” “journal,” “syslog,” “kmsg,” “journal+console,” “syslog+console,” “kmsg+console,” or “socket.” The rsyslogd service has been set to null, which is to a special file on Linux called /dev/null. If you write to /dev/null you expect your data to be blackholed or lost. This means we don’t care about the output of this service on standard out.

Finally, if you look at the [Install] section you will see a WantedBy=multi-user.target. This line instructs the systemctl command to create a symbolic link in the /etc/systemd/system/multi-user.target.wants directory to the /lib/systemd/system/rsyslog.service when a service is enabled, or remove the link if the service is disabled. More on this shortly.

Systemd unit files are complex and rely on many other dependency units. There are also countless options and settings available to systemd units. For further reading please see the following helpful links:

• https://access.redhat.com/videos/403833

• www.freedesktop.org/wiki/Software/systemd/

• www.freedesktop.org/software/systemd/man/systemd.html

• https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/chap-Managing_Services_with_systemd.html

Once your kernel has passed off to the systemd process and it has brought up your machine to a target state you will need to know how to interact with the services running on your system. We will explore that shortly when we look at the systemctl command.

Upstart: Ubuntu’s Init

The idea behind Upstart was to create a comprehensive init process that can be used to start, stop, monitor, and respond to events on behalf of services on your Linux host. It was a reworking and expansion of the SysV Init paradigm; rather than outright replacing SysV Init, the designers created Upstart to emulate SysV Init until everyone converted over.

The init process under Upstart is an event-based daemon that uses event triggers to start or stop processes. An event is a change of state that init can be informed of. The event can be the adding of a peripheral device, like plugging in a USB memory stick. The kernel can then inform init of this action by sending an event notice. This event in turn can trigger other jobs to be initiated or stopped depending on the job definitions under init’s control.

Upstart collectively calls processes under its command jobs, and you interact with those jobs using the initctl command. Jobs can either be services or tasks or abstract. Services are persistent, like a mail server, and tasks perform a function and then exit to a waiting state, like a backup program. An abstract job runs forever but has no child process. The definition files (or Upstart scripts) for jobs can be found under the /etc/init directory.

Listing 6-9 illustrates what is called a system job definition for the rsyslogd daemon . In this case, rsyslogd is a logging daemon that needs to write to the /var/log/syslog file. As logging daemon, it is important that if this gets stopped for some reason, it is good to get it to try to restart itself. Let’s run through the major points here.

Listing 6-9. /etc/init/rsyslog.conf

# rsyslog - system logging daemon
#
# rsyslog is an enhanced multi-threaded replacement for the traditional
# syslog daemon, logging messages from applications

Description      "system logging daemon"

start on filesystem
stop on runlevel [06]

expect fork
respawn

pre-start script
    /lib/init/apparmor-profile-load usr.sbin.rsyslogd
end script

script
    . /etc/default/rsyslog
    exec rsyslogd $RSYSLOGD_OPTIONS
end script

Upstart is “evented,” meaning it can react to events on your system. The start on filesystem means that when the filesystem “event” takes place, we will automatically start rsyslog too. The stop on configuration option is an event definition signalling Upstart to stop rsyslogd when a runlevel 0 or 6 event is detected. As you may know, runlevels 0, 1, 6 are special runlevels that either shut down, reboot, or put your host into maintenance mode. We expect the service to fork another process and the respawn option directs Upstart to restart the job if it is stopped unexpectedly. We can write prestart scripts that run prior to executing the main script section.

While Upstart is different from SysV and has no natural concept of runlevels, it has been made backward compatible with SysV Init scripts. These help emulate the old-style SysV scripts by executing the /etc/init.d/rc script. This will in turn run through the old /etc/rcN.d directory and start and stop services for that particular runlevel.

You manage Upstart with the initclt command and that allows you to start, stop services.

$ sudo initctl start rsyslog
$ sudo initctl stop rsyslog

Upstart doesn’t have a tool to enable or disable services, but you can use pre-start script section to source in a file that contains an ENABLED variable set. The Upstart Ubuntu Cookbook suggests something like this is the accepted way:

pre-start script

  # stop job from continuing if no config file found for daemon
  [ ! -f /etc/default/myapp ] && { stop; exit 0; }

  # source the config file
  . /etc/default/myapp

  # stop job from continuing if admin has not enabled service in
  # config file.
  [ -z "$ENABLED" ] && { stop; exit 0; }

end script

You can read more on Upstart at http://upstart.ubuntu.com/cookbook/ .

Remembering SystemV

Let’s go back in time somewhat. For a long time Linux systems were initialized by a program called SystemV Init, SysV Init , SysV, or just init. Init has since been replaced by another program called Systemd in most major distributions, including CentOS and Ubuntu. However, both of these distributions still support SysVInit startup scripts. We are going to give you a bit of a background on Init and how it works.

SysVInit has a concept of runlevels. Runlevels define what state a host should be in at a particular point. Each runlevel comprises a set of applications and services and an indicator of whether each should be started or stopped. For example, during the normal boot of your host, the init tool at each runlevel will initiate all the required sets of applications and services in that runlevel. Another example occurs when you shut down your host. When you tell your host to shut down, the init tool changes the runlevel to 0. At this runlevel all applications and services are stopped and the system is halted.

SysV has seven runlevels, ranging from 0 to 6, and each distribution uses different runlevels for different purposes. But some runlevels are fairly generic across all distributions. The common runlevels are 0, 1, and 6. You’ve already seen runlevel 0, which is used to shut down the host. Runlevel 1 is the single-user mode, or maintenance mode, that we described earlier in this chapter. Runlevel 6 is used when your host is being rebooted.

The runlevels on Ubuntu and CentOS hosts differ slightly. On Ubuntu, the runlevels 2 to 5 all run what is called multiuser mode . Multiuser mode is the mode where multiple users can sign in, not just a single console user. All required services are usually set to be started in this runlevel.

In comparison, CentOS generally starts in runlevel 5 if you have a GUI console installed or runlevel 3 for command line only. CentOS has the following runlevels :

• Runlevel 0: Shuts down the host and brings the system to a halt

• Runlevel 1: Runs in single-user (maintenance) mode, command console, no network

• Runlevel 2: Is unassigned

• Runlevel 3: Runs in multiuser mode, with network, and starts level 3 programs

• Runlevel 4: Is unassigned

• Runlevel 5: Runs in multiuser mode, with network, X Windows (KDE, GNOME), and starts level 5 programs.

• Runlevel 6: Reboots the host

$ sudo telint 3

#!/bin/sh –e

### BEGIN INIT INFO
# Provides:          postfix mail-transport-agent
# Required-Start:    $local_fs $remote_fs $syslog $named $network $time
# Required-Stop:     $local_fs $remote_fs $syslog $named $network
# Should-Start:      postgresql mysql clamav-daemon postgrey spamassassin saslauthd dovecot
# Should-Stop:       postgresql mysql clamav-daemon postgrey spamassassin saslauthd dovecot
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Postfix Mail Transport Agent
# Description:       postfix is a Mail Transport agent
### END INIT INFO

$ ls -la /usr/lib/systemd/system/runlevel3.target
lrwxrwxrwx. 1 root root 17 Feb  3 22:55 /usr/lib/systemd/system/runlevel3.target -> multi-user.target

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes

Managing Services with Systemd

As we have explained previously, in Systemd targets are used to group units so that when you boot our host to a particular target, you can expect certain services to be available and other, possibly conflicting, services to be stopped. If you change targets your services can also change from running to stopped. You can also stop and start individual services manually at any time.

Systemd has a command tool called systemctl. Systemctl is used to manage systemd resources on a local, remote, or virtual container. By issuing the --help argument we can see the list of arguments, unit file commands, machine commands, job commands, and more. To begin with, let’s look at how we can manage an individual service and then we can look at the wider system.

$ sudo systemctl status postfix.service
• postfix.service - Postfix Mail Transport Agent
   Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-04-27 20:35:10 AEST; 3h 0min ago
  Process: 642 ExecStart=/usr/sbin/postfix start (code=exited, status=0/SUCCESS)
  Process: 636 ExecStartPre=/usr/libexec/postfix/chroot-update (code=exited, status=0/SUCCESS)
  Process: 621 ExecStartPre=/usr/libexec/postfix/aliasesdb (code=exited, status=0/SUCCESS)
 Main PID: 993 (master)
   CGroup: /system.slice/postfix.service
           ├─ 993 /usr/libexec/postfix/master -w
           ├─1003 qmgr -l -t unix -u
           └─1926 pickup -l -t unix -u

Apr 27 20:35:09 au-mel-centos-1 systemd[1]: Starting Postfix Mail Transport Agent...
Apr 27 20:35:10 au-mel-centos-1 postfix/master[993]: daemon started -- version 2.10.1, configuration /etc/postfix
Apr 27 20:35:10 au-mel-centos-1 systemd[1]: Started Postfix Mail Transport Agent.

[Service]
Type=forking
PIDFile=/var/spool/postfix/pid/master.pid
EnvironmentFile=-/etc/sysconfig/network
ExecStartPre=-/usr/libexec/postfix/aliasesdb
ExecStartPre=-/usr/libexec/postfix/chroot-update
ExecStart=/usr/sbin/postfix start

$ sudo systemctl is-active postfix.service ; echo $?
active
0

$ sudo systemctl stop postfix.service

● postfix.service - Postfix Mail Transport Agent
   Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2016-04-28 11:28:04 AEST; 1s ago
  Process: 5304 ExecStop=/usr/sbin/postfix stop (code=exited, status=0/SUCCESS)
.....
Main PID: 5267 (code=killed, signal=TERM)

$ sudo systemctl start postfix.service

● postfix.service - Postfix Mail Transport Agent
   Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-04-28 11:51:45 AEST; 2s ago

$ sudo systemctl list-units --type target
UNIT                     LOAD   ACTIVE      SUB   DESCRIPTION
basic.target           loaded   active   active   Basic System
cryptsetup.target      loaded   active   active   Encrypted Volumes
getty.target           loaded   active   active   Login Prompts
local-fs-pre.target    loaded   active   active   Local File Systems (Pre)
local-fs.target        loaded   active   active   Local File Systems
multi-user.target      loaded   active   active   Multi-User System
network.target         loaded   active   active   Network
paths.target           loaded   active   active   Paths
remote-fs.target       loaded   active   active   Remote File Systems
slices.target          loaded   active   active   Slices
sockets.target         loaded   active   active   Sockets
swap.target            loaded   active   active   Swap
sysinit.target         loaded   active   active   System Initialization
timers.target          loaded   active   active   Timers

$ sudo systemctl isolate graphical.target

$ sudo systemctl disable postfix.service
Removed symlink /etc/systemd/system/multi-user.target.wants/postfix.service.

$ sudo systemctl enable postfix.service
Created symlink from /etc/systemd/system/multi-user.target.wants/postfix.service to /usr/lib/systemd/system/postfix.service.

$ sudo systemctl --type=service --state=running
UNIT                       LOAD      ACTIVE    SUB        DESCRIPTION
atd.service                loaded    active    running    Job spooling tools
crond.service              loaded    active    running    Command Scheduler
[email protected]         loaded    active    running    Getty on tty1
NetworkManager.service     loaded    active    running    Network Manager
postfix.service            loaded    active    running    Postfix Mail Transport Agent
systemd-journald.service   loaded    active    running    Journal Service
systemd-logind.service     loaded    active    running    Login Service
systemd-udevd.service      loaded    active    running    udev Kernel Device Manager
wpa_supplicant.service     loaded    active    running    WPA Supplicant daemon

[Service]
...
ExecStart=/etc/init.d/postfix start
ExecStop=/etc/init.d/postfix stop
ExecReload=/etc/init.d/postfix reload

$ sudo systemctl disable postfix
postfix.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install disable postfix

$ sudo update-rc.d postfix start defaults

$ sudo update-rc.d postfix start 23 40

$ sudo update-rc.d postfix stop 2

$ sudo update-rc.d postfix remove

$ sudo update-rc.d -f postfix remove

Systemd Timers

Systemd timers are unit files that end with *.timer suffix. They control when a service is run. It is expected that these will replace Cron for managing regularly scheduled job management.

It has advantages over Cron in that it can trigger events based on not only the wall clock (the system’s time) but also things like the period after the last event was run, on boot, on startup or a combination of these sorts of things. A timer unit file can also be used to trigger another service unit that doesn’t have a timer.

The configuration requires a [Timer] section in the unit file. Let’s take this example of the daily apt schedule that manages the daily checks for the apt cache (see Listing 6-13).

Listing 6-13. Systemd timer /lib/systemd/system/apt-daily.timer

[Unit]
Description=Daily apt activities

[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
AccuracySec=1h
Persistent=true

[Install]
WantedBy=timers.target

Here you can see that we have the usual sections for [Unit] and [Install]. The OnCalendar option specifies when we should run the service, and that can be specified by the day of the week, the year, the month, the day, and then HH:MM:SS or any number of ways! There is a full list of options in man 7 systemd.timer. Asterisks (*) mean any value and this is shorthand for daily (*-*-*). It is triggered to occur at either 06:00 or 18:00Hrs, with a 12-hour random window (RandomizedDelaySec), give or take an hour (AccuracySec). The Persistence=true says to run if it missed the last start (like the system was shutdown).

Let’s imagine we have a monitor.service file that makes a cURL POST request to a web site to tell us the system has booted, and checks in every 20 minutes after that.

[Unit]
Description=Tell the monitoring service we are up

[Timer]
OnBootSec=2min
OnUnitActiveSec=20min

[Install]
WantedBy=timers.target

We will put this configuration into a file called /etc/systemd/system/monitor.timer, next to our monitor.service file. Here, we can see that a unit file for a timer needs these three sections, [Unit], [Timer], and [Install].

The [Timer] section is where we put our timer options. We have chosen here to trigger our cURL job 2 minutes after the boot event (OnBootSec), and then every 20 minutes after the last event (OnUnitActiveSec). The times here could be expressed as 1h (1 hour), 2w (2 weeks), or a variety of other ways. These are called monotonic timers as opposed to the real-time timer we saw in Listing 6-13.

The WantedBy=timers.target in the [Install] section creates the symlinks to the timers target which enables the timer in systemd. We can enable the timer with the following:

$ sudo systemctl enable monitor.timer
Created symlink from /etc/systemd/system/timers.target.wants/monitor.timer to /etc/systemd/system/monitor.timer.

We can now list our timers with the following:

You can see our timer in Figure 6-9. You can see that it activates our monitor.service and that it was successful 18 minutes ago. You can also see the apt-daily.timer too.

Figure 6-9. Listing timers

One of the things it doesn’t do simply is trigger an e-mail in the event of failure. Some people rely on this feature and that is why they might choose Cron instead.

For more information on timers see

• https://wiki.archlinux.org/index.php/Systemd/Timers

• www.freedesktop.org/software/systemd/man/systemd.time.html

• www.freedesktop.org/software/systemd/man/systemd.timer.html

Introducing Cron

There is a last type of service management we need to show you: scheduling. You may already be familiar with the Microsoft Task Scheduler, which you can use to schedule tasks to be run once or repeated regularly on a given minute, hour, day, week, or month. The equivalent in Linux is called crontab (short for chronograph table). Its purpose is to submit tasks at set times according to the host’s clock. Tasks can be any script or application that you desire. Commonly, you will find maintenance-type tasks in the crontabs. These can be scheduled to run nightly, weekly, or monthly and perform some kind of script, like one that deletes all files in the /var/ log/httpd directory older than two months.

Cron jobs (the tasks that crontab performs) are defined in a series of scripts under the directories defined in the /etc/crontab file. These are referred to as system cron jobs. The lists of directories in the /etc/crontab file looks like this:

$ less /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01  *  *  *  *  root  run-parts  /etc/cron.hourly
02  4  *  *  *  root  run-parts  /etc/cron.daily
22  4  *  *  0  root  run-parts  /etc/cron.weekly
42  4  1  *  *  root  run-parts  /etc/cron.monthly

You should not edit this file because, being the system crontab file, it is likely to be replaced with a new version each time crontab is updated. This means any changes will be lost. Plus you could cause other problems if you make a mistake. It does, however, provide a good example of the syntax of a crontab file.

Listed at the top of the file are SHELL, PATH, MAILTO, and HOME environment variables, which we described in Chapter 4. Lines starting with # are comments and can be ignored. Further down the file, you can see five columns with either a number or *, a column with root, a column with run-parts, and finally a directory listing.

The first five columns represent minute, hour, day of the month, month, and day of the week. Let’s look at the last line:

42 4 1 * * root run-parts /etc/cron.monthly

Here, 42 is the 42nd minute of the hour, the hour is 4 (with hours being based on a 24-hour clock), and 1 represents the first day of the month. So crontab will run the last line at 04:42 a.m. on the first day of every month.

Any column with an asterisk (*) in it means all values are valid and are not restricted as to when to run. Let’s take a look at our last line again:

42 4 1 * 3 root run-parts /etc/cron.monthly

Here we’ve changed the value in the day of the week column to 3. Our job would now run at 04:42 a.m. on the first day of the month and also every Wednesday.

You can also specify automatically reoccurring jobs as follows:

*/2 4 1 * 3 root run-parts /etc/cron.monthly

Here, instead of specifying an exact time to run, we have used the notation */2. This notation tells cron to run the job every time the number of minutes is cleanly divisible by 2. This allows you to do some powerful stuff. For example, you could use */4 in the hour column, and the job would run every other minute every fourth hour, like so:

*/2 */4 1 * * root run-parts /etc/cron.monthly

The use of the comma indicates a list of values, for example

2 0,1,2,3,4 1 * * root run-parts /etc/cron.monthly

This would run the job at 12:02 a.m., 1:00 a.m., 2:00 a.m., 3:00 a.m., and 4:00 a.m.

You can also specify ranges of numbers as follows:

2 0-4,12-16 1 * * root run-parts /etc/cron.monthly

Here the command would be run at 2 minutes past the hour between the hours of 12 a.m. and 4 a.m. and between 12 p.m. and 4 p.m.

The next column, root, represents the user that this program will run as. When you add your own cron jobs (or scripts), you can set this to be any valid user.

The run-parts option is the command that is being run. run-parts is a special command that will run any executable script within a specified directory. In this case, run-parts will change to the /etc/cron.hourly directory, /etc/cron.daily directory, and so on, and run the executable script it finds there.

Let’s inspect one of the system cron directories, the /etc/cron.daily directory, for example, and examine one of the scripts already present on the Red Hat host. These are system crons that will run once every day unless otherwise defined.

$ ls -l /etc/cron.daily/
-rwxr-xr-x 1 root root  379 Dec 19  2016 0anacron
-rwxr-xr-x 1 root root  118 Oct  1 00:06 cups
-rwxr-xr-x 1 root root  180 Oct 22  2017 logrotate
-rwxr-xr-x 1 root root  114 Jan 16  2018 rpm
-rwxr-xr-x 1 root root  290 Nov 26  2016 tmpwatch

Let’s view one of these files, /etc/cron.daily/rpm, and see what is inside it.

$ less /etc/cron.daily/rpm
#!/bin/sh

/bin/rpm -qa --qf '%{name}-%{version}-%{release}.%{arch}.rpm
' 2>&1 
           | /bin/sort > /var/log/rpmpkgs

This daily executed script populates the /var/log/rpmpkgs file with a sorted list of all the RPM packages on your host.

Individual users can also create a crontab. You create and edit existing crontabs with the crontab –e command. If a crontab for your user does not exist already, this command creates a crontab file in /var/spool/cron/ username.

The syntax used in a user’s cron jobs is identical to that of the system crontab file you saw earlier, with one difference. You can only specify the user field in the system crontab file. Let’s look at an example created by the user jsmith using the –l, or list, option for crontab.

$ crontab -l
*/2 * * * * [ -e /tmp/log ] && rm -f /tmp/log

You will see a list of all the cron jobs scheduled by this user. This is a simple series of commands that first check for the existence of a file called /tmp/log, and if it exists, remove it. It is set to run once every 2 minutes (*/2).

As a privileged user, you can view another person’s cron by issuing the crontab command with the –u username option.

$ sudo crontab –u ataylor –l
1 2 * * * /usr/local/bin/changeLog.sh

You can also edit another person’s cron by issuing the same crontab command and the –u username -e options.

$ sudo crontab –u ataylor –e

This allows you to edit the crontab for the user.

You can also remove your crontab or another user’s crontab by issuing the crontab command with the –r option.

$ sudo crontab –u ataylor –r

This removes ataylor’s crontab file: /var/spool/cron/ataylor.

Your host has a service that monitors the cron jobs and any changes to them. It also executes the individual jobs when they are scheduled. This service is called crond and can be started and stopped via the systemctl commands we talked about earlier.

$ sudo systemctl start|stop|reload cron(d if CentOS)