Chapter 2. Managing Services with Systemd

Every time you start your Linux computer its initialization system launches a hundreds of processes. You can see this on your startup screen (figure Figure 2-1, press the Escape key to hide your graphical startup screen and see the startup messages.)

Linux startup messages
Figure 2-1. Linux startup messages

In olden times we had the Unix System V initialization system (SysV init), BSD init, and Linux Standard Base init (LSB) for launching processes at startup. SysV init was the most common. Those days are fading away, and now systemd is the shiny new init system for Linux. It has been adopted by all the major Linux distributions, though of course there are a number of distributions that still use the legacy init systems.

In this chapter you will learn if your Linux distribution uses systemd. You will learn what processes, threads, services, and daemons are, and how to use systemd to manage services: start, stop, enable, disable, and check status. You will become acquainted with the systemctl command, which is the systemd system and service manager.

systemd is designed to provide functionality suited to modern complex server and desktop systems, and does considerably more than the legacy init systems. It provides complete service management from startup to shutdown, starting processes at boot, on-demand after boot, and shutting down services when they are not needed. It manages functions such as system logging, auto-mounting filesystems, automatic service dependency resolution, name services, device management, network connection management, login management, and a host of other tasks.

This sounds like a lot until you realize that processes do everything on a computer, and all of this functionality used to be provided by a large assortment of other programs. systemd brings it all together in an integrated software suite that should operate the same way on all Linux systems, though as always with Linux there are some minor exceptions, such as file locations and service names, so be aware that your particular Linux may have some differences from the examples in this chapter.

systemd attempts to decrease boot times and parcel out system resources more efficiently by starting processes concurrently and in parallel, and starting only necessary services, leaving other services to start after boot as needed. A service that is dependent on other services no longer has to wait to start for those services to become available, because all it needs is a waiting Unix socket to become available. Recipe Recipe 2.9 shows how to find processes that are slowing down your system startup.

systemd binaries are written in C, which also provides some performance enhancement. The legacy inits are masses of shell scripts, and any compiled language operates faster than shell scripts.

systemd is backwards-compatible with SysV init. Most Linux distributions retain the legacy SysV configuration files and scripts, including /etc/inittab, and the /etc/rc.d/ and /etc/init.d/ directories. When a service does not have a systemd configuration file then systemd looks for a SysV configuration file. systemd is also backwards-compatible with LSB init.

systemd service files are smaller and easier to understand than SysV init files. Compare a SysV init file for sshd with its systemd service file. This is a snippet of its SysV init file:

#! /bin/sh

# Provides:		sshd
# Required-Start:	$remote_fs $syslog
# Required-Stop:	$remote_fs $syslog
# Default-Start:	2 3 4 5
# Default-Stop:
# Short-Description:	OpenBSD Secure Shell server

set -e

# /etc/init.d/ssh: start and stop the OpenBSD "secure shell(tm)" daemon

test -x /usr/sbin/sshd || exit 0
( /usr/sbin/sshd -? 2>&1 | grep -q OpenSSH ) 2>/dev/null || exit 0

umask 022

if test -f /etc/default/ssh; then
    . /etc/default/ssh

. /lib/lsb/init-functions

This goes on for a total of 162 lines. Compare to a complete systemd service file:

Description=OpenBSD Secure Shell server auditd.service

ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID


Even without reading the documentation, or knowing anything about systemd, you can understand some of what this file is supposed to do.

See Rethinking PID 1 for a detailed introduction to systemd by one of its inventors and maintainers, Lennart Poettering. Rethinking PID 1 details the rationale behind building a new init system, its architecture, advantages, and how it uses Linux existing kernel features in place of duplicating existing functionality.

2.1 Learning if your Linux uses Systemd


You need to know if your Linux distribution uses systemd, or something else.


Look for the /run/systemd/system/ directory. If this exists, then your init system is systemd.


The /run/systemd/ directory may be present on your system, if your distribution supports multiple init systems. But systemd is not the active init unless you see /run/systemd/system/.

There are several other ways to learn which init system your system is using. Try querying /sbin/init. Originally this was the SysV executable, and now most Linux distributions preserve the name and symlink it to the systemd executable. This example confirms that the init is systemd:

$ stat /sbin/init
File: /sbin/init -> /lib/systemd/systemd

On a system using SysV init it has no symlink:

$ stat /sbin/init
File: /sbin/init

The /proc/ pseudo-filesystem is an interface to your Linux kernel, and contains the current state of a running system. It is called a pseudo-filesystem because it exists only in memory, and not on disk. In this example /proc/1/exe is symlinked to the systemd executable:

$ sudo stat /proc/1/exe
File: /proc/1/exe -> /lib/systemd/systemd

On a SysV system it links to init:

$ sudo stat /proc/1/exe
File: /proc/1/exe -> /sbin/init

The /proc/1/comm file reports your active init system:

$ sudo cat /proc/1/comm

On a SysV system it reports init:

$ sudo cat /proc/1/comm

The command attached to process ID (PID) 1 is your init. PID 1 is the first process launched at startup, which then starts all other processes. You can see this with the ps command:

$ ps -p 1
  PID TTY          TIME CMD
    1 ?        00:00:00 systemd

When the init is SysV, it looks like this:

$ ps -p 1
  PID TTY          TIME CMD
    1 ?        00:00:00 init

See Recipe 2.2 for more information on PID 1.

Linux support for systemd varies. Most of the major Linux distributions, including Fedora, Red Hat, CentOS, openSUSE, SUSE Linux Enterprise, Debian, Ubuntu, Linux Mint, Arch, Manjaro, Elementary, and Mageia Linux have adopted systemd.

Some popular distributions that do not support systemd, or include it but not as the default init, are Slackware, PCLinuxOS, Gentoo Linux, MX Linux, and antiX.

See Also

  • for information on hundreds of Linux distributions.

  • man proc (5)

  • man pstree (1)

  • man ps (1)

2.2 Understanding PID 1, the Mother of All


You keep hearing about Process ID (PID) 1, and about processes, threads, services, and daemons, and need to know what it all means.


PID 1 is the mother of all processes on a Linux systems. This is the first process to start, and then it launches all other processes.

Processes are one or more running instances of a program. Every task in a Linux system is performed by a process. Processes can fork, creating independent child processes which are copies of the parent processes, each with its own unique PID, and their own allocation of system resources, such as CPU and memory. threads are lightweight processes that run in parallel and share system resources with their parents.

Daemons are non-interactive background processes, and their names traditionally end in d, for example httpd, sshd, and systemd. Services are daemons. Traditionally, Linux uses “daemons” and Microsoft Windows calls them “services”. systemd uses both terms, like this example in man 7 daemon:

A daemon is a service process that runs in the background and supervises the
system or provides functionality to other processes.

One could argue that daemons are components in a service. Arguing is fun, and so is getting on with doing stuff, so use whichever term you wish.

Every Linux system starts PID 1 first, which then launches all other processes. Use the _ps_command to list all running processes in PID order:

$ ps -ef
root         1     0  0 10:06 ?        00:00:01 /sbin/init splash
root         2     0  0 10:06 ?        00:00:00 [kthreadd]
root         3     2  0 10:06 ?        00:00:00 [rcu_gp]
root         4     2  0 10:06 ?        00:00:00 [rcu_par_gp]

The pstree command organizes this mass of information into a tree diagram. This example shows all processes, their child process, PIDs, and threads, which are enclosed in curly braces:

$ pstree -p
           │                   └─{ModemManager}(949)
           │                     ├─{NetworkManager}(989)
           │                     └─{NetworkManager}(991)
           │                      └─{accounts-daemon}(948)

The full pstree output is quite large. You can view a single process, identified by its PID, and its parents, children and threads, like the following example for the Kate text editor:

$ pstree -sp 5193

This shows that systemd(1) is Kate’s parent, bash(5218) is Kate’s child, and all the processes in curly braces are Kate’s threads.


Processes always exist in one of several states, and these states change according to system activity. The following ps example displays the PID, user, status, and command fields:

$ ps -eo pid,user,stat,comm
    1 root       Ss   systemd
    2 root       S    kthreadd
   32 root       I<   kworker/3:0H-kb
   68 root       SN   khugepaged
11222 duchess    Rl   konsole
  • R is either currently running, or waiting in the run queue.

  • l means the process is multi-threaded.

  • S is interruptable sleep, the process is waiting for an event to complete.

  • s is a session leader. Sessions are related processes managed as a unit.

  • I is an idle kernel thread.

  • < means high-priority.

  • N is low-priority.

There are several rarely-used states you can read about in man 1 ps.

See Also

  • TODO killing

  • man proc (5)

  • man pstree (1)

  • man ps (1)

2.3 Listing Services and their States with systemd


You want to list all services installed on your system, and you want to know the states of the services, whether they are running, not running, or in an error state.


systemctl, the systemd manager command, tells all. Run it with no options to see a detailed list of all loaded units. A systemd unit is any related batch of processes defined in a unit configuration file, and managed by systemd:

$ systemctl

This prints a giant pile of information, 177 active loaded units on my test system with the full unit names, status, and long descriptions. Use the tee command to store the output in a text file for easier study:

$ systemctl | tee systemctl-units.txt

Treat yourself to more information overload by listing all units, active and inactive:

$ systemctl --all

This results in 349 loaded units listed on my test system, including not-found and inactive units. How many total unit files? The following example shows five out of 322:

$ systemctl list-unit-files
UNIT FILE                                      STATE
proc-sys-fs-binfmt_misc.automount              static
-.mount                                        generated
mount                                          generated
dev-hugepages.mount                            static
home.mount                                     generated
322 unit files listed.

We are interested in service files, because Linux users and administrators interact mainly with service files, and rarely need to bother with any other type of unit file. How many are installed?:

$ systemctl list-unit-files --type=service
UNIT FILE                                  STATE
accounts-daemon.service                    enabled
acpid.service                              disabled
alsa-state.service                         static
alsa-utils.service                         masked
anacron.service                            enabled
212 unit files listed.

The above example displays the four most common states that a service can be in: enabled, disabled, static, or masked.

List only enabled services:

$ systemctl list-unit-files --type=service --state=enabled
UNIT FILE                                  STATE
accounts-daemon.service                    enabled
anacron.service                            enabled
apparmor.service                           enabled
autovt@.service                            enabled
avahi-daemon.service                       enabled
62 unit files listed.

List only disabled services:

$ systemctl list-unit-files --type=service --state=disabled
UNIT FILE                            STATE
acpid.service                        disabled
brltty.service                       disabled
console-getty.service                disabled
mariadb@.service                     disabled
12 unit files listed.

List only static services:

$ systemctl list-unit-files --type=service --state=static
UNIT FILE                              STATE
alsa-restore.service                   static
alsa-state.service                     static
apt-daily-upgrade.service              static
apt-daily.service                      static
106 unit files listed.

List only masked services:

$ systemctl list-unit-files --type=service --state=masked
UNIT FILE                    STATE
alsa-utils.service           masked
bootlogd.service             masked
bootlogs.service             masked
checkfs.service              masked
36 unit files listed.


Service unit files are in /usr/lib/systemd/system/ or /lib/systemd/system/, according to where your Linux distribution puts them. These are plain text files you can read.

  • enabled shows that the service has been activated and is managed by systemd. When a service is enabled systemd creates a symlink in /etc/systemd/system/ from the unit file in /usr/lib/systemd/system/. It can be started, stopped, reloaded, and disabled by the user with the systemctl command.

  • disabled means there is no symlink in /etc/systemd/system/, and it will not start automatically at boot. You can stop and start it manually.

  • masked means the service is linked to /dev/null/. It is completely disabled and cannot be started by any means.

  • static The unit file is a dependency of other unit files, and cannot be started or stopped by the user.

Some less-common service states you will see:

  • indirect states belong to services that are not meant to be managed by users, but are meant to be used by other services.

  • generated states indicate that the service has been converted from a non-native systemd initialization configuration file, either SysV or LSB init.

See Also

  • systemctl (1)

2.4 Querying the Status of Selected Services


You want to know the status of one service, or a few specific services.


systemctl status provides a nice little bundle of useful status information. The following example queries the sshd service:

$ systemctl status sshd
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset:
   Active: active (running) since Sun 2020-06-28 07:26:04 PDT; 2h 21min ago
  Process: 2972 ExecReload=/bin/kill -HUP $MAINPID (code=exited,
  Process: 2968 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
  Process: 940 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
 Main PID: 1031 (sshd)
    Tasks: 1
   CGroup: /system.slice/ssh.service
           └─1031 /usr/sbin/sshd -D

Jun 28 08:02:07 dreamer systemd[1]: Reloading OpenBSD Secure Shell server.
Jun 28 08:02:07 dreamer sshd[1031]: Received SIGHUP; restarting.
Jun 28 08:02:07 dreamer systemd[1]: Reloaded OpenBSD Secure Shell server.
Jun 28 08:02:07 dreamer sshd[1031]: Server listening on port 22.
Jun 28 08:02:07 dreamer sshd[1031]: Server listening on :: port 22.
Jun 28 08:15:22 dreamer systemd[1]: Reloading OpenBSD Secure Shell server.
Jun 28 08:15:22 dreamer sshd[1031]: Received SIGHUP; restarting.
Jun 28 08:15:22 dreamer systemd[1]: Reloaded OpenBSD Secure Shell server.
Jun 28 08:15:22 dreamer sshd[1031]: Server listening on port 22.
Jun 28 08:15:22 dreamer sshd[1031]: Server listening on :: port 22.

Query multiple services with a space-delimited list:

$ systemctl status mariadb.service bluetooth.service lm-sensors.service


There is a lot of useful information in this little bit of output.

The dot next to the service name is a quick status indicator. It appears in colors on most terminals. White is an inactive or deactivating state. Red is a failed or error state. Green indicates an active, reloading, or activating state.

  • Loaded: verifies that the unit file has been loaded into memory, displays its full path, the service is enabled (see the Discussion about states in Recipe 2.3), and vendor preset: disabled/enabled indicates if the installation default is to start at boot or not. When it is disabled the vendor default is to not start at boot. This only shows the vendor preference, and does not indicate if it is currently enabled or disabled.

  • Active: tells you if the service is active or inactive, and for how long it has been in that state.

  • Process: reports the PIDs and their commands and daemons.

  • Main PID: is the process number for the cgroup slice.

  • Task: Reports how many tasks the service has started. Tasks are PIDs.

  • CGroup: Shows which unit slice the service belongs to and its PID. The three default unit slices are user.slice, system.slice, and machine.slice.

Linux Control Groups (cgroups) are sets of related processes and all of their future children. In systemd, a slice is a sub-division of a cgroup, and each slice manages a particular group of processes. Run systemctl status to see a diagram of the cgroup hierarchy.

By default, service and scope units are grouped in /usr/lib/systemd/system/system.slice.

User sessions are grouped in /usr/lib/systemd/system/user.slice.

Virtual machines and containers registered with systemd are grouped in _/usr/lib/systemd/system/machine.slice.

  • The remaining lines are the most recent log entries from journalctl, the systemd log manager.

See Also

2.5 Starting and Stopping Services


You need to know how to stop and start services with systemd.


This is a job for systemctl. The following examples use the SSH daemon to demonstrate service management.

Start a service:

$ sudo systemctl start sshd.service

Stop a service:

$ sudo systemctl stop sshd.service

Stop and then restart a service:

$ sudo systemctl restart sshd.service

Reload the service’s configuration, for example you made a change to sshd_config and want to load the new configuration without restarting the service:

$ sudo systemctl reload sshd.service


All of these commands also work with multiple services, space-delimited, for example:

$ sudo systemctl start sshd.service mariadb.service firewalld.service

Some services have start, reload, stop, and other instructions in their unit files, like this example for httpd:

ExecStart=/usr/sbin/httpd/ $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

You don’t have to do anything special with this information, it is there when you are curious about how systemctl is managing a particular service.

See Also

2.6 Enabling and Disabling Services


You want a service or services to automatically start at boot, or you want to stop a service from starting at boot, or disable it completely.


Enabling a service configures it to automatically start at boot.

Disabling a service stops it from starting at boot, but it can be started and stopped manually.

Masking a service disables it so that it cannot be started at all.

The following example enables the sshd service:

$ sudo systemctl enable sshd.service
Created symlink /etc/systemd/system/ →

The output shows that enabling a service means creating a symlink from the service file in /lib/systemd/system/ to /etc/systemd/system/. This does not start the service, so you must either start it manually, or reboot.

This command disables the sshd service. It does not stop it, so you must either stop it manually after disabling it, or reboot:

$ sudo systemctl disable sshd.service
Removed /etc/systemd/system/

This command reenables the mariadb service, which disables and then enables it. This is useful for quickly resetting the symlinks:

$ sudo systemctl reenable mariadb.service
Removed /etc/systemd/system/
Removed /etc/systemd/system/mysqld.service.
Removed /etc/systemd/system/mysql.service.
Created symlink /etc/systemd/system/mysql.service →
Created symlink /etc/systemd/system/mysqld.service →
Created symlink /etc/systemd/system/ →

The following command disables the bluetooth service completely by masking it, so that it cannot be started at all:

$ sudo systemctl mask bluetooth.service
Created symlink /etc/systemd/system/bluetooth.service → /dev/null.

Unmasking the bluetooth service does not enable it, but it can be started manually:

$ sudo systemctl unmask bluetooth.service
Removed /etc/systemd/system/bluetooth.service.


When you enable, disable, mask, or unmask a service the service remains in its current state, and you must start or stop it manually, or reboot.

It is better to stop a service before disabling it, to give it a chance to write data and do whatever cleanups it needs to do.

See Also

  • systemctl (1)

2.7 Killing Processes


You want to know how to kill troublesome processes. A certain service may be unresponsive, or running away, spawning forks and causing your system to hang. Your normal stop command is not working, what do you do?


Killing a process means forcefully stopping it. On Linux systems with systemd, you should use systemctl kill. On systems without systemd use the old kill command.

systemctl kill is preferable because it stops all processes that belong to a service, and leaves no orphan processes, nor any processes that might restart the service and continue to make trouble. First try it with no options other than the service name, then check the status:

$ sudo systemctl kill mariadb
$ systemctl status mariadb
● mariadb.service - MariaDB 10.1.44 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset:
   Active: inactive (dead) since Sun 2020-06-28 19:57:49 PDT; 6s ago

The service has cleanly stopped, and the colored dot is white, indicating the service is deactivated. If this does not work, then try the nuclear option:

$ sudo systemctl kill -9 mariadb

The kill command does not recognize service or command names, but rather requires the PID of the offending process:

$ sudo kill 1234

If this does not stop it, then use the nuclear option:

$ sudo kill -9 1234


Use the top command to identify runaway processes. Run it with no options, and the processes using up the most system resources are listed at the top. Press the Q key to stop it.

$ top
top - 20:30:13 up  4:24,  6 users,  load average: 0.00, 0.03, 0.06
Tasks: 246 total,   1 running, 170 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  0.2 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16071016 total,  7295284 free,  1911276 used,  6864456 buff/cache
KiB Swap:  8928604 total,  8928604 free,        0 used. 13505600 avail Mem

 3504 carla     20   0 99.844g 177588  88712 S   2.6  1.1   0:08.68 evolution
 2081 carla     20   0 3818636 517756 177744 S   0.7  3.2   5:07.56 firefox
 1064 root      20   0  567244 148432 125572 S   0.3  0.9  12:54.75 Xorg
 2362 carla     20   0 2997732 230508 145444 S   0.3  1.4   0:40.72 Web Content

kill sends signals to processes, and the default signal is SIGTERM (signal terminate). SIGTERM is gentle, allowing processes to shut down cleanly. SIGTERM is also ignorable, and processes don’t have to pay attention to it. Signals can be identified by name or number; for most folks the numbers are easier to remember, so spelling out the default looks like this:

$ sudo kill -1 1234

kill -9 is SIGKILL. SIGKILL stops processes immediately and uncleanly, and also attempts to stop all child processes.

Killing services with systemctl kill is easier than with kill, and more reliable. You only need the service name, and don’t have to hunt down PIDs. It ensures that all processes belonging to the service are stopped, which kill cannot ensure.

There are a ton of signals that have accumulated over the years, and you can read all about them in man 7 signal. In my opinion the only relevant signals are SIGTERM and SIGKILL, but don’t let that stop you from learning more about them.

See Also

  • systemd.kill (5)

  • systemctl (1)

  • kill (1)

  • man 7 signal

2.8 Managing Runlevels with systemd


You want to know if systemd has runlevels like SysV, and how to use them.


systemd targets are similar to SysV runlevels. These are boot profiles that start your system with different options, such as multi-user mode with a graphical desktop, multi-user mode with no graphical desktop, and emergency and rescue modes to use when your default target will not boot. (See the Discussion for more information on runlevels.)

The following command checks if the system is running and reports its state:

$ systemctl is-system-running

What is the default target?

$ systemctl get-default

Get the current runlevel:

$ runlevel
N 5

Reboot to rescue mode:

$ sudo systemctl rescue

Reboot to emergency mode:

$ sudo systemctl emergency

Reboot to the default mode:

$ sudo systemctl reboot

Reboot to a different target without changing the default:

$ sudo systemctl isolate

Set a different default runlevel:

$ sudo systemctl set-default

List the runlevel target files and their symlinks on your system (on some Linux distributions the directory is /lib/systemd/system/runlevel*):

$ ls -l /usr/lib/systemd/system/runlevel*

List the dependencies in a runlevel target:

$ systemctl list-dependencies


SysV runlevels are different states that your system can boot to, for example with a graphical desktop, without a graphical desktop, and emergency runlevels to use when your default runlevel has problems and will not boot.

systemd targets approximately correspond to the legacy SysV runlevels:

  •,, halt

  •,, single-user text mode, all local filesystems mounted, root user only, no networking

  •,, multi-user text mode (no graphical environment)

  •,, multi-user graphical mode

  •,, reboot

systemctl emergency is a special target that is more restricted than rescue mode. No services, no mount points other than the root filesystem, no networking, root user only. It is the most minimal running system for debugging problems. You may see options to boot into a rescue or emergency mode in your GRUB2 bootloader screen.

systemctl is-system-running reports various system states:

  • initializing means the system has not completed startup.

  • starting means the system is in the final stages of startup.

  • running is fully operational, and all processes are started.

  • degraded means the system is operational, but one or more systemd units have failed.

  • maintenance means that either the rescue or emergency target is active.

  • stopping means that systemd is shutting down.

  • offline means that systemd is not running.

  • unknown means that there is a problem preventing systemd from determining the operational state

See Also

  • systemctl (1)

  • systemd-halt.service (8)

2.9 Diagnosing Slow Startups


systemd promises faster startups, but your system starts up slowly, and you want to find out why.


You want systemd-analyze blame. Run it with no options to see a list of system processes and how long they took to start:

$ systemd-analyze blame
         34.590s apt-daily.service
          6.782s NetworkManager-wait-online.service
          6.181s dev-sda2.device
          4.444s systemd-journal-flush.service
          3.609s udisks2.service
          2.450s snapd.service

Analyze only user processes:

systemd-analyze blame --user
          3.991s pulseaudio.service
           553ms at-spi-dbus-bus.service
           380ms evolution-calendar-factory.service
           331ms evolution-addressbook-factory.service
           280ms xfce4-notifyd.service


This may or may not be helpful, as a process that reported a long startup time may have been waiting for another process to start. It is useful to review everything that starts at boot, and perhaps find services you don’t want to starting at at boot. My favorite to disable is Bluetooth, because I don’t use it on my servers or PCs, but many Linux distros enable it by default.

See Also

  • systemd-analyze (1)

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.