Chapter 2: Understanding systemd Directories and Files
In this chapter, we'll explore the various systemd unit files and configuration files and explain the purpose of several types. We'll briefly look at some of the executable files that are associated with systemd. Along the way, we'll also look at the directories where these files live.
These are the topics we will cover in this chapter:
- Understanding the systemd configuration files
- Understanding the systemd unit files
- Understanding the systemd executables
The topics in this chapter comprise basic foundational knowledge of systemd. We'll be building upon this foundation in the chapters to come.
If you're ready, let's go.
Technical requirements
If you'd like to follow along with what I'm doing, you'll need a couple of virtual machines (VMs). Here, I'm using Ubuntu Server 20.04 for the Ubuntu side of things, and AlmaLinux 8 for the Red Hat side of things. (You'll also see me using Fedora to point out a couple of things, but you won't need a Fedora VM yourself.)
Check out the following link to see the Code in Action video: https://bit.ly/3xL4os5
Understanding the systemd configuration files
In this section, we'll look at the configuration files that control how the various components of systemd operate. If you want to follow along with your own VM, it won't much matter which distro you have because things will be mostly the same across all systemd-enabled distros. Okay—so now you're yelling at me, saying:
Mostly the same? Why, Donnie, you told us before that systemd is implemented consistently across all distros! What gives?
Well, it is consistent, in that the management and control commands are the same across all distros, but the systemd ecosystem includes several different components besides just the init system. These components are optional, and some Linux distros don't use all of them in a default configuration. Several of these components have configuration files in the /etc/systemd/ directory, as you can see here:
[donnie@localhost systemd]$ pwd
/etc/systemd
[donnie@localhost systemd]$ ls -l *.conf
-rw-r--r--. 1 root root 720 May 31 2016 bootchart.conf
-rw-r--r--. 1 root root 615 Mar 26 2020 coredump.conf
-rw-r--r--. 1 root root 1041 Mar 26 2020 journald.conf
-rw-r--r--. 1 root root 1042 Mar 26 2020 logind.conf
-rw-r--r--. 1 root root 584 Mar 26 2020 networkd.conf
-rw-r--r--. 1 root root 529 Mar 26 2020 pstore.conf
-rw-r--r--. 1 root root 764 Mar 26 2020 resolved.conf
-rw-r--r--. 1 root root 790 Mar 26 2020 sleep.conf
-rw-r--r--. 1 root root 1762 Mar 26 2020 system.conf
-rw-r--r--. 1 root root 677 Mar 26 2020 timesyncd.conf
-rw-r--r--. 1 root root 1185 Mar 26 2020 user.conf
[donnie@localhost systemd]$
The timesyncd.conf file, which you see second from the bottom in the preceding code snippet, is one of those components that you won't see everywhere. It's for the service that synchronizes the machine's time to a trusted external source. You see it here, but you won't see it on either Red Hat Enterprise Linux (RHEL) or any of RHEL's free-of-charge derivatives. That's because RHEL-type distros use an alternate time-synchronization service called chronyd, and just because you see a configuration file here for a particular systemd component doesn't necessarily mean that that component is being used. On the Fedora machine from which I took the preceding code snippet, the networkd, resolved, and timesyncd components are all disabled. (As with the RHEL distros, Fedora uses chronyd for time-keeping, but it still has the timesyncd component installed.) On the other hand, if you look at the newest versions of Ubuntu Server, you'll see that these optional components are enabled by default. (We'll see later how to tell if a service is either enabled or disabled.)
Okay—let's talk about what's in these configuration files. We'll start by looking at the system.conf file, which sets the configuration for the systemd init process. (For space reasons, I can only show part of the file here. You can view the whole file on your VM by doing less /etc/systemd/system.conf.) Here's a snippet:
[Manager]
#LogLevel=info
#LogTarget=journal-or-kmsg
#LogColor=yes
#LogLocation=no
. . .
. . .
#DefaultLimitNICE=
#DefaultLimitRTPRIO=
#DefaultLimitRTTIME=
Now, I'm not going to explain this file line by line because I don't want you to hate me for boring you to death. But seriously, in normal circumstances, you might not ever have to change any of these configuration files. If you think that you might need to do anything with them, your best bet is to read their associated man pages, which will have a breakdown of what each of these parameters is doing for you. The trick is that for most of these files, you'll have to add the systemd- text string to the front of the filename to find its man page. For example, to look at the man page for the system.conf file, type the following:
man systemd-system.conf
Also, you might have noticed that in all of these configuration files, every line is commented out. That doesn't mean that those lines have no effect. Instead, it means that these are the default parameters that are compiled in. To change something, you would uncomment the line for the desired parameter and change its value.
Pro tip
You can use the apropos command to find all man pages with a specific text string in either the man page name or man page description. For example, to find all pages that match the systemd string, just type the following: apropos systemd.
You can also type man -k systemd, which is a synonym for apropos systemd. (I got into the habit early on of always typing apropos, and I've never broken this habit.) If nothing comes up when you try this, you might have to rebuild the man page database, which you'll do by typing sudo mandb.
All right—I think we've talked enough about the configuration files. Next up, we'll talk about the systemd unit files.
Understanding the systemd unit files
Instead of using a set of complex Bash shell scripts, the systemd init system controls system and service operations with various types of unit files. Each unit file has a filename with a filename extension that describes which type of unit it is. Before we look at these files, let's see where they live.
The /lib/systemd/system/ directory is the default location for unit files that either come with the operating system or come with any packages that you might install. There might be times when you'll either need to modify some of these unit files or even create your own, but you won't do that in this directory. Instead, you'll do that in the /etc/systemd/system/ directory. Any unit files in this directory that have the same name as unit files in /lib/systemd/system/ take precedence.
Tip
You can read about unit files by typing the following: man systemd.unit.
At the bottom of this man page, you'll see where it refers you to other man pages for each specific type of unit file. You'll soon see that the trickiest part about this is having to search through the various man pages whenever you need to look up something about a particular unit-configuration parameter. To make things easier, you can look up a specific directive in the systemd.directives man page, which will direct you to the man page that contains information about that directive.
Now that you know where the unit files are, let's look at what they are.
Types of unit files
In the /lib/systemd/system directory, you'll see various types of unit files that each perform a different function. Here's a list of the more common types:
- service: These are the configuration files for services. They replace the old-fashioned init scripts that we had on the old System V (SysV) systems.
- socket: Sockets can either enable communication between different system services or they can automatically wake up a sleeping service when it receives a connection request.
- slice: Slice units are used when configuring cgroups. (We'll look at these in Part 2, Understanding cgroups.)
- mount and automount: These contain mount point information for filesystems that are controlled by systemd. Normally, they get created automatically, so you shouldn't have to do too much with them.
- target: Target units are used during system startup, for grouping units and for providing well-known synchronization points. (We'll cover these in Chapter 6, Understanding systemd Targets.)
- timer: Timer units are for scheduling jobs that run on a schedule. They replace the old cron system. (We'll work with these in Chapter 7, Understanding systemd Timers.)
- path: Path units are for services that can be started via path-based activation. (We'll cover service, path, and socket units in Chapter 3, Understanding Service, Path, and Socket Units.)
- swap: Swap units contain information about your swap partitions.
That's about it for the basic description of our unit files. We'll go into the nitty-gritty details about them in subsequent chapters.
Understanding the systemd executables
Normally, we would search for a program's executable files in either a bin/ or an sbin/ directory, and you will indeed find some of the systemd utility executable files there, but most of the systemd executables are found instead in the /lib/systemd/ directory. To save space, here's just a partial listing:
donnie@donnie-TB250-BTC:/lib/systemd$ ls -l
total 7448
-rw-r--r-- 1 root root 2367728 Feb 6 2020 libsystemd-shared-237.so
drwxr-xr-x 2 root root 4096 Apr 3 2020 network
-rw-r--r-- 1 root root 699 Feb 6 2020 resolv.conf
-rwxr-xr-x 1 root root 1246 Feb 6 2020 set-cpufreq
drwxr-xr-x 24 root root 36864 Apr 3 2020 system
-rwxr-xr-x 1 root root 1612152 Feb 6 2020 systemd
-rwxr-xr-x 1 root root 6128 Feb 6 2020 systemd-ac-power
-rwxr-xr-x 1 root root 18416 Feb 6 2020 systemd-backlight
-rwxr-xr-x 1 root root 10304 Feb 6 2020 systemd-binfmt
-rwxr-xr-x 1 root root 10224 Feb 6 2020 systemd-cgroups-agent
-rwxr-xr-x 1 root root 26632 Feb 6 2020 systemd-cryptsetup
. . .
. . .
You see that the executable for systemd itself is here, as well as the executables for the services that systemd runs as part of its own system. On some Linux distros, you'll see symbolic links in either the /bin or /usr/bin directories that point to some of the executable files here. For the most part, you won't directly interact with these files, so let's move on to something that you will interact with.
The systemctl utility is for controlling systemd, and you'll use it a lot. It's a multi-purpose tool that can do a lot of things for you. It lets you view the different units and the status of the units, and either enable them or disable them. For now, we'll look at some systemctl commands that allow you to view different types of information. Later, we'll talk about using systemctl to control and edit specific units. If you'd like to follow along, fire up a VM and start getting your hands dirty.
One thing to notice is that some systemctl commands require root privileges, and others don't. If you're just looking at system or unit information, you can do that with your normal user permissions. If you need to change a configuration, you'll need to assume the awesome powers of root. Okay—let's get started.
We'll first list the active units that systemd currently has in memory. We'll do that with the systemctl list-units command. It's a very long output, so I'll just show you the first few lines here:
[donnie@localhost ~]$ systemctl list-units
UNIT LOAD ACTIVE SUB DESCRIPTION
proc-sys-fs-binfmt_misc.automount loaded active waiting Arbitrary Executable File Formats File System Automount Point
sys-devices-pci0000:00-0000:00:17.0-ata3-host2-target2:0:0-2:0:0:0-block-sda-sda1.device loaded active plugged WDC_WDS250G2B0A-00SM50 1
sys-devices-pci0000:00-0000:00:17.0-ata3-host2-target2:0:0-2:0:0:0-block-sda-sda2.device loaded active plugged WDC_WDS250G2B0A-00SM50 2
sys-devices-pci0000:00-0000:00:17.0-ata3-host2-target2:0:0-2:0:0:0-block-sda.device loaded active plugged WDC_WDS250G2B0A-00SM50
sys-devices-pci0000:00-0000:00:1b.2-0000:02:00.1-sound-card1.device loaded active plugged GP104 High Definition Audio Controller
sys-devices-pci0000:00-0000:00:1b.3-0000:03:00.1-sound-card2.device loaded active plugged GP104 High Definition Audio Controller
. . .
. . .
This is the automount section, which shows the various devices that have been mounted. As you can see, this covers more than just storage devices.
Next, we have the mount, path, and scope units, as follows:
. . .
. . .
-.mount loaded active mounted /
boot.mount loaded active mounted /boot
dev-hugepages.mount loaded active mounted Huge Pages File System
dev-mqueue.mount loaded active mounted POSIX Message Queue File System
home.mount loaded active mounted /home
run-user-1000.mount loaded active mounted /run/user/1000
sys-fs-fuse-connections.mount loaded active mounted FUSE Control File System
sys-kernel-config.mount loaded active mounted Kernel Configuration File System
sys-kernel-debug.mount loaded active mounted Kernel Debug File System
tmp.mount loaded active mounted Temporary Directory (/tmp)
var-lib-nfs-rpc_pipefs.mount loaded active mounted RPC Pipe File System
cups.path loaded active running CUPS Scheduler
systemd-ask-password-plymouth.path loaded active waiting Forward Password Requests to Plymouth Directory Watch
systemd-ask-password-wall.path loaded active waiting Forward Password Requests to Wall Directory Watch
init.scope loaded active running System and Service Manager
session-1.scope loaded active abandoned Session 1 of user donnie
session-3.scope loaded active abandoned Session 3 of user donnie
session-4.scope loaded active running Session 4 of user donnie
. . .
. . .
Note here that there's a mount unit for each partition on your drive.
Keep scrolling down, and you'll see the same kind of display for the service, slice, socket, swap, target, and timer units. At the bottom, you'll see a brief explanation of the status codes and a short summary, as follows:
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
182 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
lines 136-190/190 (END)
Use the --all option to also see units that are not active, like so:
[donnie@localhost ~]$ systemctl list-units --all
UNIT LOAD ACTIVE SUB DESCRIPTION
● boot.automount not-found inactive dead boot.automount
proc-sys-fs-binfmt_misc.automount loaded active waiting Arbitrary Executable File Formats File System A
dev-block-8:2.device loaded active plugged WDC_WDS250G2B0A-00SM50 2
dev-disk-byx2did-atax2dWDC_WDS250G2B0Ax2d00SM50_181202802064.device loaded active plugged WDC_WDS250G2B0A-00SM50
dev-disk-byx2did-atax2dWDC_WDS250G2B0Ax2d00SM50_181202802064x2dpart1.device loaded active plugged WDC_WDS250G2B0A-00SM50 1
dev-disk-byx2did-atax2dWDC_WDS250G2B0Ax2d00SM50_181202802064x2dpart2.device loaded active plugged WDC_WDS250G2B0A-00SM50 2
. . .
. . .
That was luck. We found an inactive unit right at the very top.
You can also view specific types of units with the -t option. For example, to see just the service units, run the following command:
[donnie@localhost ~]$ systemctl list-units -t service
UNIT LOAD ACTIVE SUB DESCRIPTION
abrt-journal-core.service loaded active running Creates ABRT problems from coredumpctl messages
abrt-oops.service loaded active running ABRT kernel log watcher
abrt-xorg.service loaded active running ABRT Xorg log watcher
abrtd.service loaded active running ABRT Automated Bug Reporting Tool
alsa-state.service loaded active running Manage Sound Card State (restore and store)
atd.service loaded active running Deferred execution scheduler
auditd.service loaded active running Security Auditing Service
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
. . .
. . .
You can view the other units in the same way.
Now, let's say that we just want to see the services that are dead. We can do that with the --state option, like so:
[donnie@localhost ~]$ systemctl list-units -t service --state=dead
UNIT LOAD ACTIVE SUB DESCRIPTION
abrt-vmcore.service loaded inactive dead Harvest vmcores for ABRT
alsa-restore.service loaded inactive dead Save/Restore Sound Card State
auth-rpcgss-module.service loaded inactive dead Kernel Module supporting RPCSEC_GSS
● autofs.service not-found inactive dead autofs.service
blk-availability.service loaded inactive dead Availability of block devices
dbxtool.service loaded inactive dead Secure Boot DBX (blacklist) updater
dm-event.service loaded inactive dead Device-mapper event daemon
dmraid-activation.service loaded inactive dead Activation of DM RAID sets
dnf-makecache.service loaded inactive dead dnf makecache
dracut-cmdline.service loaded inactive dead dracut cmdline hook
. . .
. . .
By running systemctl --state=help, you'll see a list of all of the different states that you can view for the different unit types.
In addition to seeing the units that are currently in memory, you can also see the unit files that are installed on the system by running the following command:
[donnie@localhost ~]$ systemctl list-unit-files
UNIT FILE STATE
proc-sys-fs-binfmt_misc.automount static
-.mount generated
boot.mount generated
dev-hugepages.mount static
dev-mqueue.mount static
home.mount generated
proc-fs-nfsd.mount static
. . .
. . .
session-1.scope transient
session-3.scope transient
session-4.scope transient
abrt-journal-core.service enabled
abrt-oops.service enabled
abrt-pstoreoops.service disabled
abrt-vmcore.service enabled
abrt-xorg.service enabled
abrtd.service enabled
. . .
. . .
Here, you see some things that may seem rather strange. At the top, you see some mount files that are in a generated state. These files live in the /run/systemd/units/ directory and are automatically generated by systemd. To create these mount files, systemd reads the /etc/fstab file every time you either boot the machine or manually reload the fstab file.
Unit files in a static state are ones that you can neither enable nor disable. Rather, other units will call in these static units as dependencies.
Unit files in a transient state deal with things that are, well, transient. Here, we see three scope units that are managing three user sessions. When a user logs out of a session, one of these units will disappear.
And of course, units that are in an enabled state will automatically start upon booting the machine, and units that are in a disabled state won't.
To see if just one individual unit is either enabled or active, you can use the is-enabled and is-active options with systemctl. A while back, I told you that the networkd, resolved, and timesyncd services were all disabled on my Fedora machine. Here's how to prove that:
[donnie@localhost ~]$ systemctl is-enabled systemd-timesyncd
disabled
[donnie@localhost ~]$ systemctl is-enabled systemd-networkd
disabled
[donnie@localhost ~]$ systemctl is-enabled systemd-resolved
disabled
[donnie@localhost ~]$
And here's how to prove that they're not active:
[donnie@localhost ~]$ systemctl is-active systemd-timesyncd
inactive
[donnie@localhost ~]$ systemctl is-active systemd-resolved
inactive
[donnie@localhost ~]$ systemctl is-active systemd-timesyncd
inactive
[donnie@localhost ~]$
On the other hand, the NetworkManager service is enabled and active on my Fedora machine, as you can see here:
[donnie@localhost ~]$ systemctl is-enabled NetworkManager
enabled
[donnie@localhost ~]$ systemctl is-active NetworkManager
active
[donnie@localhost ~]$
Now, I'll leave it to you to verify all of this on the Ubuntu machine.
You can also see information about just one type of unit file. Here, we'll just look at information about the swap unit files:
[donnie@localhost units]$ systemctl list-unit-files -t swap
UNIT FILE STATE
dev-mapper-fedora_localhostx2dx2dlivex2dswap.swap generated
1 unit files listed.
[donnie@localhost units]$
Just as it did with the mount unit files, systemd generated this file by reading the /etc/fstab file.
Earlier, I showed you the /etc/systemd/system.conf file, which sets the global configuration for systemd. With the show option, you can see the actual running configuration by doing systemctl show. Here's the partial output:
[donnie@localhost ~]$ systemctl show
Version=v243.8-1.fc31
Features=+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified
Architecture=x86-64
Tainted=local-hwclock
FirmwareTimestampMonotonic=0
LoaderTimestampMonotonic=0
KernelTimestamp=Thu 2021-03-11 11:58:01 EST
KernelTimestampMonotonic=0
. . .
. . .
DefaultLimitRTPRIOSoft=0
DefaultLimitRTTIME=infinity
DefaultLimitRTTIMESoft=infinity
DefaultTasksMax=4608
TimerSlackNSec=50000
DefaultOOMPolicy=stop
Use the --property= option to view just one item, like so:
[donnie@localhost ~]$ systemctl show --property=DefaultLimitSIGPENDING
DefaultLimitSIGPENDING=15362
[donnie@localhost ~]$
There is a man page for systemctl, and you're welcome to peruse it. But if you just need a quick reference, run systemctl -h.
All right—I think that's enough for now. So, let's wrap this chapter up and put a bow on it, shall we?
Summary
Okay—we've hit the ground running and have covered quite a few concepts. We covered the various types of configuration files and unit files and saw where they live. We ended by using the systemctl command to view information about our running system.
In the next chapter, we'll expand on this by showing you the inner workings of the service, path, and socket unit files. I'll see you there.
Questions
- Which of the following commands tells you what the running systemd configuration is?
a. systemctl list
b. systemctl show
c. systemd show
d. systemd list
- Which of the following statements is true?
a. To configure your drive partitions, you need to hand-configure the mount units.
b. The mount units for your drive partitions get generated automatically when systemd reads the fstab file.
c. The mount units for your drive partitions are static units.
d. No mount units are needed for your drive partitions.
- Which of the following will tell you if the NetworkManager service is running?
a. systemctl active NetworkManager
b. systemd active NetworkManager
c. systemd enabled NetworkManager
d. systemctl is-enabled NetworkManager
e. systemctl is-active NetworkManager
Answers
- b
- b
- e
Further reading
systemd units and unit files:
https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files