Chapter 3: Understanding Service, Path, and Socket Units

In this chapter, we'll examine the inner workings of the service, path, and socket unit files. We'll examine the parts that are in each and look at some of the parameters that you can set. Along the way, I'll give you some pointers about how to find information about what the various parameters are doing for you.

In this chapter, we will cover the following topics:

• Understanding service units
• Understanding socket units
• Understanding path units

At some point in your Linux administrator career, you could be tasked with modifying existing units or creating new ones. The knowledge in this chapter can help you with that. So, if you're ready, let's go.

Technical requirements

As always, I'll be doing the demos on an Ubuntu Server 20.04 virtual machine and an Alma Linux 8 virtual machine. Feel free to fire up your own virtual machines to follow along.

Check out the following link to see the Code in Action video: https://bit.ly/2ZQBHh6

Understanding service units

Service units are the equivalent of init scripts on old SysV systems. We'll use them to configure our various services, which we used to call daemons in the old days. A service can be pretty much anything that you want to start automatically and run in the background. Examples of services include Secure Shell, your web server of choice, a mail server, and various services that are required for proper system operation. While some service files can be short and sweet, others can be fairly lengthy, with more options enabled. To read about all of these options, just type the following:

man systemd.directives

The descriptions for all of the parameters that you can set are spread over several different man pages. This systemd.directives man page is an index that will direct you to the proper man page for each parameter.

Rather than trying to explain every parameter that service files can use, let's look through a few example files and explain what they're doing.

Understanding the Apache service file

We'll start with the service file for the Apache web server. On my Ubuntu Server 20.04 virtual machine, it is the /lib/systemd/system/apache2.service file. The first thing to note is that service unit files are divided into three sections. The top section is the [Unit] section, which contains parameters that can be placed in any type of unit file. It looks like this:

[Unit]

Description=The Apache HTTP Server

After=network.target remote-fs.target nss-lookup.target

Documentation=https://httpd.apache.org/docs/2.4/

Here, we see these three parameters:

• Description=: Okay, this one should be fairly self-explanatory. All it does is tell the human user what the service is. The systemctl status command pulls its description information from this line.
• After=: We don't want Apache to start until certain other things have happened. We haven't talked about target files yet, but that's okay. For now, just know that we want to prevent Apache from starting until after the network, any possible attached remote filesystems, and the Name Switch Service are available.
• Documentation=: Here's another one that's self-explanatory. It just shows where to find the Apache documentation.

To read about the options that you can place in the [Unit] section of any unit file, just type the following:

man systemd.unit

Next, we have the [Service] section, where things get a bit more interesting. It contains parameters that can only be placed in a service unit file, and looks like this:

[Service]

Type=forking

Environment=APACHE_STARTED_BY_SYSTEMD=true

ExecStart=/usr/sbin/apachectl start

ExecStop=/usr/sbin/apachectl stop

ExecReload=/usr/sbin/apachectl graceful

PrivateTmp=true

Restart=on-abort

In this particular file, we see these parameters:

• Type=: There are several different service types that you'll see described in the systemd.service man page. In this case, we have the forking type, which means that the first Apache process that starts will spawn a child process. When Apache startup is complete and the proper communication channels have been set up, the original process—the parent process—will exit and the child process will carry on as the main service process. When the parent process exits, the systemd service manager will finally recognize the service as having fully started. According to the man page, this is the traditional behavior for Unix services, and systemd just carries on the tradition.
• Environment=: This sets an environmental variable that affects the behavior of the service. In this case, it tells Apache that it was started by systemd.
• ExecStart=, ExecStop=, and ExecReload=: These three lines just point the way to the Apache executable file, and specify the command arguments for starting, stopping, and reloading the service.
• PrivateTmp=: Many services write temporary files for various reasons, and you're probably used to seeing them in the /tmp/ directory that everyone can access. Here though, we see a cool systemd security feature. When set to true, this parameter forces the Apache service to write its temporary files to a private /tmp/ directory that nobody else can access. So, if you're concerned that Apache might write sensitive information to its temporary files, you'll want to use this feature. (You can read more about this feature, as well as other security features, on the systemd.exec man page.) Also, note that if you leave this parameter out altogether, it will default to false, which means that you won't have this protection.
• Restart=: Sometimes, you might want a service to automatically restart if it stops. In this case, we're using the on-abort parameter, which just means that if the Apache service were to crash with an unclean signal, systemd would automatically restart it.

Okay, that's it for the [Service] section. Let's move on to the [Install] section, which looks like this:

[Install]

WantedBy=multi-user.target

The nomenclature for this seems a bit weird because it doesn't seem like we're installing anything here. What this actually does is control what happens when you enable or disable a unit. In this case, we're saying that we want the Apache service to be enabled for the multi-user.target unit, which will cause the service to automatically start when the machine boots into the multi-user target. (We'll cover targets and the boot-up process later. For now, just understand that the multi-user target is when the machine is fully booted and ready for use. For you SysV veterans, the target in this case is akin to a SysV runlevel.)

Understanding the Secure Shell service file

For something a bit different, let's look at the service file for the Secure Shell service, which on this Ubuntu machine is the /lib/systemd/system/ssh.service file. Here's the [Unit] section:

[Unit]

Description=OpenBSD Secure Shell server

Documentation=man:sshd(8) man:sshd_config(5)

After=network.target auditd.service

ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

In the [Unit] section, we see the ConditionPathExists= parameter, which we didn't see before. It checks for either the existence or non-existence of a file. In this case, we see an exclamation point (!) in front of the path to the file, which means that we're checking for the non-existence of the named file. If systemd finds it there, it won't start the Secure Shell service. If we were to remove the exclamation point, then systemd would only start the service if the file were there. So, if we wanted to prevent the Secure Shell service from starting, all we'd have to do is create a dummy file in the /etc/ssh/ directory, like so:

sudo touch /etc/ssh/sshd_not_to_be_run

I'm not sure how useful this feature really is, because it's just as easy to simply disable the service if you don't want it to run. But, if you think that you might ever need this, it's there for you.

Next up is the [Service] section:

[Service]

EnvironmentFile=-/etc/default/ssh

ExecStartPre=/usr/sbin/sshd -t

ExecStart=/usr/sbin/sshd -D $SSHD_OPTS

ExecReload=/usr/sbin/sshd -t

ExecReload=/bin/kill -HUP $MAINPID

KillMode=process

Restart=on-failure

RestartPreventExitStatus=255

Type=notify

RuntimeDirectory=sshd

RuntimeDirectoryMode=0755

In the [Service] section, we see a few new parameters:

• EnvironmentFile=: This parameter causes systemd to read a list of environmental variables from the specified file. The minus sign (-) in front of the path to the file tells systemd that if the file doesn't exist, don't worry about it and start the service anyway.
• ExecStartPre=: This tells systemd to run a specified command before it starts the service with the ExecStart= parameter. In this case, we want to run the sshd -t command, which tests the Secure Shell configuration to ensure that it's valid.
• KillMode=: I've already told you that one of the beauties of systemd is its ability to stop all processes of a service if you have to send a kill signal to it. That's the default behavior if you don't include this parameter in your service file. Sometimes though, you might not want that. By setting this parameter to process, a kill signal will only kill the main process for the service. All other associated processes will remain running. (You can read more about this parameter on the systemd.kill man page.)
• Restart=: This time, instead of automatically restarting a stopped service on-abort, it will now restart it on-failure. So, in addition to restarting the service because of an unclean signal, systemd will also restart this service because of an unclean exit code, a timeout, or a watchdog event. (A watchdog, in case you're wondering, is a kernel feature that can restart a service upon some sort of unrecoverable error.)
• RestartPreventExitStatus=: This prevents the service from automatically restarting if a certain exit code is received. In this case, we don't want the service to restart if the exit code is 255. (For more information about exit codes, see the $EXIT_CODE, $EXIT_STATUS_ section of the systemd.exec man page.)
• Type=: For this service, the type is notify, instead of forking as we saw in the previous example. This means that the service will send a notification message when the service has finished starting. After it sends the notification message, systemd will continue loading the follow-up units.
• RuntimeDirectory= and RuntimeDirectoryMode=: These two directives create a runtime directory under the /run/ directory, and then set the permissions value for that directory. In this case, we're setting the 0755 permission on the directory, which means that it will have read, write, and execute permissions for the directory's owner. Everyone else will only have read and execute permissions.

Finally, here's the [Install] section:

[Install]

WantedBy=multi-user.target

Alias=sshd.service

In the [Install] section, we see the Alias= parameter, which can be quite handy. That's because certain services can have different names on different Linux distros. For example, the Secure Shell service is sshd on Red Hat-type systems and just ssh on Debian/Ubuntu systems. By including this Alias=sshd.service line, we can control the service by specifying either name.

Understanding the timesyncd service file

For the last example, I want to show you the service file for the timesyncd service. This is the /lib/systemd/system/systemd-timesyncd.service file. First, the [Unit] section:

[Unit]

Description=Network Time Synchronization

Documentation=man:systemd-timesyncd.service(8)

ConditionCapability=CAP_SYS_TIME

ConditionVirtualization=!container

DefaultDependencies=no

After=systemd-sysusers.service

Before=time-set.target sysinit.target shutdown.target

Conflicts=shutdown.target

Wants=time-set.target time-sync.target

For this file, I mainly just want to focus on the security-related parameters. In the [Unit] section, there's the ConditionCapability= parameter, which I'll explain in a moment. The Wants= line, which isn't security-related, defines the dependency units for this service. If these dependency units aren't running when this service gets started, then systemd will attempt to start them. If they fail to start, this service will still go ahead and start anyway.

Next, we'll look at the [Service] section, where we'll see more security-related parameters. (For space reasons, I can only place part of the file here, so feel free to view it on your own virtual machine.):

[Service]

AmbientCapabilities=CAP_SYS_TIME

CapabilityBoundingSet=CAP_SYS_TIME

ExecStart=!!/lib/systemd/systemd-timesyncd

LockPersonality=yes

MemoryDenyWriteExecute=yes

. . .

. . .

ProtectSystem=strict

Restart=always

RestartSec=0

RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

RestrictNamespaces=yes

RestrictRealtime=yes

RestrictSUIDSGID=yes

RuntimeDirectory=systemd/timesync

StateDirectory=systemd/timesync

SystemCallArchitectures=native

SystemCallErrorNumber=EPERM

SystemCallFilter=@system-service @clock

Type=notify

User=systemd-timesync

WatchdogSec=3min

The AmbientCapabilities= and the CapabilityBoundingSet= parameters are all set to CAP_SYS_TIME, as is the ConditionCapability= parameter in the [Unit] section. Toward the end of the [Service] section, we see the User=systemd-timesync line, which tells systemd to run this service under a non-privileged account. But, setting the system time requires root privileges, which the systemd-timesync user doesn't have. We can fix that by assigning a root-level kernel capability to this user. In this case, we're allowing this user to set the system time, but nothing else. Some systems though, might not be able to implement the AmbientCapabilities= directive. So, the double-exclamation points (!!) in the ExecStart= line tell systemd to run the indicated service with minimum privileges. Be aware that this double-exclamation point option only takes effect if the system can't deal with the AmbientCapabilities= directive.

Note

You can read more about kernel capabilities by typing man capabilities. An important thing to understand about kernel capabilities is that they can vary across different CPU architectures. So, the set of capabilities that can be used with an ARM CPU won't be the same as the set of capabilities on an x86_64 CPU.

Read down through the rest of the [Service] section, and you'll see a lot of parameters that are obviously for enhancing security. I'm not going to go over all of them, because for most of them, you can tell what they're doing just by looking at their names. For the few that aren't so obvious, I would encourage you to consult the man pages. These security settings are a powerful feature, and you can see here that they're pretty much doing the same job as a mandatory access control system.

And finally, we have the [Install] section:

[Install]

WantedBy=sysinit.target

Alias=dbus-org.freedesktop.timesync1.service

The main thing to see here is that this service is wanted by the sysinit.target, which means that it will come up during the system initialization process.

We've only scratched the surface for what we can do with service files. But there are so many different parameters that scratching the surface is all we can reasonably expect to do. Your best bet is to skim over the man pages to get a good feel for things and to consult the man pages whenever you have questions.

Next, we'll cover socket units. (Fortunately, that section won't need to be quite as long.)

Understanding socket units

The socket unit files are also in the /lib/systemd/system/ directory, and their filenames end with .socket. Here's a partial list of them on one of my Ubuntu Server machines:

donnie@ubuntu20-10:/lib/systemd/system$ ls -l *.socket

-rw-r--r-- 1 root root  246 Jun  1  2020 apport-forward.socket

-rw-r--r-- 1 root root  102 Sep 10  2020 dbus.socket

-rw-r--r-- 1 root root  248 May 30  2020 dm-event.socket

-rw-r--r-- 1 root root  197 Sep 16 16:52 docker.socket

-rw-r--r-- 1 root root  175 Feb 26  2020 iscsid.socket

-rw-r--r-- 1 root root  239 May 30  2020 lvm2-lvmpolld.socket

-rw-r--r-- 1 root root  186 Sep 11  2020 multipathd.socket

-rw-r--r-- 1 root root  281 Feb  2 08:21 snapd.socket

-rw-r--r-- 1 root root  216 Jun  7  2020 ssh.socket

. . .

. . .

-rw-r--r-- 1 root root  610 Sep 20 10:16 systemd-udevd-kernel.socket

-rw-r--r-- 1 root root  126 Aug 30  2020 uuidd.socket

donnie@ubuntu20-10:/lib/systemd/system$

The socket units can do a couple of things for us. First, they can take the place of the legacy inetd and xinetd superserver daemons that were on the old SysV systems. This means that instead of having a server daemon run full-time, even when it isn't needed, we can leave it shut down most of the time, and only start it when the system detects an incoming network request for it. For a simple example, let's look at the ssh.socket file on an Ubuntu machine:

[Unit]

Description=OpenBSD Secure Shell server socket

Before=ssh.service

Conflicts=ssh.service

ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Socket]

ListenStream=22

Accept=yes

[Install]

WantedBy=sockets.target

Even though this socket file gets installed by default, it's not enabled by default. On a default configuration of Ubuntu, the Secure Shell service runs all the time. In the [Unit] section, we see these two interesting directives:

• Before=ssh.service: This tells systemd to start the socket before starting the Secure Shell service.
• Conflicts=ssh.service: This tells systemd to not allow the Secure Shell service to run normally if this socket is enabled. If you were to enable this socket, the normal SSH service would get shut down.

In the [Socket] section, we see that the socket listens on port 22/tcp, which is the default port for Secure Shell. The Accept=yes line is a bit deceiving because it doesn't mean exactly what you would think. It really means that the service will spawn a new instance for every incoming connection. According to the systemd.socket man page, this setting should only be used for services that were designed to work under the old inetd and xinetd schemes. For better performance, new services should be designed to not behave like this.

To demonstrate how this works, I first want to show you that the ssh service on my Ubuntu VM is running normally:

donnie@ubuntu20-10:~$ sudo systemctl is-active ssh

active

donnie@ubuntu20-10:~$

So, it's active, which means that it's running as a normal daemon. Now, let's enable ssh.socket, and then look at the difference:

donnie@ubuntu20-10:~$ sudo systemctl enable --now ssh.socket

Created symlink /etc/systemd/system/sockets.target.wants/ssh.socket → /lib/systemd/system/ssh.socket.

donnie@ubuntu20-10:~$ sudo systemctl is-active ssh

inactive

donnie@ubuntu20-10:~$

So, as soon as I enable this socket, the Conflicts= line automatically shuts down the ssh service. But I can still connect to this machine because the socket will automatically start the SSH service just long enough to service the connection request. When the service is no longer needed, it will automatically go back to sleep.

Secondly, note that this socket doesn't mention which service to start, or where its executable file is. That's because the socket, when activated, will just pull that information from the ssh.service file. You don't have to tell it to do that, because the default behavior for any socket file is to get its information from a service file that has the same prefix in the filename.

Finally, socket units can enable communication between operating system processes. For example, a socket can take messages from various system processes and pass them to the logging system, as we see here in this systemd-journald.socket file:

[Unit]

Description=Journal Socket

Documentation=man:systemd-journald.service(8) man:journald.conf(5)

DefaultDependencies=no

Before=sockets.target

. . .

. . .

IgnoreOnIsolate=yes

[Socket]

ListenStream=/run/systemd/journal/stdout

ListenDatagram=/run/systemd/journal/socket

SocketMode=0666

PassCredentials=yes

PassSecurity=yes

ReceiveBuffer=8M

Service=systemd-journald.service

We see here that instead of listening to a network port, this socket listens for TCP output from /run/systemd/journal/stdout, and for UDP output from /run/systemd/journal/socket. (The ListenStream= directive is for TCP sources, and the ListenDatagram= directive is for UDP sources. The systemd.socket man page doesn't make that clear, so you have to do some DuckDuckGo searching to find this out.)

There's no Accept=yes directive here, because, unlike the Secure Shell service that we saw earlier, the journald service doesn't need to spawn a new instance for every incoming connection. By leaving this setting out, it defaults to a value of no.

The PassCredentials=yes line and the PassSecurity=yes line cause the sending process to pass security credentials and security context information to the receiving socket. These parameters also default to no if you leave them out. To enhance performance, the ReceiveBuffer= line sets aside 8 MB of buffer memory.

Finally, the Service= line specifies the service. According to the systemd.socket man page, this can only be used if Accept=no is set. The man page also says that this usually isn't needed, because by default the socket will still reference the service file that has the same name as the socket. But if you do use this, it might pull in some extra dependencies that it might not otherwise pull in.

Understanding path units

You can use a path unit to have systemd monitor a certain file or directory to see when it changes. When systemd detects that the file or directory has changed, it will activate the specified service. We'll use the Common Unix Printing System (CUPS) as an example.

In the /lib/systemd/system/cups.path file, we see this:

[Unit]

Description=CUPS Scheduler

PartOf=cups.service

[Path]

PathExists=/var/cache/cups/org.cups.cupsd

[Install]

WantedBy=multi-user.target

The PathExists= line tells systemd to monitor a specific file for changes, which in this case is the /var/cache/cups/org.cups.cupsd file. If systemd detects any changes to this file, it will activate the printing service.

Summary

All right, we've made it through another chapter, which is a good thing. In this chapter, we examined the structure of the service, socket, and path unit files. We saw the three sections of each type of unit and looked at some of the parameters that we can define for each of those sections. Of course, it's pretty much impossible to explain every single available parameter, so I've just shown you a few examples. And I'll show you more examples in the next few chapters.

An important skill for any IT administrator is knowing how to look up things that you don't know. That can be a bit of a challenge with systemd, because things are spread out over quite a few man pages. I've given you some tips on how to use the man pages to find what you need, which will hopefully be of some help.

The next skill you'll want to acquire is that of controlling service units, which is the topic of the next chapter. I'll see you there.

Questions

1. Which kind of unit monitors files and directories for changes?

   a. system

   b. file

   c. path

   d. timer

   e. service

2. A socket unit can:

   a. automatically notify the user if a network request comes in

   b. automatically set up communication between Linux and Windows machines

   c. listen for network connections, and act as a firewall

   d. automatically start a network service when it detects a connection request for that service

3. What is the purpose of the [Install] section?

   a. It defines what other packages are to be installed when you install a service.

   b. It defines what happens when you enable or disable a unit.

   c. It defines parameters that are specific to an install unit.

   d. It defines parameters that are specific to a service unit.

Answers

1. c
2. d
3. b

Further reading

Systemd socket units:

https://www.linux.com/training-tutorials/end-road-systemds-socket-units/

The difference between ListenStream= and ListenDatagram=:

https://unix.stackexchange.com/questions/517240/systemd-socket-listendatagram-vs-listenstream

Monitoring paths and directories:

https://www.linux.com/topic/desktop/systemd-services-monitoring-files-and-directories/