systemd defines itself as a system and service manager. The project was initiated in 2010 by Lennart Poettering and Kay Sievers to create an integrated set of tools for managing a Linux system including an init daemon. It also includes device management (udev) and logging, among other things. Some would say that it is not just an init program, it is a way of life. It is state of the art, and still evolving rapidly. systemd is common on desktop and server Linux distributions, and is becoming popular on embedded Linux systems too, especially on more complex devices. So, how is it better than System V init for embedded systems?
- Configuration is simpler and more logical (once you understand it), rather than the sometimes convoluted shell scripts of System V
init,systemdhas unit configuration files to set parameters - There are explicit dependencies between services rather than a two digit code that merely sets the sequence in which the scripts are run
- It is easy to set the permissions and resource limits for each service, which is important for security
systemdcan monitor services and restart them if needed- There are watchdogs for each service and for s
ystemditself - Services are started in parallel, reducing boot time
A complete description of systemd is neither possible nor appropriate here. As with System V init, I will focus on embedded use-cases, with examples based on the configuration produced by Yocto Fido, which has systemd version 219. I will give a quick overview and then show you some specific examples.
The default init in Yocto Fido is System V. To select systemd, add these lines to your configuration, for example, in conf/local.conf:
DISTRO_FEATURES_append = " systemd" VIRTUAL-RUNTIME_init_manager = "systemd"
Note that the leading space is important! Then rebuild.
Buildroot has systemd as the third init option. It requires glibc as the C library, and kernel version 3.7 or later with a particular set of configuration options enabled. There is a complete list of dependencies in the README file in the top level of the systemd source code.
Before I describe how systemd init works, I need to introduce these three key concepts.
Firstly, a target is a group of services, similar to, but more general than, a SystemV runlevel. There is a default target which is the group of services that are started at boot time.
Secondly, a service is a daemon that can be started and stopped, very much like a SystemV service.
Finally, a unit is a configuration file that describes a target, a service, and several other things. Units are text files that contain properties and values.
You can change states and find out what is going on by using the systemctl command.
The basic item of configuration is the unit file. Unit files are found in three different places:
/etc/systemd/system: Local configuration/run/systemd/system: Runtime configuration/lib/systemd/system: Distribution-wide configuration
When looking for a unit, systemd searches the directories in that order, stopping as soon as it finds a match, allowing you to override the behavior of a distribution-wide unit by placing a unit of the same name in /etc/systemd/system. You can disable a unit completely by creating a local file that is empty or linked to /dev/null.
All unit files begin with a section marked [Unit] which contains basic information and dependencies, for example:
[Unit] Description=D-Bus System Message Bus Documentation=man:dbus-daemon(1) Requires=dbus.socket
Unit dependencies are expressed though Requires, Wants, and Conflicts:
Requires: A list of units that this unit depends on, which is started when this unit is startedWants: A weaker form ofRequires: the units listed are started but the current unit is not stopped if any of them failConflicts: A negative dependency: the units listed are stopped when this one is started and, conversely, if one of them is started, this one is stopped
Processing the dependencies produces a list of units that should be started (or stopped). The keywords Before and After determine the order in which they are started. The order of stopping is just the reverse of the start order:
Before: This unit should be started before the units listedAfter: This unit should be started after the units listed
In the following example, the After directive makes sure that the web server is started after the network:
[Unit] Description=Lighttpd Web Server After=network.target
In the absence of Before or After directives, the units will be started or stopped in parallel with no particular ordering.
A service is a daemon that can be started and stopped, equivalent to a System V service. A service is a type of unit file with a name ending in .service, for example, lighttpd.service.
A service unit has a [Service] section that describes how it should be run. Here is the relevant section from lighttpd.service:
[Service] ExecStart=/usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf -D ExecReload=/bin/kill -HUP $MAINPID
These are the commands to run when starting the service and restarting it. There are many more configuration points you can add in here, so refer to the man page for systemd.service.
Now we can see how systemd implements the bootstrap. systemd is run by the kernel as a result of /sbin/init being symbolically linked to /lib/systemd/systemd. It runs the default target, default.target, which is always a link to a desired target such as multi-user.target for a text login or graphical.target for a graphical environment. For example, if the default target is multi-user.target, you will find this symbolic link:
/etc/systemd/system/default.target -> /lib/systemd/system/multi-user.target
The default target may be overridden by passing system.unit=<new target> on the kernel command line. You can use systemctl to find out the default target, as shown here:
# systemctl get-default multi-user.target
Starting a target such as multi-user.target creates a tree of dependencies that bring the system into a working state. In a typical system, multi-user.target depends on basic.target, which depends on sysinit.target, which depends on the services that need to be started early. You can print a graph using systemctl list-dependencies.
You can also list all the services and their current state using systemctl list-units --type service, and the same for targets using systemctl list-units --type target.
Using the same simpleserver example as before, here is a service unit:
[Unit] Description=Simple server [Service] Type=forking ExecStart=/usr/bin/simpleserver [Install] WantedBy=multi-user.target
The [Unit] section only contains a description so that it shows up correctly when listed using systemctl and other commands. There are no dependencies; as I said, it is very simple.
The [Service] section points to the executable, and has a flag to indicate that it forks. If it were even simpler and ran in the foreground, systemd would do the daemonizing for us and Type=forking would not be needed.
The [Install] section makes it dependent on multi-user.target so that our server is started when the system goes into multi-user mode.
Once the unit is saved in /etc/systemd/system/simpleserver.service, you can start and stop it using the systemctl start simpleserver and systemctl stop simpleserver commands. You can use this command to find its current status:
# systemctl status simpleserver
simpleserver.service - Simple server
Loaded: loaded (/etc/systemd/system/simpleserver.service; disabled)
Active: active (running) since Thu 1970-01-01 02:20:50 UTC; 8s ago
Main PID: 180 (simpleserver)
CGroup: /system.slice/simpleserver.service
└─180 /usr/bin/simpleserver -n
Jan 01 02:20:50 qemuarm systemd[1]: Started Simple server.At this point, it will only start and stop on command, as shown. To make it persistent, you need to add a permanent dependency to a target. That is the purpose of the [Install] section in the unit, it says that when this service is enabled it will become dependent on multi-user.target, and so will be started at boot time. You enable it using systemctl enable, like this:
# systemctl enable simpleserver Created symlink from /etc/systemd/system/multi-user.target.wants/simpleserver.service to /etc/systemd/system/simpleserver.service.
Now you can see how dependencies are added at runtime without having to edit any unit files. A target can have a directory named <target_name>.target.wants which can contain links to services. This is exactly the same as adding the dependent unit to the [Wants] list in the target. In this case, you will find that this link has been created:
/etc/systemd/system/multi-user.target.wants/simpleserver.service /etc/systemd/system/simpleserver.service
If this is were an important service you might want to restart if it failed. You can accomplish that by adding this flag to the [Service] section:
Restart=on-abort
Other options for Restart are on-success, on-failure, on-abnormal, on-watchdog, on-abort, or always.
Watchdogs are a common requirement in embedded devices: you need to take action if a critical service stops working, usually by resetting the system. On most embedded SoCs, there is a hardware watchdog which can be accessed via the /dev/watchdog device node. The watchdog is initialized with a timeout at boot and then must be reset within that period, otherwise the watchdog will be triggered and the system will reboot. The interface with the watchdog driver is described in the kernel source in Documentation/watchdog, and the code for the drivers is in drivers/watchdog.
A problem arises if there are two or more critical services that need to be protected by a watchdog. systemd has a useful feature that distributes the watchdog between multiple services.
systemd can be configured to expect a regular keepalive call from a service and take action if it is not received, in other words, a per-service software watchdog. For this to work, you have to add code to the daemon to send the keepalive messages. It needs to check for a non-zero value in the WATCHDOG_USEC environment variable and then call sd_notify(false, "WATCHDOG=1") within that time (a period of half of the watchdog timeout is recommended). There are examples in the systemd source code.
To enable the watchdog in the service unit, add something like this to the [Service] section:
WatchdogSec=30s Restart=on-watchdog StartLimitInterval=5min StartLimitBurst=4 StartLimitAction=reboot-force
In this example, the service expects a keepalive every 30 seconds. If it fails to be delivered, the service will be restarted, but if it is restarted more than four times in five minutes, systemd will force an immediate reboot. Once again, there is a full description of these settings in the systemd manual.
A watchdog like this takes care of individual services, but what if systemd itself fails, or the kernel crashes, or the hardware locks up. In those cases, we need to tell systemd to use the watchdog driver: just add RuntimeWatchdogSec=NN to /etc/systemd/system.conf. systemd will reset the watchdog within that period, and so the system will reset if systemd fails for some reason.
systemd has a lot of features that are useful in embedded Linux, including many that I have not mentioned in this brief description such as resource control using slices (see the man page for systemd.slice(5) and systemd.resource-control(5)), device management (udev(7)) and system logging facilities (journald(5)).
You have to balance that with its size: even with a minimal build of just the core components, systemd, udevd, and journald, it is approaching 10 MiB of storage, including the shared libraries.
You also have to keep in mind that systemd development follows the kernel closely, so it will not work on a kernel more than a year or two older than the release of systemd.