Chapter 10. Managing Processes

The following topics are covered in this chapter:

• Introduction to Process Management

• Managing Shell Jobs

• Using Common Command-Line Tools for Process Management

• Using top to Manage Processes

• Using tuned to Optimize Performance

The following RHCSA exam objectives are covered in this chapter:

• Identify CPU/memory-intensive processes and kill processes

• Adjust process scheduling

Process management is an important task for a Linux administrator. In this chapter, you learn what you need to know to manage processes from a perspective of daily operation of a server. You’ll learn how to work with shell jobs and generic processes. You’ll also be introduced to system performance optimization using tuned.

“Do I Know This Already?” Quiz

The “Do I Know This Already?” quiz allows you to assess whether you should read this entire chapter thoroughly or jump to the “Exam Preparation Tasks” section. If you are in doubt about your answers to these questions or your own assessment of your knowledge of the topics, read the entire chapter. Table 10-1 lists the major headings in this chapter and their corresponding “Do I Know This Already?” quiz questions. You can find the answers in Appendix A, “Answers to the ‘Do I Know This Already?’ Quizzes and ‘Review Questions.’”

1. Which of the following is not generally considered a type of process? (Choose two.)

a. A shell job

b. A cron job

c. A daemon

d. A thread

2. Which of the following can be used to move a job to the background?

a. Press &

b. Press Ctrl-Z and then type bg

c. Press Ctrl-D and then type bg

d. Press Ctrl-Z, followed by &

3. Which key combination enables you to cancel a current interactive shell job?

a. Ctrl-C

b. Ctrl-D

c. Ctrl-Z

d. Ctrl-Break

4. Which of the following statements are true about threads? (Choose two.)

a. Threads cannot be managed individually by an administrator.

b. Multithreaded processes can make the working of processes more efficient.

c. Threads can be used only on supported platforms.

d. Using multiple processes is more efficient, in general, than using multiple threads.

5. Which of the following commands is most appropriate if you’re looking for detailed information about the command and how it was started?

a. ps ef

b. ps aux

c. ps

d. ps fax

6. Of the following nice values, which will increase the priority of the selected process?

a. 100

b. 20

c. -19

d. -100

7. Which of the following shows correct syntax to change the priority for the current process with PID 1234?

a. nice -n 5 1234

b. renice 5 1234

c. renice 5 -p 1234

d. nice 5 -p 1234

8. Which of the following commands cannot be used to send signals to processes?

a. kill

b. mkill

c. pkill

d. killall

9. Which of the following commands would you use from top to change the priority of a process?

a. r

b. n

c. c

d. k

10. Which of the following commands will set the current performance profile to powersave?

a. tuneadm profile set powersave

b. tuned-adm profile powersave

c. tuneadm profile --set powersave

d. tuned-adm profile --set powersave

Foundation Topics

Introduction to Process Management

For everything that happens on a Linux server, a process is started. For that reason, process management is among the key skills that an administrator has to master. To do this efficiently, it is important to know which type of process you are dealing with. A major distinction can be made between three process types:

• Shell jobs are commands started from the command line. They are associated with the shell that was current when the process was started. Shell jobs are also referred to as interactive processes.

• Daemons are processes that provide services. They normally are started when a computer is booted and often (but certainly not in all cases) run with root privileges.

• Kernel threads are a part of the Linux kernel. You cannot manage them using common tools, but for monitoring of performance on a system, it’s important to keep an eye on them.

When a process is started, it can use multiple threads. A thread is a task started by a process and that a dedicated CPU can service. The Linux shell does not offer tools to manage individual threads. Thread management should be taken care of from within the command.

To manage a process efficiently, it is paramount that you know what type of process you are dealing with. Shell jobs require a different approach than the processes that are automatically started when a computer boots.

Managing Shell Jobs

When a user types a command, a shell job is started. If no particular measures have been taken, the job is started as a foreground process, occupying the terminal it was started from until it has finished its work. As a Linux administrator, you need to know how to start shell jobs in the foreground or background and what you can do to manage shell jobs.

Running Jobs in the Foreground and Background

By default, any executed command is started as a foreground job. That means that you cannot do anything on the terminal where the command was started until it is done. For many commands, that does not really matter because the command often takes a little while to complete, after which it returns access to the shell from which it was started. Sometimes it might prove useful to start commands in the background. This makes sense for processes that do not require user interaction and take significant time to finish. A process that does require user interaction will not be able to get that when running in the background, and for that reason will typically stall when moved to the background. You can take two different approaches to run a process in the background.

If you know that a job will take a long time to complete, you can start it with an & behind it. This immediately starts the job in the background to make room for other tasks to be started from the command line. To move the last job that was started in the background back as a foreground job, use the fg command. This command immediately, and with no further questions, brings the last job back to the foreground. If multiple jobs are currently running in the background, you can move a job back to the foreground by adding its job ID, as shown by the jobs command.

A job might sometimes have been started that takes (much) longer than predicted. If that happens, you can use Ctrl-Z to temporarily stop the job. This does not remove the job from memory; it just pauses the job so that it can be managed. Once paused, you can continue it as a background job by using the bg command. An alternative key sequence that you can use to manage shell jobs is Ctrl-C. This stops the current job and removes it from memory.

A related key combination is Ctrl-D, which sends the End Of File (EOF) character to the current job. The result is that the job stops waiting for further input so that it can complete what it was currently doing. The result of pressing Ctrl-D is sometimes similar to the result of pressing Ctrl-C, but there is a difference. When Ctrl-C is used, the job is just canceled, and nothing is closed properly. When Ctrl-D is used, the job stops waiting for further input and next terminates, which often is just what is needed to complete in a proper way.

Managing Shell Jobs

When moving jobs between the foreground and background, it may be useful to have an overview of all current jobs. To get such an overview, use the jobs command. As you can see in Table 10-2, this command gives an overview of all jobs currently running as a background job, including the job number assigned to the job when starting it in the background. These job numbers can be used as an argument to the fg and bg commands to perform job management tasks. In Exercise 10-1, you learn how to perform common job management tasks from the shell.

Managing Parent-Child Relations

When a process is started from a shell, it becomes a child process of that shell. In process management, the parent-child relationship between processes is very important. The parent is needed to manage the child. For that reason, all processes started from a shell are terminated when that shell is stopped. This also offers an easy way to terminate processes no longer needed.

Processes started in the background will not be killed when the parent shell from which they were started is killed. To terminate these processes, you need to use the kill command, as described later in this chapter.

Using Common Command-Line Tools for Process Management

On a Linux server, many processes are usually running. On an average server or desktop computer, there are often more than 100 active processes. With so many processes being active, things may go wrong. If that happens, it is good to know how noninteractive processes can be stopped or how the priority of these processes can be adjusted to make more system resources available for other processes.

Understanding Processes and Threads

Tasks on Linux are typically started as processes. One process can start several worker threads. Working with threads makes sense, because if the process is very busy, the threads can be handled by different CPUs or CPU cores available in the machine. As a Linux administrator, you cannot manage individual threads; you can manage processes, though. It is the programmer of the multithreaded application that has to define how threads relate to one another.

Before talking about different ways to manage processes, it is good to know that there are two different types of background processes: kernel threads and daemon processes. Kernel threads are a part of the Linux kernel, and each of them is started with its own process identification number (PID). When managing processes, it is easy to recognize the kernel processes because they have a name that is between square brackets. Example 10-1 shows a list of a few processes as output of the command ps aux | head (discussed later in this chapter), in which you can see a couple of kernel threads.

As an administrator, it is important to know that kernel threads cannot be managed. You cannot adjust their priority; neither is it possible to kill them, except by taking the entire machine down.

[root@server3 ~]# ps aux | head
USER  PID %CPU %MEM    VSZ  RSS TTY  STAT  START  TIME   COMMAND
root    1  0.0  0.4 252864 7792 ?    Ss    08:25  0:02   /usr/lib/
  systemd/systemd --switched-root --system --deserialize 17
root    2  0.0  0.0      0    0 ?    S    08:25  0:00   [kthreadd]
root    3  0.0  0.0      0    0 ?    I<   08:25  0:00   [rcu_gp]
root    4  0.0  0.0      0    0 ?    I<   08:25  0:00   [rcu_par_gp]
root    6  0.0  0.0      0    0 ?    I<   08:25  0:00   [kworker/0:
                                                          0H-kblockd]
root    8  0.0  0.0      0    0 ?    I<   08:25  0:00   [mm_percpu_wq]
root    9  0.0  0.0      0    0 ?    S    08:25  0:00   [ksoftirqd/0]
root   10  0.0  0.0      0    0 ?    I    08:25  0:00   [rcu_sched]
root   11  0.0  0.0      0    0 ?    S    08:25  0:00   [migration/0]

Using ps to Get Process Information

The most common command to get an overview of currently running processes is ps. If used without any arguments, the ps command shows only those processes that have been started by the current user. You can use many different options to display different process properties. If you are looking for a short summary of the active processes, use ps aux (as you saw in Example 10-1). If you are looking for not only the name of the process but also the exact command that was used to start the process, use ps -ef (see Example 10-2). Alternative ways to use ps exist as well, such as the command ps fax, which shows hierarchical relationships between parent and child processes (see Example 10-3).

[root@server3 ~]# ps -ef
UID     PID   PPID  C STIME TTY      TIME     CMD
root      1      0  0 08:25 ?      00:00:02  /usr/lib/systemd/systemd
                                               --switched-root
                                               --system --deserialize
                                               17
...
root  34948      2  0 12:16 ?      00:00:00  [kworker/0:1-events]
root  34971   1030  0 12:17 ?      00:00:00  sshd: root [priv]
root  34975  34971  0 12:17 ?      00:00:00  sshd: root@pts/2
root  34976  34975  0 12:17 pts/2  00:00:00  -bash
root  35034      1  0 12:17 pts/2  00:00:00  sleep 3600
root  35062      2  0 12:20 ?      00:00:00  [kworker/u256:2]
root  35064      2  0 12:20 ?      00:00:00  [kworker/0:3-cgroup_
                                               destroy]
root  35067      2  0 12:20 ?      00:00:00  [kworker/1:2-events_
                                               freezable_power_]
root  35087    939  0 12:21 ?      00:00:00  sleep 60
root  35088  33127  0 12:22 pts/1  00:00:00   ps -ef

[root@server3 ~]# ps fax
   PID TTY       STAT   TIME  COMMAND
     2 ?         S      0:00  [kthreadd]
     3 ?         I<     0:00  \_ [rcu_gp]
     4 ?         I<     0:00  \_ [rcu_par_gp]
...
  2460 ?         Ssl    0:00  \_ /usr/bin/pulseaudio --daemonize=no
  2465 ?         Ssl    0:00  \_ /usr/bin/dbus-daemon --session
                                --address=systemd: --nofork
                                --nopidfile --systemd-activation --
  2561 ?         Ssl    0:00  \_ /usr/libexec/at-spi-bus-launcher
  2566 ?         Sl     0:00  |   \_ /usr/bin/dbus-daemon --config-
                                file=/usr/share/defaults/at-spi2/
                                accessibility.conf --nofork
  2569 ?         Sl     0:00  \_ /usr/libexec/at-spi2-registryd
                                --use-gnome-session
  2589 ?         Ssl    0:00  \_ /usr/libexec/xdg-permission-store
  2594 ?         Sl     0:00  \_ /usr/libexec/ibus-portal
  2704 ?         Sl     0:00  \_ /usr/libexec/dconf-service
  2587 ?         Sl     0:00  /usr/libexec/ibus-x11 --kill-daemon
  2758 ?         Sl     0:00  /usr/bin/gnome-keyring-daemon --daemonize
                                --login
  2908 tty3      Sl     0:00  /usr/libexec/ibus-x11 --kill-daemon
  2936 ?         Ssl    0:00  /usr/libexec/geoclue
  3102 tty3      Sl+    0:00  /usr/libexec/gsd-printer
  3173 tty3      Sl+    0:12  /usr/bin/vmtoolsd -n vmusr
  3378 ?         Ssl    0:00  /usr/libexec/fwupd/fwupd
  3440 ?         Ss     0:00  gpg-agent --homedir /var/lib/fwupd/gnupg
                                --use-standard-socket --daemon
  3455 ?         S      0:00  /usr/libexec/platform-python /usr/
                                libexec/rhsmd
 33093 ?         Ss     0:00  /usr/lib/systemd/systemd --user
 33105 ?         S      0:00  \_ (sd-pam)
 33117 ?         S<sl   0:00  \_ /usr/bin/pulseaudio --daemonize=no
 33123 ?         Ssl    0:00  \_ /usr/bin/dbus-daemon --session
                                --address=systemd: --nofork
                                --nopidfile --systemd-activation --
 35034 pts/2     S      0:00  sleep 3600

An important piece of information to get out of the ps command is the PID. Many tasks require the PID to operate, and that is why a command like ps aux | grep dd, which will show process details about dd, including its PID, is quite common. An alternative way to get the same result is to use the pgrep command. Use pgrep dd to get a list of all PIDs that have a name containing the string "dd".

Adjusting Process Priority with nice

When Linux processes are started, they are started with a specific priority. By default, all regular processes are equal and are started with the same priority, which is the priority number 20, as shown by utilities like top. In some cases, it is useful to change the default priority that was assigned to the process when it was started. You can do that using the nice and renice commands. Use nice if you want to start a process with an adjusted priority. Use renice to change the priority for a currently active process. Alternatively, you can use the r command from the top utility to change the priority of a currently running process.

Changing process priority may make sense in two different scenarios. Suppose, for example, that you are about to start a backup job that does not necessarily have to finish fast. Typically, backup jobs are rather resource intensive, so you might want to start the backup job in a way that does not annoy other users too much, by lowering its priority.

Another example is where you are about to start a very important calculation job. To ensure that it is handled as fast as possible, you might want to give it an increased priority, taking away CPU time from other processes.

On earlier Linux versions, it could be dangerous to increase the priority of one job too much, because of the risk that other processes (including vital kernel processes) might be blocked out completely. On current Linux kernels, that risk is minimized for these reasons:

• Modern Linux kernels differentiate between essential kernel threads that are started as real-time processes and normal user processes. Increasing the priority of a user process will never be able to block out kernel threads or other processes that were started as real-time processes.

• Modern computers often have multiple CPU cores. A single-threaded process that is running with the highest priority will never be able to get beyond the boundaries of the CPU it is running on.

When using nice or renice to adjust process priority, you can select from values ranging from –20 to 19. The default niceness of a process is set to 0 (which results in the priority value of 20). By applying a negative niceness, you increase the priority. Use a positive niceness to decrease the priority. It is a good idea not to use the ultimate values immediately. Instead, use increments of 5 and see how it affects the application.

Let’s take a look at examples of how to use nice and renice. The command nice -n 5 dd if=/dev/zero of=/dev/null & starts an infinite I/O-intensive job, but with an adjusted niceness so that some room remains for other processes as well. To adjust the niceness of a currently running process, you need the PID of that process. The following two commands show how ps aux is used to find the PID of the dd job from the previous example. Next, you see how the renice command is used to change the niceness of that command:

1. Use ps aux | grep dd to find the PID of the dd command that you just started. The PID is in the second column of the command output.

2. Use renice -n 10 -p 1234 (assuming that 1234 is the PID you just found).

Note that regular users can only decrease the priority of a running process. You must be root to give processes increased priority.

Sending Signals to Processes with kill, killall, and pkill

Before starting to think about using the kill command or sending other signals to processes, it is good to know that Linux processes have a hierarchical relationship. Every process has a parent process, and as long as it lives, the parent process is responsible for the child processes it has created. In older versions of Linux, killing a parent process would also kill all of its child processes. In RHEL 8, if you kill a parent process, all of its child processes become children of the systemd process.

The Linux kernel allows many signals to be sent to processes. Use man 7 signals for a complete overview of all the available signals. Three of these signals work for all processes:

• The signal SIGTERM (15) is used to ask a process to stop.

• The signal SIGKILL (9) is used to force a process to stop.

• The SIGHUP (1) signal is used to hang up a process. The effect is that the process will reread its configuration files, which makes this a useful signal to use after making modifications to a process configuration file.

To send a signal to a process, you use the kill command. The most common use is the need to stop a process, which you can do by using the kill command followed by the PID of the process. This sends the SIGTERM signal to the process, which normally causes the process to cease its activity and close all open files.

Sometimes the kill command does not work because the process you want to kill can ignore it. In that case, you can use kill -9 to send the SIGKILL signal to the process. Because the SIGKILL signal cannot be ignored, it forces the process to stop, but you also risk losing data while using this command. In general, it is a bad idea to use kill -9:

• You risk losing data.

• Your system may become unstable if other processes depend on the process you have just killed.

There are some commands that are related to kill: killall and pkill. The pkill command is a bit easier to use because it takes the name rather than the PID of the process as an argument. You can use the killall command if multiple processes using the same name need to be killed simultaneously.

Using killall was particularly common when Linux environments were multiprocessing instead of multithreading. In a multiprocessing environment where a server starts several commands, all with the same name, it is not easy to stop these commands one by one based on their individual PID. Using killall enables you to terminate all these processes simultaneously.

In a multithreaded environment, the urge to use killall is smaller. Because there is often just one process that is generating several threads, all these threads are terminated anyway by stopping the process that started them. You still can use killall, though, to terminate lots of processes with the same name that have been started on your server. In Exercise 10-2, you practice using ps, nice, kill, and related utilities to manage processes.

Using top to Manage Processes

A convenient tool to manage processes is top. For common process management tasks, top is great because it gives an overview of the most active processes currently running (hence the name top). This enables you to easily find processes that might need attention. From top, you can also perform common process management tasks, such as adjusting the current process priority and killing processes. Figure 10-1 shows the interface that appears when you start top.

Among the information that you can conveniently obtain from the top utility is the process state. Table 10-3 provides an overview of the different process states that you may observe.

Now that you know how to use the kill and nice commands from the command line, using the same functionality from top is even easier. From top, type k; top then prompts for the PID of the process you want to send a signal to. By default, the most active process is selected. After you enter the PID, top asks which signal you want to send. By default, signal 15 for SIGTERM is used. However, if you want to insist a bit more, you can type 9 for SIGKILL. Now press Enter to terminate the process.

To renice a running process from top, type r. You are first prompted for the PID of the process you want to renice. After entering the PID, you are prompted for the nice value you want to use. Enter a positive value to decrease process priority or a negative value to increase process priority.

Another important parameter you can get from top is the load average. The load average is expressed as the number of processes that are in a runnable state (R) or in a blocking state (D). Processes are in a runnable state if they currently are running, or waiting to be serviced. Processes are in a blocking state if they are waiting for I/O. The load average is shown for the last 1, 5, and 15 minutes, and you can see the current values in the upper-right corner of the top screen. Alternatively, you can use the uptime command to show current load average statistics (see Example 10-4).

[root@server3 ~]# uptime
 12:43:03 up  4:17,  3 users,  load average: 4.90, 0.98, 0.19

As a rule of thumb, the load average should not be higher than the number of CPU cores in your system. You can find out the number of CPU cores in your system by using the lscpu command. If the load average over a longer period is higher than the number of CPUs in your system, you may have a performance problem. In Exercise 10-3 you investigate the load average statistics and learn how to manage load average.

Using tuned to Optimize Performance

To offer the best possible performance right from the start, RHEL 8 comes with tuned. It offers a daemon that monitors system activity and provides some profiles. In the profiles, an administrator can automatically tune a system for best possible latency, throughput, or power consumption.

Based on the properties of an installed system, a tuned profile is selected automatically at installation, and after installation it’s possible to manually change the current profile. Administrators can also change settings in a tuned profile. Table 10-4 gives an overview of the default profiles.

It is relatively easy to create custom profiles. Also, when installing specific packages, profiles may be added. So you may find that some additional performance profiles exist on your server.

To manage the performance profile, the tuned-adm command is provided. It talks to the tuned daemon, so before you can use it, run systemctl enable --now tuned to start the tuned profile. Next, use tuned-adm to find out which profile currently is selected. For an overview of profiles available on your server, type tuned-adm list. To select another profile, type tuned-adm profile profile-name. The tuned service can also recommend a tuned profile for your system: use tuned-adm recommend. In Exercise 10-4 you can practice working with tuned.

Summary

Managing processes is a common task for a Linux system administrator. In this chapter, you learned how to look up specific processes and how to change their priority using nice and kill. You have also learned how to use tuned to select the performance profile that best matches your server’s workload.

Exam Preparation Tasks

As mentioned in the section “How to Use This Book” in the Introduction, you have several choices for exam preparation: the end-of-chapter labs; the memory tables in Appendix B; Chapter 26, “Final Preparation”; and the practice exams.

Review All Key Topics

Review the most important topics in the chapter, noted with the Key Topic icon in the outer margin of the page. Table 10-5 lists a reference of these key topics and the page number on which each is found.

Complete Tables and Lists from Memory

Print a copy of Appendix B, “Memory Tables” (found on the companion website), or at least the section for this chapter, and complete the tables and lists from memory. Appendix C, “Memory Tables Answer Key,” includes completed tables and lists to check your work.

Define Key Terms

Define the following key terms from this chapter and check your answers in the glossary:

job

process

background

foreground

nice

kill

signal

PID

thread

tuned

profile

zombie

Review Questions

The questions that follow are meant to help you test your knowledge of concepts and terminology and the breadth of your knowledge. You can find the answers to these questions in Appendix A.

1. Which command gives an overview of all current shell jobs?

2. How do you stop the current shell job to continue running it in the background?

3. Which keystroke combination can you use to cancel the current shell job?

4. A user is asking you to cancel one of the jobs he has started. You cannot access the shell that user currently is working from. What can you do to cancel his job anyway?

5. Which command would you use to show parent-child relationships between processes?

6. Which command enables you to change the priority of PID 1234 to a higher priority?

7. On your system, 20 dd processes are currently running. What is the easiest way to stop all of them?

8. Which command enables you to stop the command with the name mycommand?

9. Which command do you use from top to kill a process?

10. What is required to select a performance profile that best matches your system needs?

End-of-Chapter Lab

In the end-of-chapter lab, you apply some of the most important process management tasks. Use the tools that you find the most convenient to perform these labs.

Lab 10.1

1. Launch the command dd if=/dev/zero of=/dev/null three times as a background job.

2. Increase the priority of one of these commands using the nice value -5. Change the priority of the same process again, but this time use the value -15. Observe the difference.

3. Kill all the dd processes you just started.

4. Ensure that tuned is installed and active, and set the throughput-performance profile.