The following topics are covered in this chapter:

• Understanding the Role of the Linux Kernel

• Working with Kernel Modules

• Upgrading the Linux Kernel

The Linux kernel is the heart of the Linux operating system. It takes care of many things, including hardware management. In this chapter, you learn all you need to know about the Linux kernel from an RHCSA perspective. In fact, you even learn a bit more. This chapter includes information about topics that are not on the current list of RHCSA objectives. I think it is good to know about these topics anyway. Any serious Linux administrator should be able to deal with issues related to the topics discussed in this chapter.

“Do I Know This Already?” Quiz

The “Do I Know This Already?” quiz allows you to assess whether you should read this entire chapter thoroughly or jump to the “Exam Preparation Tasks” section. If you are in doubt about your answers to these questions or your own assessment of your knowledge of the topics, read the entire chapter. Table 16-1 lists the major headings in this chapter and their corresponding “Do I Know This Already?” quiz questions. You can find the answers in Appendix A, “Answers to the ‘Do I Know This Already?’ Quizzes and ‘Review Questions.’”

1. What causes a tainted kernel?

a. A kernel driver that is not available as an open source driver

b. A driver that was developed for a different operating system but has been ported to Linux

c. A driver that has failed

d. An unsupported driver

2. What is the name of the command that shows kernel events since booting?

a. logger

b. dmesg

c. klogd

d. journald

3. Which command enables you to find the actual version of the kernel that is used?

a. uname -r

b. uname -v

c. procinfo -k

d. procinfo -l

4. Which command shows the current version of RHEL you are using?

a. uname -r

b. cat /proc/rhel-version

c. cat /etc/redhat-release

d. uname -k

5. What is the name of the process that helps the kernel to initialize hardware devices properly?

a. systemd-udevd

b. hwinit

c. udev

d. udevd

6. Where does your system find the default rules that are used for initializing new hardware devices?

a. /etc/udev/rules.d

b. /usr/lib/udev/rules.d

c. /usr/lib/udev.d/rules

d. /etc/udev.d/rules

7. Which command should you use to unload a kernel module, including all of its dependencies?

a. rmmod

b. insmod -r

c. modprobe -r

d. modprobe

8. Which command enables you to see whether the appropriate kernel modules have been loaded for hardware in your server?

a. lsmod

b. modprobe -l

c. lspci -k

d. lspci

9. Where do you specify a kernel module parameter to make it persistent?

a. /etc/modules.conf

b. /etc/modprobe.conf

c. /etc/modprobe.d/somefilename

d. /usr/lib/modprobe.d/somefilename

10. Which statements about updating the kernel are not true?

a. The yum update kernel command will install a new kernel and not update it.

b. The yum install kernel command will install a new kernel and keep the old kernel.

c. The kernel package should be set as a yum-protected package to ensure that after an update the old kernel is still available.

d. After you have installed a new kernel version, you must run the grub2-mkconfig command to modify the GRUB 2 boot menu so that it shows the old kernel and the newly installed kernel.

Foundation Topics

Understanding the Role of the Linux Kernel

The Linux kernel is the heart of the operating system. It is the layer between the user who works with Linux from a shell environment and the hardware that is available in the computer on which the user is working. The kernel manages the I/O instructions it receives from the software and translates them into the processing instructions that are executed by the central processing unit and other hardware in the computer. The kernel also takes care of handling essential operating system tasks. One example of such a task is the scheduler that makes sure any processes that are started on the operating system are handled by the CPU.

Understanding the Use of Kernel Threads and Drivers

The operating system tasks that are performed by the kernel are implemented by different kernel threads. Kernel threads are easily recognized with a command like ps aux. The kernel thread names are listed between square brackets (see Example 16-1).

[root@server1 ~]# ps aux | head -n 20
USER   PID %CPU %MEM    VSZ   RSS TTY   STAT   START  TIME COMMAND
root   1   1.8  0.6   52980  6812 ?      Ss    11:44  0:02 /usr/lib/
systemd/systemd --switched-root --system --deserialize 23
root   2   0.0   0.0    0    0 ?     S   11:44  0:00  [kthreadd]
root   3   0.0   0.0    0    0 ?     S   11:44  0:00  [ksoftirqd/0]
root   4   0.0   0.0    0    0 ?     S   11:44  0:00  [kworker/0:0]
root   5   0.0   0.0    0    0 ?     S<  11:44  0:00  [kworker/0:0H]
root   6   0.0   0.0    0    0 ?     S   11:44  0:00  [kworker/u128:0]
root   7   0.1   0.0    0    0 ?     S   11:44  0:00  [migration/0]
root   8   0.0   0.0    0    0 ?     S   11:44  0:00  [rcu_bh]
root   9   0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/0]
root   10  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/1]
root   11  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/2]
root   12  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/3]
root   13  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/4]
root   14  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/5]
root   15  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/6]
root   16  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/7]
root   17  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/8]
root   18  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/9]
root   19  0.0   0.0    0    0 ?     S   11:44  0:00  [rcuob/10]

Another important task of the Linux kernel is hardware initialization. To make sure that this hardware can be used, the Linux kernel uses drivers. Every piece of hardware contains specific features, and to use these features, a driver must be loaded. The Linux kernel is modular, and drivers are loaded as kernel modules, which you read about more later in this chapter.

In some cases, the availability of drivers is an issue because hardware manufacturers are not always willing to provide open source drivers that can be integrated well with the Linux kernel. That can result in a driver that does not provide all the functionality that is provided by the hardware.

If a manufacturer is not willing to provide open source drivers, an alternative is to work with closed source drivers. Although these make it possible to use the hardware in Linux, the solution is not ideal. Because a driver performs privileged instructions within the kernel space, a badly functioning driver may crash the entire kernel. If this happens with an open source driver, the Linux kernel community can help debug the problem and make sure that the issue is fixed. If it happens with a closed source driver, the Linux kernel community cannot do anything. But, a proprietary driver may provide access to features that are not provided by its open source equivalent.

To make it easy to see whether a kernel is using closed source drivers, the concept of the tainted kernel is used. A tainted kernel is a kernel that contains closed source drivers. The concept of tainted kernels helps in troubleshooting drivers. If your RHEL 8 kernel appears to be tainted, Red Hat support can identify it as a tainted kernel and recognize which driver is tainting it. To fix the problem, Red Hat might ask you to take out the driver that is making it a tainted kernel.

Analyzing What the Kernel Is Doing

To help analyze what the kernel is doing, some tools are provided by the Linux operating systems:

• The dmesg utility

• The /proc file system

• The uname utility

The first utility to consider if you require detailed information about the kernel activity is dmesg. This utility shows the contents of the kernel ring buffer, an area of memory where the Linux kernel keeps its recent log messages. An alternative method to get access to the same information in the kernel ring buffer is the journalctl --dmesg command, which is equivalent to journalctl -k. In Example 16-2, you can see a part of the result of the dmesg command.

[   8.153928] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   8.154289] sd 0:0:1:0: Attached scsi generic sg1 type 0
[   8.154330] sd 0:0:2:0: Attached scsi generic sg2 type 0
[   8.154360] sd 0:0:3:0: Attached scsi generic sg3 type 0
[   8.154421] sr 4:0:0:0: Attached scsi generic sg4 type 5
[   8.729016] ip_tables: (C) 2000-2006 Netfilter Core Team
[   8.850599] nf_conntrack version 0.5.0 (7897 buckets, 31588 max)
[   8.939613] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   9.160092] Ebtables v2.0 registered
[   9.203710] Bridge firewalling registered
[   9.586603] IPv6: ADDRCONF(NETDEV_UP): eno16777736: link is not
  ready
[   9.587520] e1000: eno16777736 NIC Link is Up 1000 Mbps Full
  Duplex, Flow Control: None
[   9.589066] IPv6: ADDRCONF(NETDEV_CHANGE): eno16777736: link becomes
  ready
[  10.689365] Rounding down aligned max_sectors from 4294967295 to
  4294967288
[  5158.470480] Adjusting tsc more than 11% (6940512 vs 6913395)
[21766.132181] e1000: eno16777736 NIC Link is Down
[21770.391597] e1000: eno16777736 NIC Link is Up 1000 Mbps Full
  Duplex, Flow Control: None
[21780.434547] e1000: eno16777736 NIC Link is Down

In the dmesg output, all kernel-related messages are shown. Each message starts with a time indicator that shows at which specific second the event was logged. This time indicator is relative to the start of the kernel, which allows you to see exactly how many seconds have passed between the start of the kernel and a particular event. (Notice that the journalctl -k / --dmesg commands show clock time, instead of time that is relative to the start of the kernel.) This time indicator gives a clear indication of what has been happening and at which time it has happened.

Another valuable source of information is the /proc file system. The /proc file system is an interface to the Linux kernel, and it contains files with detailed status information about what is happening on your server. Many of the performance-related tools mine the /proc file system for more information.

As an administrator, you will find that some of the files in /proc are very readable and contain status information about the CPU, memory, mounts, and more. Take a look, for instance, at /proc/meminfo, which gives detailed information about each memory segment and what exactly is happening in these memory segments.

A last useful source of information is the uname command. This command gives different kinds of information about your operating system. Type, for instance, uname -a for an overview of all relevant parameters of uname -r to see which kernel version currently is used. This information also shows when using the hostnamectl status command, which shows useful additional information as well (see Example 16-3).

[root@server1 ~]# hostnamectl status
   Static hostname: server1.example.com
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 5aa095b495ed458d934c54a88078c165
           Boot ID: b7273a66ba254358b566639e329c350d
    Virtualization: vmware
  Operating System: Red Hat Enterprise Linux 8.0 (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8.0:GA
            Kernel: Linux 4.18.0-80.el8.x86_64
      Architecture: x86-64

Working with Kernel Modules

In the old days of Linux, kernels had to be compiled to include all drivers that were required to support computer hardware. Other specific functionality needed to be compiled into the kernel as well. Since the release of Linux kernel 2.0 in the mid-1990s, kernels are no longer compiled but modular. A modular kernel consists of a relatively small core kernel and provides driver support through modules that are loaded when required. Modular kernels are very efficient, as they include only those modules that really are needed.

Understanding Hardware Initialization

The loading of drivers is an automated process that roughly goes like this:

1. During boot, the kernel probes available hardware.

2. Upon detection of a hardware component, the systemd-udevd process takes care of loading the appropriate driver and making the hardware device available.

3. To decide how the devices are initialized, systemd-udevd reads rules files in /usr/lib/udev/rules.d. These are system-provided rules files that should not be modified.

4. After processing the system-provided udev rules files, systemd-udevd goes to the /etc/udev/rules.d directory to read any custom rules if these are available.

5. As a result, required kernel modules are loaded automatically, and status about the kernel modules and associated hardware is written to the sysfs file system, which is mounted on the /sys directory. The Linux kernel uses this pseudo file system to track hardware-related settings.

The systemd-udevd process is not a one-time-only process; it continuously monitors plugging and unplugging of new hardware devices. To get an impression of how this works, as root you can type the command udevadm monitor. This lists all events that are processed while activating new hardware devices. For instance, if you plug in a USB device while this command is active, you can see exactly what’s happening. Press Ctrl-C to close the udevadm monitor output.

Example 16-4 shows output of the udevadm monitor command. In this command, you can see how features that are offered by the hardware are discovered automatically by the kernel and systemd-udevd working together. Each phase of the hardware probing is concluded by the creation of a file in the /sys file system. Once the hardware has been fully initialized, you can also see that some kernel modules are loaded.

[root@server2 ~]# udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[132406.831270] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1 (usb)
KERNEL[132406.974110] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1/1-1:1.0 (usb)
UDEV [132406.988182] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1 (usb)
KERNEL[132406.999249] add /module/usb_storage (module)
UDEV [132407.001203] add /module/usb_storage (module)
KERNEL[132407.002559] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1/1-1:1.0/host33 (scsi)
UDEV [132407.002575] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1/1-1:1.0 (usb)
KERNEL[132407.002583] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1/1-1:1.0/host33/scsi_host/host33 (scsi_host)
KERNEL[132407.002590] add /bus/usb/drivers/usb-storage (drivers)
UDEV [132407.004479] add /bus/usb/drivers/usb-storage (drivers)
UDEV [132407.005798] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1/1-1:1.0/host33 (scsi)
UDEV [132407.007385] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1/1-1:1.0/host33/scsi_host/host33 (scsi_host)
KERNEL[132408.008331] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1/1-1:1.0/host33/target33:0:0 (scsi)
KERNEL[132408.008355] add /devices/pci0000:00/0000:00:11.0/0000:02:
  04.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0 (scsi)
...
KERNEL[132409.381930] add          /module/fat (module)
KERNEL[132409.381951] add          /kernel/slab/fat_cache (slab)
KERNEL[132409.381958] add          /kernel/slab/fat_inode_cache (slab)
KERNEL[132409.381964] add          /module/vfat (module)
UDEV [132409.385090]  add          /module/fat (module)
UDEV [132409.385107]  add          /kernel/slab/fat_cache (slab)
UDEV [132409.385113]  add          /kernel/slab/fat_inode_cache (slab)
UDEV [132409.386110]  add          /module/vfat (module)

Managing Kernel Modules

Linux kernel modules normally are loaded automatically for the devices that need them, but you will on rare occasions have to load the appropriate kernel modules manually. A few commands are used for manual management of kernel modules. Table 16-2 provides an overview.

An alternative method of loading kernel modules is through the /etc/modules-load.d directory. In this directory, you can create files to load modules automatically that are not loaded by the systemd-udevd method already. For default modules that should always be loaded, this directory has a counterpart in /usr/lib/modules-load.d.

The first command to use when working with kernel modules is lsmod. This command lists all kernel modules that currently are used, including the modules by which this specific module is used. Example 16-5 shows the output of the first ten lines of the lsmod command.

[root@server1 ~]# lsmod | head
Module                  Size   Used by
nls_utf8               16384   1
isofs                  45056   1
fuse                  126976   3
rfcomm                 90112   6
xt_CHECKSUM            16384   1
ipt_MASQUERADE         16384   1
xt_conntrack           16384   1
ipt_REJECT             16384   1
nft_counter            16384   16

If you want to have more information about a specific kernel module, you can use the modinfo command. This gives complete information about the specific kernel modules, including two interesting sections: the alias and the parms. A module alias is another name that can also be used to address the module. The parms lines refer to parameters that can be set while loading the module. (In the section “Managing Kernel Module Parameters” later in this chapter, you learn how to work with kernel module parameters.) Example 16-6 shows partial output of the modinfo e1000 command.

[root@server1 ~]# modinfo e1000
filename:       /lib/modules/4.18.0-80.el8.x86_64/kernel/drivers/net/
                  ethernet/intel/e1000/e1000.ko.xz
version:        7.3.21-k8-NAPI
license:        GPL
description:    Intel(R) PRO/1000 Network Driver
author:         Intel Corporation, <[email protected]>
rhelversion:    8.0
srcversion:     A2C44EC5D0B865EE9C972D5
alias:          pci:v00008086d00002E6Esv*sd*bc*sc*i*
...
depends:
intree:         Y
name:           e1000
vermagic:       4.18.0-80.el8.x86_64 SMP mod_unload modversions
sig_id:         PKCS#7
signer:         Red Hat Enterprise Linux kernel signing key
sig_key:        20:16:97:CB:B1:7E:D5:A0:A7:3C:0B:40:E2:54:80:2D:83:F1:
                  72:5A
sig_hashalgo:   sha256
signature:      97:3A:58:F6:4E:B7:F1:CE:44:50:65:5E:84:41:D8:A7:A3:
                  A2:2D...
parm:           TxDescriptors:Number of transmit descriptors (array
                  of int)
parm:           RxDescriptors:Number of receive descriptors (array
                  of int)
parm:           Speed:Speed setting (array of int)
parm:           Duplex:Duplex setting (array of int)
parm:           AutoNeg:Advertised auto-negotiation setting (array
                  of int)
parm:           FlowControl:Flow Control setting (array of int)
parm:           XsumRX:Disable or enable Receive Checksum offload
                  (array of int)
parm:           TxIntDelay:Transmit Interrupt Delay (array of int)
parm:           TxAbsIntDelay:Transmit Absolute Interrupt Delay
                  (array of int)
parm:           RxIntDelay:Receive Interrupt Delay (array of int)
parm:           RxAbsIntDelay:Receive Absolute Interrupt Delay
                  (array of int)
parm:           InterruptThrottleRate:Interrupt Throttling Rate
                  (array of int)
parm:           SmartPowerDownEnable:Enable PHY smart power down
                  (array of int)
parm:           copybreak:Maximum size of packet that is copied to
                  a new buffer on receive (uint)
parm:           debug:Debug level (0=none,...,16=all) (int)

To manually load and unload modules, you can use the modprobe and modprobe -r commands. On earlier Linux versions, you may have used the insmod and rmmod commands. These should no longer be used because they do not load kernel module dependencies. In Exercise 16-1, you learn how to manage kernel modules using these commands.

Checking Driver Availability for Hardware Devices

On modern Linux servers, many hardware devices are supported. On occasion, you might find that some devices are not supported properly because their modules are not currently loaded. The best way to find out whether this is the case for your hardware is by using the lspci command. If used without arguments, it shows all hardware devices that have been detected on the PCI bus. A very useful argument is -k, which lists all kernel modules that are used for the PCI devices that were detected. Example 16-7 shows sample output of the lspci -k command.

[root@server1 ~]# lspci -k | head
00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX
  Host bridge (rev 01)
        Subsystem: VMware Virtual Machine Chipset
        Kernel driver in use: agpgart-intel
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/
  DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
        Subsystem: VMware Virtual Machine Chipset
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE
  (rev 01)
        Subsystem: VMware Virtual Machine Chipset
        Kernel driver in use: ata_piix
        Kernel modules: ata_piix, ata_generic
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
        Subsystem: VMware Virtual Machine Chipset
        Kernel modules: i2c_piix4
00:07.7 System peripheral: VMware Virtual Machine Communication
  Interface (rev 10)
        Subsystem: VMware Virtual Machine Communication Interface
        Kernel driver in use: vmw_vmci
        Kernel modules: vmw_vmci
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
        Subsystem: VMware SVGA II Adapter
        Kernel driver in use: vmwgfx
        Kernel modules: vmwgfx
00:10.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030
  PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01)
        Subsystem: VMware LSI Logic Parallel SCSI Controller
        Kernel driver in use: mptspi
        Kernel modules: mptspi
00:11.0 PCI bridge: VMware PCI bridge (rev 02)
00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
        Kernel driver in use: pcieport

If you discover that PCI devices were found for which no kernel modules could be loaded, you are probably dealing with a device that is not supported. You can try to find a closed source kernel module, but you should realize that that might endanger the stability of your kernel. A much better approach is to check with your hardware vendor that Linux is fully supported before you purchase specific hardware.

Managing Kernel Module Parameters

Occasionally, you might want to load kernel modules with specific parameters. To do so, you first need to find out which parameter you want to use. If you have found the parameter you want to use, you can load it manually, specifying the name of the parameter followed by the value that you want to assign. To make this an automated procedure, you can create a file in the /etc/modprobe.d directory, where the module is loaded including the parameter you want to be loaded. In Exercise 16-2 you see how to do this using the cdrom kernel module.

Upgrading the Linux Kernel

From time to time, you need to upgrade the Linux kernel. When you upgrade the Linux kernel, a new version of the kernel is installed and used as the default kernel. The old version of the kernel file will still be available, though. This ensures that your computer can still boot if in the new kernel nonsupported functionality is included. To install a new version of the kernel, you can use the command yum upgrade kernel. The yum install kernel command also works. Both commands install the new kernel beside the old kernel.

The kernel files for the last four kernels that you have installed on your server will be kept in the /boot directory. The GRUB 2 boot loader automatically picks up all kernels that it finds in this directory. This allows you to select an older kernel version while booting, which is useful if the newly installed kernel doesn’t boot correctly.

Summary

In this chapter, you learned how to work with the Linux kernel. You learned that the Linux kernel is modular and how working with kernel modules is important. You also learned how to manage kernel modules and how kernel modules are managed automatically while working with new hardware.

Exam Preparation Tasks

As mentioned in the section “How to Use This Book” in the Introduction, you have several choices for exam preparation: the end-of-chapter labs; the memory tables in Appendix B; Chapter 26, “Final Preparation”; and the practice exams.

Review All Key Topics

Review the most important topics in the chapter, noted with the Key Topic icon in the outer margin of the page. Table 16-3 lists a reference of these key topics and the page number on which each is found.

Complete Tables and Lists from Memory

Print a copy of Appendix B, “Memory Tables” (found on the companion website), or at least the section for this chapter, and complete the tables and lists from memory. Appendix C, “Memory Tables Answer Key,” includes completed tables and lists to check your work.

Define Key Terms

Define the following key terms from this chapter and check your answers in the glossary:

kernel

module

dmesg

udev

sysfs

proc

tainted kernel

Review Questions

The questions that follow are meant to help you test your knowledge of concepts and terminology and the breadth of your knowledge. You can find the answers to these questions in Appendix A.

1. Which command shows the current version of the kernel that is used on your computer?

2. Where do you find current version information about your RHEL 8 installation?

3. Which command shows a list of kernel modules that currently are loaded?

4. Which command enables you to discover kernel module parameters?

5. How do you unload a kernel module?

6. What can you do if you get an error message while trying to unload a kernel module?

7. How do you find which kernel module parameters are supported?

8. Where do you specify kernel module parameters that should be used persistently?

9. Assuming that the cdrom module has a parameter debug, which must be set to 1 to enable debug mode, which line would you include in the file that will automatically load that module?

10. How do you install a new version of the kernel?

End-of-Chapter Lab

In the end-of-chapter lab, you install a new version of the kernel and work with kernel modules.

Lab 16.1

1. Find out whether a new version of the kernel is available. If so, install it and reboot your computer so that it is used.

2. Use the appropriate command to show recent events that have been logged by the kernel.

3. Locate the kernel module that is used by your network card. Find out whether it has options. Try loading one of these kernel module options manually; if that succeeds, take the required measures to load this option persistently.