Chapter 13. Configuring Logging

Image

The following topics are covered in this chapter:

The following RHCSA exam objectives are covered in this chapter:

  • Locate and interpret system log files and journals

  • Preserve system journals

Analyzing log files is an important system administrator task. If anything goes wrong on a Linux system, the answer is often in the log files. On RHEL 8, two different log systems are used side by side, and it is important to know which information can be found where. This chapter teaches you all about it. You learn how to read log files, how to configure rsyslogd and journald, and how to set up your system for log rotation so that you can prevent your disks from being completely filled up by services that are logging too enthusiastically.

“Do I Know This Already?” Quiz

The “Do I Know This Already?” quiz allows you to assess whether you should read this entire chapter thoroughly or jump to the “Exam Preparation Tasks” section. If you are in doubt about your answers to these questions or your own assessment of your knowledge of the topics, read the entire chapter. Table 13-1 lists the major headings in this chapter and their corresponding “Do I Know This Already?” quiz questions. You can find the answers in Appendix A, “Answers to the ‘Do I Know This Already?’ Quizzes and ‘Review Questions.’

Table 13-1 “Do I Know This Already?” Section-to-Question Mapping

Foundation Topics Section

Questions

Understanding System Logging

1–3

Configuring rsyslogd

4–7

Rotating Log Files

8

Working with journald

9–10

1. Which of the following statements about journald is not true?

a. journald logs kernel messages.

b. journald writes to the journal, which by default does not persist between boots.

c. journald is a replacement of rsyslogd.

d. To read files from the journal, the journalctl command is used.

2. Which log would you read to find messages related to authentication errors?

a. /var/log/messages

b. /var/log/lastlog

c. /var/log/audit/audit.log

d. /var/log/secure

3. Which log would you read to find information that relates to SELinux events?

a. /var/log/messages

b. /var/log/lastlog

c. /var/log/audit/audit.log

d. /var/log/secure

4. What is the name of the rsyslogd configuration file?

a. /etc/rsyslog.conf

b. /etc/sysconfig/rsyslogd.conf

c. /etc/sysconfig/rsyslog.conf

d. /etc/rsyslog.d/rsyslogd.conf

5. You need to change the startup behavior of the rsyslogd service. Which of the following describes the recommended approach to do so?

a. Include the startup parameter in the main rsyslog configuration file.

b. Create a snap-in file in the directory /etc/rsyslog.d and specify the required parameters in there.

c. Change the Systemd unit file in /usr/lib/systemd/system to include the required startup parameter.

d. Use the SYSLOGD_OPTIONS line in the file /etc/sysconfig/rsyslog and include the startup parameter here.

6. In the rsyslog.conf file, which of the following destinations refers to a specific rsyslogd module?

a. -/var/log/maillog

b. /var/log/messages

c. :omusrmsg:*

d. *

7. Which facility is the best solution if you want to configure the Apache web server to log messages through rsyslog?

a. daemon

b. apache

c. syslog

d. local0-7

8. You want to maximize the file size of a log file to 10 MB. Where do you configure this?

a. Create a file in /etc/logrotate.d and specify the maximal size in that file.

b. Put the maximal size in the logrotate cron job.

c. Configure the destination with the maximal size option.

d. This cannot be done.

9. Which directory is used to store the journald journal?

a. /var/log/journal

b. /var/run/journal

c. /run/log

d. /run/log/journal

10. What do you need to do to make the journald journal persistent?

a. Create the directory /var/log/journal, set appropriate permissions, and reboot your machine.

b. Open /etc/sysconfig/journal and set the PERSISTENT option to yes.

c. Open the /etc/systemd/journald.conf file and set the PERSISTENT option to yes.

d. Create the /var/log/journal file and set appropriate permissions.

Foundation Topics

Understanding System Logging

Most services used on a Linux server write information to log files. This information can be written to different destinations, and there are multiple solutions to find the relevant information in system logs. No less than three different approaches can be used by services to write log information:

  • Direct write: Some services write logging information directly to the log files—even some important services such as the Apache web server and the Samba file server.

  • rsyslogd: rsyslogd is the enhancement of syslogd, a service that takes care of managing centralized log files. Syslogd has been around for a long time.

  • journald: With the introduction of Systemd, the journald log service systemd-journald has been introduced also. This service is tightly integrated with Systemd, which allows administrators to read detailed information from the journal while monitoring service status using the systemctl status command.

Understanding the Role of rsyslogd and journald

Key topic

On RHEL 8, journald (which is implemented by the systemd-journald daemon) provides an advanced log management system. journald collects messages from the kernel, the entire boot procedure, and services and writes these messages to an event journal. This event journal is stored in a binary format, and it can be queried using the journalctl command. The journalctl command enables you to access a deep level of detail about messages that are logged, as it is an integrated part of Systemd and, as such, receives all messages that have been generated by Systemd units.

Key topic

Because the journal that is written by journald is not persistent between reboots, messages are also forwarded to the rsyslogd service, which writes the messages to different files in the /var/log directory. rsyslogd also offers features that do not exist in journald, such as centralized logging and filtering messages by using modules. Numerous modules are available to enhance rsyslog logging, such as output modules that allow administrators to store messages in a database.

In the current state of RHEL 8, journald is not a replacement for rsyslog; it is just another way of logging information. journald is tightly integrated with Systemd; therefore, it logs everything that your server is doing. rsyslogd adds some services to it. In particular, it takes care of writing log information to specific files (that will be persistent between reboots), and it allows you to configure remote logging and log servers.

Apart from rsyslogd and systemd-journald, there is the auditd service. This service provides auditing, an in-depth trace of what specific services, processes, or users have been doing. Configuration of auditing is beyond the scope of the RHCSA exam, but you’ll notice that SELinux, for instance, logs detailed messages to the auditd service.

To get more information about what has been happening on a machine running RHEL, administrators have to take three approaches:

  • Monitor the files in /var/log that are written by rsyslogd.

  • Use the journalctl command to get more detailed information from the journal.

  • Use the systemctl status <unit> command to get a short overview of the last significant events that have been logged by Systemd units through journald. This command shows the status of services, as well as the last couple of lines that have been logged. Example 13-1 shows an example where this command clearly indicates what went wrong while starting a service.

Example 13-1 Using systemctl status to Show Relevant Log Information

[root@server1 ~]# systemctl status sshd -l
   sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled;
             vendor preset: enabled)
   Active: active (running) since Sat 2019-06-08 03:34:56 EDT;
             55min ago
     Docs: man:sshd(8)
           man:sshd_config(5)
 Main PID: 1055 (sshd)
    Tasks: 1 (limit: 11363)
   Memory: 5.5M
   CGroup: /system.slice/sshd.service
            Image1055 /usr/sbin/sshd -D [email protected],
                [email protected],ae>

Jun 08 03:34:56 server1.example.com systemd[1]: Starting OpenSSH
  server daemon...
Jun 08 03:34:56 server1.example.com sshd[1055]: Server listening on
  0.0.0.0 port 22.
Jun 08 03:34:56 server1.example.com sshd[1055]: Server listening on ::
  port 22.
Jun 08 03:34:56 server1.example.com systemd[1]: Started OpenSSH server
  daemon.
Jun 08 03:57:38 server1.example.com sshd[3368]: Accepted password for
  root from 192.168.4.1 port 5470>
Jun 08 03:57:38 server1.example.com sshd[3368]: pam_unix(sshd:
  session):session opened for user root

Reading Log Files

Apart from the messages that are written by journald to the journal and which can be read using the journalctl command, on a Linux system you’ll also find different log files in the directory /var/log. Most of the files in this directory are managed by rsyslogd, but some of the files are created directly by specific services. You can read these files by using a pager utility such as less.

The exact number of files in the /var/log directory will change, depending on the configuration of a server and the services that are running on that server. Some files, however, do exist on most occasions, and as an administrator, you should know which files they are and what kind of contents can be expected in these files. Table 13-2 provides an overview of some of the standard files that are created in this directory.

Key topic

Table 13-2 System Log Files Overview

Log File

Explanation

/var/log/messages

The most commonly used log file, it is the generic log file where most messages are written to.

/var/log/dmesg

Contains kernel log messages.

/var/log/secure

Contains authentication-related messages. Look here to see which authentication errors have occurred on a server.

/var/log/boot.log

Look here for messages that are related to system startup.

/var/log/audit/audit.log

Contains audit messages. SELinux writes to this file.

/var/log/maillog

Look here for mail-related messages.

/var/log/samba

Provides log files for the Samba service. Notice that Samba by default is not managed through rsyslog, but writes directly to the /var/log directory.

/var/log/sssd

Contains messages that have been written by the sssd service, which plays an important role in the authentication process.

/var/log/cups

Contains log messages that were generated by the print service CUPS.

/var/log/httpd/

Directory that contains log files that are written by the Apache web server. Notice that Apache writes messages to these files directly and not through rsyslog.

Understanding Log File Contents

As an administrator, you need to be able to interpret the contents of log files. For example, Example 13-2 shows partial content from the /var/log/messages file.

Example 13-2 /var/log/messages Sample Content

[root@server1 ~]# tail -10 /var/log/messages
Jun  8 03:57:46 server1 journal[2682]: Received error from DBus
  search provider org.gnome.Software.desktop: Gio.IOErrorEnum:
  Timeout was reached
Jun  8 03:58:20 server1 journal[3078]: failed to get featured apps:
  no apps to show
Jun  8 03:58:20 server1 journal[3078]: Only 0 apps for popular list,
  hiding
Jun  8 03:58:20 server1 journal[2105]: Skipping refresh of rhel-8-for-
  x86_64-highavailability-beta-source-rpms: cannot update repo 'rhel-
  8-for-x86_64-highavailability-beta-source-rpms': Cannot download
  repomd.xml: Curl error (42): Operation was aborted by an application
  callback for https://cdn.redhat.com/content/beta/rhel8/8/x86_64/
  highavailability/source/SRPMS/repodata/repomd.xml [Callback
  aborted]; Last error: Curl error (42): Operation was aborted by
  an application callback for https://cdn.redhat.com/content/beta/
  rhel8/8/x86_64/highavailability/source/SRPMS/repodata/repomd.xml
  [Callback aborted]
Jun  8 03:58:20 server1 journal[3078]: hiding category graphics
  featured applications: found only 0 to show, need at least 9
Jun  8 03:58:23 server1 journal[2105]: Skipping refresh of rhel-8-
  for-x86_64-supplementary-beta-debug-rpms: cannot update repo 'rhel-
  8-for-x86_64-supplementary-beta-debug-rpms': Cannot download repomd.
  xml: Curl error (42): Operation was aborted by an application
  callback for https://cdn.redhat.com/content/beta/rhel8/8/x86_64/
  supplementary/debug/repodata/repomd.xml [Callback aborted]; Last
  error: Curl error (42): Operation was aborted by an application
  callback for https://cdn.redhat.com/content/beta/rhel8/8/x86_64/
  supplementary/debug/repodata/repomd.xml [Callback aborted]
Jun  8 03:58:46 server1 chronyd[907]: Selected source 5.200.6.34
Jun  8 03:59:16 server1 systemd[2571]: Starting Mark boot as
  successful...
Jun  8 03:59:16 server1 systemd[2571]: Started Mark boot as
  successful.
Jun  8 04:00:16 server1 systemd[3374]: Starting Mark boot as
  successful...

As you can see in Example 13-2, each line that is logged has specific elements:

  • Date and time: Every log message starts with a timestamp. For filtering purposes, the timestamp is written as military time.

  • Host: The host the message originated from. This is relevant because rsyslogd can be configured to handle remote logging as well.

  • Service or process name: The name of the service or process that generated the message.

  • Message content: The content of the message, which contains the exact message that has been logged.

To read the content of a log file, you can use a pager utility, like less, or you can live monitor what is happening in the log file, as described in the next section.

Live Log File Monitoring

When you are configuring services on Linux, it might be useful to see in real time what is happening. You could, for example, open two terminal sessions at the same time. In one terminal session, you configure and test the service. In the other terminal session, you see in real time what is happening. The tail -f <logfile> command shows in real time which lines are added to the log file. Exercise 13-1 in the following section shows a small example in which tail -f is used. When monitoring a log file with tail -f, the trace remains open until you press Ctrl-C to close it.

Using logger

Most services write information to the log files all by themselves or through rsyslogd. The logger command enables users to write messages to rsyslog from the command line or a script. Using this command is simple. Just type logger, followed by the message you want to write to the logs. The logger utility, in this way, offers a convenient solution to write messages from scripts. This allows you to have a script write to syslog if something goes wrong.

When using logger, you can also specify the priority and facility to log to. The command logger -p kern.err hello writes hello to the kernel facility, for example, using the error priority (discussed later in this chapter). This option enables you to test the working of specific rsyslog facilities. In Exercise 13-1, you use tail -f to monitor a log file in real time and use logger to write log messages to a log file.

Exercise 13-1 Using Live Log Monitoring and logger

  1. Open a root shell.

  2. From the root shell, type tail -f /var/log/messages.

  3. Open a second terminal window. In this terminal window, type su - student to open a subshell as user student.

  4. Type su - to open a root shell, but enter the wrong password.

  5. Notice that nothing appears in /var/log/messages. That is because login-related errors are not written here.

  6. From the student shell, type logger hello. You’ll see the message appearing in the /var/log/messages file in real time.

  7. In the tail -f terminal, press Ctrl-C to stop tracing the messages file.

  8. Type tail -20 /var/log/secure. This shows the last 20 lines in /var/log/secure, which also shows the messages that the su - password errors have generated previously.

Configuring rsyslogd

To make sure that the information that needs to be logged is written to the location where you want to find it, you can configure the rsyslogd service through the /etc/rsyslog.conf file. In this file, you find different sections that allow you to specify where and how information should be written.

Understanding rsyslogd Configuration Files

Like many other services on RHEL 8, the configuration for rsyslogd is not defined in just one configuration file. The /etc/rsyslog.conf file is the central location where rsyslogd is configured. From this file, the content of the directory /etc/rsyslog.d is included. This directory can be populated by installing RPM packages on a server. When looking for specific log configuration, make sure to always consider the contents of this directory also.

If specific options need to be passed to the rsyslogd service on startup, you can do this by using the /etc/sysconfig/rsyslog file. This file by default contains one line, which reads SYSLOGD_OPTIONS="". On this line, you can specify rsyslogd startup parameters. The SYSLOGD_OPTIONS variable is included in the Systemd configuration file that starts rsyslogd. Theoretically, you could change startup parameters in this file, as well, but that is not recommended. (See Chapter 17, “Managing and Understanding the Boot Procedure,” for more details about Systemd configuration.)

Understanding rsyslog.conf Sections

The rsyslog.conf file is used to specify what should be logged and where it should be logged. To do this, you’ll find different sections in the rsyslog.conf file:

  • #### MODULES ####: rsyslogd is modular. Modules are included to enhance the supported features in rsyslogd.

  • #### GLOBAL DIRECTIVES ####: This section is used to specify global parameters, such as the location where auxiliary files are written or the default timestamp format.

  • #### RULES ####: This is the most important part of the rsyslog.conf file. It contains the rules that specify what information should be logged to which destination.

Understanding Facilities, Priorities, and Log Destinations

To specify what information should be logged to which destination, rsyslogd uses facilities, priorities, and destinations:

  • A facility specifies a category of information that is logged. rsyslogd uses a fixed list of facilities, which cannot be extended. This is because of backward compatibility with the legacy syslog service.

  • A priority is used to define the severity of the message that needs to be logged. When specifying a priority, by default all messages with that priority and all higher priorities are logged.

  • A destination defines where the message should be written. Typical destinations are files, but rsyslog modules can be used as a destination as well, to allow further processing through a rsyslogd module.

Example 13-3 shows an example of the RULES section in rsyslog.

Example 13-3 Example of the RULES Section in rsyslog.conf

#### RULES ####


# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                    /dev/console

# Log anything (except mail) of level info or higher.
# Do not log private authentication messages!
*.info;mail.none;authpriv.none;cron.none   /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                 /var/log/secure

# Log all the mail messages in one place.
mail.*                                     -/var/log/maillog


# Log cron stuff
cron.*                                     /var/log/cron

# Everybody gets emergency messages
*.emerg                                    :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                             /var/log/spooler

In Example 13-3, you can see how different facilities and priorities are used to define locations where information can be logged. The available facilities and priorities are fixed and cannot be added to. Table 13-3 shows which facilities are available, and Table 13-4 shows a list of all priorities.

When specifying a destination, a file is often used. If the filename starts with a hyphen (as in -/var/log/maillog), the log messages will not be immediately committed to the file but instead will be buffered to make writes more efficient. Device files can also be used, such as /dev/console. If this device is used, messages are written in real time to the console. On modern servers, this often does not make sense, because administrators often log in remotely and do not see what is happening on the server console.

Key topic

Table 13-3 rsyslogd Facilities

Facility

Used by

auth/authpriv

Messages related to authentication.

cron

Messages generated by the crond service.

daemon

Generic facility that can be used for nonspecified daemons.

kern

Kernel messages.

lpr

Messages generated through the legacy lpd print system.

mail

Email-related messages.

mark

Special facility that can be used to write a marker periodically.

news

Messages generated by the NNTP news system.

security

Same as auth/authpriv. Should not be used anymore.

syslog

Messages generated by the syslog system.

user

Messages generated in user space.

uucp

Messages generated by the legacy UUCP system.

local0-7

Messages generated by services that are configured by any of the local0 through local7 facilities.

The syslog facilities were defined in the 1980s, and to guarantee backward compatibility, no new facilities can be added. The result is that some facilities still exist that basically serve no purpose anymore, and some services that have become relevant at a later stage do not have their own facility. As a solution, two specific facility types can be used. The daemon facility is a generic facility that can be used by any daemon. In addition, the local0 through local7 facilities can be used.

If there are services that do not have their own rsyslogd facility that need to write log messages to a specific log file anyway, these services can be configured to use any of the local0 through local7 facilities. You next have to configure the services to use these facilities as well. The procedure you follow to do that is specific to the service you are using. Then you need to add a rule to the rsyslog.conf file to send messages that come in through that facility to a specific log file.

To determine which types of messages should be logged, different severities can be used in rsyslog.conf lines. These severities are the syslog priorities. Table 13-4 provides an overview of the available priorities in ascending order.

Key topic

Table 13-4 rsyslogd Priorities

Priority

Description

debug

Debug messages that will give as much information as possible about service operation.

info

Informational messages about normal service operation.

notice

Informational messages about items that might become an issue later.

warning / warn

Something is suboptimal, but there is no real error yet.

err /error

A noncritical error has occurred.

crit

A critical error has occurred.

alert

Used when the availability of the service is about to be discontinued.

emerg/panic

Message generated when the availability of the service is discontinued.

When a specific priority is used, all messages with that priority and higher are logged according to the specifications used in that specific rule. If you need to configure logging in a detailed way, where messages with different priorities are sent to different files, you can specify the priority with an equals sign (=) in front of it, as in the following line, which will write all cron messages with only the debug priority to a specific file with the name /var/log/cron.debug. The - in front of the line specifies to buffer writes so that information is logged in a more efficient way.

cron.=debug -/var/log/cron.debug

Tip

There is no need to learn the names of rsyslogd facilities and priorities by heart. They are all listed in man 5 rsyslog.conf. On the exam, you have access to the man pages, so this information will be easily accessible.

Exercise 13-2 shows how to change rsyslog.conf. You configure the Apache service to log messages through syslog, and you create a rule that logs debug messages to a specific log file.

Exercise 13-2 Changing rsyslog.conf Rules

  1. By default, the Apache service does not log through rsyslog but keeps its own logging. You are going to change that. To start, type yum install -y httpd to install the Apache service.

  2. After installing the Apache service, open its configuration file /etc/http/conf/httpd.conf and add the following line to it:

    ErrorLog    syslog:local1
  3. Type systemctl restart httpd.

  4. Create a line in the rsyslog.conf file that will send all messages that it receives for facility local1 (which is now used by the httpd service) to the file /var/log/httpd-error.log. To do this, include the following line:

    local1.error        /var/log/httpd-error.log
  5. Tell rsyslogd to reload its configuration, by using systemctl restart rsyslog.

  6. All Apache error messages will now be written to the httpd-error.log file.

  7. From the Firefox browser, go to http://localhost. As no index.html page exists yet, this will be written to the error log.

  8. Create a snap-in file that logs debug messages to a specific file as well. To do this, type echo "*.debug /var/log/messages-debug"> /etc/rsyslog.d/debug.conf.

  9. Again, restart rsyslogd using systemctl restart rsyslog.

  10. Use the command tail -f /var/log/messages-debug to open a trace on the newly created file.

  11. Type logger -p daemon.debug "Daemon Debug Message". You’ll see the debug message passing by.

  12. Press Ctrl-C to close the debug log file.

Rotating Log Files

To prevent syslog messages from filling up your system completely, the log messages can be rotated. That means that when a certain threshold has been reached, the old log file is closed and a new log file is opened. The logrotate utility is started periodically through the crond service to take care of rotating log files.

When a log file is rotated, the old log file is typically copied to a file that has the rotation date in it. So, if /var/log/messages is rotated on June 8, 2019, the rotated filename will be /var/log/messages-20190608. As a default, four old log files are kept on the system. Files older than that period are removed from the system automatically.

Warning

Log files that have been rotated are not stored anywhere; they are just gone. If your company policy requires you to be able to access information about events that have happened more than five weeks ago, for example, you should either back up log files or configure a centralized log server where logrotate keeps rotated messages for a significantly longer period.

The default settings for log rotation are kept in the file /etc/logrotate.conf (see Example 13-4).

Example 13-4 /etc/logrotate.conf Sample Content

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here

The most significant settings used in this configuration file tell logrotate to rotate files on a weekly basis and keep four old versions of the file. You can obtain more information about other parameters in this file through the man logrotate command.

If specific files need specific settings, you can create a configuration file for that file in /etc/logrotate.d. The settings for that specific file overwrite the default settings in /etc/logrotate.conf. You will find that different files exist in this directory already to take care of some of the configuration files.

Working with journald

The systemd-journald service stores log messages in the journal, a binary file that is temporarily stored in the file /run/log/journal. This file can be examined using the journalctl command.

Using journalctl to Find Events

The easiest way to use journalctl is by just typing the command. It shows that recent events have been written to the journal since your server last started. The result of this command is shown in the less pager, and by default you’ll see the beginning of the journal. Because the journal is written from the moment your server boots, this shows boot-related log messages. If you want to see the last messages that have been logged, you can use journalctl -f, which shows the last lines of the messages where new log lines are automatically added. You can also type journalctl and use (uppercase) G to go to the end of the journal. Also note that the search options / and ? work in the journalctl output. Example 13-5 shows a partial result of this command.

Example 13-5 Watching Log Information Generated by journald

-- Logs begin at Sat 2019-06-08 04:45:34 EDT, end at Sat 2019-06-08
  04:56:11 EDT. --
Jun 08 04:45:34 server1.example.com kernel: Linux version 4.18.0-80.
  el8.x86_64 ([email protected]>
Jun 08 04:45:34 server1.example.com kernel: Command line: BOOT_
  IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-80.e>
Jun 08 04:45:34 server1.example.com kernel: Disabled fast string
  operations
Jun 08 04:45:34 server1.example.com kernel: x86/fpu: Supporting XSAVE
  feature 0x001: 'x87 floating po>
Jun 08 04:45:34 server1.example.com kernel: x86/fpu: Supporting XSAVE
  feature 0x002: 'SSE registers'
Jun 08 04:45:34 server1.example.com kernel: x86/fpu: Supporting XSAVE
  feature 0x004: 'AVX registers'
Jun 08 04:45:34 server1.example.com kernel: x86/fpu: Supporting XSAVE
  feature 0x008: 'MPX bounds regi>
Jun 08 04:45:34 server1.example.com kernel: x86/fpu: Supporting XSAVE
  feature 0x010: 'MPX CSR'
Jun 08 04:45:34 server1.example.com kernel: x86/fpu: xstate_offset[2]:
  576, xstate_sizes[2]:  256
Jun 08 04:45:34 server1.example.com kernel: x86/fpu: xstate_offset[3]:
  832, xstate_sizes[3]:   64
Jun 08 04:45:34 server1.example.com kernel: x86/fpu: xstate_offset[4]:
  896, xstate_sizes[4]:   64
Jun 08 04:45:34 server1.example.com kernel: x86/fpu: Enabled xstate
  features 0x1f, context size is 96>

What makes journalctl a flexible command is that its many filtering options allow you to show exactly what you need. Exercise 13-3 shows some of the most interesting options.

Exercise 13-3 Discovering journalctl

  1. Type journalctl. You’ll see the content of the journal since your server last started, starting at the beginning of the journal. The content is shown in less, so you can use common less commands to walk through the file.

  2. Type q to quit the pager. Now type journalctl --no-pager. This shows the contents of the journal without using a pager.

  3. Type journalctl -f. This opens the live view mode of journalctl, which allows you to see new messages scrolling by in real time. Press Ctrl-C to interrupt.

  4. Type journalctl and press the Tab key twice. This shows specific options that can be used for filtering. Type, for instance, journalctl _UID=1000 to show messages that have been logged for your student user account.

  5. Type journalctl -n 20. The -n 20 option displays the last 20 lines of the journal (just like tail -n 20).

  6. Type journalctl -p err. This command shows errors only.

  7. If you want to view journal messages that have been written in a specific time period, you can use the --since and --until commands. Both options take the time parameter in the format YYYY-MM-DD hh:mm:ss. Also, you can use yesterday, today, and tomorrow as parameters. So, type journalctl --since yesterday to show all messages that have been written since yesterday.

  8. journalctl allows you to combine different options, as well. So, if you want to show all messages with a priority err that have been written since yesterday, use journalctl --since yesterday -p err.

  9. If you need as much detail as possible, use journalctl -o verbose. This shows different options that are used when writing to the journal (see Example 13-6). All these options can be used to tell the journalctl command which specific information you are looking for. Type, for instance, journalctl _SYSTEMD_UNIT=sshd.service to show more information about the sshd Systemd unit.

  10. Type journalctl --dmesg. This shows kernel-related messages only. Not many people use this command, as the dmesg command gives the exact same result.

In the preceding exercise, you typed journalctl -o verbose to show verbose output. Example 13-6 shows an example of the verbose output. As you can see, this provides detailed information for all items that have been logged, including the PID, the ID of the associated user and group account, the command that is associated, and more. This verbose information may help you in debugging specific Systemd units.

Example 13-6 Showing Detailed Log Information with journalctl -o verbose

[root@server1 ~]# journalctl _SYSTEMD_UNIT=sshd.service -o verbose
-- Logs begin at Sat 2019-06-08 04:45:34 EDT, end at Sat 2019-06-08
  05:01:40 EDT. --
Sat 2019-06-08 04:45:52.633752 EDT [s=53e57e2481434e078e8306367dc5645c;
  i=898;b=f35bb68348284f9ead79c3>
    _BOOT_ID=f35bb68348284f9ead79c3c6750adfa1
    _MACHINE_ID=5aa095b495ed458d934c54a88078c165
    _HOSTNAME=server1.example.com
    PRIORITY=6
    _UID=0
    _GID=0
    _SYSTEMD_SLICE=system.slice
    _CAP_EFFECTIVE=3fffffffff
    _TRANSPORT=syslog
    SYSLOG_FACILITY=10
    SYSLOG_IDENTIFIER=sshd
    SYSLOG_PID=1211
    MESSAGE=Server listening on 0.0.0.0 port 22.
    _PID=1211
    _COMM=sshd
    _EXE=/usr/sbin/sshd
    _CMDLINE=/usr/sbin/sshd -D [email protected],
       [email protected],aes256->
    _SELINUX_CONTEXT=system_u:system_r:sshd_t:s0-s0:c0.c1023
    _SYSTEMD_CGROUP=/system.slice/sshd.service
    _SYSTEMD_UNIT=sshd.service
    _SYSTEMD_INVOCATION_ID=728a7dfecd7d436387dcd6e319c208c7
    _SOURCE_REALTIME_TIMESTAMP=1559983552633752
Sat 2019-06-08 04:45:52.634696 EDT [s=53e57e2481434e078e8306367dc5645c;
  i=899;b=f35bb68348284f9ead79c3>
    _BOOT_ID=f35bb68348284f9ead79c3c6750adfa1
lines 1-26

Preserving the Systemd Journal

By default, the journal is stored in the file /run/log/journal. The entire /run directory is used for current process status information only, which means that the journal is cleared when the system reboots. To make the journal persistent between system restarts, you should make sure that a directory /var/log/journal exists.

Storing the journal permanently requires setting the Storage=auto parameter in /etc/systemd/journal.conf. This parameter can have different values:

  • Storage=auto The journal will be written on disk if the directory /var/log/journal exists.

  • Storage=volatile The journal will be stored only in the /run/log/journal directory.

  • Storage=persistent The journal will be stored on disk in the directory /var/log/journal. This directory will be created automatically if it doesn’t exist.

  • Storage=none No data will be stored, but forwarding to other targets such as the kernel log buffer or syslog will still work.

Even when the journal is written to the permanent file in /var/log/journal, that does not mean that the journal is kept forever. The journal has built-in log rotation that will be used monthly. Also, the journal is limited to a maximum size of 10% of the size of the file system that it is on, and it will stop growing if less than 15% of the file system is still free. If that happens, the oldest messages from the journal are dropped automatically to make room for newer messages. To change these settings, you can modify the file /etc/systemd/journald.conf, as shown in Example 13-7 (along with other parameters you can set).

Example 13-7 Setting journald Parameters Through /etc/systemd/journald.conf

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug

Making the journald journal permanent is not hard to do. Exercise 13-4 shows how to proceed.

Key topic

Exercise 13-4 Making the journald Journal Permanent

  1. Open a root shell and type mkdir /var/log/journal.

  2. Before journald can write the journal to this directory, you have to set ownership. Type chown root:systemd-journal /var/log/journal, followed by chmod 2755 /var/log/journal.

  3. Next, you can either reboot your system (restarting the systemd-journald service is not enough) or use the killall -USR1 systemd-journald command.

  4. The Systemd journal is now persistent across reboots. If you want to see the log messages since the last reboot, use journalctl -b.

Summary

In this chapter, you learned how to configure logging. You read how the rsyslogd and journald services are used on RHEL 8 to keep log information, and you learned how to manage logs that are written by these services. You also learned how to configure log rotation and make the journal persistent.

Exam Preparation Tasks

As mentioned in the section “How to Use This Book” in the Introduction, you have several choices for exam preparation: the end-of-chapter labs; the memory tables in Appendix B; Chapter 26, “Final Preparation”; and the practice exams.

Review All Key Topics

Review the most important topics in the chapter, noted with the Key Topic icon in the outer margin of the page. Table 13-5 lists a reference of these key topics and the page number on which each is found.

Key topic

Table 13-5 Key Topics for Chapter 13

Key Topic Element

Description

Page

Paragraph

journald explanation

288

Paragraph

rsyslogd explanation

288

Table 13-2

System log files overview

290

Table 13-3

rsyslogd facilities

295

Table 13-4

rsyslogd priorities

296

Exercise 13-4

Making the journald journal permanent

303

Complete Tables and Lists from Memory

Print a copy of Appendix B, “Memory Tables” (found on the companion website), or at least the section for this chapter, and complete the tables and lists from memory. Appendix C, “Memory Tables Answer Key,” includes completed tables and lists to check your work.

Define Key Terms

Define the following key terms from this chapter and check your answers in the glossary:

journald

journalctl

rsyslogd

facility

priority

destination

log rotation

Review Questions

The questions that follow are meant to help you test your knowledge of concepts and terminology and the breadth of your knowledge. You can find the answers to these questions in Appendix A.

1. Which file is used to configure rsyslogd?

2. Which configuration file contains messages related to authentication?

3. If you do not configure anything, how long will it take for log files to be rotated away?

4. Which command enables you to log a message from the command line to the user facility, using the notice priority?

5. Which line would you add to write all messages with a priority of info to the file /var/log/messages.info?

6. Which configuration file enables you to allow the journal to grow beyond its default size restrictions?

7. Which command enables you to see new messages in the journal scrolling by in real time?

8. Which command enables you to see all journald messages that have been written for PID 1 between 9:00 a.m. and 3:00 p.m.?

9. Which command enables you to see journald messages since the last reboot on a system where a persistent journal has been configured?

10. Which procedure enables you to make the journald journal persistent?

End-of-Chapter Lab

You have now learned how to work with logging on Red Hat Enterprise Linux 8 and know how to configure rsyslogd and journald. You can now complete the end-of-chapter lab to reinforce these newly acquired skills.

Lab 13.1

1. Configure the journal to be persistent across system reboots.

2. Make a configuration file that writes all messages with an info priority to the file /var/log/messages.info.

3. Configure logrotate to keep ten old versions of log files.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.144.248.24