Understanding journald

You'll find the journald logging system on any Linux distro that uses the systemd ecosystem. Instead of sending its messages to text files, journald sends messages to binary files. Instead of using normal Linux text file utilities to extract information, you have to use the journalctl utility. At the time of writing, no Linux distro that I know of has made the complete transition to journald. Current Linux distros that use systemd run journald and rsyslog side by side. Currently, the default is for journald log files to be temporary files that get erased every time you reboot the machine. (You can configure journald to make its log files persistent, but there's probably not much point as long as we still need to keep the old rsyslog files.)

A new feature of RHEL 8/CentOS 8 is that journald, instead of rsyslog, is now what actually collects log messages from the rest of the operating system. But rsyslog is still there, collecting the messages from journald and sending them to the old-fashioned rsyslog text files. So, the way you do log file management hasn't really changed.

It will likely take a few more years to completely transition away from rsyslog. One reason is that third-party log aggregation and analysis utilities, such as LogStash, Splunk, and Nagios, are still set up to read text files instead of binary files. Another reason is that, at this point, using journald as a remote, central log server is still in a proof-of-concept stage that isn't ready for production use. So, for now, journald isn't a suitable substitute for rsyslog.

To view the journald log file in its entirety, use the journalctl command. With Ubuntu, the person who installed the operating system has been added to the adm group, which allows that person to use journalctl without sudo or root privileges. Any users who are added later would only be able to see their own messages. In fact, here's what happened for Frank:

frank@ubuntu4:~$ journalctl
Hint: You are currently not seeing messages from other users and the system.
Users in groups 'adm', 'systemd-journal' can see all messages.
Pass -q to turn off this notice.
-- Logs begin at Tue 2019-11-26 17:43:28 UTC, end at Tue 2019-11-26 17:43:28 UTC. --
Nov 26 17:43:28 ubuntu4 systemd[10306]: Listening on GnuPG cryptographic agent and passphrase cache.
Nov 26 17:43:28 ubuntu4 systemd[10306]: Reached target Timers.
Nov 26 17:43:28 ubuntu4 systemd[10306]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
. . .
. . .
Nov 26 17:43:28 ubuntu4 systemd[10306]: Reached target Basic System.
Nov 26 17:43:28 ubuntu4 systemd[10306]: Reached target Default.
Nov 26 17:43:28 ubuntu4 systemd[10306]: Startup finished in 143ms.
frank@ubuntu4:~$

To see messages from either the system or from other users, these new users would have to be added to either the adm or the systemd-journal group or granted the proper sudo privileges. With RHEL/CentOS, no users are automatically added to either the adm or systemd-journal group. So, initially, only users who have sudo privileges can view the journald logs.

Doing either journalctl or sudo journalctl, as appropriate, automatically opens the log in the more pager. (The journalctl man page says that it uses less, but it lies.) So, instead of advancing through the file with the page down key as you would with less, you'll have to use the spacebar. What you'll see looks pretty much the same as what you'd see in the normal rsyslog log files, with the following exceptions:

  • Long lines run past the right-hand edge of the screen. To see the rest of the lines, use the right cursor key.
  • You'll also see color-coding and highlighting to make different types of messages stand out. Messages of ERROR level and higher are in red, while messages from NOTICE level up to ERROR level are highlighted with bold characters.

There are lots of options that can display different types of information in various formats. For example, to only see messages about the SSH service on CentOS, use the --unit option, like so:

[donnie@localhost ~]$ sudo journalctl --unit=sshd
-- Logs begin at Tue 2019-11-26 12:00:13 EST, end at Tue 2019-11-26 15:55:19 EST. --
Nov 26 12:00:41 localhost.localdomain systemd[1]: Starting OpenSSH server daemon...
Nov 26 12:00:42 localhost.localdomain sshd[825]: Server listening on 0.0.0.0 port 22.
Nov 26 12:00:42 localhost.localdomain sshd[825]: Server listening on :: port 22.
Nov 26 12:00:42 localhost.localdomain systemd[1]: Started OpenSSH server daemon.
Nov 26 12:22:08 localhost.localdomain sshd[3018]: Accepted password for donnie from 192.168.0.251 port 50797 ssh2
Nov 26 12:22:08 localhost.localdomain sshd[3018]: pam_unix(sshd:session): session opened for user donnie by (uid=0)
Nov 26 13:03:33 localhost.localdomain sshd[4253]: Accepted password for goldie from 192.168.0.251 port 50912 ssh2
Nov 26 13:03:34 localhost.localdomain sshd[4253]: pam_unix(sshd:session): session opened for user goldie by (uid=0)
[donnie@localhost ~]$

You can't use the grep utility with these binary logs, but you can search for a string with the -g option. By default, it's case-insensitive and finds your desired text string even when it's embedded in another text string. Here, we see it finding the text string, fail:

[donnie@localhost ~]$ sudo journalctl -g fail
-- Logs begin at Tue 2019-11-26 12:00:13 EST, end at Tue 2019-11-26 15:57:19 EST. --
Nov 26 12:00:13 localhost.localdomain kernel: NMI watchdog: Perf event create on CPU 0 failed with -2
Nov 26 12:00:13 localhost.localdomain kernel: acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
Nov 26 12:00:14 localhost.localdomain kernel: [drm:vmw_host_log [vmwgfx]] *ERROR* Failed to send log
Nov 26 12:00:14 localhost.localdomain kernel: [drm:vmw_host_log [vmwgfx]] *ERROR* Failed to send log
. . .

There are lots more options besides just these. To see them, just do this:

man journalctl

Now that you've seen the basics of using both rsyslog and journald, let's look at a cool utility that can help to ease the pain of doing log reviews.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.128.199.162