It doesn't matter whether you're talking about the syslog and auth.log files on Debian/Ubuntu or the messages and secure files on RHEL/CentOS. On either system, the files are the same, just with different names. The system log files and the authentication log files have the same basic structure and are all plaintext files. This makes it easy to search for specific information with tools that are already built into Linux. It doesn't really matter which virtual machine (VM) we use for this, other than to keep the names of the files straight.

To begin, let's look at a simple message from the system log:

Jul  1 18:16:12 localhost systemd[1]: Started Postfix Mail Transport Agent.

Here's the breakdown:

• Jul  1 18:16:12: This is the date and time that the message was generated.
• localhost: This is the hostname of the machine that generated the message. This is important because one Linux machine can serve as the central log repository for other Linux machines. By default, messages from other machines will just get dumped into the same log file that the local machine uses. So, we need this field to let us know what's happening on each machine.
• systemd[1]: This is the service that generated the message. In this case, it was the systemd daemon.
• The rest of the line is the specific message.

There are several ways to extract information from the text-mode log files. For now, we'll just open the files in less, as in this example:

sudo less syslog

Then, to search for a specific text string, hit the / key, type in the string that you want to find, and hit Enter.

So, what kind of security-related information can we expect to find in these files? To start, let's look at the permissions on the server's private SSH keys:

donnie@orangepione:/etc/ssh$ ls -l
total 580
. . .
-rw-------+ 1 root root   1679 Feb 10  2019 ssh_host_rsa_key
-rw-r--r--  1 root root    398 Feb 10  2019 ssh_host_rsa_key.pub
donnie@orangepione:/etc/ssh$

This private key, the ssh_host_rsa_key file, has to have permissions set for only the root user. But, the + sign at the end of the permissions settings denotes that someone has set an access-control list (ACL) on that file. getfacl will show us exactly what's going on:

donnie@orangepione:/etc/ssh$ getfacl ssh_host_rsa_key
# file: ssh_host_rsa_key
# owner: root
# group: root
user::rw-
user:sshdnoroot:r--
group::---
mask::r--
other::---
donnie@orangepione:/etc/ssh$

So, someone has created the sshdnoroot user and assigned it the read permission for the server's private SSH keys. Now, if I try to restart the OpenSSH daemon, it will fail. A peek into the system log—the syslog file, in this case—will tell me why:

Mar 13 12:47:46 localhost sshd[1952]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 13 12:47:46 localhost sshd[1952]: @ WARNING: UNPROTECTED PRIVATE KEY FILE! @
Mar 13 12:47:46 localhost sshd[1952]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 13 12:47:46 localhost sshd[1952]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Mar 13 12:47:46 localhost sshd[1952]: It is required that your private key files are NOT accessible by others.
Mar 13 12:47:46 localhost sshd[1952]: This private key will be ignored.
Mar 13 12:47:46 localhost sshd[1952]: key_load_private: bad permissions
Mar 13 12:47:46 localhost sshd[1952]: Could not load host key: /etc/ssh/ssh_host_rsa_key

So, the SSH daemon won't start if someone other than the root user has any access permissions for the server's private keys. But how did this happen? Let's search through the authentication file—auth.log, in this case—to see whether there's a clue:

Mar 13 12:42:54 localhost sudo:   donnie : TTY=tty1 ; PWD=/etc/ssh ; USER=root ; COMMAND=/usr/bin/setfacl -m u:sshdnoroot:r ssh_host_ecdsa_key ssh_host_ed25519_key ssh_host_rsa_key

Ah, so that donnie character did this. Why, this is an outrage! Fire that guy immediately! Oh wait, that's me. On second thought, let's not fire him. But seriously, this shows the value of forcing users to use sudo instead of allowing them to do everything from the root shell. If I had done this from the root shell, the authentication log would have shown where I logged in as the root user, but it wouldn't have shown anything I did as the root user. With sudo, every root-level action gets logged, along with who did it.

There are several ways to obtain specific information from the log files. These include the following:

• Using the search feature of the less utility, as I mentioned earlier
• Using grep to search for text strings through either one file or multiple files at once
• Writing scripts in languages such as bash, Python, or awk

Here's an example of using grep:

sudo grep 'fail' syslog

In this case, I'm searching through the syslog file for all lines that contain the text string, fail. By default, grep is case-sensitive, so this command won't find any instances of fail with uppercase letters. Also, by default, grep finds text strings that are embedded within other text strings. So, in addition to just finding fail, this command will also find failed, failure, or any other text string that contains the text string fail. 

To make the search case-insensitive, add the -i option, like so:

sudo grep -i 'fail' syslog

This will find all forms of fail in either uppercase or lowercase letters. To only search for the fail text string, and to exclude where it's embedded in other text strings, use the -w option, like so:

sudo grep -w 'fail' syslog

You can combine the two options like this:

sudo grep -iw 'fail' syslog

In general, if you don't know exactly what you're looking for, start off with a more generic search that will probably show you too much. Then, narrow things down until you find what you want.

Now, this is all good when you just want to search through the log files for specific information. But it's rather tedious when you need to do your daily log review. Later on, I'll show you a tool that will make that much easier. For now, let's look at the binary log files.