An administrator running a network needs to understand what is happening on that network, making an understanding of logs essential. Not only do logs help determine how the network is functioning, they can also provide clues to the activities of malicious actors on a network. However, because an attacker that gains root or administrative privileges can modify any logs saved on that system, an administrator needs to know how to set up a distributed logging system so that logs on one system are stored on a different system.
This chapter starts with the basics of logging on Linux, including the syslog standard and a brief introduction to the common daemons (syslog and rsyslog) that were commonly used between 2011 and 2017. The chapter also describes systemd-journal, which is the logging service provided by systemd. The reader will learn how to configure each, both for local logging and as part of a distributed logging system. Different techniques to enable the spoofing of log entries locally and over the network are provided. The auditd daemon can provide detailed information about file access, program execution, and network connections.
Windows uses a fundamentally different approach to logging. Windows uses audit policies to determine what is to be recorded, and the chapter covers how to configure these using the Advanced Audit Policy Configuration tools. Windows logs can be queried with the built-in Event Viewer tool; they can also be queried with PowerShell scripts. Windows includes tools that can be used to view the logs on other Windows systems; it also can use subscriptions to aggregate logs from different systems in one location. Examples of group policy for these alternatives are developed.
The open source tool NXLog is introduced; NXLog can be configured to forward logs on a Windows system to a Linux system using syslog.
Logging in Linux
Older Linux system use syslog as their preferred method for managing logs. Later systems that use systemd use systemd-journald but generally retain their syslog server, in part to provide features that are not yet available in systemd-journald.
Syslog
Syslog has been an informal standard for many years; the syslog protocol was codified in RFC 3164 and then updated in RFC 5424. Syslog is used on most, but not all, Linux systems. A syslog daemon is used by CentOS, Mint, and Ubuntu; a syslog daemon is also used by default on OpenSuSE systems up through OpenSuSE 13.1; on OpenSuSE 13.2 and later, it is available but not installed by default. There are different daemons that implement the syslog standards, including syslog and rsyslog. Syslog is centered around messages that are issued by programs or other systems and stored either locally or on other systems.
Syslog Messages
Syslog messages contain a plain text message, a timestamp, and either the hostname or the IP address of the sending system. They include two additional values - a facility, which identifies the type of message; and a priority, which determines its importance. Allowable facilities include
auth: Used for security/authorization messages. (Code 4)
authpriv: Used for security/authorization messages. Also known as security, though that name is deprecated. (Code 10)
cron: Used for the cron scheduler. (Code 9)
daemon: Used for system daemons without separate facility values. (Code 3)
ftp: Used for the ftp server. (Code 11)
kern: Used solely for kernel messages. (Code 0)
local0, local1, ..., local7: Available for local system use. (Codes 16-23)
lpr: Used for the print subsystem. (Code 6)
mail: Used for the mail server. (Code 2)
news: Used for the news server (NNTP; see, e.g., RFC 977). (Code 7)
syslog: Used for messages generated by the log server itself. (Code 5)
user: Default facility; used for generic messages. (Code 1)
uucp: Used for the (now obsolete) Unix to Unix Copy system (UUCP). (Code 8)
The priorities are, in decreasing order of severity:
emerg (emergency) (Code 0)
alert (Code 1)
crit (critical) (Code 2)
err (error) (Code 3)
warning (Code 4)
notice (Code 5)
info (informational) (Code 6)
debug (Code 7)
Log messages can be stored locally in files, broadcast to all users, or sent over the network to one or more receiving hosts. Here is a short snippet of the log file /var/log/syslog on an Ubuntu 13.10 system showing three typical log messages.
Jul 30 12:48:55 crescent anacron[887]: Job `cron.weekly' started
Jul 30 12:48:55 crescent anacron[2562]: Updated timestamp for job `cron.weekly' to 2017-07-30
These messages were sent by the (local) host named crescent in the early afternoon of July 30, and all are related to the anacron scheduler.
Different daemons are used to implement syslog logging on Linux systems. The most common daemon in use on CentOS/Mint/OpenSuSE/Ubuntu is rsyslog. The exceptions are the various CentOS 5 systems that use the older syslog daemon.
Configuring the rsyslog Daemon on CentOS 6
As an example of an rsyslog configuration, consider a CentOS 6.8 system. On this distribution, the primary configuration file for rsyslog is /etc/rsyslog.conf. That file contains four main sections: a list of loaded modules, a list of global directives, a list of rules, and a collection of forwarding rules. The global directives section includes a line to include *.conf files contained in the subdirectory /etc/rsyslog.d. The rules section has the content shown in Listing 10-1.
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
This sends messages from any facility of priority level info or higher to /var/log/messages, with the exceptions that no messages from the mail, authpriv, or cron facility should be sent.
The directive for mail has the structure
mail.* -/var/log/maillog
This sends all messages from the mail facility to the file /var/log/maillog. The dash before the file name indicates that the file does not need to be synced after every write. Automatic file sync has been disabled by default since version 3 of rsyslogd, while CentOS 6.8 uses version 5.8.10, so the dash here has no effect. This default behavior can be changed by uncommenting the corresponding directive in the global section of /etc/rsyslog.conf, which has the default content
# File syncing capability is disabled by default. This feature is usually not required, not useful and an extreme performance hit
#$ActionFileEnableSync on
Most of the destinations listed are local files; one exception is the destination for messages of priority emergency (*.emerg) where the destination is simply “*”. These messages are sent to all users using wall. The wall command can also be used outside of syslog; for example:
Configuring the rsyslog Daemon on Other Distributions
Configuration of rsyslog on CentOS 7 follows the same lines; the primary configuration file remains /etc/rsyslog.conf that is broken into the same four sections. The global directives section includes the files /etc/rsyslog.d/*.conf. The rules section is essentially the same, though the lines for emergency messages have been modified.
# Everybody gets emergency messages
*.emerg :omusrmsg:*
This has the same effect as the older directive to send messages to all users; only the syntax has been updated.
On OpenSuSE systems, the situation varies with the release. On older systems like OpenSuSE 12.1, configuration is spread across three files. The primary file is /etc/rsyslog.conf, while /etc/rsyslog.d/remote.conf contains configuration information for remote logging, and /etc/rsyslog.early.conf contains those statements for rsyslog that are safe to run before the network or remote file systems become available. On later systems like OpenSuSE 13.1, there are only two configuration files; the file rsyslog.early.conf is not present. On OpenSuSE 13.2 and later, the rsyslog daemon is not installed by default, and the logging is handled solely by systemd-journald.
When OpenSuSE uses rsyslog, it sends files to different locations than CentOS. For example, on OpenSuSE 13.1, the configuration file /etc/rsyslog.conf contains the series of directives
mail.* -/var/log/mail
mail.info -/var/log/mail.info
mail.warning -/var/log/mail.warn
mail.err /var/log/mail.err
These send mail related messages to different files depending on the message’s priority. It also contains the directive
*.*;mail.none;news.none -/var/log/messages
This ensures that most log messages are sent to /var/log/messages.
On Mint and Ubuntu systems, the rsyslog daemon is configured in three files. The main file is /etc/rsyslog.conf; this includes the files /etc/rsyslog.d/*.conf. There are two files in that subdirectory; the first is 20-ufw.conf that configures how the system handles log messages sent by the firewall UFW. The other, 50-default.conf, is similar to the rules section from Listing 10-1. One major difference is the location of the logs; this file contains directives of the form
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
On Mint and Ubuntu systems, the default primary log file is /var/log/syslog.
Changes to syslog configuration can be made by changing the appropriate configuration file; for example, to configure Mint 18.1 to log most messages to /var/log/messages add the line below to /etc/rsyslog.d/50-default.conf
After this change, most log messages have two different destinations: /var/log/syslog and /var/log/messages. When a log message can be sent to more than one destination, all destinations receive the log message.
Configuring the syslog Daemon
CentOS 5 systems use the older syslog daemon. The configuration file is /etc/syslog.conf. The contents of that file match what has been presented from the rules section of the CentOS 6.8 configuration file /etc/rsyslog.conf in Listing 10-1 and can be customized in substantially the same ways.
Systemd-journald
Most distributions that use systemd also use systemd-journald to store logs on the system. Distributions that have taken this approach include CentOS 7.0 and later, Mint 18 and later, OpenSuSE 12.3, and later, and Ubuntu 15.04 and later.
Like syslog and rsyslog, systemd-journald classifies messages by facility and priority, using the same values. Unlike syslog and rsyslog, systemd-journald does not use plain text files to store data; instead the data is stored in binary format. As such, specialized tools are needed to read the logs.
Reading systemd-journald Logs with journalctl
Running the command journalctl without any flags provides a paged view of the systemd-journald logs available on the system, beginning with the oldest records. On a Mint 18 system, for example, the result is
jmaxwell@kalliope ~ $ sudo journalctl
-- Logs begin at Tue 2017-08-01 20:27:41 EDT, end at Tue 2017-08-01 20:31:55 EDT
Aug 01 20:27:41 kalliope systemd-journald[324]: Runtime journal (/run/log/journal/) is 1.2M, max 10.0M, 8.7M free.
Aug 01 20:27:41 kalliope kernel: Initializing cgroup subsys cpuset
Aug 01 20:27:41 kalliope kernel: Initializing cgroup subsys cpu
Aug 01 20:27:41 kalliope kernel: Initializing cgroup subsys cpuacct
Aug 01 20:27:41 kalliope kernel: Linux version 4.4.0-21-generic (buildd@lgw01-06 (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu
Aug 01 20:27:41 kalliope kernel: KERNEL supported cpus:
Aug 01 20:27:41 kalliope kernel: Intel GenuineIntel
Aug 01 20:27:41 kalliope kernel: AMD AuthenticAMD
Aug 01 20:27:41 kalliope kernel: NSC Geode by NSC
lines 1-10
This pager view can be navigated with the arrow keys, page up/down keys, or the spacebar.
It is possible to filter the logs before they are displayed by the pager. To filter by time, the user can use the options --since or --until. As an example, the user can try
-f to follow the logs and update the result with new log entries as they occur.
As an example, OpenSuSE 13.2 uses systemd-journald and runs the SSH server sshd by default. The user can see just these logs with the command
merak:~ # journalctl -u sshd
The paged output has the structure
Aug 01 20:28:34 merak sshd-gen-keys-start[1321]: Checking for missing server keys in /etc/ssh
Aug 01 20:28:36 merak sshd[1405]: Server listening on 0.0.0.0 port 22.
Aug 01 20:28:36 merak sshd[1405]: Server listening on :: port 22.
As another example, Ubuntu 16.04 also uses systemd-journald and the server sshd can be installed. However, the administrator cannot use the -u switch to check for SSH log entries; indeed, doing so leads to the result
-- Logs begin at Fri 2018-02-16 19:38:42 EST, end at Fri 2018-02-16 20:11:01 EST. --
Feb 16 19:38:55 elpis sshd[877]: Server listening on 0.0.0.0 port 22.
Feb 16 19:38:55 elpis sshd[877]: Server listening on :: port 22.
... Output deleted ...
As another example, suppose the administrator wants to read the log entries associated with the user egalois. They first find the UID with the command ID
-- Logs begin at Sat 2017-01-14 15:52:05 EST, end at Tue 2017-08-01 22:00:01 EDT. --
Jan 14 15:54:06 linux-zmw6 pulseaudio[2018]: [pulseaudio] alsa-util.c: Disabling timer-based scheduling because runing inside a VM.
Jan 14 15:54:06 linux-zmw6 pulseaudio[2018]: [pulseaudio] sink.c: Default and alternate sample rates are the same.
Jan 14 15:54:06 linux-zmw6 pulseaudio[2018]: [pulseaudio] alsa-util.c: Disabling timer-based scheduling because running inside a VM.
Non-root users can also query the logs using journalctl.
Configuring systemd-journald
The primary configuration file for systemd-journald is the file /etc/systemd/journald.conf. As an example, on CentOS 7.2-1511, this file has the content shown in Listing 10-2.
... Output Deleted ...
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.
[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
Listing 10-2
A portion of the file /etc/systemd/journald.conf on CentOS 7.2-1511
The directives in this file have been commented out, as they represent the default settings. Two settings in this file are of particular importance. The first of these is the value of the Storage parameter, which here is set to auto. With this setting, log data is stored in the directory /run/log/journal. This data is kept only until the system reboots; then the old log data is deleted, and new logs created. This is the same behavior if Storage were set to volatile. When Storage is set to auto though, there is an additional complication. If the directory /var/log/journal exists, then systemd-journald will store its logs in this directory instead. Moreover, if this directory exists, then log data will be kept across system reboots.
The second key directive is ForwardToSyslog, which on this CentOS 7.2-1511 system is set to no. Logs sent through systemd-journald on CentOS 7.2-1511 are not also copied to rsyslog. On Mint and Ubuntu systems, this is generally set to yes, so both systemd-journald and rsyslog record the logs. This is also the case for some OpenSuSE systems. However, on the most recent OpenSuSE releases, although ForwardToSyslog is set to yes in /etc/systemd/journald.conf, the rsyslog daemon is not actually running, so nothing is done with the forwarded logs.
Systemd-journald data is kept only until the system reboots. Copies of these logs are sent to rsyslog, which are kept beyond system reboots.
OpenSuSE 12.3, 13.1
Storage=auto
The directory /var/log/journal does not exist,
ForwardToSyslog=yes
Systemd-journald data is kept only until the system reboots. Copies of these logs are sent to rsyslog, which are kept beyond system reboots.
OpenSuSE 13.2, 42.1, 42.2, 42.3
Storage=auto
The directory /var/log/journal exists,
ForwardToSyslog=yes
Systemd-journald data is kept across system reboots. The rsyslog service is not running and records no logs.
To change the configuration, the administrator needs to edit the configuration file /etc/systemd/journald.conf, then restart the service with a command like
wei:~ # systemctl restart systemd-journald
wei:~ # journalctl --since "1 minute ago"
-- Logs begin at Sat 2017-01-14 16:00:55 EST, end at Wed 2017-08-02 21:31:35 EDT. --
Aug 02 21:31:35 wei systemd-journal[2502]: Journal stopped
Aug 02 21:31:35 wei systemd-journal[2519]: Permanent journal is using 120.0M (max allowed 1.4
Aug 02 21:31:35 wei systemd-journal[2519]: Journal started
Spoofing Log Messages
An unprivileged local user can generate arbitrary fake or malicious logs. Consider the tool logger, which is installed on most Linux systems.
egalois@wei:~> man logger
LOGGER(1) User Commands LOGGER(1)
NAME
logger - a shell command interface to the syslog(3) system log module
SYNOPSIS
logger [options] [message]
DESCRIPTION
logger makes entries in the system log. It provides a shell
command interface to the syslog(3) system log module.
Using this tool, a user can craft log messages with specified facilities and priorities.
egalois@wei:~> logger -p kern.alert "I can write my own log entries?"
This is true whether the system uses syslog or systemd-journald.
egalois@wei:~> journalctl --since "1 minute ago"
-- Logs begin at Sat 2017-01-14 16:00:55 EST, end at Wed 2017-08-02 21:17:08 EDT. --
Aug 02 21:17:08 wei egalois[2244]: I can write my own log entries?
Attackers that find this interface insufficiently flexible can write programs that directly interact with the logging system. Documentation to do so is included in the man (3) page for syslog. Consider the C program shown in Listing 10-3.
#include<syslog.h>
int main(int argc, char* argv[])
{
const char log_ident[] = "named [31337]";
const int log_option = LOG_NDELAY ;
const int log_facility = LOG_SYSLOG;
openlog(log_ident, log_option, log_facility);
syslog(LOG_CRIT, "I just experienced a critical error!");
closelog();
return(0);
}
Listing 10-3
C code log.c; a program to send a custom local syslog message
After this program is run, it may appear to a system administrator that the named process, with PID 31337, just had a critical error. On a CentOS 6.8 system for example, one has
Aug 2 21:27:40 scheat named [31337]: I just experienced a critical error!
Systemd-journald is more careful about distinguishing the log data from the source of the data. If the same program is run on OpenSuSE 42.1, then the log reads as follows.
egalois@wei:~> journalctl --since "1 minute ago"
-- Logs begin at Sat 2017-01-14 16:00:55 EST, end at Wed 2017-08-02 21:20:10 EDT. --
Aug 02 21:20:10 wei log[2327]: named [31337]: I just experienced a critical error!
auditd
Chapter 3 showed how to install and use aureport to search for login attempts. The aureport tool uses the results generated by the auditd daemon. The auditd daemon can be configured to collect more data; for example, it can be used to record changes to key system files, note when programs are executed from directories, or even log all outbound network connections.
Recall that not every system includes auditd in the default installation; for example, on Ubuntu systems, it needs to be installed with the command
jmaxwell@siegena:~$ sudo apt-get install auditd
On OpenSuSE 12.3 system, the corresponding installation command is
menklent:~ # zypper install audit
Creating Auditd Rules
The default configuration files for auditd are in the directory /etc/audit. The location of the audit file is specified in /etc/audit/audit.conf, and generally this has the default value /var/log/audit/audit.log. The rules are specified in the file /etc/audit/audit.rules. On some systems, like CentOS 7.0-1406, this file is generated automatically from the content in /etc/audit/rules.d; that directory has a single file, /etc/audit/rules.d/audit.rules.
As an example of a typical rule file /etc/audit/audit.rules, Listing 10-4 shows the situation on Mint 17.
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.
# First rule - delete all
-D
# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 320
# Feel free to add below this line. See auditctl man page
Listing 10-4
The default configuration file /etc/audit/rules.d/audit.rules on Mint 17
As the configuration file explains, each line is an auditing rule, and the structure of the line matches the flags that would be passed to auditctl on the command line. Allowable flags include the following:
-w path Watch file system object at this path.
-p[r|w|x|a] The permissions that will trigger the event - read, write, execute, or attribute change.
-S syscall Watch for a kernel system call. The syscall can be specified by name or by number.
-F [option=value] Options for a syscall. The arguments to the syscall are passed as a0, a1, a2, a3. The system’s architecture is passed as arch, with arch=b64 for 64-bit systems and arch=b32 for 32-bit systems. Other options include the system’s exit code from the syscall (exit), the effective user ID (euid), the effective group IP (egid), and the process ID (pid).
-k A keyword associated with this rule.
For example, suppose that the administrator wants to generate an audit entry any time the file /etc/shadow is changed. Then they can add the following line to /etc/audit/audit.rules
-w /etc/shadow -pw -k change_in_shadow_file
To generate an entry every time a program in the directory /tmp is run, the administrator can add the rule
-w /tmp -px -k execution_in_tmp
To log all outbound network connections, one approach the administrator can take is to log every time the system makes a system call that uses the network. The audited syscall depends on the system’s architecture. On 32-bit systems, outbound network connections begin with socketcall. The first argument of socketcall is the type of network call; for a call to connect() the first argument to socketcall has the value 3. The rule to log these connections then is
-a always,exit -F arch=b32 -S socketcall -F a0=3 -k outbound_connection
On a 64-bit system, the socket syscall is used. The first argument to a connect syscall is the connection type; for an IPv4 connection, this has the value 2, while IPv6 has the value 10. The second argument is the socket type; type 1 is for TCP connections and type 2 for UDP. The rule to log outbound IPv4 TCP connections then is
In each case, the required argument -a always,exit is needed to ensure that this rule is added to the syscall exit list and an entry generated every time the syscall is made.
Once the changes have been made in /etc/audit/audit.rules, the administrator can restart the audit service so that the new rules are used.
[root@enif ~]# service auditd restart
Stopping logging: [ OK ]
Redirecting start to /bin/systemctl start auditd.service
The collection of the running rules can be found by running the command
Errors in the syntax of audit rules will be noted in the system logs; they may not be reported when the service is restarted.
Pre-built Rules
The auditd system also includes some sample rule sets. On an Ubuntu 16.04 system, for example, the directory /usr/share/doc/auditd/examples contains a rule set based on the Controlled Access Protection Profile (CAPP), another rule set based on the Labeled Security Protection Profile (LSPP), another rule set based on the National Industrial Security Program Operating Manual Chapter 8 (NISPOM), and one based on the Secure Technical Implementation Guide (STIG). To implement one of these pre-built rule sets, the rules need to be uncompressed.
Before being used, the rule set should be carefully read and properly configured. For example, the STIG rule set on Ubuntu 16.04 needs to have the 32-bit rules commented out if the rules are to be used on a 64-bit system. When the process is complete, the edited file can be used in place of /etc/audit/audit.rules.
Mint stores the sample rules in the same directory as Ubuntu systems. On CentOS systems, these sample rules are available in the directory /usr/share/doc/audit-xx.yy.zz where xx.yy.zz matches the installed version of auditd. On OpenSuSE systems, they are in the directory /usr/share/doc/packages/audit.
Reading the Audit Log
Suppose that the administrator uses the custom rule for the change in the shadow file. Attempts to add users to the system will be recorded in the audit log file, which by default is /var/log/audit/audit.log. That file would have content of the following general form.
Here the administrator can see that the change in the shadow file was caused by the program /usr/sbin/useradd running as the root user (uid=0).
If the administrator uses the custom rule for generating entries in the log whenever a program in /tmp is run, then an auditd log entry of the following form will be generated.
In this example, the administrator can see that the program /usr/bin/python2.7 with PID 8175 was used with the argument program.py by the user with ID=1000. The id command shows that this is the user cgauss.
Suppose that the administrator also loaded the rule to log outbound TCP connections and that a user launched Firefox then browsed to http://google.com. A typical audit log entry on a 64-bit system might have the form
Manually reading through the many log entries recorded by auditd can be time consuming. One approach is to use the ausearch command. The ausearch command can be run with the -k flag to specify the keyword. In the example, the rule to watch the file /etc/shadow had the keyword change_in_shadow_file; the administrator can run
This shows the first two of a long sequence of audit log entries created when the root user updated the shadow file and created a new user. These can be grouped in a more readable format by sending the output to aureport.
49. 08/15/2017 23:42:15 /etc/gshadow.8577 open yes /usr/sbin/useradd cgauss 5326
Remote Logging
The syslog daemons (syslog and rsyslog) allow for logs to be sent to remote destinations. Support for remote logging is not provided by systemd-journald.
Sending Logs with Syslog
To configure either syslogd or rsyslogd to send logs to a remote system over the default UDP/514, in the configuration file (Listing 10-1) instead of providing a file name as a destination, provide the IP address of the destination system, preceded by “@”. Consider the directive
*.* @10.0.2.99
This sends all messages, regardless of facility or priority, to the host 10.0.2.99 via UDP/514. Add this line to /etc/rsyslog.d/50-default.conf on an Ubuntu 14.04 system and restart the rsyslog service. A subsequent Wireshark capture shows the syslog data in transit.
Notice that the traffic is sent in plain text, unencrypted. If a netcat listener is running on the target (and the proper port is opened in the firewall), then it receives the log messages.
[root@scheat ~]# nc -l -u -v 514
Connection from 10.0.3.48 port 514 [udp/syslog] accepted
The first component of the received syslog message, <85>, represents the facility and the priority for the message. It is formed by multiplying the code number for the facility by eight and adding the priority. In this example, the facility is authpriv (code 10) and the priority is notice (code 5) yielding 85. Wireshark parsed this code for the user and displayed it as part of the packet capture.
One disadvantage of using UDP as a protocol is that data transfer is unreliable. The rsyslog (but not syslog) daemon permits the user to send logging data to remote hosts using TCP. To send log messages via TCP/514 on an rsyslog-based system like Ubuntu 14.04, add the directive below to /etc/rsyslog.d/50-default.conf and restart the daemon.
*.* @@10.0.2.99
The doubled @@ symbols tell rsyslog to use TCP.
On rsyslog (TCP or UDP), the port number is specified by appending a colon and the port number to the IP address. For example, the directive below sends logs to 10.0.2.28 via TCP/1514.
* @@10.0.2.28:1514
Note that on CentOS systems, if SELinux is in enforcing mode, then by default it blocks attempts by rsyslog to send data via TCP on ports other than 514.
Whether the administrator is using TCP or UDP, an attacker on the local network that can sniff the traffic would be able to read these log entries as they traverse the network.
Receiving Logs with syslog
Not only can syslog daemons send logs to remote sites, they can be configured to process the results, storing the results locally in files or forwarding them on to other hosts. On a system running rsyslog (like a Mint 17.2 system), to allow rsyslog to receive log messages from UDP/514, uncomment the following lines in /etc/rsyslog.conf
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
To allow rsyslog to receive log messages from TCP/514, uncomment the lines
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
In each case, the appropriate port must be opened in the firewall and the rsyslog service needs to be restarted. The situation for other rsyslog systems is similar; the preferred configuration file for OpenSuSE is /etc/rsyslog.d/remote.conf while for Ubuntu and CentOS it is /etc/rsyslog.conf.
Some systems, including Ubuntu 12.04, suffer from a bug where they are unable to use rsyslog to listen on TCP ports less than 1024; attempts to do so fail with the log entry
Oct 11 10:12:17 Bubble rsyslogd-2077: Could not create tcp listener, ignoring port 514. [try http://www.rsyslog.com/e/2077 ]
The syslog daemon can be used to process log files received remotely, though solely through UDP/514. This is controlled through a flag passed to the daemon on program start. On CentOS 5, the file /etc/sysconfig/syslog contains these flags. To do so, include the -r flag in SYSLOG_OPTIONS.
# Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
SYSLOGD_OPTIONS="-r -m 0"
Save the file and restart the daemon.
Spoofing Remote Logs
Systems that accept remote logs are at an even greater danger of receiving spoofed log entries. Suppose that the host 10.0.2.32 is listening for logs on UDP/514. Consider the Python script shown in Listing 10-5.
#!/usr/bin/python
from scapy.all import IP,UDP,Raw,send
import time
priority = 3 # error
facility = 1 # user
code = '<' + str(8 * facility + priority) +'>'
timestamp = time.strftime("%b %d %H:%M:%S")
message = "Host named [31337] I just experienced a critical error"
Python script log_spoof.py that sends syslog messages to a target, spoofing the source IP
When run, this sends a properly formatted syslog message to the target spoofing the source address as 10.0.2.32 (which is not the IP address of the sending system!) The receiving log server records the entry
[root@alderamin ~]# tail -n1 /var/log/messages
Aug 3 22:22:09 Host named [31337] I just experienced a critical error.
An attacker that has identified the log server(s) for a network can try to hide their activity by sending noise to hide in. Another option is to send enough data to exhaust the storage capacity of the log server(s).
Log Rotation
Logs cannot be kept indefinitely; as they continue to expand in size, they will eventually consume all system resources. This is the case even if attackers are not deliberately trying to fill the logs. The logrotate tool is used on Linux systems to compress, archive, rotate, and delete log files generated by syslog or rsyslog. Configuration directives for logrotate are contained in the file /etc/logrotate.conf. As a typical example, consider the portion of that file on a CentOS 6.6 system shown in Listing 10-6.
# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
dateext
# uncomment this if you want your log files compressed
#compress
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0600 root utmp
rotate 1
}
# system-specific logs may be also be configured here.
Listing 10-6
The file /etc/logrotate.conf on CentOS 6.6
Logs are rotated each week, and four weeks of older logs are kept, uncompressed. Additional directives for individual log files are provided by files in the directory /etc/logrotate.d; these can override the default values in /etc/logrotate.conf.
The logrotate tool itself is called by a cron job; on the example CentOS 6.6 system, the actual script is /etc/cron.daily/logrotate.
Log Rotation with systemd-journald
When systemd-journald is used to store log files permanently by setting Storage=auto in /etc/systemd/journald.conf on a system where the directory /var/log/journal exists, then systemd-journald distinguishes between the runtime journal and the permanent journal. The latter contain the permanently stored logs; they are limited in size to 10% of the size of the partition that contains the permanent journal.
Consider an example OpenSuSE 42.1 system with a 15G drive.
wei:~ # df -h /var/log/journal/
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 15G 6.4G 7.9G 45% /var/log
A check of systemd-journald shows that the maximum size of the permanent journal is 1.4G.
Active: active (running) since Thu 2017-08-03 23:30:38 EDT; 19min ago
Docs: man:systemd-journald.service(8)
man:journald.conf(5)
Main PID: 397 (systemd-journal)
Status: "Processing requests..."
CGroup: /system.slice/systemd-journald.service
└─397 /usr/lib/systemd/systemd-journald
Aug 03 23:30:38 wei systemd-journal[397]: Runtime journal is using 6.2M (max allowed 49.6M, trying to leave 74.4M free of 490.3M available ➤ current limit 49.6M).
Aug 03 23:30:38 wei systemd-journal[397]: Runtime journal is using 6.2M (max allowed 49.6M, trying to leave 74.4M free of 490.3M available ➤ current limit 49.6M).
Aug 03 23:30:38 wei systemd-journal[397]: Journal started
Aug 03 23:30:44 wei systemd-journal[397]: Permanent journal is using 120.0M (max allowed 1.4G, trying to leave 2.1G free of 7.8G available ➤ current limit 1.4G).
Aug 03 23:30:46 wei systemd-journal[397]: Time spent on flushing to /var is 1.578035s for 454 entries.
The maximum size of the permanent journal can be controlled through the option SystemMaxUse in /etc/systemd/journald.conf; see the man page for journald.conf for details.
Logging in Windows
Windows systems take a fundamentally different approach to logging. The primary tool for viewing logs on Windows systems is Event Viewer (Figure 10-1). On Windows Server 2008 R2, launch it by navigating Start ➤ Administrative Tools ➤ Event Viewer. On Windows Server 2012, 2012 R2, and 2016, Event Viewer is available from the tools menu on Server Manager. On Desktop systems like Windows 7 and 8, Event Viewer can be started from the Control Panel, navigating through System and Security. It can also be run from the command line, as eventvwr.msc.
Figure 10-1
Windows Event Viewer on Windows 2012 R2
There are four primary Windows logs - application logs, security logs, system logs, and setup logs. The setup log contains information from the system installation process. The system log records data from Windows itself, while the application log is used by programs. Events in the security log are often called audits, and are generated by a range of security events, including logon/logoff, privilege usage, and object access. Applications and services have their own logs separate from the Windows logs.
Audit policies determine what is recorded in the security log. These can be configured in three different ways: via local policy, via group policy, and directly from the command line with the tool auditpol.exe. To modify local security policy, from the Control Panel, navigate to System and Security ➤ Administrative Tools ➤ Local Security Policy (Figure 10-2).
Figure 10-2
Local security policy on Windows 10
There are two types of audit policy settings, the basic policies in Security Settings ➤ Local Policies ➤ Audit Policy, and the advanced settings in Security Settings ➤ Advanced Audit Policy Configuration ➤ System Audit Policies. These settings are handled differently, and changes should not be made in both locations; indeed, Microsoft goes so far as to say “Using both advanced and basic audit policy settings can cause unexpected results.”3
To configure Advanced Audit Policy from an administrator command line, use the tool auditpol. To see the categories and their current effective settings, run the following command.
C:Windowssystem32>auditpol /get /category:*
System audit policy
Category/Subcategory Setting
System
Security System Extension No Auditing
System Integrity Success and Failure
IPsec Driver No Auditing
Other System Events Success and Failure
Security State Change Success
Logon/Logoff
Logon Success
Logoff Success
Account Lockout Success
IPsec Main Mode No Auditing
IPsec Quick Mode No Auditing
IPsec Extended Mode No Auditing
Special Logon Success
Other Logon/Logoff Events No Auditing
Network Policy Server Success and Failure
User / Device Claims No Auditing
Group Membership No Auditing
... Output Deleted ...
Account Logon
Kerberos Service Ticket Operations No Auditing
Other Account Logon Events No Auditing
Kerberos Authentication Service No Auditing
Credential Validation No Auditing
Changing policies from the command line can be accomplished with commands like
Advanced audit policies can also be changed via group policy. From the Group Policy Editor, navigate Computer Configuration ➤ Policies ➤ Windows Settings ➤ Security Settings ➤ Advanced Audit Policy Configuration ➤ Audit Policies (Figure 10-3).
Figure 10-3
Advanced audit policy configuration from group policy on Windows Server 2012 R2
Viewing Windows Logs
Windows Event Viewer (Figure 10-1) provides a reasonable interface to the various Windows logs, allowing searches and filtering. Suppose that an administrator wants to quickly review the logs to determine if a new user has been added to the system. One way to do so is to create a Custom View. From the action pane in Windows Event Viewer (Figure 10-1), select “Create Custom View.” The administrator looks for events logged at any time in the Windows Security Log. The Event ID for user creation is 4720, and the administrator is looking for successful events. The configuration of the resulting Custom View has the format in Figure 10-4.
Figure 10-4
Creating a custom view in Event Viewer on Windows Server 2012 R2
The resulting custom view can be named and saved, for example, as Security Log Event 4720. A check of the navigation pane on Figure 10-1 shows this saved in the list of custom views. Notice that the view shows that in March 2017, several new domain accounts were created.
Using PowerShell to View Logs
Although custom views in Event Viewer are useful, there are times when an administrator needs more control over how the logs are parsed. One excellent way to search the logs is by using PowerShell.
The following PowerShell script (Listing 10-7) searches the security logs for any instance of the string “A user account was created” after the indicated date.
$start = get-date 3/8/2017
$secevents = get-eventlog -logname Security -Message "*A user account was created*" -after $start
$secevents | format-list -property *
Listing 10-7
Powershell script Logs.ps1, used to search the security log for events containing the phrase “A user account was created” after a given start date
When this script is run by an administrator, the log entries for the new account are displayed to the screen.
This entry shows the creation of an account for the user tstafford. A subsequent search of the Security Logs can be made with a PowerShell script like Listing 10-8.
PowerShell script LogSearch.ps1 to search the security log for events from the user tstafford after a given date
This finds the log entries with the new user tstafford and stores them in a plain text file for subsequent analysis.
PsLogList
Chapter 8 demonstrated a brute force attack against the password of a domain user, where 88,396 passwords were used by the Metasploit module auxiliary/scanner/smb/smb_login to try to log in to a known domain administrator account. This attack leaves traces across the Windows logs.
Failed login attempts are recorded by the Security log as Event ID 4625. It is possible to use PowerShell to look through the logs for such events, but a brute force attack leaves many such events. Instead, a better approach is simply to count the number of failed login attempts on a given day. This can be done quickly and easily with the Sysinternals tool psloglist.exe.
PsLoglist v2.71 - local and remote event log viewer
Copyright (C) 2000-2009 Mark Russinovich
Sysinternals - www.sysinternals.com
9
This command looks through the security log (the -s switch) for events with id 4625 (the -i switch) that occurred before 6/3/2017 (the -b switch) and on or after 6/2/2017 (the -a switch). This is then piped to the find command, which counts (/c) the number of times the null string "" does not appear (/v). Effectively, this counts the number of lines in the output. The first line states the source of the log, so this result shows that there were 8 failed login attempts on 6/3/2017. The brute force attack the next day is easily spotted.
PsLoglist v2.71 - local and remote event log viewer
Copyright (C) 2000-2009 Mark Russinovich
Sysinternals - www.sysinternals.com
88400
The PowerShell cmdlet Get-EventLog can be used to find the entries in the security log after 6/3/2017 and before the end of 6/4/2017 of type 4625 (Failed log on attempt), then grab the first one, and show the details in this log entry.
This event is generated when a logon request fails. It is generated on the computer where access was attempted.
... Output Deleted ...
The defender now knows that a brute force attack appears to have been launched from 10.0.2.2, targeting the domain administrator jbach.
Clearing Logs
An attacker with administrative privileges can clear logs using PowerShell via the command
PS C:Windowssystem32> Clear-EventLog -log Application, Security, System
This clears the application, security, and system logs. A subsequent check of the security log shows that it contains a single entry with EventID 1102 indicating that the log was cleared.
1374 Oct 10 16:40 SuccessA... Microsoft-Windows... 1102 The audit log was cleared...
Defenders can examine the security logs for EventID 1102; this can be done via Event Viewer (Figure 10-1) or via a PowerShell command.
PS C:Windowssystem32> Get-EventLog -LogName Security | where {$_.eventID -eq 1102}
Index Time EntryType Source InstanceID Message
----- ---- --------- ------ ---------- -------
1374 Oct 10 16:40 SuccessA... Microsoft-Windows... 1102 The audit log was cleared...
Creating Logs
A user can use the command eventcreate to create entries in the various logs. For example, to create an entry in the application log of type error with ID 500, a user can run
C:Usersanders>eventcreate /L Application /T ERROR /ID 500 /D "This is a custom error"
SUCCESS: An event of type 'ERROR' was created in the 'Application' log with 'EventCreate' as the source.
810 Aug 12 18:57 Error EventCreate 500 This is a custom error
Auditing File Access
Like auditd on Linux, Windows can generate log entries when selected files are accessed / modified / changed. To illustrate the process, create a file, say on the Desktop named test.txt. Navigate test.txt (right-click) ➤ Properties ➤ Security ➤ Advanced ➤ Auditing, then authenticate (Figure 10-5). Click add to create an auditing entry. Each auditing entry has two components. The first is the collection of users that are being audited. It is important to be broad; if a user is not explicitly listed in an auditing entry, then their access remains unaudited. The second component of an auditing entry are the types of file access that are to be audited. These follow the usual Windows file permissions. Audits can be generated if a user successfully uses privileges on a file, or if a user attempts to access a file without the necessary permissions, or both.
Figure 10-5
Configuring auditing on a file on Windows 8.1
Once the changes have been made and applied to the file properties, make some changes to the file and check the security log for the results.
1489 Aug 11 19:20 SuccessA... Microsoft-Windows... 4616 The system time was changed....
1488 Aug 11 19:20 SuccessA... Microsoft-Windows... 4616 The system time was changed....
1487 Aug 11 19:20 SuccessA... Microsoft-Windows... 4634 An account was logged off....
1486 Aug 11 19:20 SuccessA... Microsoft-Windows... 4672 Special privileges assigned to new logon....
Though auditing has been correctly configured on the file, no entries appear in the security log. Although the process explained so far set the auditing policy for the file, Windows ignores those settings unless file-level auditing is enabled in the system’s audit policy. Verify that the required settings have not (yet) been enabled:
Another option is to configure this via group policy. To do so, create or edit a group policy; navigate Computer Configuration ➤ Policies ➤ Windows Settings ➤ Security Settings ➤ Advanced Audit Policies ➤ Audit Policies. Select Object Access ➤ Audit File System and select the desired setting.
Subsequent modification of the audited file yields the expected entries in the security log.
1509 Aug 11 19:22 SuccessA... Microsoft-Windows... 4663 An attempt was made to access an object....
1508 Aug 11 19:22 SuccessA... Microsoft-Windows... 4663 An attempt was made to access an object....
1507 Aug 11 19:22 SuccessA... Microsoft-Windows... 4656 A handle to an object was requested....
1506 Aug 11 19:22 SuccessA... Microsoft-Windows... 4656 A handle to an object was requested....
Rotating Windows Logs
Windows logs are kept at a fixed size. The system administrator determines what should occur when the full size is reached; either older events can be overwritten (the default), or the file can be archived, or the administrator can be required to manually clear the log. This is controlled through the properties of the log; these can be found in Event Viewer. Right-click on a log and select Properties to obtain a dialog box like Figure 10-6.
Figure 10-6
Properties of the security log on Windows Server 2016
Remote Windows Logs
A Windows network administrator generally has more than a single system to monitor. There are two ways the administrator can examine logs across a range of systems. One way is to store the logs on the individual machines and use the networking features of Event Viewer or PowerShell to examine the logs on a remote system. The other option is to collect the logs from multiple systems together on one or more log servers.
Remotely Viewing Logs
Following Chapter 7, it is possible to use Event Viewer on one computer to view the logs on a second computer provided Remote Procedure Calls are allowed between the systems. From Event Viewer, select Action ➤ Connect to Another Computer. Be sure to select Event Viewer (Local) in the navigation pane, or the option to connect to another computer will not appear in the Action menu. Enter the remote system name and the account details (if different) for the other machine. (See Figure 7-4)
Provided the Remote Registry service is also running on the remote system (cf. Chapter 7), it is also possible for an administrator to use PowerShell to view the logs on another computer by specifying the computer name in the command.
1912 Aug 12 08:22 SuccessA... Microsoft-Windows... 4634 An account was lo...
1911 Aug 12 08:22 SuccessA... Microsoft-Windows... 4634 An account was lo...
1910 Aug 12 08:22 SuccessA... Microsoft-Windows... 4624 An account was su...
1909 Aug 12 08:22 SuccessA... Microsoft-Windows... 4672 Special privilege...
Windows Event Collector
Suppose that an administrator wants to set up a Windows host that will collect logs from other Windows systems. To do so, the administrator first configures Windows Remote Management (WinRM), following the techniques in Chapter 7.
Next, on the system that will serve as the destination for the logs, the administrator starts Event Viewer and navigates to Subscriptions. The first time this is done, the administrator is provided with a dialog box (Figure 10-7) saying that that the Windows Event Collector Service must be running and configured; enable the service.
Figure 10-7
Windows 10 dialog box for the Windows Event Collector Service
Right-click on Subscriptions, and then select Create Subscription (Figure 10-8). Give the subscription a name and a description. The user can choose from the various logs as destinations, but the default destination, Forwarded Events, is a reasonable choice. For the subscription type, choose Collector Initiated, and select the source computer(s). Be sure to test the connection; if the connectivity test fails, then it is likely that there is a problem with the WinRM service or the firewall. Select the events that are to be forwarded; these can be filtered by the log or the source and can be further filtered by level, category, user, or keyword. By default, a machine account is used to connect to the remote computer to collect the logs; this account usually does not have sufficient privileges to do so. Press the advanced button and select a user and password that have such privileges. When the process is complete, right-click on the subscription and select its runtime status; no errors should be reported.
Figure 10-8
Configuration of a Windows subscription on Windows 10
Once the subscription is configured, test the configuration by using eventcreate to generate a log entry while on the remote system. Log entries in the Forwarded Events log can also be accessed via PowerShell using the cmdlet Get-WinEvent.
Sysmon is part of the Sysinternals suite. It provides detailed low-level logging of processes, files, registry use, and network connections.
To install Sysmon, the program is run with the flag -i.
c:Program FilesSysinternalsSuite>Sysmon.exe -i
System Monitor v6.00 - System activity monitor
Copyright (C) 2014-2017 Mark Russinovich and Thomas Garnier
Sysinternals - www.sysinternals.com
Sysmon installed.
SysmonDrv installed.
Starting SysmonDrv.
SysmonDrv started.
Starting Sysmon..
Sysmon started.
To uninstall it, the program is run with the -u flag. The default installation does not log network connections; running the program with the -n flag instructs Sysmon to log network connections. Configuration for Sysmon can be included in a configuration file; these can be included during installation as an argument to the -i flag or updated later as an argument to the -c flag. If Sysmon is run with the -c flag and no argument, the current configuration is dumped.
c:Program FilesSysinternalsSuite>Sysmon.exe -c
System Monitor v6.00 - System activity monitor
Copyright (C) 2014-2017 Mark Russinovich and Thomas Garnier
Sysinternals - www.sysinternals.com
Current configuration:
- Service name: Sysmon
- Driver name: SysmonDrv
- HashingAlgorithms: SHA1
- Network connection: disabled
- Image loading: disabled
- CRL checking: disabled
- Process Access: disabled
No rules installed
Once Sysmon is running, it stores its events in the log Applications and Services Logs ➤ Microsoft ➤ Windows ➤ Sysmon ➤ Operational.
The events logged by Sysmon include the following:
Process Creation (ID 1)
Network Connection (ID 3)
Sysmon Service State Changed (ID 4)
Process Terminated (ID 5)
Driver Loaded (ID 6)
Create Remote Thread (ID 8)
File Creation (ID 11)
Registry Creation/Deletion (ID 12), Value Changed (ID 13), Key or Value Rename (ID 14)
WMI Events (ID 19, 20, 21)
There are also some specialized event types designed to detect the kind of behavior expected from malware; for example, if a process changes the time a file was created, Sysmon records an event with ID 2.
Using PowerShell to Query Sysmon Logs
Event Viewer (Figure 10-1) can be used to examine the events produced by Sysmon. It is also possible to use PowerShell to perform more sophisticated searches through these logs. To grab the events in the Sysmon logs, an administrator can use the Get-WinEvent cmdlet
8/13/2017 11:51:11 AM 5 Information Process terminated:...
8/13/2017 11:50:46 AM 5 Information Process terminated:...
8/13/2017 11:50:45 AM 2 Information File creation time changed:...
8/13/2017 11:50:41 AM 5 Information Process terminated:...
8/13/2017 11:50:41 AM 5 Information Process terminated:...
Suppose that the administrator wants more than just this high-level overview, but instead wants full details from the most recent process creation event (ID 1). In this case the administrator can run a command like
This provides the administrator with a wealth of information, including the time the process was created, the user that created the process, its command line, its PID as well as the command line, and PID for the parent process.
An administrator may wish to examine these entries in more detail; for example, they may wish to enumerate the programs that have been run by a user. Though this data is available in the output from this PowerShell command, getting access to it in the script is non-trivial. This is because much of the interesting data is contained in the Message field, which is not a simple data structure. One way to get useful access to this data is to cast the data as XML. Consider Listing 10-9, which is a PowerShell script to search through the Sysmon logs for process creation entries (ID 1).
$events = Get-WinEvent -FilterHashTable @{logname='Microsoft-Windows-Sysmon/Operational'; id =1}
foreach($event in $events) {
$eventXML = [xml]$event.ToXml()
for($i=0; $i -le 20; $i++){
$eventXML.Event.EventData.Data[$i]
}
""
}
Listing 10-9
PowerShell script to examine the structure of a Sysmon event after conversion to XML
The output from this script are a series of entries, one for each event, with structure like
To access this data, the administrator can use the fields in the XML structure. Consider Listing 10-10 that loops through all the process creation events (ID 1) in the Sysmon logs; for each such event, it extracts the program image, the command line, the user that started the process, the integrity level of the process, the PID of the process, and both the name and PID of the parent process. This script simply displays these to the screen, but this information could be used in a variety of ways.
$events = Get-WinEvent -FilterHashTable @{logname='Microsoft-Windows-Sysmon/Operational'; id =1}
foreach($event in $events) {
$eventXML = [xml]$event.ToXml()
$image = $eventXML.Event.EventData.Data | where {$_.name -eq "Image"}
$commandline = $eventXML.Event.EventData.Data |
where {$_.name -eq "CommandLine"}
$user = $eventXML.Event.EventData.Data | where {$_.name -eq "User"}
PowerShell script to extract data from a process creation entry in the Sysmon logs
Sysmon Configuration
In its default configuration, Sysmon logs all process creation events. However, to an administrator defending a system or network, not all process creation events are of interest. For example, standard programs that run in the background need not be noted, whether they are standard Windows services or updaters for programs like Adobe Flash. At the same time, some network connections are more suspicious than others. A network connection on TCP/80 from Internet Explorer probably does not need to be recorded, while a network connection from a program running from c:WindowsTemp is very much of interest.
SwiftOnSecurity has created a sample configuration file for Sysmon and made it available at https://github.com/SwiftOnSecurity/sysmon-config. It is well commented and easy to customize further. To use the configuration file, an administrator can download the repository, them install it using Sysmon with the -c flag.
Copyright (C) 2014-2017 Mark Russinovich and Thomas Garnier
Sysinternals - www.sysinternals.com
Loading configuration file with schema version 3.30
Configuration file validated.
Configuration updated.
A subsequent dump of the configuration file shows that the changes have been made.
c:Program FilesSysinternalsSuite>Sysmon.exe -c
System Monitor v6.00 - System activity monitor
Copyright (C) 2014-2017 Mark Russinovich and Thomas Garnier
Sysinternals - www.sysinternals.com
Current configuration:
- Service name: Sysmon
- Driver name: SysmonDrv
- HashingAlgorithms: MD5,SHA256
- Network connection: enabled
- Image loading: disabled
- CRL checking: disabled
- Process Access: disabled
Rule configuration (version 0.00):
- ProcessCreate onmatch: exclude
CommandLine filter: begin with value: 'C:Windowssystem32DllHost.exe /Processid'
CommandLine filter: is value: 'C:Windowssystem32SearchIndexer.exe /Embedding'
Image filter: end with value: 'C:WindowsSystem32CompatTelRunner.exe'
Image filter: is value: 'C:WindowsSystem32MusNotification.exe'
... Output Deleted ...
Key features of this configuration include the following:
Sysmon will not log process creation for a collection of programs likely to be safe.
Changes in file creation time generated by Microsoft OneDrive are ignored.
Some network connections are logged. These include programs launched from C:Users, C:ProgramData and C:WindowsTemp. Windows programs like notepad.exe and PowerShell.exe that connect to the network are logged.
Remote thread creation is logged only for important processes, including C:WindowsSystem32svchost.exe, C:WindowsSystem32wininit.exe, and C:WindowsSystem32winlogon.exe.
File creation is logged in critical locations, including the Start Menu and key directories including C:WindowsTasks and certain subdirectories of C:WindowsSystem32 or C:WindowsSysWOW64. Creation of certain file types is logged regardless of the location; these include .bat scripts, .vbs scripts, and .ps1 scripts.
Registry changes in locations likely to be used by malware are logged.
Because the configuration file is XML, an administrator can customize it further.
Installing and Configuring Sysmon Across the Domain
Sysmon is sufficiently valuable that it should be installed and configured on each system in the domain. Unfortunately, the command-line installation process described so far is designed for a single host. There are ways to deploy Sysmon across the domain. One method is to set up a file share that contains both Sysmon and the configuration file, then use psexec to perform the installation.
To do so, start by creating a directory (say on a domain controller) that will be used to share both the Sysmon program and the configuration files. Chapter 13 covers how to set up file servers in general; however, for this purpose it is possible to proceed with a much simpler solution. Create the directory C:Sysmon and copy the two installation files along with a configuration file (say the SwiftOnSecurity configuration file) into this directory. Right-click on the directory, select the Sharing tab, and then the Advanced Sharing button. Select Share this Folder (Figure 10-9).
Figure 10-9
Sharing the directory C:Sysmon. Windows Server 2008 R2 shown.
Next, obtain a list of the hosts on the domain; this can be done with a command like
c:UserssrevinDesktop>wmic /NAMESPACE:\rootdirectoryldap PATH ds_computer GET ds_dnshostname
DS_dNSHostName
venera.venus.test
fornax.venus.test
gabie.venus.test
uorsar.venus.test
hestia.venus.test
UT.venus.test
Remove the header (DS_dNSHostName) and save the result as a text file, say hosts.txt. Pass this list of hosts to psexec and run the installer specifying the shared folder as the location of the executable and the location of the configuration file.
Here the server that holds the executable (named sysmon.exe) and the configuration file (named config.xml) is named \venera. Note that psexec is run with the -s flag so that the remote process is run with System privileges; these are needed for the installation. Note also that the Sysmon installation automatically accepted the end user license agreement and is configured to record network traffic (with the -n flag).
Integrating Windows and Linux Logs
It is possible to aggregate logs from both Windows and Linux systems on the same host using a variety of commercial tools. One open source tool that can forward Windows event logs to Linux systems using the syslog protocol is NXLog. It is available for download at http://nxlog.org/products/nxlog-community-edition/download, including a Windows installer.
Once NXLog is installed on a Windows system, it must be configured. The primary configuration file is located at C:Program Files (x86)
xlogconf
xlog.conf on 64-bit systems and at C:Program Files
xlogconf
xlog.conf on 32-bit systems. To use NXLog to send Windows logs to a Linux system, changes need to be made to this configuration file. First, the ROOT variable needs to be properly set; this is the path to the NXLog directory. To use syslog for the output format, the corresponding syslog extension (xm_syslog) needs to be loaded. Finally, the output module needs to be configured with the destination log server (e.g., 10.0.9.190), port (e.g., TCP/514), and told how to configure the output (syslog). The result is an nxlog.conf file (on a 64-bit Windows 2012 system) like Listing 10-11.
define ROOT C:Program Files (x86)
xlog
Moduledir %ROOT%modules
CacheDir %ROOT%data
Pidfile %ROOT%data
xlog.pid
SpoolDir %ROOT%data
LogFile %ROOT%data
xlog.log
<Extension syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_msvistalog
</Input>
<Output out>
Module om_tcp
Host 10.0.9.190
Port 514
Exec to_syslog_bsd();
</Output>
<Route 1>
Path in => out
</Route>
Listing 10-11
Example configuration file C:Program Files (x86)
xlogconf
xlog.conf on Windows 10
NXLog is configured to start automatically, but once changes are made to the configuration file, it should be restarted. Navigate Control Panel ➤ System and Security ➤ Administrative Tools ➤ Services. Select the NXLog service, right-click, and select start or restart as appropriate.
Once started, NXLog begins to send syslog formatted log messages to the selected destination. These follow the syslog standards, and so are in plain text.
Though the license for NXLog is an open source license, it is not one of the traditional open source licenses (GPL, BSD, MIT, Apache), and is not currently on the list of licenses approved by the Open Source Initiative (OSI). The NXLog public license is available at http://nxlog.org/nxlog-public-license, while the list of licenses approved by OSI is at http://opensource.org/licenses/alphabetical.
Michael Haag at https://github.com/MHaggis/sysmon-dfir has a collection of resources for Sysmon, including links to configuration files and ways to incorporate Sysmon into other tools like Elk and Splunk.
This approach may be the easier to understand. It grabs all the events from the specified log, then filters those to determine which have the value of EventID set to 4625 (which corresponds to a failed logon attempt).
This same approach can also be used with Get-WinEvent; for example, an administrator can run the command
The problem with this approach, though, is performance. This grabs all the entries in the log before filtering them. If, as is often the case, there are many entries, it can take quite a bit of time to grab them and then filter them.
A faster approach is to use a FilterHashTable; this is the approach taken in Listing 10-9. There the administrator uses the command
Get-WinEvent -FilterHashTable @{logname='Microsoft-Windows-Sysmon/Operational'; id =1}
A careful reader also likely noticed that the way that Listing 10-9 and Listing 10-10 looks for data from the Sysmon operational log is odd. The explanation for this is in the way that Sysmon stores data. Launch Event Viewer and navigate to the Sysmon operational log (Event Viewer ➤ Applications and Services Logs ➤ Microsoft ➤ Windows ➤ Sysmon ➤ Operational). Select a log entry with EventID 1 and examine the details in XML view. A typical result has the following structure.
