Microsoft Sysmon

Microsoft System Monitor (Sysmon) is a utility used to log events that are especially interesting to security analysts. These events include creation of new processes, services, files, and network connections. Sysmon logs more detail in the Event Viewer than default Microsoft logs. Windows logging is very noisy by default. The same is true of Sysmon logs. It is imperative the SOC leaders identify specific use cases Sysmon logging. Without filtering the logs collected, tens of thousands of events can be collected from a single endpoint in an hour. The capacity for log retention and the ability to filter logs for useful data become impractical. Figure 5-5 shows 114 Security Events were generated during startup of the Windows 10 machine.

These events show process creations and logons. Figure 5-5 shows 114 events were logged during system startup.

Figure 5-6 shows the System logs. During startup, 162 event logs were recorded.

Figure 5-7 shows command-line use where a new process creation is attempted.

In Figure 5-8, the command used is displayed on line 6. The screenshot shows the Event ID 1, new process creation. The Sysmon log shows the hash of the command, ParentProcessId, ParentImage and ParentCommandLine.

Figure 5-9 displays a command executed to turn on all Windows firewall profiles. They were off and then turned back on.

The command was captured under Event type 1, a process creation. On line 12 the command entered on the command line is shown. If an attacker turned off the firewall profiles, that would be shown too.

The parent process ID is an important piece of information. When investigating a potential malware outbreak, collecting the parent process helps the team determine the extent of an infection. It also can confirm the existence of malware when processes run from unusual parent processes. As Figure 5-11 shows, compiling a list of other endpoints running the parent process ID creates and inventory of infected endpoints.

How Snort Works

In the forward to Snort IDS and IPS toolkit, Stephen Northcutt said the most effective analysts are the ones who understand how Snort works internally. Here we will break that down in enough detail for business and security leaders to understand what the security operations team should be doing with IDS solutions. Figure 5-21 shows the process a packet goes through with Snort.

Snort processes packets one by one. If Snort gets behind on packet processing, limited buffering cannot prevent packets from dropping and not being processed. Dropped packets cause traffic of interest to be missed. This applies if Snort is running in passive mode, receiving copies of traffic from a network device. If Snort is running in inline mode, then packets are not dropped, but network performance issues may occur.

Packets are decoded based on the type. An Ethernet packet is passed into the decoder where once the Ethernet portion of the packet is decoded it passes to the IP layer. If virtual LANS (vLANs) are used, 802.1Q decoding occurs. This is the protocol used for vLANs.

If the IP protocol used is IPv4, then Snort decodes this layer as such. Then based on the transport layer protocol used, TCP, ICMP, or UDP decoding comes next. This process prepares the packet for the preprocessor and rules engine. Common preprocessors used once packets are decoded include fragmented packets are reassembled, verification packets are part of an established session, and preprocessors exist to examine the application layers and find incorrect implementations of protocols. Once the pre-processing is complete, packets are evaluated against the rules in Snort and actions are taken based on the rule header: alert, log, drop, and so on.

Snort Rules

Some of these groups are stand-alone like app (detect), content (replace), exploit(kit), NetBIOS, SQL, and X11. Others have multiple categories of rules. Those are documented in Figure 5-22.

The nice thing about this set-up in Snort is the ease of configuration when it comes to certain types of rules. If, for instance, a standard exists within the entity to only use Internet Explorer (IE) in the environment, then the rules for Chrome, Firefox, and others can be disabled allowing for less overhead by the rules engine when examining packets. The only alert necessary might be to detect uses of other Internet browsers. That way, no rule in those categories triggers and wastes analysts’ time. Same can be said for the non-Windows-based operating systems. If no operating systems outside of Windows are used, then none of those rules need to be enabled making the processing of packets more efficient.

Running Snort

Most times when used as a component of continuous monitoring, Snort will analyze traffic directed to it on a specified interface. Snort is also useful in examining traffic for investigative purposes. If an entity possesses captured traffic of interest that came from a part of the entity not monitored by Snort, the captured traffic can be analyzed by reading it into the solution.

The implementation of Snort used for the following demonstrations was configured to run on the SANS (SIFT) workstation. This workstation was developed as part of SANS Digital Forensics and Incident Response (DFIR) offerings.¹¹ To run Snort against a PCAP, the analyst must be in the directory where Snort is located or specify the path to Snort. The path to the configuration file also needs to be specified on the command line.

Figure 5-23 shows the command used to analyze a traffic already captured with Snort. Brad Duncan at Malware-Traffic-Analysis¹² provided a capture of Emotet and TrickBot traffic in May 2019. The PCAP was analyzed by Snort using the following command:

snort -c /etc/snort/snort.conf -r <packet capture> -l /var/log/snort

Once the command is run, Snort initializes and prepares to process the PCAP file. Figure 5-24 shows 5227 rules exist. It is possible to see the breakdown of the rules by protocol, source, and destination as well.

In Figure 5-25, we see 17,431 packets were processed in less than two seconds. The third section from the top shows all packets were processed and none were dropped.

All the packets were Ethernet and IPv4. The majority were TCP protocol since only 16 were identified as UDP.

Figure 5-26 contains statistics about actions Snort took with the packets and the beginning of preprocessor statistics starting with Frag3 preprocessor.

Snort allowed all the packets to pass. None were blocked or replaced. 1891 were whitelisted by Snort. Figure 5-27 shows the preprocessor statistics.

Snort counted 75 total sessions, 67 TCP and 8 UDP. The stats also show 1709 segments were queued and released.

The next several figures highlight statistics of packets processed by Snort’s preprocessors. The first category in Figure 5-28 is the HTTP Inspect preprocessor. HTTP Inspect identifies HTTP fields and normalizes them for processing by the rules engine. Snort identified 24 HTTP request headers with 20 POST methods and 4 GET methods. In a client-server architecture, POST methods represent the client providing data to the server. A simple example is a user filling in form fields on a web site. A GET request method example is when someone requests a download of content from the web site.

The next section shows preprocessor stats for SMTP, mail traffic. This traffic capture did not contain mail traffic based on Snort’s analysis. Snort also did not identify traffic related to Distributed Computing Environment or Remote Procedure Call (DCE/RPC).

The next section shows that 447 Secure-Socket-Layer (SSL) packets were decoded by Snort. Snort identified the Client Hello, Server Hello, and certificate exchange 62 times. These are in Figure 5-28.

Snort Alerts

Since 209.134.25.170 was an address of interest in the Zeek conn.log, a rule can be written to detect connections to this address. It is also possible through threat intelligence to learn of this address as an indicator of Emotet traffic. The rule is simple to write:

alert tcp $HOME_NET any -> 209.134.25.170 80 (msg: "Potential Emotet/Trickbot Traffic"; type threshold, track by_dst, count 5, seconds 3600; sid: 10000001; rev:001;)

This rule looks for any endpoint and any port making an outbound connection attempt to the address and port listed in the rule. A threshold was set whereby only five alerts are generated in any given 60-minute time frame.

Using the following command, the PCAP can be analyzed to confirm the rule detects connections to the address of interest:

snort -A console -K none -q -c /etc/snort/snort.conf -r 2019-05-01-Emotet-Infection-with-Trickbot.pcap

This command process prevents the startup and statistics returned in Figures 5-23 through 5-27 from displaying. It also outputs the alerts to the screen instead of a file in the /var/log/snort directory. Figure 5-29 displays the alerts generated. The customized rule was triggered. It shows the five connections within a one-hour period. Two other rules from the Community Rules set also triggered. These were in the malware rules and detected potential Trickbot CNC traffic. This rule detected the traffic from 185.198.57.70 on port 443 which is used for secure/encrypted HTTP traffic. This was rule 44402 that triggered.

The first ten alerts were triggered by the custom rule created earlier. It shows the following information:

05/01-14:28:03.435059  [**] [1:10000001:1] Potential Emotet/Trickbot Traffic [**] [Priority: 0] {TCP} 10.5.1.102:49208 -> 209.134.25.170:80

The last two alerts came from the Snort rule set detecting Trickbot traffic.

05/01-15:08:44.093367  [**] [1:44402:1] MALWARE-CNC Win.Trojan.Trickbot self-signed certificate exchange [**] [Classification: A Network Trojan was detected] [Priority: 1] {TCP} 185.198.57.70:443 -> 10.5.1.102:49273

Figure 5-30 shows the snippet of the rule that caused this alert to generate. The rule looks for TCP connections flowing into the network. This is the rule header that was matched. The content portion looks for specific fast pattern content. The string [email protected] content match is an example of writing efficient Snort rules because the engine can immediately look for and detect this content.

When alerts are generated, the first thing an analyst should do is review the rule to understand why Snort triggered the alert. Loading the Emotet/TrickBot traffic into Wireshark allows the analyst to verify the content causing the alert.

It is possible for Snort to capture packets when alerts are triggered so analysts can view the packet in question. Settings are also available to capture the preceding and succeeding packets. That setting was not utilized during this walkthrough.

Since we are looking for a specific connection coming from 185.198.57.70, a good first step is going to the statistics drop-down and highlighting Conversations shown in Figure 5-31.

Once the two endpoints of interest are found, right-clicking the line makes it easy to filter the traffic to only the two of interest. This is shown in Figure 5-32.

To further filter the traffic, right-click on an instance where 185.198.57.70 is the source IP address. This filters the display to only traffic originating from 185.198.57.70. Now it is easier to find the packet in question. This is displayed in Figure 5-33.

Highlighting the third line, a TLSv1 Server Hello connection turns out to be the packet we are looking for. Figure 5-34 is the details pane where analysts can see contents of the payload if not obfuscated by encryption. Since this was the Server Hello, the Client Hello flowed in the opposite direction, so it is not displayed, the encrypted tunnel is not set up, and the payload contents are visible.

At this point, it seems there is an indicator of TrickBot activity on this endpoint.

Other IDS/IPS Solutions

Besides the Snort and Bro solutions we discussed for IDS and network traffic analysis, there are many other solutions available for healthcare entities building security operations centers. One open source solution not discussed in detail is Suricata. One of the differences between Suricata and Snort is that Suricata detects protocol usage based on what is in the traffic and not relying on ports identified in the traffic. For instance, seeing POST and GET requests causes Suricata to identify the traffic as HTTP, similar to what Zeek does.

Selecting an IDS/IPS solution depends on a number of factors: cost, support, internal resources, and so on. Open source solutions provide the benefits of lower initial investment. Some provide strong community support or have entities that provide paid support. If the organization has resources with deep understanding of how the solution works, then opportunities exist for customization. Zeek is a great example. Someone with strong understanding and scripting skills with the Zeek language creates customized signatures. Commercial solutions must be evaluated based on fit within the organization.

Inbound Layer 7 Protocols

This answers the question, what protocols are used coming into the network and why? It is easy to capture this traffic via Zeek. Analyzing traffic via egress and ingress points develops a picture of Layer 7 protocols in use. A simple table showing the protocols used and counting the number of instances makes it easier to spot unusual traffic. Based on the infrastructure supporting the Zeek implementation, the table can show these occurrences over a week or month.

Outbound Layer 7 Protocols

Again, using Zeek or another traffic analysis/capture tool, it is possible to uncover what protocols are in use with egress traffic. Naturally, there are protocols analysts would not expect to see such as Remote Desktop Protocol (RDP) connecting to external endpoints. This type of view allows the SOC to quickly see unusual one-off protocol uses. Low occurrence protocols become interesting to investigate.

Top Outbound Talkers

Any solution capturing flow traffic can present this data to analysts. Again, Zeek is perfect for this job. Flow information found in the conn.log captures flow information. Analysts can pull this information directly from Zeek logs or by sending the log data to a SIEM and creating views there.

Connections over Ports 25 and 53

Outbound traffic using these two ports should only be seen coming from specific servers. Port 25 is the Simple Mail Transfer Protocol (SMTP) and should only have traffic coming from the mail server. Port 53 using either UDP or TCP is the Domain Name System (DNS) protocol used to map domain names to IP addresses. Internally specific servers run DNS services, and those servers are the only ones that reach out to authoritative servers. Any other endpoints connecting on these ports outbound require investigation.

Hosts per Domain

Some C2 sites use multiple subdomains, so tracking the number of hosts per domain creates detection opportunities when new domains are detected with unusually high numbers of hosts. There is no magic number for what is considered a high number of hosts; analysts are looking for anomalies. Again, new domains not seen before are of interest and so would domains with more hosts than others on the list.

Communication on Ports Used by Known Malware

The work done by MITRE ATT&CK in Chapter 3 documented specific ports used by the software Deep Panda historically employed. Infections by Emotet and Trickbot use specific ports for communication. It is possible to monitor outbound communications on those specific ports and investigate any uses. This type of alert is low-hanging fruit since ports used by malware is not high on the Pyramid of Pain. Changing the port used by malware is not a difficult coding change.

URL with High Entropy

When it comes to HTTP traffic, monitoring the Universal Resource Locator (URL) for high entropy/randomness. C2 domains can quickly change. Attackers sometimes employ random domain generators. There are tools available for the SOC conducting this analysis. Mark Baggett, SANS instructor and creator of the Automating Information Security with Python, created two tools to calculate the frequency scores of domains seen. One version calculates the scores when an analyst manually executes this tool against list of domains. The other creates an automated process.¹⁴ Each does the same thing; it’s a matter of whether the organization can take advantage of the automated process.

Monitoring this randomness regularly makes it easier to spot abnormally random domains. There are different ways to calculate entropy for this section depending on the solution implemented. We will discuss a few examples in the SOF-ELK and Splunk sections.

New Domain Connections

Using historical analysis, the SIEM can monitor for connections to domains never seen before. A list of domains visited is kept in a file, and when a domain name is seen on the wire, the lookup file is referenced to see if the domain exists. If not, then it is logged, or an alert is generated for the SOC analysts to investigate the new domain connection. Over time there should be fewer and fewer domains visited never seen before. Andrew Dauria wrote a blog post called “Finding NEW Evil: Detecting New Domains with Splunk” that outlines how Splunk users can monitor for this activity.¹⁵

User Agent String Analysis

Monitoring for user agent strings falls in the middle of the Pyramid of Pain, under the network-based indicators. Sometimes interesting indicators are found. Some attackers place information about the malware, version, and so on in the user agent string to keep track of what malware is reporting. Other times, configuration information about the infected endpoint is sent via the user agent string. Attackers also use user agent strings that are common and expected.

Another indicator to monitor for is user agent (UA) strings that do not belong on the network. For instance, if the SOC is monitoring a primarily Windows-based environment, it would be unusual to see UA strings associated with iPhone or Android devices. This might detect attackers connecting out with traffic where the UA string was set to a non-Windows string for a different intrusion and never changed when used in the Windows environment. It also detects malicious insider behavior. If Windows 10 does not have Bluetooth disabled, an end user can pair his or her phone with the laptop/desktop and move files between the two endpoints. Any use of HTTP by the device paired with a Windows machine creates an unusual UA string on that network segment.

Beacon Detection

Malware often connects with its host server. It receives instructions and sends data during the intrusion. Detecting beacons is as simple as monitoring for connections to known domains and IP addresses used for C2. It also involves detecting randomly generated domain names. Detecting on high-frequency connections to domains with many subdomains and domains considered highly random are other indicators of this type of communication.

NXDomain Returns

Sometimes attackers take down C2 servers when not in use. This can happen during maintenance periods or periods when malware is not active. Yes, these attack groups do plan for outages to conduct maintenance. Examples exist where the DNS records for the domains residing on those servers point 127.0.0.1, or no response is given resulting in an NXDomain answer to the DNS query. A spike in these occurrences might indicate an endpoint attempting to communicate with a C2 server.

Logins to Multiple Devices

There are specific groups where multiple logins across the network are expected. Then there are users where remote access to one or many endpoints is highly unusual. Or logons by accounts not recognized might indicate something unusual occurring. This monitoring should not focus only on user accounts but service accounts as well.

Login Attempt to Disabled Accounts

Most intruders use resources available in the network. Abusing disabled accounts is one way to use resources of an entity against itself. Attackers might be inclined to think this activity is not monitored and attempt to gain access to disabled accounts. Monitoring this activity often yields instances where certain services trigger failed logins when attempting to sync. Microsoft’s Active Directory Federated Services (ADFS) is common. Consistent monitoring allows the analyst to become familiar with what is expected and detect what is out of the ordinary.

New Process Creation with Entropy

To run malware must create new processes. Sometimes these processes closely resemble processes already running with very subtle differences. Viewing a list of processes running on an endpoint, svchost is a common occurrence. Svchost allows multiple services to share a process to save on computing resources. Malware may create a new process called svch0st, changing the o to a zero. It would not be hard to overlook such an occurrence. Attackers not as concerned with stealth might create new processes with randomly generated process names. Detecting new process creation and measuring the level of randomness is another opportunity to detect malicious activity.

New Services Created

Windows Event ID 7045 means a new service was created. Monitoring these for execution in unusual directories, like the temp directory or highly random names, may indicate the process helps malware persist.

Registry Key Changes

Registry key changes are used to maintain a foothold for malware. In Chapter 3, we saw how Sakula, a malware variant used by Deep Panda, makes changes to registry settings to survive reboot. Specifically, autorun registries are where Sakula makes changes. With the right endpoint monitoring, these changes might be detectable. There are some forms of malware that install themselves in the AutoStart registry and operate without installing files making detection more difficult.

New Scheduled Tasks

As stated before, malware needs to run, and hiding it within scheduled tasks is one way attackers do this. By monitoring these new tasks, analysts can see if new tasks are created that are randomly generated. Attackers will at times use randomly generated names for tasks, and if these fall outside the norms based on random scores than baselines in the network, then investigation into the new processes is warranted.

DNS Queries by Type

Earlier in the chapter, the types of DNS requests were discussed. The SOC team should track numbers of requests by type over time looking for spikes in TXT and NULL requests. These types of requests are used to place plaintext and encoded data in available fields for exfiltration.

Review for Connections with More POST Than GET Bytes

This is another indicator of possible data exfiltration. During normal operations, an HTTP connection expects to transfer more data via GET requests than POST requests. This behavior is unusual and needs to be investigated.

POST Requests over Five Minutes

This is another event/alert that works best with mature baselines of network activity. Elevated POST methods over five minutes affords the SOC team another opportunity to detect data exfiltration or at the very least C2 connections.