There are three primary operating systems in use in the enterprise: Microsoft Windows, Linux, and Sun Solaris. Linux and Sun Solaris are more similar than Windows, and the approach to monitoring is similar as well. The primary difference being the use of DLLs and a registry in Windows while the other two operating systems use a complex set of files to run the operating system with all having a kernel at the center of the architecture.

When monitoring operating systems, these distinctions must be understood in order to implement the correct monitoring and provide output that is human readable and actionable. The most common method of operating system security monitoring is file activity/integrity monitoring. This is a solution that monitors access and modifications to critical operating system files and registry keys in the case of Windows.

When a change occurs, regardless if benign or malicious, it is recorded along with the forensic audit trail information to track the change to an individual or process responsible. Initially the use of FIM will produce a significant amount of data until tuned; as every change will be recorded, not all needs a response. Once tuned, the data can be invaluable in properly monitoring changes to the operating system and supporting files.

The following are the Windows files to monitor:

The following are the Linux files to monitor:

Any detected changes should be investigated; and if there is a false positive or a constantly changing file, like a log file, it may be necessary to ignore these changes to reduce the amount of data to review. Additionally, all review must occur at a level above those able to make changes to enforce separation of duties and the risk of collusion. There are open source tools such as OSSEC (http://www.ossec.net/) and Tripwire for Linux (http://sourceforge.net/projects/tripwire/) that focus on FIM across the operating system and all other files resident on the system.

Using FIM is an effective method to detect a security event on a system where files are added, deleted, or modified in response to malicious activities common to virus, Trojan, and exploit activity that will trigger an event in a properly configured FIM solution.

Where the solution has little value is memory; no files are manipulated in any way so the detection engine would not identify the threat. Solutions that address this issue are generically called application whitelisting solutions and control what has the permission to run based on a signature captured for legitimate applications. If the application that is attempting to run is not a known and trusted version of the application, it will be blocked from running. This method also detects and blocks instances where the executable of a legitimate program has been altered even in system memory.

There are several tools that can be used; and if outputted data is correlated in some manner, can provide a good indication on whether there may be a system issue. It is recommended to script common Linux tools such as ps , netstat , and tcpdump to see snapshots and real-time information on running processes and open network connections. Monitoring processes on Windows is just as easy with several tools such as perfmon , taskmgr , and netstat to monitor aspects of the operating system that may provide detail on security events affecting the system.

Commercial FIM solutions such as Tripwire (http://www.tripwire.com) support several operating systems, network devices, and security devices. This makes it a good candidate to evaluate for the enterprise environment. Using a combination of open source and commercial solutions may be the best option and has to be assessed for each unique environment. As with any security tool, it is important to have the supporting processes, policies, and standards in place to ensure effective use of the solutions and intended security as purposefully implemented.

In order to have an effective implementation, a phased approach may be the best to ease the impacted internal organizations into the process. Referencing the trust models developed with matrices of critical data and applications, a list of critical and sensitive systems can serve as the starting point for an overall enterprise implementation. Engaging the impacted organizations in decisions on solutions and the process to handle the output of such implementations can drive the correct solution selection and drive a more mature program. This is highlighted because in most enterprise environments there are a plethora of agents running on systems, and adding more without providing an initial value will make a FIM solution implementation all the more challenging. Several compliance bodies are mandating this type of solution is implemented and can be the catalyst to building a system security monitoring capability (if non-existent), adding a critical component to enterprise data security.

A host-based intrusion detection system (HIDS) is very similar to network-based intrusion detection, however, it is specific to the host or set of hosts where it is implemented. In a similar fashion of signature, anomaly, and behavioral analysis, HIDS performs the same function on network systems instead of the network layer. The benefit is more finely-tuned policies that minimize the noise of traditional network-based IDS where detection is for anything and everything that is protected by the IDS.

To make the point clear, IDS simply detects but does not mitigate. However, IDS can operate in an active response (send RST packets, and so on) making it essentially an IPS. For this section of the book, the term HIDS will be used referring the solutions with active response capabilities.

Having an HIDS solution implemented on enterprise systems provides unparalleled intelligence on what attacks and other network anomalies (malicious and benign) are being targeted at enterprise systems. It is sometimes difficult to determine if an attack detected at network IPS made it to the intended target; with HIDS, this is no longer unknown. If detected, it reached the intended target or set of targets. Most commonly used HIDS solutions perform more functions than intrusion detection. Often application whitelisting and FIM are features that can be leveraged bringing more value to the implementation by providing more security capabilities and reducing system overhead required for multiple agents.

There are some unique offerings in the area of host malware; intrusion detection that builds on trust is an integral theme of this book. The solution is able to leverage a number of sources and analyses to determine if the executable or file has benign or malicious intent and actions. The solution, Bit9 (https://www.bit9.com/), while not technically an HIDS, has a unique approach that has characteristics of HIDS, in addition to their special proprietary capabilities. While the intent of this book is not to dive too much into vendor specific attributes, there are a few solutions that are well known for their approach and warrant mention where the solution may be of interest for further investigation and evaluation.

When evaluating the implementation, an HIDS solution or any other host-based solution, it is important to consider management of the solution, level of provided support, and benefit to the overall security architecture, all of which helps to determine the total cost of ownership (TCO). There must be enough value and benefit that the TCO is acceptable. Because HIDS is an endpoint solution, management of the solution must be decided, both technically and organizationally. This issue remains a point of debate between security teams and system support staff. Commonly, IT security will be responsible for the policies implemented and configuration while the system support staff will maintain the software, including patches, and ensure it remains functioning.

As an industry, the move to HIDS has been a slow process, however, with emerging threats easily outwitting traditional methods of security, implementing HIDS-like solutions may be the only method of systems' security monitoring and protection effective enough to protect enterprise data.