The success of any kind of forensic investigation hinges on the preparation. As we have seen, logs are the mother lode of information and without them, network forensics would be seriously crippled. Criminals also realize this. Once a perpetrator has gained access to our network, one of the first things they try to do is cover the tracks. The first step in this process is getting rid of the logs that document their activity in first attempting and then succeeding in breaching the security of the network. To counter this risk, sensible log management processes have to be in place.
In every organization, there are a multitude of operating systems, a variety of security software, and a large number of applications; each of which generate logs. All this makes log management very complicated. Some of the problems associated with log management are as shown in the following:
- Multiple log sources: Logs can be generated on multiple hosts throughout the organization. Further, some applications can generate multiple logs, for example, authentication information can go into one log and application usage could be stored in another.
- Inconsistent log file structure: Different sources use different formats for their log files. Logs can be stored in Comma Separated Values (CSV), tab-separated values (TSV), databases, Extensible Markup Language (XML), syslog, Simple Network Markup Protocol (SNMP), and binary (as in the case of Windows) files.
- Inconsistent data field representations: Different logs represent different content differently. Some logs would identify a host using its IP address, while another would use the username. This could make it difficult to correlate the two though they represent the same entity. Another issue is how the data values are presented, for example, dates could be stored in the MMDDYYYY or MM-DD-YYYY format. Similarly, one system could log the use of the File Transfer Protocol (FTP), while another could just identify it by its port number—21.
- Inconsistent timestamps: While just about every log entry has a timestamp, the system generating the timestamp references its own internal system clock. Therefore, if the internal clock is inaccurate, the timestamp will also be inaccurate. This is a recipe for disaster. In the event of inconsistency between multiple security systems logs, an analysis of the sequence of events in a security breach can become seriously messed up.
- Large volumes of data: Any organization of a reasonable size can generate large volumes of logs in its network. Multiple sources and limited local storage compound the problem. Log file sizes are usually limited by the number of entries or size. However, this may not suit us from an investigation perspective. Therefore, it becomes imperative to have a way of collecting and storing logs for longer periods in separate locations.
The essential components of a forensic-friendly log management system include log management infrastructure and log management planning and policies.
Network forensic investigations heavily rely on the logs to unravel a case. This is usually post mortem or after the event. If effective infrastructure is not in place and the required logs are unavailable, the investigation can be derailed or at the very least, seriously hampered.
A log management infrastructure typically has a layered three-tier structure as shown in the following image:
Log management infrastructure is required to perform a number of functions in a manner that does not change (or affect the integrity of) logs in any way.
Some of the typical log management infrastructure functions are as follows:
For any forensic investigation exercise to be meaningful, the organization needs to plan ahead and define its requirements and goals relating to the log management.
These can be influenced by the regulatory environment, such as Health Insurance Portability and Accountability Act (HIPAA), Payment Card Industry Data Security Standard (PCI DSS), and so on. From the perspective of business continuity, a balance has to be maintained between the time and resources required and the reduction of the organization's risk profile. Aspects of log management that need to be considered when defining policies and procedures are as follows:
|
Log management aspect |
Issues to be addressed |
|---|---|
|
Roles and responsibilities |
Roles and Responsibilities of the following need to be clearly defined:
|
|
Establish logging policies |
Mandatory requirements and recommendations need to be defined for the following:
Which hosts & type of hosts will log? Which host components (OS / service / applications) will log? Which events to log (security events, logon attempt, and so on)? What information to be logged for each event (user name and source IP for logons)? What will be the logging frequency? (every time, every 100 times, and so on)
Which hosts will transfer to log management servers? What data/entries should be transferred to the infrastructure? When and how often should this be done? What methods/protocols should be used for the transmission? (Bandwidth requirements for the transmission will need to be looked at.) How will the confidentiality, integrity, and availability (CIA) be maintained during this process?
How often will logs be rotated? How long do the logs have to be stored? How will the legal preservation requests be handled? How much storage space will be required? (Daily load, peak load, and so on need to be studied and catered for.) What processes will be followed for secure deletion/disposal? How will the confidentiality, integrity, and availability be maintained during the storage and disposal processes? How often will each type of log be analyzed? What are the steps to be followed once a suspicious activity is identified? Who will be permitted to access the log data and who will follow these steps? How will the accidental exposure of confidential data in logs (such as e-mail content/passwords) be handled? How will the confidentiality, integrity, and availability of log analysis and reports be protected? |
Now that we understand the fundamentals behind the logs and their management, let's see how to analyze the network logs.