Each team identified from previous meetings for building enterprise support for incident response will need to identify resources with areas of expertise that can be committed to incident response in the event the process is triggered. The capacity in which they are engaged is dependent on the severity of the incident and may serve in an advisory role for less severe incidents. Each assigned resource must be made aware of the responsibility of being a member of the incident response team and respond within agreed service-level agreements (SLAs).
The confidentiality of security incidents is as important as a forensic investigation and should be treated as such until the full impact of the incident is understood and communication should be sourced from the communications role in the incident response plan.
Each team member will need to know the correct procedures for already defined incident types, but also be agile enough to take the correct action if an unplanned type of incident occurs. The more the incident response at any level is practiced, the more the process will be refined and the team members will become more proficient, making less mistakes, ultimately reducing the impact of the incident with outcomes that will better secure the enterprise.
The individuals selected on each team will be the decision of the management for each team with management being the first contact for the team and defined in the contacts section of the incident response plan. There may be areas of expertise within each team and those with specializations will be documented to ensure that the right resource is assigned to the incident. Each role will need to know how to interact with the other roles and ensure that the established communications protocol is followed so that all pertinent information is recorded in the incident. In order to build an incident response team it is necessary to define roles, responsibilities, contacts, and supporting procedures.
Each team involved in incident response has a defined and important role in ensuring that the incident is handled correctly and efficiently. There may be several individuals from a team selected for participation on the incident response team. Each team will be leveraged for the pertinent expertise within the team and responsibilities assigned accordingly. Common IT enterprise and business teams with incident response roles are:
- Desktop support
- Systems support
- Applications support
- Database support
- Network support
- Information security
- HR, legal, and public relations
The desktop support team will have expertise in desktop operating systems and applications at the user level. This team should also have knowledge of the deployed security tools on user desktops and be able to analyze their output as it relates to the incident. Common installed security tools on desktops include antivirus, anti-malware, and firewalls. In addition to security tools, system logs can be a valuable resource for incident investigation.
The systems support team is very similar to the desktop team in regards to the operating system and security tools implemented. However, the scope is increased as, typically, servers host critical business applications, databases, and other data that is most commonly the target of malicious incidents. In the case of non-malicious incidents, the aforementioned holds true and introduces potentially greater impact to the business. This team must be aware of data, applications, and processes that each system stores, runs, and manages respectively. This team may have more detailed procedures that need to be developed in order to properly respond to an incident due to the added visibility of server systems including additional notifications and approvals to take action. Another consideration is offloading services running on a compromised or altered system that requires a disaster recovery and continuity plan that can be enacted in an acceptable time frame.
The applications teams may have direct or indirect tasks associated with responding to a security incident. Typically, applications will leverage a system in the enterprise but it may be the mechanism used to perpetrate the incident. The team's knowledge of the application including its function, code, and logs will be an invaluable resource when responding to an incident and finding the root cause or method(s) used in the incident. Coordination between the applications and systems team is imperative to further reduce impact on the business in response to an incident. Often the applications team will already have a load balancing or migration process for moving an instance of an application and you should know what the risks and ramifications are when doing so. Understanding the function of the application and the associated processes can also give a perspective of possible scope of the security incident and the data that is at risk. Having this team or teams involved in the development of the security incident response is critical and will ensure complete coverage for business processes.
The database team is the largest custodian of enterprise data and knows where the most critical data resides in databases. Web applications, critical applications, and other business critical processes most commonly rely on backend database infrastructure to store their data. The end goal of typical security incidents is obtaining enterprise data, whatever that represents for the specific enterprise. This team must be involved to document which databases contain what data and which applications leverage the infrastructure. Additionally, this team must be trained to know what unauthorized or unexpected database accesses look like in logs, in order to aid the incident response team and identify what data was accessed, altered, deleted, and possibly exfiltrated. Without this knowledge and expertise, data that was just accessed and exfiltrated may go unnoticed and the guessing game of what occurred will leave the enterprise in an uncertain state.
The network team is responsible for getting traffic across the network and therefore has visibility of all the traffic traversing the network. Some network teams are responsible for not only the routers and switches but also the firewalls at the network edge and within the enterprise network. The expertise of how traffic flows on the network and the advanced protocol knowledge will help decipher the network communications of the security incident. Reviewing firewall logs and firewall policies can provide detailed information on how access from the source to destination occurred. This information can be valuable for lessons learned and provides possible areas of improvement, or to drive a new design requirement to further segment and protect valuable assets. Another service the network team may be able to provide are packet captures of traffic that traversed the network related to the security incident. It is common for network teams to have packet capture technology implemented for troubleshooting and monitoring network latency and connectivity. These tools can be an invaluable source of data. Additionally, if the incident is ongoing, the network team can make the necessary network changes to mitigate the threat at the network layer by changing the routing, implementing the firewall rules, and using router ACLs. The network and security teams should already have a good working relationship and know the methods to stop any threat at the network layer. Working with common nomenclature and existing processes for day-to-day operations should create a synergistic approach to security incident response. This team should already have intimate knowledge of all connections to the enterprise network and know the impact of any network-based incident and what effect the changes to the network will have on these connections.
The information security team has a unique role in security incident response, not only in responding but in leading and in the management of security incident response. After all, security incident response is a primary role of the IT security team. This team must be aware of the environment that they are protecting, the applications, the data, and the processes. They must know what to look for, providing guidance to other teams. The team must be very knowledgeable of protocols and the technologies in use in order to know where to point teams and focus the investigation. A forensic capability may also be a function provided by the security team if the expertise is present in-house. A brief note on the forensic aspect of security incident response is provided in the Supporting procedures section of this chapter. Whether this capability exists in-house or not, the information security team must ensure the forensic soundness of the investigation until it is determined that the need for forensic analysis is not required. Beyond providing guidance on what to look for and where, the security team must work with the other teams to stop an on-going threat or help determine how to stop the incident from occurring again. Adjustments to policies, standards, and processes may be the end result of enacted incident response.
These non-IT business teams are critical to the incident response process and must be involved in the development of the team and plan. There may be instances where the incident requires the involvement of one or more of these teams. The scenarios include:
- Employee, contractor, consultant, and so on (HR)
- Website defacement (PR)
- Theft (legal)
This should give an idea of why these teams need to be involved in the incident response process. IT security should have a relationship with each team as many functions that IT security may perform will require guidance from each of these teams. A few examples are forensics and legal, policy violation and HR, and publicly visible incidents and PR. It is important that the team members on these teams understand the IT aspect of their practice and are actively involved during incident response.
There are other responsibilities outside of IT that may need to be defined and leveraged depending on the type of incident and based on agreed incident severity and priority. These responsibilities include but are not limited to public relations, both internal and external, for press releases if necessary, legal, and HR. Generally, responsibilities will be assigned in alignment with expertise and the associated team member. An example is malicious data destruction on a file server. The responsibility assigned to the system's support function would be to facilitate a full assessment and determine through logs and other system data what happened, who took the action, and what data was affected. In this scenario the system's support team would lead with the other teams supporting as necessary. Each team should have a clear understanding about what they are responsible for, providing an expertise and action perspective for the incident response. Example roles and responsibilities are outlined in the following table:
|
Roles |
Responsibilities |
|---|---|
| |
| |
| |
| |
| |
| |
|
Legal |
|
|
HR |
|
|
Public relations |
|
When an incident occurs, whether critical, major, or minor, it must be assessed and given a severity and that severity should have an expected response time. This is common practice for general incident management in the enterprise. The response time should take into consideration realistic time frames for an individual to answer the call for action and be in a position to take action. It is important to note that each severity type must be defined and understood by all parties. The times defined must be acceptable to senior management with proper expectations set. A sample SLA table to give an idea of what this may look like for an enterprise incident response plan is as shown here:
|
Incident severity |
Response time |
|---|---|
|
Critical |
30 minutes phone, 1 hour action |
|
Major |
1 hour phone, 2 hours action |
|
Minor |
2 hours phone, 3 hours action |
Time-to-resolution (TTR) may be unknown and may only be an estimate depending on the type of incident. There are some cases where the time to remediation is known, for instance, restoring lost data to a server. This is a common practice that an estimate can easily provide. However, a more severe incident, such as a complete compromise of a critical system, may take significantly more time, as forensic analysis will be required, and eventually a complete rebuild of the system, data, and applications. The defined service-level agreements for response and action can and may need to be adjusted based on observed response times as incidents are generated and resolved.
Knowing who to call and when is probably the most critical decision when an incident is or has taken place and an action must be taken. For this reason, it is a critical step in the incident response plan to create a complete contact list that is accessible for the first responders to initiate the necessary procedures to resolve the incident.
Typically, management representatives are the first point of contact for critical and major severity incidents, while minor incidents would follow a less urgent communication path and be resolved with little or no cross-team interaction at or below the management level. It must be stated that, in some cases, what is initially deemed a minor incident becomes a more severe incident after initial investigation. At this time, management should be contacted as per the communication plan in the incident response plan.
Not only will the management of the various teams be listed as contacts, but non-IT roles need to be listed as well. These include HR, legal, public relations, and other senior management members that are business aligned and not directly involved with IT but are dependent on or affected by the incident. The decision to communicate the incident to senior management should be the call of the IT management members and must be in accordance with the incident response plan communication guidelines.
A typical contact list will include the following:
- Senior leadership (CIO, directors, and so on)
- Legal representative
- HR representative
- Corporate communications representative
- Desktop manager
- Desktop responder
- Systems manager
- Systems responder
- Database manager
- Database responder
- Application manager (may be several teams)
- Application responder (may be several teams)
- Network manager
- Network responder
- IT security manager
- IT security responder
For reference you can go to www.cert.org/archive/pdf/csirt-handbook.pdf. There may be contacts outside of the enterprise depending on the response implementation, internal or external. Additionally, third-party contact communication may be a function of a member or team on the list; the third-party contact information may not reside on the contact list to effectively control external communication of an incident. Contacting the CEO and board may be handled in the same manner. This can reduce liability to the enterprise in the event of a critical incident that may have far reaching consequences that should be handled with diplomacy and at the right echelon within the organization.
Each team responsible for incident response should have defined procedures for responding appropriately. The procedures should be documented and peer reviewed to ensure that the intended outcome is achieved. Forensic soundness may be of concern depending on the type of incident and the procedures should have this requirement as a basis for the developed procedures. It is important that the procedures are as detailed as possible to ensure consistency and repeatable execution regardless of who from the team actually performs the procedures.
In order to build the responding procedures, each team will need to complete one or more scenario exercises based on predefined incident types and carefully document caveats, challenges, and success criteria. The process of developing procedures may highlight operational areas that need to be developed or refined to better respond to a security incident. There may be some actions that will be cross-functional in nature and will require coordination amongst one or more teams to respond to the incident, and reduce further impact to the enterprise. An example would be an application move and change; this scenario may require the database and network teams to facilitate a successful application move to another system, possibly on another network segment. The database team would have to account for changes in the incoming application connection and the network team would have to account for VLAN changes, routing, and possibly firewall rules to permit the new connectivity while shutting down the previous network access to the application.
Procedures developed by the teams must be peer reviewed to ensure that the actions will produce the intended outcome, impact on the enterprise is reduced, and that the actions follow forensically sound methods. The procedures to be implemented during an incident may require a change management ticket even in the event of a security incident in order to obtain approval and notify all parties that could be impacted by the changes.
Forensics has been mentioned previously in this chapter as this capability is critical for any incident response team. There is significant complexity in a properly set up forensic capability and it may be outsourced if the in-house team does not have the expertise. Forensics may be used for legal gathering of incident artifacts for the prosecution of a perpetrator or to simply investigate the root cause and determine a method to thwart future incidents. Key concepts for forensics include treating the evidence the same as one would for a murder case; having the correct tools and expertise to perform forensic investigations. Training on the forensic tools is highly recommended, as time can be of the essence and completing a forensic investigation in a timely fashion may be warranted. If the aforementioned are not the intent of the enterprise, then outsourcing the forensic capability would be the better option. The enterprise incident response team will have the responsibility to understand basic forensics and how to fulfill the requirements of a first responder team without destroying forensic evidence. Having a contracted service for forensics in the event of an incident will be costly to the enterprise. Furthermore, the incident plan must account for when to enact the costly service. A hybrid approach may be the most cost-effective and functionally correct method for implementing a forensic capability.
In the hybrid model, the enterprise builds a forensic capability, but once it is determined that the incident affects a certain type of system or criticality, then the third party is brought in to perform the forensic investigation. The internal team handles all other investigations and any decision to enact the third party will instruct the internal team to take the agreed forensically sound steps to stop the immediate threat. There are several ways to approach the forensic capability in the enterprise and the decision to use a specific model must be carefully assessed based on current expertise, the budget to invest in the capability and training, and risk reduction leveraging a certified third party.