Chapter 17. On Threat Intelligence
In this chapter, I will discuss the consumption and processing of threat intelligence. Threat intelligence is a process of sharing data about attacks—victims of attacks or investigators share contextual information. Threat intelligence can comprise a variety of data sources, including geolocation data, reputation information (often gussied-up geolocation data), and information on attacker techniques, malware signatures, and vulnerabilities.
I have divided this chapter into two major sections. In the first section, I discuss threat intelligence source data: the type of information that comprises threat intelligence, and formats you can expect to receive this information in. In the second section, I discuss the process of setting up a threat intelligence program for an organization.
Defining Threat Intelligence
For our purposes, I am going to define threat intelligence data as contextual data collected from multiple sources to improve response. By contextual data, I mean that threat intelligence is data collected to enhance event-based data such as IDS alerts or flow data. Threat intelligence data is collected and synthesized from multiple sources; this includes actions more related to conventional intelligence gathering. Finally, threat intelligence data is used to improve incident response—it provides information for hardening networks, identifies indicators of higher-risk attacks, and provides a means for operations teams to identify common threads.
I must emphasize an important point here: threat intelligence data is supplementary and contextual. You cannot run a detection program on threat intelligence alone; there must be some set of event data to apply threat intelligence to. A threat intelligence feed without a primary data source (incidents, network traffic) is useless. A threat intelligence feed that isn’t relevant to the types of data you’re collecting is equally useless.
Raw threat intelligence data appears in network feeds, in the same way that conversations appear in the air—it doesn’t become a product until someone analyzes it and packages it. In this sense, “threat intelligence” is a term now applied to formerly informal or largely corporate processes of information sharing. That’s a cynical way of stating a good thing; in the last few years, there’s been an increasing recognition that threat intelligence needs to be treated more systematically, and that information security isn’t a purely technical effort. This includes the establishment of standards with some traction (e.g., STIX and TAXII), the increasing acceptance of threat intel platforms (e.g., MISP), and the development of open organizations for sharing threat intel (the Information Sharing and Analysis Centers, or ISACs in particular have taken charge on this).
Data Types
Threat intelligence is a wide, undefined topic, and it’s an area where people are very eager to sell data. In this section, I am going to (loosely) classify threat intelligence data along several different qualities. Specifically, these qualities are the type of intelligence delivered, the maturity of the data, its origin, and its format.
Types of threat intelligence data
We will consider three major types of threat intelligence data: network reputation information (e.g., IP addresses, URLs), IOCs (e.g., malware hashes), and TTPs (tools, techniques, and procedures).
Network reputation information refers to data that scores URLs, IP addresses, or other indicators for hostility. A reputation threat feed will generally consist of timed and dated lists of keys, and an explanation for why a key is marked as a threat. As a rule, reputation data is something that you can directly plug into your access control lists (ACLs) or network filters, or automatically annotate your SIEM data with.
A free example of this type of information is the rules provided by the Emerging Threats database, a free repository of threat intelligence encoded as lists of IP addresses and IDS rulesets. This information is subdivided into different categories, such as botnet command and control, and various named groups.
Indicators of compromise (IOCs) really describe anything that can be used to indicate that a host has been compromised by malware. IOC data is forensic and largely host-based, including information such as malware hashes or credentials.
TTPs are usually reports describing the methods and techniques used by attackers, meant to be read and processed by people for situational awareness. These can be general, perhaps signaling awareness of a new form of attack, or quite specific, detailing behavioral signatures for particular malware groups.
Finally, there is an implicit category of threat intelligence data that I think of as secondary. Secondary data is threat intelligence that is delivered to platforms within your organization for their use. The classic example is antivirus signatures—when you subscribe to an AV service, you receive regularly updated intelligence delivered to your AV clients.
Maturity and format of threat intelligence data
A key workflow problem for any threat intelligence organization is the process of collecting, vetting, and reformatting the data. Threat intelligence data comes in a number of distinct formats, most of which are not directly accessible to the end users.
In addition, we must discuss the current efforts toward developing standards. In the last decade or so, there have been a number of efforts to develop threat intelligence formats. The first of these were the IODEF (Incident Object Description Exchange Format) and IDMEF (Intrusion Detection Message Exchange Format) standards developed by the IETF in the mid-2000s.
Several providers have also created their own formats. The most notable of these is MISP, the malware intelligence sharing platform, originally built by CIRCL (Luxembourg’s CERT). MISP is an open source project with a thriving community. Similar projects include OpenIOC, an outgrowth of Mandiant’s (now part of FireEye) forensics tools, and AlienVault’s Open Threat Exchange (OTX) standard.
The other major project is a collection of OASIS-backed standards grouped under the moniker STIX (Structured Threat Information Expression). The actual project includes several standards; STIX itself is the standard for representing IOCs, while TAXII (Trusted Automated Exchange of Intelligence Information) is the transport protocol.
So, with all these standards, what ones to use? Probably both MISP and STIX—with IOC data, the syntax of how the data is recorded is usually less important than the semantics. All IOC reporting standards are some form of key/value pairing, and the more important questions are what the keys are, not how they’re stored, and whether there are interoperability tools.
With that in mind, here are some common types of IOC fields you can expect:
- File hashes
-
Hashes of specific files associated with a compromise. Given the rapid pace at which hashes are replaced and move (see Chapter 6), IOC formats usually specify multiple common hashes. These include MD5, SHA1, SHA256, and the longer formats (like SHA384 and SHA512). IOCs will usually hash the malware files, but also may hash filenames.
- Network addresses
-
Domain names, IP addresses. These are often categorized as botnet sites, command and control servers, drop sites, and the like.
- Target information
-
Email addresses, financial information (ABA routing numbers, credit card numbers, etc.).
- Windows data
-
Portable Executable (PE) hashes, Registry information, and other Windows-specific data.
A final note: the tools are, to be honest, less important than the communities. The intent of all of these standards is to provide a way to share information that is more structured than an email, but the standards are useless without intelligence to populate them. In the case of the MISP and STIX standards, you should really pay the most attention to their resources pages to find the organizations that are sharing data, and which ones are most relevant to your organization.
Provenance of threat intelligence data
The first and oldest sources of threat intelligence data are mailing lists and publicly shared resources curated by private organizations such as the Spamhaus Project and the various national CERTs. These organizations have a distinct producer/consumer separation, and may be government or privately funded.
Information sharing organizations are another common source of this data—but I have to emphasize the word sharing. These organizations have a clear concept of ingroup and outgroup, and if you want information from them, you have to find some way to join the group. The most easily accessible and directly useful for many organizations are the various information sharing and analysis centers;1 these groups share information about sector-specific threats.
There are also proprietary threat intelligence feeds: subscription services that deliver threat intelligence data. These services, when you join them, provide a curated collection of threat intelligence information that is updated and maintained on a regular basis. This information is usually the best formatted, but it is also a total black box. You do not know where the information comes from.
Finally, there’s “hidden” threat intelligence data. When I say this data is “hidden,” what I mean is that you’re buying it through another source—for example, you are getting threat intelligence via your AV and antispam systems, and may be getting it via other appliances as well.
Creating a Threat Intelligence Program
An effective threat intelligence program will focus on techniques for effectively incorporating threat intelligence into the operational workflow with minimal pain and frustration for the ops team. The responsibilities of the analysis team include identifying the goals and domain for threat intelligence data, and determining what type of data to purchase and apply.
Identifying Goals
The easiest way to waste money on threat intelligence data is to let the vendors decide what threat intelligence you buy. The first step the threat intelligence team needs to focus on is determining what the data will be used for, what data is relevant, and what is redundant.
When discussing what the data will be used for, there are a couple of basic goals for the team to consider. These are:
- Improving hardening
-
In this context, this means developing stronger and more rigorous defenses. Hardening requires that the ops team have a history of effectively aggressively blocking material—they have the experience, they have the policy support, and they have user buy-in. Network reputation information and up-to-date rule feeds are useful examples of this category of defenses.
- Annotating attack data
-
Reputation feeds and network IOCs are useful sources for annotating attack data, which helps analysts determine in turn whether an attack represents a significant threat to the organization. Reputation feeds are usually reasonably easy to convert into actionable data such as IDS signatures or firewall rules, since they are generally just lists of IP addresses or names anyway. IOCs require more contextual information, understanding what the indicated address means and how it appears.
- Supporting forensic investigations
-
Malware IOCs support investigations by providing clues that the team can use to identify how a system is compromised and additional cues for finding compromised system. For example, if a specific malware hash is an IOC for an attack, and that same attack has a specific command and control, they can query traffic for the command and control address to identify other compromised hosts.
- Keeping current
-
For mature organizations, threat intelligence services that keep you up-to-date on current attacker TTPs are a vital tool for identifying potential new threats.
As a rule of thumb, annotation is usually the easiest task to start with—annotating an IDS feed with high-quality threat information can help an analyst get through false alerts more quickly and focus on the higher-priority alerts. Forensic investigation support assumes you have a team that is conducting forensic investigations and malware analysis.
In addition to operational impact, determine what sector you’re involved in and how that impacts your threat intelligence needs. In particular, consider:
-
Your national affiliations.
-
Your industrial sector.
-
What category of attacks you expect to deal with. Are you expecting to be the target of spear-phishing campaigns? Do you have a large number of industrial systems? Do you build/prototype software?
-
Your software/hardware inventory.
-
Whether you’re already purchasing hidden threat intelligence (e.g., through regularly updated antispam or antivirus tools).
Underpinning all of these issues is the question: what are you going to do with the data? If you don’t run an active remediation program, then bleeding-edge IOCs aren’t going to help much.
Starting with Free Sources
Start with the newspaper—the easiest way to start collecting threat intelligence data is to read publicly available feeds of threat data. This includes reports from organizations like Cisco’s Talos, Mandiant, FireEye, and the like. In addition, use news sites such as Dark Reading as a jumping-off point.
The next step is to look into the organizations that provide threat intelligence data for your goals and sectors. If you have an industrial or national CERT, start checking to see what intelligence it provides. If there’s a relevant ISAC for your organization, see about joining it. Also check organizations such as FIRST. Mailing lists such as full-disclosure (for malware) and NANOG (for networking issues) are critical.
Determining Data Output
The next phase of your threat intelligence project’s maturity involves staging up the data you’re acquiring for use. Based on the four classes of data that I discussed previously, you can expect the output to be as follows:
-
Hardening data will translate into access controls. This includes firewall and ACL rules, as well as specialized alerts such as SiLK scripts based on the type of data.
-
Annotation data is going to be fed to the analyst, so you should expect this data to be as close to the analyst as possible: well-maintained databases plugged into your SIEM, or merging the information with event data as it comes online.
-
IOC data will need to be searched; for that purpose, you need some kind of platform for managing and interchanging IOCs. Examples include MISP.
-
TPP data needs to be distributed. Regular roundtable meetings discussing the current state of the practice and reviewing the latest intelligence are a reasonable approach. Note that TPPs may be most effective as user briefings—for example, presentations to your ops team to provide them with up-to-date threats, or presentations to your C-suite to explain what problems the ops team is facing. An effective threat intelligence program will require multiple outputs for multiple audiences, delivered at varying levels of detail.
Purchasing Sources
As security companies mature, they tend to offer threat intelligence services. There’s a simple reason for this: threat intelligence is gathered by observing attacks in action, and if your company is spending all of its time dissecting attacks, examining malware, and collecting information on how the internet operates, threat intelligence is a perfectly logical product. A good number of network security devices are hardware platforms for delivering threat intelligence—what you pay for, basically, is expertise.
With that in mind, the savvy consumer is going to quickly recognize that a lot of companies are selling threat intelligence data and aren’t going to ask you questions about whether the data is relevant. Your responsibility as the consumer is to figure out what sources are germane to you and seek them out.
Questions to ask when evaluating the data:
-
Does the data overlap other types I’ve already acquired? It’s a fair step to begin by collecting threat intel feeds within orthogonal domains (for example, TTPs for situational awareness, then IP reputation data, then a malware feed, then IOCs).
-
Does the data overlap data I’m getting elsewhere? For example, if I purchase an IOC feed, am I getting better results than I do from a mailing list or other public source? Using the free data as a reference point is useful here.2
-
Are the feeds relevant and timely? Keep track of how much the ops team uses threat intelligence results—if an IOC feed isn’t helping your ops team, consider whether it’s worth paying for.
Brief Remarks on Creating Threat Intelligence
This chapter is primarily focused on threat intelligence from a consumer perspective. I view creating data as a complementary task after you have a working program running. To create threat intelligence data, you have to have threats to learn from, which means both that you’re observing threats and that they’re ones you don’t have prior knowledge of. So, the prerequisites for creating threat intelligence include:
-
A threat intelligence program mature enough to determine whether what you’re dealing with is novel.
-
Manpower to investigate threats. This likely involves at least a working forensics and remediation program (more than just patch and restore).
-
Channels for sharing the data. If your threat intelligence is mature enough to determine novelty, you should be in contact with ISACs and other organizations to share your data with.
-
Enough incidents to generate intelligence.
On the last point, while you can generate intelligence from being hit repeatedly, if you run a large enough network, you can also instrument your network to proactively collect and share threat intelligence. In particular, you can collect dark space data, run honeypots to collect intelligence, scan your web logs for IOCs, and share that information. The end state of many information security companies is threat intelligence, because they’re capitalizing on the most basic resource needed to start creating threat intelligence: a plethora of different attacks to examine.
Further Reading
-
The ACM Workshops on Information Sharing and Collaborative Security (WISCS) are an excellent source of research and technology on threat intelligence.
-
H. A. Slatman, “A Curated List of Awesome Threat Intelligence Resources,” available at https://github.com/hslatman/awesome-threat-intelligence. This is a giant curated list of threat intelligence feeds and tools, well worth perusing.
-
J. Dietle, Effective Threat Intelligence: Building and Running an Intel Team for Your Organization (CreateSpace Independent Publishing, 2016).
1 A comprehensive list of ISACs is available at https://www.nationalisacs.org/member-isacs.
2 One thing to note in passing: even if all the threat intelligence feed does is repackage public sources, you may want to ask whether you want to spend your time repackaging the public sources.