Before diving into how to operate against security threats with Elastic ML, let's provide a bit of context about the challenges that security teams face in terms of the volume and variety of data.

Even modestly sized enterprises can collect over a terabyte of data per day and keep that data for a 6-month retention period. This is an ingest rate of about 25,000 events per second if we consider that an event's size, on average, is 800 bytes. Bigger enterprises may collect 100,000 or more events per second.

For a security professional, the daunting challenge is not just simply coping with this amount of data and looking for the known bad behaviors that may exist, it is also challenging to spot what is anomalous compared to normal behaviors and being able to correlate multiple anomalies together to understand the entire life cycle of an emerging threat. Plus, you have to do all of this while simultaneously assessing the potential impact for the organization. Thus, it's not just about taking action on a single IoC, but rather tracing back from IoC to IoC to understand the source of an attack and take action from there to prevent any further similar events.

Timing is very important from the point where the IoC has been identified, because the more time it takes to find the source, the more vulnerable the organization assets are. This means that security teams can't wait to get an answer; real time, or near-real time to stay realistic, is mandatory for threat hunting from ingestion to anomaly detection and analytics.

We went through most of these requirements in the previous chapter, Chapter 4, IT Operational Analytics and Root Cause Analysis, but security analytics brings another level of expectations from the use profile standpoint that could not fit in a traditional security solution, such as Security Information and Event Management (SIEM), which is not scalable by nature.

In the field, we can see more and more legacy SIEM projects adopting the SIEM augmentation pattern, allowing the security team to combine the benefits of SIEM in terms of data collection, compliance reporting, and incident handling process, but also the benefits of the Elastic Stack in terms of being able to have all the data in one place, detect anomalies, and threat hunt in real time. Eventually, those SIEM solutions are replaced by Elastic when the functional needs are slowly implemented with features within the stack.

This migration path becomes even more natural with the help of Elastic ML, whose configuration and operation are agnostic of the type of time series or use case implemented. In other words, it can cope with versatility and a variety of data to analyze in real time, leaving the heavy lifting of data management to Elasticsearch.

Two key points stand out here:

• Having all of the data easily accessible in one place makes it possible to search, analyze, correlate, and gain insight with less time and effort
• The process of that analysis should be agnostic of the data types, offering flexibility over use cases, including ones that have yet to be conceived

In many legacy security projects, this is not the case: data is spread across multiple systems or datastores, which severely hinders the process of finding threats in the first place, but also relating detected threats to a potential overall attack. Legacy security solutions, such as intrusion detection system (IDS) platforms, are extremely efficient in detecting known threats based on a knowledge base of existing rules. However, being too specific limits the flexibility of detection, potentially missing events when the variety of data grows. Even worse, being too specific with a set of known detection rules enables the adversaries to know what types of behaviors are being looked for in advance, allowing the modification of behaviors to stay under the radar and avoid detection. We will see how the Elastic Stack enables the security team to operate flexibly and face complex and versatile attacks later.