Multi-Tenant Cloud Environment

Threat hunting helps reduce the operation overload as well as boosts identifying threats in an on-premises environment and prioritizes remediation planning.

Cloud computing is more common than ever, not only for organizations using different cloud services, but also using multiple CSPs, which is also known as a multi-cloud environment. In other words, you have one organization or community using several CSPs in once place. These CSPs could be private, public, or even hybrid cloud. On the contrary, one cloud provider serves a variety of organizations and each organization or customer is known as the tenant; this is called a multi-tenant environment.

In this section, we discuss the common risks in multi-cloud and multi-tenant environments and how threat hunting helps to address these risks.

Multi-tenant is defined as multiple customers or organizations accessing the same CSP, whereby different systems (IaaS, PaaS, SaaS) from different companies are hosted on one pool of physical servers. Multi-tenant architectures apply to both public cloud and private cloud environments, allowing each tenant's data to be separated from each other. There are several multi-tenancy model types, all with varying levels of complexity and costs. Multi-tenancy is one of the main criteria in defining cloud computing, where different customers access the secured and isolated, abstracted system without interfering with each other or even noticing other users. One benefit of multi-tenanting is that many users can get access to one server and use the resources at maximum level with lower costs, all at the same time.

In a multi-tenant environment, customers enjoy the affordable cost and integration via APIs with other applications. Most of the asset maintenance and updating/upgrading is under the CSP roles and responsibilities. It is important that the shared responsibility model be discussed at the contract level. On the contrary, multi-tenant offers limited customization, security changes, and upgrades dependencies and the vendor lock-in issue.

In a nutshell, if customers need a particular customization or security features to be applied, it depends on CSP approval and their roadmap. Other CSPs might provide the same customization, maybe with a lower cost on demand without waiting for the CSP roadmap.

In addition to these operational challenges, there are a number of cyber risks and compliance issues that led to many organizations being hesitant to adopt this new approach and migrate to the cloud. Many of the cyber risks in the multi-tenant (single CSP) environment also exist in a multi-cloud environment, which is discussed in the next section.

Such challenges drive many customers and organizations to look into the multi-cloud environments that address such limitations.

Threat Hunting in Multi-Cloud and Multi-Tenant Environments

An organization should know the risk of migrating to the cloud, be it multi-cloud or multi-tenant cloud, in advance, such as data security and privacy, shared model responsibility, increased dependencies on vendors, poor configuration, and access management, especially on admin and privileged users, unpatched VMs and platforms/infra, and many more. Of course, many security controls have been put in place, such as separation of duties, network security tools, integration and single management console and dashboard, IDS, IPS, encryptions, and other next-generation fantasy stuff.

However, if there is no proper monitoring considered, regardless of how secure the CSPs and on-premise platforms are, it will be difficult to reduce the cyber risk and identify them before an incident takes place or identify the APT attacks that could potentially be in the environment.

As discussed earlier in this chapter, threat hunting and threat modeling should be applied in the cloud environment as well.

As organizations migrate from a physical infrastructure/on-premise environment to a cloud environment, threat identification will be more challenging due to difficulties in compliance and configuration transparency, remote data sources and infrastructures, core security capabilities, and the number of APIs. In a nutshell, as the attack surface is expanding, threat hunting requires more attention, so the challenges related to threat hunting in a multi-cloud and/or multi-tenant environment must be discussed.

Now consider the challenges of a multi-tenant environment whereby different customer data types are hosted; the threat hunter could possibly breach the data privacy if certain data protection controls as well as role-based access controls (RBAC) are not in place.

On the other hand, threat-hunting complexity increases on a multi-cloud environment when you have heterogeneous infrastructures, services, vendors with different security postures, unbalanced security postures, different security controls in place where proper integration might not be in place, and more. Poor visibility on data at rest, in transit, and in process prevents the threat hunters from performing the threat hunting at all levels, and this is limited based on the access provided by the CSP and customers.

While customers can define application logs to be pumped into the SIEM or log collectors for threat hunting and incident management correlation, a number of the logs such as hypervisor logs, packet details, or system logs might not be shared with the customers.

Another challenge that threats hunters usually face in a modern cloud environment is that many organizations are moving to containerization and serverless approaches, whereby the instance constantly turns off and on whenever needed. This is typically a case where the organization is heavily invested in DevOps tools.

One of the main benefits of the cloud is you pay as you go. While the customer only pays for the amount of resources used, such as bandwidth, data, and processing resources, threat hunters should choose the required data carefully, as it will impact cost.

There are different types of threat-hunting commercial products and services offered by different vendors and service providers that could meet these challenges. Security-as-a-Service is one such approach that allows customers to subscribe and deploy into their multi-tenant cloud environment to boost their incident response and threat-hunting capabilities.

During threat-hunting maturity readiness, it is advisable that you build a successful threat-hunting program by starting small. As the maturity level increases, you then expand the scope of threat hunting to avoid overwhelming your hunters with thousands of events. In a multi-cloud or multi-tenant environment, the number of alerts received by log collectors and hunters increases tremendously over a period of time. Using the reliable threat source, you will now have an opportunity to use automation in threat-hunting for building and maintaining various use cases and hypothesis in a timely manner.

Scope and Type of SOC

There is no one-size-fits-all approach when it comes to establishing and creating an SOC. An SOC should be scaled to either the global footprint of the organization or to the span of control for the business sector that operates the SOC.

Many large corporations have a main SOC, called a Global Security Operations Center (GSOC), which could be supported by smaller (child) SOCs around the globe.

These smaller SOCs also provide alerts and threat feeds to the main GSOC. There is always some redundancy built into SOCs, so if one is offline, another one can manage the load and sustain operations.

Services, Not Just Monitoring

Define all your SOC requirements and then develop a roadmap. Which services do you want to create within your SOC? It's not just about monitoring a few logs and events from your infrastructure, which used to be the case in a traditional SOC.

Modern SOCs provide a number of proactive services, which are resource-intensive operation tasks. This includes:

• Security management and event monitoring
• Security orchestration, automation, and response (SOAR)
• Log management
• Incident response
• Vulnerability management and penetration testing
• Managed defense and red team/blue team service
• Threat hunting and threat monitoring
• ICS and SCADA security monitoring
• Business resiliency
• Controls and compliance reporting based on certain regulatory requirements, such as PCI-DSS, HIPAA, SOX, and FISMA

SOC Model

The SOC model can include an onsite (in-house) dedicated SOC, outsourced (remote) SOC, or both.

Today's cybersecurity market is full of small, medium, large, and boutique managed service providers who offer you specific services for your security monitoring based on your specific needs. This includes threat hunting, penetration testing, red team/blue team, etc.

Each model has its own pros and cons. You choose from remote monitoring and analysis, a dedicated SOC center operated on your premises, or, for maximum security and cost effectiveness, a hybrid solution, which combines both.

Define a Process for Identifying and Managing Threats

This is the most crucial step and a key process one must embark upon when building an SOC, which is a threat modeling exercise. Threat modeling entails answering the following questions:

• Which threats are applicable to my organization and do I care the most about?
• What do these threats look like and for which sets of assets?
• How does SOC identify, detect, and block these threats?
• What run book and playbook do I need to create?

Once these questions are answered, playbooks are built in order to document how to respond, set severity, and escalate these specific threat types. Other important processes to consider are shift time and models.

Tools and Technologies to Empower SOC

The tooling in the SOC (see Figure 2.4) is a mixture of centralized breadth capabilities and specialized tools that enable high-quality alerts and an end-to-end investigation and remediation experience.

One key element for managing an SOC is to ensure that the technology and platforms sync well with the information systems. You want to build a tool chest of software that can perform security audits, log analysis, penetration tests, and port scans. There are many commercial systems that can provide intrusion prevention, intrusion detection, and analyses.

You should have a good service/help desk ticketing system, documentation system, and inventory system. You should also stay on top of all the security trends by connecting to websites and security threat feeds that will update you on current events and on the threat landscape.

People (Specialized Teams)

You need to organize the SOC with specialized teams, allowing them to better develop and apply deep expertise, which supports the overall goals of reducing time to acknowledge and remediate threats.

Figure 2.5 represents the key SOC functions within Microsoft: threat intelligence, incident management, and SOC analyst tiers.

You need a highly trained team of security analysts who are familiar with security-based alerts and scenarios. As security threats are constantly changing, analysts need to adapt and think outside the box when it comes to solving problems. Attacks can come in a variety of different forms and types, so having people who can learn on the fly is important.

Finding the right people can be challenging. You may need to evaluate other options, such as outsourcing (via managed security service providers, known as MSSPs) or even hiring specialists to provide surge incident response (IR) support. I believe a hybrid of these options functions quite effectively.

In a SANS Incident Response report, 61% of respondents called upon their own surge staff to manage serious incidents and 58% had a specialized response team.

Cyberthreat Detection

The first cybercrime took place many decades ago, when Bob Thomas developed the first computer worm in 1971. It showed a popup message on victims' computers that read “I'm the creeper, catch me if you can.” Following that, cybercrimes started to take place one after another. The first Denial of Service (DOS) attack happened in 1989, by launching a worm attack and slowing the Internet for a few days. The estimated damage was about $100,000, up to $10mil.

During those early days, organizations were always looking for a way to fix the unknown and unpredicted incident. Gradually, software and security vendors come up with patches and hot fixes and anti-hacking, antivirus, and malware software and hardware was invented.

Over time, with advancing technologies, threat detection improved, using known attacks and viruses, malware signatures, and indicators of compromise (IOCs). All companies invested in signature-based detection control mechanisms. A lot of security products and server providers made a great deal of money by selling the latest signatures.

Following that, the speed and accuracy of identifying the known signature and IOCs became the winning point. That is where Intrusion Detection and Prevention Systems (IDPS) came in. Soon, organizations realized that while signature-based methods are perfectly accurate and fast enough (thanks to high-speed processors and memories), they could only mitigate against known signatures and IOCs. Hackers (and even script kiddies with minor changes in their code) could manage to penetrate the network and perform their malicious activity. See Figure 2.6.

Thereafter, threat detection from signature-based options changed to behavior and anomaly based detection. Technologies expanded and a variety of detection methods were introduced by security researchers.

Threat detection is no longer about identifying IOCs; in fact, now it's all about hunting the threats in the wild. Not to mention that regardless of the amount of investments and complex technologies and creative and strategic security defense controls, intruders can still manage to break into your network and remain undetected. Threat hunting allows you to proactively look for known and unknown threats and the modus operandi to mitigate the risk and prevent another headline in the news. Threat hunters constantly look for suspicious activity, codes, and unauthorized activity prior to data exfiltration, ransomware attacks, or Advanced Persistent Threats (APTs).

To avoid operational day-to-day activity and meeting the OPs deadline and SLAs, this role should be absolutely separate from operational duties and mainly focus on data collection and hunting. It should define and refine new procedures and automate and maintain them to increase efficiency.

Threat hunters may be part of SecOps organization, which may also depend on the size of an organization. They may also sit between the SecOps and enterprise and security architects teams and the business units.

Threat-Hunting Goals and Objectives

Threat hunting is like fishing in a big ocean. You need to know which ocean you are in and how much you know about it—how deep can you go, what type of fish are you planning to fish, what is your existing equipment, how skillful are you, and more. However, knowing what fish you are looking for is as important as knowing what type of threats you are looking for in your environment to hunt.

Threat hunting has been introduced to meet different goals and objectives. For instance, based on my experience, there is no organization that can claim they are running with fully 100% patched systems free of any vulnerabilities or misconfigurations. Not to mention that there are, from time to time, unknown assets in organizations that just pop up like mushrooms. On the other hand, there are many companies running on legacy infra and apps that are EOL and EOS. Businesses have accepted the risk and are running the operation by relying on existing mitigation controls.

One of the main objectives of the CISO should be to get to know the crown jewels and vulnerable systems and prioritize them based on system criticality. Then, based on threat modeling, prepare the hunting objectives and scope. Scoping is discussed in more detail in "Threat Modeling and the SOC" later in the chapter.

The next objective is to reduce the false positives and noise in the SOC and prioritize the events by relying on threat hunting and threat intelligence IOCs. Security Operations Centers usually receive thousands of false and true positive events and alerts. Threat hunting helps the SOC team prioritize which events to investigate and respond to. In complex and large organizations with heterogeneous software, applications and middleware, databases, operating systems and mainframes, and hardware and network devices, there are tons of patches to be deployed. Threat hunting, with the aid of reliable IOCs and TTPs, helps the threat and vulnerability management team prioritize which security patch should be deployed first and which vulnerability should be remediated.

Threat hunting helps the incident response and digital forensics team identify where the first compromise was initiated, what other machines are involved in this series of exploits, and when, how, and what the modus operandi was.

CISOs and security operations teams should take into account the lesson-learned strategy and define the goal of threat hunting. There is a saying that all companies have been compromised, the only difference is whether they know it or not.

Research and news show that many cyberattacks take place when the cybercriminal and attackers had presence in the organization network months in advance. While some attackers may act noisy and launch ransomware and DDOS attacks, either destroying the assets or being caught by security control systems, many APTs successfully evade such security controls and silently coexist in the environment without raising a flag or alarm. Threat hunters, using the TTP and known vulnerabilities and recent attack news and intel, should look for possible incidents and inform the CISO by taking precautionary action.

The output of such goals helps the remediation team to know, among reams of the missing patches, which patch and vulnerability should be prioritized, which system configuration should be fixed first, and identify the gap in process documents and standards. Hence, threat hunting is not just another checkbox for regulatory requirements or best practices to check. It requires a certain level of skill, organization security defense maturity, and the right equipment in place.

Senior management should also understand that the effectiveness of threat hunting is based on the available visibility, maturity level, and defined scope. Hence, it's best to define the scope based on the crown jewels and relevant threat actor present in their industry. As maturity levels improve, the visibility and coverage should be expanded. In that case, the ROI will be more justifiable.

Threat Modeling and SOC

Threat modeling is an important task when planning and building a successful SOC. Think about threat modeling as a practice to understanding your adversaries, their methods for attacking your organization, and how you will identify and respond to the attacks. Threat modeling will help you define scope, determine interesting log sources, select appropriate tools and technologies, and many other aspects to make SOC successful.

Make threat modeling exercises a standard practice to model and remodel threats. In the absence of proper threat modeling, you will end up wasting significant time, money, and energy in collecting irrelevant logs and investigating events that don't matter much. Threat modeling will also help you create appropriate use cases and filter out noise. Threat modeling details are covered in the following sections.

Invest in People

Because threat hunting is concerned with emerging threats rather than known attack methods, people take the lead. It's therefore important that they have the time and authority to research and pursue hypotheses. This isn't possible if they are bogged down with security alerts. Many SOCs, including those at Microsoft, establish a three-tier model to address known and unknown threats. Tier-1 and Tier-2 analysts respond to alerts. Tier-3 analysts conduct research focused on revealing undiscovered adversaries. See Figure 2.7. You can learn more about how Microsoft organizes its SOC in “Lessons Learned From The Microsoft SOC—Part 2a: Organizing People.”

Security Analysis

Threat hunters should be able to understand and work with network packets, parse the IOC feed, work with log correlation tools and SIEM, use security appliances such as firewall and IDS, reverse engineering malware, and know about exploits. They must constantly update their knowledge of threat actors, attack tools, and tactics. Similar to any other security analyst role, they must know how different operations work, what their critical files and processes are, and what their basic network protocols and services run. For instance, a threat hunter should be able to profile every department's normal behavior, like working hours, usual activities, and required tools, especially administrative tools such as PowerShell. By knowing what is considered “normal,” anomalies can be identified with a minimum of false positives.

Data Analysis

Threat hunters should be able to know how to combine different data together in a structured and unstructured manner. They need visual demonstrations, machine learning tools, elastic searches, and relevant skills. Due to the high volume of data that hunters deal with, experience and knowledge in Machine Learning (ML) helps them by training the ML tools about normal behavior, cluster the known, bad, and questionable activities, and finally group them for further profiling and investigations.

Programming Languages

With most IT and security roles, basic skills in programming and scripting is always in good demand. However, if you are expecting your team to be able to customize and automate the procedures, reverse engineer and perform data analysis, having knowledge of scripting languages such as Python, Perl, and C/C++ is mandatory.

Analytical Mindset

Hunters should have an analytical mindset to be able to develop different hypotheses and build various use cases. They need to analyze the output and tune their processes and procedures.

Soft Skills

Threat hunters need to talk with different technical and nontechnical coworkers from time to time. Soft skills help them build an efficient relationship and obtain the required info and support. Threat hunters also present reports to management about the latest states. Knowing how to write technical and nontechnical reports tailored for different audiences is highly desirable. Based on the seniority level of this role, deep knowledge and practical hands-on skills may vary.

Outsourcing

Many organizations, due to a lack of security talent pools, prefer to outsource managed security services. While managed services provide advanced technologies and rich intel feeds, human factors play a critical role in threat hunting. CISOs need to ensure that threat hunting is not a one-day contract kind of job. Consultants can't simply walk in, set up the platforms, run the exercise, wash their hands, and walk away. It requires the constant presence of hunters who know the business and IT environment and work on cases and artifacts 24/7.

To summarize, based on my experience, threat hunters require the following core skills:

• A mindset of curiosity
• Log analysis and general analytical skills
• Understanding of normal network behavior
• Understanding of normal endpoint user and application behavior
• Understanding the threat landscape and the use of CTI
• System administrator experience across Windows/Linux/common security products

Foundational Metrics

• Scope: One of the key metrics that needs to be defined is the total number of organizational assets and the number of included assets into the threat-hunting scope. You can break down this metric into critical and very important systems and others. In that case management always has a proper overview of their crown jewels. To demonstrate the improvement and expanding of the scope over time, always keep the trend liner charts.
• Visibility: Visibility metrics are important. Dropping the number of assets from what have been defined could be a red flag. Threat hunters need to keep the number in mind at all times, and if there are any changes, take the required action.
• Functionality metrics: Installed security tools and sensors should function correctly and maintain a healthy state on all assets. For instance, say the antivirus or EDR agent is not working properly, meaning it either failed to report back to the console, didn't get the latest update signature, or couldn't send back the logs to the server or execute the administrative commands. This metric is one of the important ones, because security tools not functioning could be a sign of compromise as well. Many hackers not only bypass security tools, but they also sometimes disable them to avoid making any noise at the SOC level.

Low compliance rate of any of these metrics leads to inaccurate threat-hunting reports.

Operational Metrics

• Number of hunted items vs. number of incidents
• Number of open hunting investigation vs. number of closed based on defined SLAs
• Number of hunted items based on environment and business criticality
• Detecting time
• Number of total hypothesized vs. verified hypothesis
• Number of hunts based on the threat intelligence feeds
• Number of automated procedures vs. manual procedures
• Duration of each hunting process end to end (categorized based on automated and manual)
• Total relevant threat actors specific to an industrial, directly targeted the organization vs. number of defined procedures and used cases
• Total number of reported hunts vs. number of open and closed issues based on hunting (remediation)
• Duration of remediation from the time the hunt has been reported
• Used technique for hunting (% of technique's effectiveness)
• Data source used for each hunt
• Type of finding and root cause analysis (e.g., broken process, system malfunction, human error, misconfiguration, data breach, and other cyber incident categories)
• Type of vulnerabilities

These metrics can be used for different purposes. In addition to having oversight and monitoring the threat-hunting program, they also show how effectively the program is serving the organization. On the other hand, they can help the CISO obtain strategic oversight as well. For instance, most of the reports show the number of security agents having a malfunction, which would be an alarming message to infra and the SecOps team. The following section explains how threat hunting can help the effectiveness of other compliance functional and operational reports.

On the other hand, the total number of reported hunts in comparison to timely remediation of them shows how proactively another team is acting. Reporting hunts does not prevent the cybercriminals; remediation does.

The total number of use cases and procedures demonstrates how the threat-hunting team operates—passively or proactively. Similar to SOCs, many of the defined use cases are outdated and obsolete with invalid signatures and nonexistent IPs, focusing on low-risk items.

Meeting the investigation SLA, threat-hunting lead, SOC, and CISO should determine whether automation is in place and if the number of resources and staff are adequate enough to meet the objective.

The patch compliance report shows the overall patch deployment state, representing most identified compromise could potentially demonstrate the accuracy and effectiveness of the patch management program as well. This same strategy can be applied to security configuration or vulnerability management reports as well.

The number of human errors, such as initiated cyber incidents, is identified due to social engineering or phishing attacks. Maybe the information security awareness program is not proactive enough and should be revised (see Table 2.2).

Ultimately, the value of any metric is how useful it is to the recipient, often a senior manager such as a CISO, so all metrics should be developed in collaboration between the threat-hunting team and its relevant senior managers.

Adopt organizationally relevant metrics, such as Table 2.2, to drive improvements and show the return on security investment (ROSI) over time.