Chapter 12: Analyzing and Detecting Crypto-Mining Attacks

Crypto-mining attacks are becoming more notable as blockchain and cryptocurrency are becoming more and more popular. Cryptocurrency is earned as the transaction fee of decentralized transactions on a blockchain for utilizing compute resources. The process of earning cryptocurrency for validating transactions using compute resources is called crypto-mining and is conducted by a software called a crypto-miner. Security researchers have found hacking incidents related to various crypto-miner binaries running within victims' infrastructures. The default openness of Kubernetes clusters and the availability of the extensive compute power required for mining makes Kubernetes clusters a perfect target for crypto-mining attacks. The complexity of Kubernetes clusters also makes crypto-mining activities hard to detect.

Since we've already been introduced to different Kubernetes built-in security mechanisms and open source tools to secure Kubernetes clusters, we'll now look at how to use them in a concrete scenario. In this chapter, we will first analyze a couple of known crypto-mining attacks, then we will discuss the detection mechanisms for crypto-mining attacks by using open source tools. Last but not least, we will recap the topics we discussed in previous chapters and see how they should be applied to defend our environment against attacks in general.

The following topics will be covered in this chapter:

• Analyzing crypto-mining attacks
• Detecting mining attacks
• Defending against attacks

Analyzing crypto-mining attacks

In this section, we will first provide a brief introduction to crypto-mining attacks, and then we will analyze some publicly disclosed crypto-mining attacks. We hope that you are aware of the crypto-mining attack patterns as well as the flaws that make the attack possible.

An introduction to crypto-mining attacks

Blockchain forms the basis for cryptocurrency. In short, blockchain is a chain of digital assets represented as blocks. These blocks have information about the transaction and who was involved in the transaction as a digital signature. Each cryptocurrency is associated with a blockchain. The process of verifying transactional records is called mining. Mining adds history to the blockchain to ensure that blocks cannot be modified in the future. Mining is designed to be resource-intensive to ensure the decentralized property of a blockchain. By successfully mining blocks, miners earn the transaction fee that is associated with the transaction. So, if you have a laptop or PC, you can use it to mine cryptocurrency, too; but most likely, you will need some dedicated GPUs or specialized hardware, such as Field-Programmable Gate Arrays (FPGA) and Application-Specific Integrated Circuit (ASIC) in order to do a good job of mining. The availability of resources in Kubernetes clusters makes them an ideal target for attackers to earn cryptocurrency.

A crypto-mining attack is just like free riding on Wi-Fi. Just as your network bandwidth will be shared by the free rider, some (or most) of your CPU or computing resources will be occupied by the mining processes without your consent. The impact is also similar. If the Wi-Fi free rider is downloading movies via BitTorrent using your Wi-Fi network, you may have a poor experience while watching Netflix. When there is a mining process running, other applications running in the same node will be severely impacted as the mining process may occupy the CPU most of the time.

Crypto-mining attacks have become one of the most appealing attacks to hackers as it is an almost guaranteed way of gaining some benefits out of a successful intrusion. Thieves come only to steal or destroy. If disruption is not the goal of the intrusion, a crypto-mining attack is probably one of the main choices for hackers.

At least two ways for hackers to launch a crypto-mining attack on a target victim have been reported. One is through application vulnerabilities, such as cross-site scripting, SQL injection, remote code execution, and more, so that the hacker gains access to the system, then downloads and executes the miner binary. The other way is through a malicious container image. When a container is created from the image that contains the mining binary, the mining process starts.

Although there are different types of crypto-mining binaries available on the internet, in general, the mining process is computation heavy and occupies a lot of CPU cycles. The mining process sometimes joins a mining pool in order to carry out mining in a collaborative way.

Next, let's look at a couple of crypto-mining attacks that have occurred in the real world. We will discuss the flaws that made the attacks possible and we will look at the attack patterns.

The crypto-mining attack on Tesla's Kubernetes cluster

A crypto-mining attack on Tesla's Kubernetes cluster occurred in 2018 and was reported by RedLock. Although the attack took place quite a while ago, there are at least two things we can learn from it—the flaw that made the attack possible and the attack patterns.

The flaw

The hacker infiltrated the Kubernetes dashboard, which was not protected by a password. From the dashboard, the hacker gained some important secrets to access the Amazon S3 buckets.

The attack patterns

The hackers did a pretty good job of hiding their footprint so that they could avoid being detected. The following are a few patterns worth mentioning:

• The mining process did not occupy too many CPU cycles, so the CPU usage of the pod was not too high.
• Unlike most crypto-mining cases, the mining process did not join any well-known mining pools. Instead, it had its own mining server, which sat behind Cloudflare, a Content Delivery Network (CDN) service.
• The communication between the mining process and the mining server was encrypted.

With the preceding maneuver, the hacker purposely tried to hide the crypto-mining pattern so that they could evade detection.

Graboid – a crypto-worm attack

This crypto-worm attack was discovered by the Palo Alto Network Unit42 research team in late 2019. Although the attack was not directed against Kubernetes clusters, this was aimed at Docker daemons, which is one of the foundation pillars in a Kubernetes cluster. In one of the attack steps, the toolkit downloaded images containing a crypto-mining binary from Docker Hub and launched. This step can also be applied to Kubernetes clusters too.

The flaw

The Docker engine was exposed to the internet while it was configured without authentication and authorization. The attacker could easily take full control of the Docker engine.

The attack patterns

Once the hacker took control over the Docker engine, they started downloading a malicious image and launched a container. The following are a few patterns worth mentioning regarding the malicious container:

• The malicious container contacted the command and control server to download some malicious scripts.
• The malicious container contained a Docker client binary, which was used to control other insecure Docker engines.
• The malicious container initiated commands via the Docker client to other insecure Docker engines to download and launch another image, which contained the crypto-mining binary.

According to Shodan, a search engine for internet-connected devices, more than 2,000 Docker engines were exposed to the internet. The preceding steps were repeated so that the crypto-mining worm spread.

Lessons learned

To recap what we have discussed about the two known crypto-mining attacks, misconfiguration is one of the major issues that make hacking easy. Crypto-mining has some typical patterns—for example, the mining process will communicate with mining pools and the mining process usually occupies a lot of CPU cycles. However, hackers may purposely disguise their mining behavior to evade detection. Once hackers get into the pod, they can start the contacting command and control server to download and execute the mining binary; on the other hand, they can also start reconnaissance. It would be easy for them to make a lateral move if the security domain in your Kubernetes cluster is not properly configured. Next, let's use the open source tools we introduced in previous chapters to detect typical crypto-mining activities in Kubernetes clusters.

Detecting crypto-mining attacks

In this section, we are going to talk about detecting crypto-mining activities in the Kubernetes cluster with some of the open source tools we introduced in earlier chapters. We detect crypto-mining activities based on the known patterns of crypto-mining: high CPU usage, communicating to mining pools, the executed command line of miner, and the binary signature. Note that each individual measure has its own limitations. Combining them improves the efficiency of detection for sure. However, there are still some advanced crypto-mining attacks, such as the one that attacked Tesla. It's necessary for you to work with your security team to apply a comprehensive detection strategy for your Kubernetes cluster to cover all kinds of intrusion.

In order to demonstrate each tool to detect crypto-mining, we simulate a victim nginx pod:

$ kubectl get pods -n insecure-nginx

NAME                              READY   STATUS    RESTARTS   AGE

insecure-nginx-8455b6d49c-z6wb9   1/1     Running   0          163m

Inside the nginx pod, there is a miner binary located in the /tmp directory:

root@insecure-nginx-8455b6d49c-z6wb9:/# ls /tmp

minerd2  perg

minerd2 is the mining binary. We can assume that minerd2 is either seeded in the image or downloaded from a command and control server. First, let's see how monitoring the CPU usage can help detect crypto-mining activities.

Note

It is not recommended that you run crypto-mining binaries in your production servers. This is for educational purposes only.

Monitoring CPU utilization

As we discussed in Chapter 10, Real-Time Monitoring and Resource Management of a Kubernetes Cluster, resource management and resource monitoring are crucial to maintaining a service's availability. Crypto-mining usually occupies tons of CPU cycles, which leads to the CPU usage of a container or a pod reaching a significantly higher level. Let's take a look at an example by comparing the CPU usage before and after crypto-mining happens within an nginx pod:

Figure 12.1 – The CPU usage of an nginx pod before mining happens in Grafana metrics

The preceding screenshot shows the CPU usage of the insecure-nginx pod monitored by Prometheus and Grafana. In general, the maximum CPU usage rate is less than 0.1. When the crypto-mining binary is executed, you will find that the CPU usage skyrockets:

Figure 12.2 – The CPU usage of an nginx pod after mining happens

The CPU usage rate hikes from an average rate of 0.07 to around 2.4. No matter what happens behind the scenes, such a huge CPU usage hike should get your attention immediately. It's also quite obvious that even with this CPU surge, it doesn't mean there is a crypto-mining binary running inside the pod. The CPU surge can be caused by some other reasons, too.

On the flip side, if the hacker purposely restricts the crypto-mining attack progress, as was the case with the attack on Tesla, there may only be a little hike in the CPU that is hard to notice. Next, let's look at how Falco can help detect crypto-mining activities.

Detecting network traffic to a mining pool

One typical crypto-mining process behavior is where the mining process works collaboratively with other mining processes within the same mining pool for the purpose of mining efficiently. The mining processes communicate with the mining pool server during mining.

In Falco's default rules, there is one rule to detect outbound connections to known mining pools. Let's take a closer look at this rule. First, there are predefines lists for mining ports and the mining domain (https://github.com/falcosecurity/falco/blob/master/rules/falco_rules.yaml#L2590):

- list: miner_ports

  items: [

        25, 3333, 3334, 3335, 3336, 3357, 4444,

        5555, 5556, 5588, 5730, 6099, 6666, 7777,

        7778, 8000, 8001, 8008, 8080, 8118, 8333,

        8888, 8899, 9332, 9999, 14433, 14444,

        45560, 45700

    ]

- list: miner_domains

  items: [

      "Asia1.ethpool.org","ca.minexmr.com", "monero.crypto-pool.fr",

      ...

      "xmr-jp1.nanopool.org","xmr-us-east1.nanopool.org",

      "xmr-us-west1.nanopool.org","xmr.crypto-pool.fr",

      "xmr.pool.minergate.com"

      ]

Then, there is a predefined macro for network connections to the preceding miner ports and miner domains:

- macro: minerpool_other

  condition: (fd.sport in (miner_ports) and fd.sip.name in (miner_domains))

Besides the minerpool_other macro, there are two other macros for HTTP and HTTPS connections, respectively—minerpool_http and minerpool_https—and all of them combine to get the main detection logic:

- macro: net_miner_pool

  condition: (evt.type in (sendto, sendmsg) and evt.dir=< and (fd.net != "127.0.0.0/8" and not fd.snet in (rfc_1918_addresses)) and ((minerpool_http) or (minerpool_https) or (minerpool_other)))

The net_miner_pool macro is then used by the Detect outbound connections to common miner pool ports rule to detect outbound connections to miner domains:

# The rule is disabled by default.

# Note: Falco will send DNS requests to resolve miner pool domains which may trigger alerts in your environment.

- rule: Detect outbound connections to common miner pool ports

  desc: Miners typically connect to miner pools on common ports.

  condition: net_miner_pool and not trusted_images_query_miner_domain_dns

  enabled: true

  output: Outbound connection to IP/Port flagged by cryptoioc.ch (command=%proc.cmdline port=%fd.rport ip=%fd.rip container=%container.info image=%container.image.repository)

  priority: CRITICAL

  tags: [network, mitre_execution]

If there is a crypto-mining process running and communicating to the miner domains defined in the list, the alert will be triggered, as follows:

19:46:37.939287649: Critical Outbound connection to IP/Port flagged by cryptoioc.ch (command=minerd2 -a cryptonight -o stratum+tcp://monero.crypto-pool.fr:3333 -u 49TfoHGd6apXxNQTSHrMBq891vH6JiHmZHbz5Vx36nLRbz6WgcJunTtgcxno G6snKFeGhAJB5LjyAEnvhBgCs5MtEgML3LU -p x port=37110 ip=100.97.244.198 container=k8s.ns=insecure-nginx k8s.pod=insecure-nginx-8455b6d49c-z6wb9 container=07dce07d5100 image=kaizheh/victim) k8s.ns=insecure-nginx k8s.pod=insecure-nginx-8455b6d49c-z6wb9 container=07dce07d5100 k8s.ns=insecure-nginx k8s.pod=insecure-nginx-8455b6d49c-z6wb9 container=07dce07d5100

The Detect outbound connections to common miner pool ports rule is straightforward. If there is an alert generated by this rule, you should address it as high-priority. The limitation of the rule is also obvious; you will have to keep the mining domain and mining ports updated. If there is a new mining domain available or a new mining server port is used and they are not added to the Falco list, then the rule will miss detecting the crypto-mining activities. Note that the rule is disabled by default. As Falco needs to send DNS requests to resolve the miner pool domains, these DNS requests will be alerted by some cloud providers. A side note is that an open source tool such as Hubble from Cilium can help monitor network traffic.

Another approach is to use the whitelist approach. If you know the target port or IP blocks in the outbound connections of your microservices, you can create Falco rules to alert any outbound connection's destination IPs or ports that are not on the whitelist. The following is an example:

- list: trusted_server_addresses

  items: [...]

- list: trusted_server_ports

  items: [...]

- rule: Detect anomalous outbound connections

  desc: Detect anomalous outbound connections

  condition: (evt.type in (sendto, sendmsg) and container and evt.dir=< and (fd.net != "127.0.0.0/8" and not fd.snet in (trusted_server_addresses) or not fd.sport in (trusted_server_ports)))

  output: Outbound connection to anomalous IP/Port(command=%proc.cmdline port=%fd.rport ip=%fd.rip container=%container.info image=%container.image.repository)

  priority: CRITICAL

The preceding rule alerts any outbound connection to IP addresses or ports outside of trusted_server_ports or trusted_server_addresses. Given that the attack happened in Tesla, Falco will alert that there is an anomalous connection, even though the IP address looks normal. Next, let's look at another Falco rule to detect potential crypto-mining activities based on patterns in the command line.

Detecting launched crypto-mining processes

Stratum mining protocol is the most common protocol for the mining process to communicate with mining servers. Some mining binaries allow users to specify protocols to communicate with the mining pool server when executed.

In Falco's default rules, there is one to detect the crypto binaries' execution based on keywords in the command line:

- rule: Detect crypto miners using the Stratum protocol

  desc: Miners typically specify the mining pool to connect to with a URI that begins with 'stratum+tcp'

  condition: spawned_process and proc.cmdline contains "stratum+tcp"

  output: Possible miner running (command=%proc.cmdline container=%container.info image=%container.image.repository)

  priority: CRITICAL

  tags: [process, mitre_execution]

The Detect crypto miners using the Stratum protocol rule will raise an alert if Falco detects any processes launched with stratum+tcp and is specified in the process's command line. The output looks as follows:

19:46:37.779784798: Critical Possible miner running (command=minerd2 -a cryptonight -o stratum+tcp://monero.crypto-pool.fr:3333 -u 49TfoHGd6apXxNQTSHrMBq891vH6JiHmZHbz5Vx36 nLRbz6WgcJunTtgcxnoG6snKFeGhAJB5LjyAEnvhBgCs5MtEgML3LU -p x container=k8s.ns=insecure-nginx k8s.pod=insecure-nginx-8455b6d49c-z6wb9 container=07dce07d5100 image=kaizheh/victim) k8s.ns=insecure-nginx k8s.pod=insecure-nginx-8455b6d49c-z6wb9 container=07dce07d5100 k8s.ns=insecure-nginx k8s.pod=insecure-nginx-8455b6d49c-z6wb9 container=07dce07d5100

The minerd2 -a cryptonight -o stratum+tcp://monero.crypto-pool.fr:3333 -u 49TfoHGd6apXxNQTSHrMBq891vH6JiHmZHbz5Vx36nLRbz6Wgc JunTtgcxnoG6snKFeGhAJB5LjyAEnvhBgCs5MtEgML3LU -p x command line that was executed contains the stratum+tcp keyword. That's why the alert was triggered.

Like other name-based detection rules, the limitation of the rule is obvious. If the crypto binary execution does not contain stratum+tcp, the rule will not be triggered.

The preceding rule uses the blacklist approach. Another way is to use a whitelist approach if you know the processes that are going to run in the microservices. You can define a Falco rule to raise an alert when any process that is not on the trusted list is launched. The following is an example of this:

- list: trusted_nginx_processes

  items: ["nginx"]

- rule: Detect Anomalous Process Launched in Nginx Container

  desc: Anomalous process launched inside container.

  condition: spawned_process and container and not proc.name in (trusted_nginx_processes) and image.repository.name="nginx"

  output: Anomalous process running in Nginx container (command=%proc.cmdline container=%container.info image=%container.image.repository)

  priority: CRITICAL

  tags: [process]

The preceding rule will alert any anomalous process launched in an nginx container, which includes the crypto-mining processes. Last but not least, let's look at how image scanning tools can help detect the existence of crypto-mining binaries through integrating with malware feed services.

Checking the binary signature

Crypto-mining binaries can sometimes be recognized as malware. Like traditional anti-virus software, we can also check the hash value of running binaries against the malware feeds. With the help of an image scanning tool, such as Anchore, we can get the file's hash values:

root@anchore-cli:/# anchore-cli --json image content kaizheh/victim:nginx files | jq '.content | .[] | select(.filename=="/tmp/minerd2")'

{

  "filename": "/tmp/minerd2",

  "gid": 0,

  "linkdest": null,

  "mode": "00755",

  "sha256": "e86db6abf96f5851ee476eeb8c847cd73aebd0bd903827a362 c07389d71bc728",

  "size": 183048,

  "type": "file",

  "uid": 0

}

The hash value of the /tmp/minerd2 file is e86db6abf96f5851ee476eeb8c847 cd73aebd0bd903827a362c07389d71bc728. Then, we can check the hash value against VirusTotal, which provides malware feed service:

$ curl -H "Content-Type: application/json" "https://www.virustotal.com/vtapi/v2/file/report?apikey=$VIRUS_FEEDS_API_KEY&resource=e86db6abf96f5851ee476eeb8c847cd73aebd0bd903827a 362c07389d71bc728" | jq .

$VIRUS_FEEDS_API_KEY is your API key to access the VirusTotal API service, which then provides the following report:

{

  "scans": {

    "Fortinet": {

      "detected": true,

      "version": "6.2.142.0",

      "result": "Riskware/CoinMiner",

      "update": "20200413"

    },

    ...

    "Antiy-AVL": {

      "detected": true,

      "version": "3.0.0.1",

      "result": "RiskWare[RiskTool]/Linux.BitCoinMiner.a",

      "update": "20200413"

    },

  },

  ...

  "resource": "e86db6abf96f5851ee476eeb8c847cd73aebd0bd903827a362c07389d71bc 728",

  "scan_date": "2020-04-13 18:22:56",

  "total": 60,

  "positives": 25,

  "sha256": "e86db6abf96f5851ee476eeb8c847cd73aebd0bd903827a362c07389d71bc 728",

}

The VirusTotal report shows that /tmp/minerd2 has been reported as malware by 25 different feed sources, such as Fortinet and Antiy AVL. By integrating an image scanning tool and malware feeds service in your CI/CD pipeline, you can help detect malware at an early stage in the development life cycle. However, the downside of this single approach is that you will miss the crypto-mining attack if the mining binary is downloaded from the command and control server into a running pod. Another limitation is that if the feed server doesn't have any information about the crypto binary, you will definitely miss it.

We have talked about four different approaches to detect crypto-mining attacks. Each of these approaches has its own advantages and limitations; it would be ideal to apply some of these approaches together to improve their detection capability and detection efficacy.

Next, let's recap what we've discussed in this book, and comprehensively use this knowledge to prevent attacks in general.

Defending against attacks

In the previous section, we talk about a few ways of detecting crypto-mining activities. In this section, we will talk about defending against attacks in general by securing Kubernetes clusters. So, this involves more than just defending against a particular attack, but defending against all kinds of attacks. The four major defense areas are Kubernetes cluster provisioning, build, deployment, and runtime. First, let's talk about securing Kubernetes cluster provisioning.

Securing Kubernetes cluster provisioning

There are multiple ways to provision Kubernetes clusters such as kops and kubeadm. No matter which tool you use to provision a cluster, each Kubernetes component needs to be configured securely. Use kube-bench to benchmark your Kubernetes cluster and improve the security configurations. Make sure that RBAC is enabled, the --anonymous-auth flag is disabled, network connections are encrypted, and so on. The following are the key areas we covered in Chapter 6, Securing Cluster Components, and Chapter 7, Authentication, Authorization, and Admission Control:

• Properly configuring authentication and authorization for the Kubernetes control plane, kubelet, and so on
• Securing communication between Kubernetes components—for example, communication between kube-apiserver, kubelet, kube-apiserver, and etcd
• Enabling data encryption at rest for etcd
• Ensuring you do not launch unnecessary components, such as the dashboard
• Making sure all the necessary admission controllers are enabled while the deprecated ones are disabled

With the Kubernetes clusters securely provisioned, there are fewer chances for hackers to hack into your Kubernetes cluster easily, as was the case with Tesla's clusters (where the dashboard did not require authentication). Next, let's talk about securing the build.

Securing the build

Securing Kubernetes clusters also includes securing microservices. Securing microservices has to start at the beginning of the CI/CD pipeline. The following are some key countermeasures, as discussed in Chapter 8, Securing Kubernetes Pods, and Chapter 9, Image Scanning in DevOps Pipelines to secure microservices at the build stage:

• Address vulnerabilities discovered by image scanning tools properly for your microservices so that the possibility of a successful intrusion through exploiting application vulnerabilities is slim.
• Benchmark Dockerfiles to improve security configuration for images. Make sure no sensitive data is stored in the image, that all the dependent packages are updated, and so on.
• Scan executable files in the image to make sure no malware is seeded inside the image.
• Configure Kubernetes security contexts properly for workloads. Follow the principle of least privileges, limit access to system resources, such as using host-level namespaces, host paths, and so on, and remove unnecessary Linux capabilities, only granting the ones that are required.
• Do not enable an auto-mount service account. If no service account is required for the workload, don't create a service account for it.
• Follow the principle of least privileges, try to understand the tasks your workloads are carrying out, and only grant the required privileges to the service account.
• Follow the principle of least privileges, try to estimate the resource usage for workloads, and apply proper resource requests and limits to workloads.

Of course, securing the build can also be expanded to secure the entire CI/CD pipeline, such as source code management and CI/CD components. However, that is beyond the scope of this book. We will only suggest the options we think are most relevant to securing your Kubernetes clusters. Next, let's talk about securing deployment.

Securing deployment

We've already talked about different kinds of admission controllers in Kubernetes clusters in Chapter 7, Authentication, Authorization, and Admission Control, and Chapter 8, Securing Kubernetes Pods, and the need to use them properly with an example of an image-scanning admission controller (Chapter 9, Image Scanning in DevOps Pipelines). Using admission controllers and other built-in mechanisms serves as a great security gatekeeper for your workloads. The following are some key counter-measures:

• Apply network policies for namespaces and workloads. This could either be to restrict access to workloads (inbound network policies) or to implement the principle of least privileges (outbound network policies). When given a workload, if you know the destination IP block for outbound connection, you should create a network policy for that workload to restrict its outbound connection. The outbound network policy should block any traffic with a destination beyond the whitelisted IP block, such as downloading a crypto-mining binary from command and control server.
• Use Open Policy Agent (OPA) to ensure only images from trusted image registries are allowed to run in the cluster. With this policy, OPA should block any images from untrusted sources from running. For example, malicious images that contain crypto-mining binaries may reside in Docker Hub, so you should never consider Docker Hub as a trusted image registry.
• Use image-scanning admission controllers to ensure only images compliant with the scanning policy are allowed to run in the cluster. We already talked about this in Chapter 9, Image Scanning in DevOps Pipelines. New vulnerabilities may be discovered and the vulnerabilities' databases will be updated when you deploy workloads. It is necessary to scan before deploying.
• Use OPA or pod security policies to ensure workloads with limited Linux capabilities and restricted access to the host-level namespaces, host paths, and so on.
• It would be ideal to have AppArmor enabled on worker nodes and for each image that is deployed to have an AppArmor profile applied to it. Confining AppArmor profiles is done when workloads deploy, although the actual protection happens during runtime. A good use case is to build an AppArmor profile to whitelist the allowed processes when you know the processes that are running inside the container so that other processes, such as crypto-mining processes, will be blocked by AppArmor.

Do leverage the power of admission controllers and build a gatekeeper for your workload's deployment. Next, let's talk about securing workloads in runtime.

Securing runtime

Most likely, your Kubernetes clusters are the front battlefield to fight against hackers. Although we discussed different tactics to secure the build and deployment, all of these tactics ultimately aim to reduce the attack surface in the Kubernetes clusters. You cannot simply close your eyes and assume everything is going to be fine in your Kubernetes cluster. That's why we talk about resource monitoring in Chapter 10, Real-Time Monitoring and Resource Management of a Kubernetes Cluster, and auditing, secret management, detection, and forensics in Chapter 11, Defense in Depth. To recap what was covered in those two chapters, the following are the key counter-measures to secure runtime:

• Deploy decent monitor tools, such as Prometheus and Grafana, to monitor resource usage in your Kubernetes cluster. This is critical to ensure the availability of services and also, attacks such as crypto mining may trigger surges in CPU usage.
• Enable Kubernetes' audit policy to log Kubernetes events and activities.
• Ensure high availability across your infrastructure, Kubernetes components, and workloads.
• Use decent secret management tools, such as Vault, to manage and provision secrets for microservices.
• Deploy decent detection tools, such as Falco, to detect suspicious activities in Kubernetes clusters.
• It would be ideal to have forensics tools to collect and analyze suspicious events.

You may notice that securing communication among microservices is not mentioned. Service meshes are a hot topic that could help secure communication among microservices and beyond. However, service meshes are not covered in this book for two reasons:

• A service mesh introduces performance overhead to workloads and Kubernetes clusters, so they are not yet a perfect solution to secure communication among services.
• From an application security standpoint, it is easy to enforce service listening on port 443 with a CA-signed certificate so that the communication is encrypted. If microservices also perform authentication and authorization, then only trusted microservices can access authorized resources. A service mesh is not an irreplaceable solution to secure communication among services.

To defend against attacks to Kubernetes clusters, we need to secure the provisioning, build, deployment, and runtime of our Kubernetes clusters from end to end. They should all be considered as equally important as the strength of your defense is determined by your weakest link.

Summary

In this chapter, we went through a couple of the crypto-mining attacks that occurred over the last two years that brought a lot of attention to the need for securing containerized environments. Then, we showed you how to detect crypto-mining attacks with different open source tools. Last but not the least, we talked about how to defend your Kubernetes clusters against attacks in general by recapping what we discussed in previous chapters.

We hope you understand the core concepts of securing a Kubernetes cluster, which means securing the cluster provisioning, build, deployment, and runtime stages. You should also feel comfortable with starting to use Anchore, Prometheus, Grafana, and Falco.

As we know, Kubernetes is still evolving and it's not perfect. In the next chapter, we're going to talk about some known Kubernetes Common Vulnerabilities and Exposures (CVEs) and some mitigations that can protect your cluster against unknown variations. The purpose of the following chapter is to prepare you to be able to respond to handling any Kubernetes CVEs discovered in the future.

Questions

• What was the flaw that made a crypto-mining attack possible in Tesla's Kubernetes cluster?
• If you were the DevOps of Tesla, what would you do to prevent the crypto-mining attack?
• When you see CPU usage surge in a container, can you conclude that there has been a crypto-mining attack?
• Can you think of a crypto-mining process that can bypass the Detect crypto miners using the Stratum protocol Falco rule?
• What are the four areas you need to secure in order to secure your Kubernetes cluster?

Further reading

Refer to the following links for more information on the topics covered in this chapter:

• The Tesla crypto-mining attack: https://redlock.io/blog/cryptojacking-tesla
• The crypto-worm attack: https://unit42.paloaltonetworks.com/graboid-first-ever-cryptojacking-worm-found-in-images-on-docker-hub/
• Prometheus: https://prometheus.io/docs/introduction/overview/
• Falco: https://falco.org/docs/
• The VirusTotal API: https://developers.virustotal.com/v3.0/reference
• The crypto-mining attack analysis: https://kromtech.com/blog/security-center/cryptojacking-invades-cloud-how-modern-containerization-trend-is-exploited-by-attackers
• Hubble: https://github.com/cilium/hubble