The term “cloud computing” in the Information Technology (IT) world refers to the ability to run virtual instances of compute resources in a centralized data center with racks of physical servers and machines managed and offered as multitenant services by third parties. This new endowment to spin up on-demand compute resources in the cloud powered many businesses to host applications in a matter of minutes (if not seconds) at scale with the minimal skill set required to manage the resources yet without compromising resiliency. The term “cloud native” refers to the framework and architecture that introduces the ability of the applications to be hosted in such cloud environments to realize the maximum benefits offered by cloud computing.
In this chapter, we will start by sprucing up the readers’ knowledge about the virtualization concept and explaining various capture techniques in different cloud environments. We would like to highlight that this chapter is not about control or management plane captures to debug cloud-native deployments but to capture the traffic from instances hosted as virtual machines and containers.
Evolution of Virtualization and Cloud
Can we simplify the implementation of compute resources?
Can we optimize the resource utilization without compromising resiliency?
Can we optimize the cost of the resources and bring down the server when not in need?
While the preceding queries are explained with compute resources, these are all applicable for storage and network resources as well. Such CAPEX and OPEX challenges identified and raised by the IT professionals introduced the need for the new virtualization concept.
Basics of Virtualization
While the concept of virtualization became very popular in the last couple of decades, the technique was explored in the early 1960s by the development of a virtualization hypervisor called Control Program by IBM.
Paravirtualization
Full virtualization
Hardware-assisted full virtualization
Nested virtualization
Paravirtualization is the technique used earlier that requires customizing the operating system to be hosted as virtual machines. The customized operating system is capable of making custom resource requests to the underlying hardware. As it could be noted, this is not a scalable approach as this requires customizing every operating system and making it compatible with different types of underlying hardware resources.
Full virtualization, on the other hand, introduces the hypervisor that acts as a medium between the guest operating system and the underlying hardware resources. Full virtualization eliminates the need for customizing the operating system and drastically improves the types of OS hosted as virtual machines.
The hardware-assisted full virtualization approach introduces the capability of natively supporting the virtualization by the processors that significantly improve the performance of the hosted virtual machines. Most of the recent processors, including the latest Apple processors, support hardware-assisted full virtualization.
Nested virtualization is an approach that allows the installation of a hypervisor layer on top of an existing virtual machine, thereby enabling the users to host virtual machines on a virtual machine.
Some of the processors may require additional firmware upgrades or patches to support hardware-assisted full virtualization. When a virtual machine is instantiated on a host without hardware assistance, the performance is degraded, resulting in a poor user experience.
Hypervisor – Definition and Types
Managing the underlying hardware and emulating virtual instances of CPU (vCPU), memory (vMem), and storage (vStorage)
Managing the hypercall request for underlying hardware resources from the guest operating system
Managing and scheduling the work on virtual processors across the host
Type 1 Hypervisor (native)
Type 2 Hypervisor (hosted)
Type 1 Hypervisor, also known as the native or bare-metal hypervisor, is a software or firmware that is directly installed on top of the bare metal. As Type 1 Hypervisor does not require any operating system and runs directly on top of bare metal, it offers better performance compared to its counterpart. Linux KVM is one such Type 1 Hypervisor that is open source and so developed and managed by the open community.
Type 2 Hypervisor, also known as hosted hypervisor, is a software installed on a physical or a virtual machine with a guest operating system. This hypervisor leverages the nested virtualization concept and allows to host virtual entities on top of another virtual entity. VirtualBox and VMware Fusion are a few such Type 2 Hypervisors.
There were some discussions to embed the hypervisor within the system BIOS itself. This type of hypervisor is referred to as Type 3.
Virtualization – Virtual Machines and Containers
In this section, we will spruce up the readers’ knowledge about different virtualization by-products, such as virtual machines and containers.
Virtual Machines
Containers
The container engine or the manager that is responsible for hosting and managing the containers in the user space. The kernel features such as cgroup and namespace are used to provide isolation and privacy between the containers that are sharing the same operating system. Docker and LXC are well-known container platforms used in the industry. While LXC is used only to Linux containers, the Docker platform is OS agnostic and can be used in different types of applications as containers.
Different cloud service providers have different portals to configure and host the virtual machines. This book or chapter does not intend to help the readers learn each such portal and assume that the readers are familiar with the portal to perform the capture in machines hosted in different provider environments.
In the subsequent sections, we will discuss how to capture the traffic in different cloud environments, such as AWS and GCP, and cloud-native environments, such as Docker and Kubernetes.
Traffic Capture in AWS Environment
When one or more EC2 instances are hosted in the AWS environment, there are different options to capture the traffic. While one simple option is to use the native way of logging in to the EC2 instance and leverage the native tools such as tcpdump to capture the traffic, it is not different from how we capture in a physical server. However, such native option requires per-instance configuration to capture the traffic and does not provide a holistic view of the entire network. If the requirement is to capture the traffic to more than one instance in the same virtual private cloud (VPC), this native option is not sufficient.
In this section, we will discuss the traffic mirroring ability introduced in the AWS portal to configure the capture, the source, and the target along with the filters to specifically capture the traffic from one or more instances.
VPC Traffic Mirroring
The SourceVM is hosted with one interface eth0 with Internet access. This is the interface from which bidirectional traffic is required to be captured. The ClientVM is hosted in the same VPC with one interface eth0, with Internet access. In our example, we use this VM to generate ICMP traffic to the SourceVM. The WiresharkTarget is the target VM to which the traffic from eth0 of SourceVM will be mirrored and captured. This VM is hosted with two interfaces where eth0 is with Internet access, while eth1 is used as the target interface to which the mirrored traffic from SourceVM will be replicated. Let us now look into the configuration and procedure to mirror the traffic in this scenario.
Now the setup is ready to mirror the traffic from the eth0 interface of SourceVM and mirror the traffic to eth1 of WiresharkTarget VM. By logging in to WiresharkTarget VM and using the tcpdump tool, we will be able to capture the traffic from eth1 which is the mirrored traffic from eth0 of SourceVM.
While the preceding example scenario was explained with one source port, multiple ports can be configured to mirror the traffic to the same target port.
The mirrored traffic from the VPC is encapsulated using UDP port 4789 before forwarding to the target port. So, it is essential to configure the relevant security group rules to allow this UDP port.
Traffic Capture in GCP Environment
Identify the source compute instances from where the packet needs to be mirrored and captured.
Create a new compute instance to act as the target machine with Wireshark installed.
Create a new unmanaged instance group and associate the target machine.
Create a new UDP load balancer with the packet mirroring flag set and associate the instance group as the backend.
Create the packet mirroring policy by setting the source as the instances from where the traffic should be mirrored and the target as the load balancer.
The Instance group field in the backend is used to configure the unmanaged instance group we created and associated the targetvm. It is also essential to enable the Packet Mirroring flag in the frontend configuration. Without this flag, the load balancer will not be listed to be configured in the mirror policy section.
All the network and subnet in our example configuration are left to default. If the network or the subnet differs in your scenarios or production environment, the relevant network and subnet must be chosen.
In our example scenario, both the source and the target machines are in the same VPC network, and so the respective option is chosen to select the VPC network. The mirror source is set to the instance, and sourcevm is selected to mirror the traffic. The collector destination is set to the frontend created as part of the UDP load balancer, and finally the filter policies are created to choose all the traffic or selective traffic to be mirrored.
The setup is now ready to mirror the traffic from the sourcevm and replicate the traffic to the frontend of the load balancer which in turn will replicate it to the backend instance targetvm. By logging in to the targetvm and using the tcpdump tool, we will be able to capture the traffic from sourcevm.
Most of the cloud providers such as AWS and GCP offer different types of third-party capturing tools as part of the marketplace. While in this book we used an Ubuntu instance as the target machine, any capturing tool and service such as Fortigate or Cisco Stealthwatch may be used as the target where the replicated traffic will be consumed for analytics purposes.
Traffic Capture in Docker Environment
Docker default bridge
Docker user-defined bridge
Host network
The traffic can also be captured from the host interface (eth0 in our example) using the same tcpdump command. But as mentioned in the preceding section, the source address of the traffic will be changed to the host interface address, and so it is hard to differentiate if the traffic is originally from the host or translated by the host. So, it is a lot easier to capture the traffic from the docker0 interface.
Traffic Capture in Kubernetes Environment
While Docker is the popular container platform to host application containers, the use of the command line to instantiate and manage applications on a per-container basis lacks dynamic lifecycle management of the application containers. For example, if one of the containers is stuck or down, manual intervention is required to identify the failed container instance and reinstantiate the application container. However, in a production-grade environment, it is common to run a voluminous number of containers running different applications, and it is humanly impossible to monitor and manually manage all such containers. What we need is a dynamic orchestration platform that can automate the deployment of containers, management, and scaling aspects with minimal or no manual intervention.
The Kubernetes architecture comprises a minimum of one master node and one or more worker nodes as control plane components, collectively referred to as a cluster. Within the cluster, any application is hosted as a Pod, which is a group of one or more containers. The init process or other service agents can be hosted as a container along with the application container within the Pod. The master node acts as the control plane component that manages all the worker nodes and then manages the Pod scheduling and scaling. In such container applications hosted as a Pod within the cluster, it is essential to perform traffic capture for various troubleshooting purposes. While one option to capture the traffic is to log in to each worker node and use the tcpdump command with the docker0 interface as input, this is already covered in the previous section. In this section, we will discuss another approach using the ksniff plugin that can be embedded within the kubectl, which is the command-line interface for Kubernetes to host and manage the Pods.
The captured Wireshark file can be offloaded from the master node for analytics purposes.
During the time of this chapter authoring, while ksniff is the better option to capture the traffic from the Pod, it is not officially managed by CNCF.
Summary
In this chapter, we spruced up the knowledge of the readers about virtualization and the evolution of cloud computing that is now offered as a new service by different cloud providers. We explained the different virtualization types and the hypervisor types to realize the benefits of virtualization.
We further discussed about different by-products of virtualization, such as virtual machines and containers, and the ability to orchestrate and manage those virtual entities.
With this background, we explained how to capture the traffic on different cloud providers with examples captured from virtual machines hosted on AWS and Google Cloud environments. We then explained how to capture the traffic from the Docker container platform and Kubernetes cluster.
References for This Chapter
Ramdoss, Yogesh, Nainar, Nagendra Kumar. Containers in Cisco IOS-XE, IOS-XR, and NX-OS: Orchestration and Operations, First Edition. Cisco Press, 2020.
https://docs.aws.amazon.com/vpc/latest/mirroring/what-is-traffic-mirroring.html