Chapter 11. HyperFlex Workload Optimization and Efficiency
This chapter covers the following key topics:
Enterprise Workload Issues: Describes the workload issues that enterprises face, including reactive mode to growth, lack of visibility into their applications, overprovisioning, and cloud creep.- HyperFlex with Cisco Tetration: Describes better visibility into the traffic flows and the service chain between the applications.
- Cisco Workload Optimizer: Covers the automation software product that monitors the performance consumption of applications and matches the resources with the needs of the application.
- Cisco AppDynamics: An application and business monitoring platform that allows enterprises to monitor their business applications and transactions and make sure they are delivering the best performance. The product gives root cause analysis of performance issues at the code level.
HyperFlex is surrounded by a suite of software products to optimize the workload and enhance application performance and efficiency in the data center while minimizing the cost of operation. In an environment where workloads are growing at almost 30% per year, the human factor in information technology (IT) organizations cannot keep up. Also, IT budgets are falling short in keeping up with the infrastructure growth. The combination of growing workloads and lower budgets is a recipe for disaster.
Enterprise Workload Issues
Enterprises are challenged today by rolling out applications as fast as possible to grow their business and capture new customers. With this comes the challenge of managing the rollout and provisioning cycles, using private or public clouds, and monitoring the application performance for customer satisfaction. The main issues facing enterprises in efficiently growing workloads follow:
- IT reactive mode to growth: As data center applications grow, the nature of any business requires fast turnaround time in deploying applications to meet revenue goals. IT administrators are in a reactive mode trying to fix a moving vehicle. More applications mean more resources. Add standalone servers or blades, purchase more hypervisor licenses, initiate more virtual machines or containers, and include more central processing units (CPUs), memory, storage, and network.
- Overprovision everything: As IT administrators launch their applications, they operate under the “just in case” model and overprovision everything. An application that needs 1 vCPU, 2 GB of RAM, and 500 GB of storage gets 2 vCPUs, 4 GB of RAM, and 1 TB of storage “just in case” the application needs more resources next year. The result is an imbalance between idle resources committed to lightweight applications while heavyweight applications are starved for resources.
- Cloud creep: As chief information officers are pressured to reduce budgets, their first safe haven is the cloud. Enterprises spending too much money running and growing their data centers want to offload the workload and the work to the cloud. IT administrators just take their applications and dump them on Amazon Web Services (AWS), Azure, or Google cloud “as is.” This means that any inefficiencies they had in their data centers translate into a lot of resources and money spent in the cloud. And now instead of just having an inefficient data center model, the enterprise topped it with an inefficient cloud model.
- Lack of visibility: One of the main issues leading to workload inefficiencies and hence higher capital expenditures (CAPEX) and operational expenditures (OPEX) is the lack of visibility into the workload and the resources that it needs. Most of the time, IT does not know why an application is performing poorly, and its first reaction is to add more resources to it, which can cause lower performance. There is a lack of visibility in how the applications interact with each other and why performance sometimes suffers. Applications that have all the resources they need still perform poorly.
Also, as applications move around from private data centers to other geographical locations and into public clouds, the business does not really know whether the user experience remained the same, got better, or became worse.
Cisco HyperFlex addresses all of these issues via a suite of products, with the main focus on enhancing the application performance while minimizing the cost of resources. This chapter discusses three main products that work hand in hand with HyperFlex to reach that goal.
HyperFlex with Cisco Tetration
In plain terms, Tetration allows an IT administrator to monitor the Internet Protocol (IP) flows between the different applications of the data center. This allows you to monitor which application is talking to which application, what ports and services are being used, what users are accessing the application, what users can access the application, and what the service flow is between the applications.
In addition, as applications are moving between private and public clouds, Cisco Tetration allows the user to monitor traffic to and from the application regardless of the underlying infrastructure.
Based on such information, the IT administrator has a clear map of the applications and the flow of services that are actually running. Cisco Tetration falls under the umbrella of what Cisco calls intent-based networking, meaning that the product captures how the user “intends” the network to behave in the form of policies between the different endpoints. Tetration then checks the user policy versus what is happening in the network and alerts the user whether the network complies with the user intent.
Users might have conflicting policies. An example might be when the Applications team sets allow and deny rules that conflict with the Database team. Cisco Tetration resolves these conflicts. So, if two teams set conflicting policies between two endpoints—say one team allows port 80 and the other denies port 80—then conflict is flagged and resolved based on which team has a configured higher priority.
The information from Cisco Tetration is exported and leveraged toward enforcing additional functions either at the network layer or at the application layer.
On the network layer, information from the Tetration platform forms the basis of network profiles, allowing the users to define policies to be enforced inside the network switches. These policies define service flows between the applications. This is discussed in Chapter 17, “Application-Centric Infrastructure.”
Alternatively, information from the Cisco Tetration platform can be leveraged to offer workload optimization services. Knowing where the application sits in the network and what other applications it interacts with helps move the workload around to optimize the use of resources. This is discussed in this chapter under “Cisco Workload Optimizer.”
Let’s start with the basic Tetration architecture, as seen in Figure 11-1.
Figure 11-1 Tetration Architecture Overview
The three main components of the architecture are the data collection, the Tetration analytics cluster, and open access.
Data Collection
The data collection component defines the different methods to collect the data. Tetration captures only the headers of IP flows between two endpoints. Tetration does not capture the payload itself. When data is captured and sent to the analytics engine, it is encrypted for added security. Tetration relies on multiple methods for data collection:
- Hardware sensors: These are basically advanced application-specific integrated circuits (ASICs) inside the Nexus 9000 switches that collect IP flows at wire speed with no impact on the CPUs. The ASICs also annotate the data with switch-specific context information and provide the analytics engine with telemetry data.
- Software sensors: These are lightweight software binaries that are loaded inside the operating system where the application runs. The software runs in the user space of the operating system (OS), which could be running on bare-metal servers or as a guest OS inside the virtual machine (VM). The sensors stream data to the analytics engine, and they annotate the data with information gathered from the host, such as users and processes. Also, the software sensors enforce policy inside the host itself. The software sensors are based on Linux VMs, Windows Server VMs, the host OS, or other.
- Third-party sources: A third point of collection is from Encapsulated Remote Switched Port Analyzer (ERSPAN) agents. This is done by having collection points on the network, where a flow is copied and encapsulated inside a generic routing encapsulation (GRE) tunnel across L3 networks to reach the Tetration appliance. Data flows are augmented with information from third-party sources. Examples are configuration from network services such as load balancers, or IP addresses from an IP address management (IPAM) device, or possible asset tagging where the flow is tagged as a production flow, an operation flow, and so on. This allows filters to be applied on the flows to enforce policy.
Tetration Analytics Cluster
Another piece of the Tetration architecture is the Tetration analytics cluster. This is a Big Data–type appliance that receives the data collected from the software and hardware sensors and analyzes it. The cluster comes in different form factors. The hardware appliances come as a 39 rack unit (RU) large form factor platform (C1-Tetration) for data centers with more than 5,000 servers or an 8 RU small form factor platform for data centers with fewer than 5,000 servers. The analytics cluster also comes as a software-based virtual appliance Tetration-V that targets smaller installations that do not exceed 1,000 servers. The Tetration-V runs on-premises on the customers’ own servers but manages and monitors workloads from public clouds such as Amazon Web Services (AWS) and Microsoft Azure. Cisco also launched Tetration as a software as a service (SaaS). The SaaS offering makes it easier for customers to use the service on per-license basis and to handle large and small environments without initial investment in hardware or software.
Open Access
This identifies the different methods where the user accesses information from the analytics engine. Data is accessed via different methods such as a web graphical user interface (GUI), which allows a user with limited Big Data knowledge easy access to the information. Other methods are event notification using Apache Kafka, which allows subscription and streaming of records. Access to the cluster is also done via Representational State Transfer application programming interfaces (REST APIs) and the Tetration apps. The Tetration apps allow the user to write Python or Scala programming language scripts that are executed in the Tetration platform with full access to the collected data.
Using the Data
Cisco Tetration displays an application map and shows the interaction between the different endpoints and all the flows within a cluster. Because agents are running inside the OS of the application, they collect information such as the processes running, the process IDs, usernames that are attached to the processes, and so on.
The Tetration appliance uses advanced algorithms to have a better and more accurate insight into the application and its components. Here are some examples of the output of the Tetration appliance:
- Grouping of endpoints into different tiers such as web tiers, app tiers, and so on.
- A whitelist policy recommendation for applications. A whitelist is a list of approved applications that are permitted to be present in the network. The appliance also monitors and enforces the whitelist.
- Forensic analysis of the data using search filters.
- Impact of applying a certain policy on the network and how it affects the existing applications.
In addition, Tetration offers a powerful filtering engine that instantly accesses any piece of information that the user requires. Different parameters can be applied to the filter to capture the flows. Examples of such parameters are duration of the flow, IP addresses, ports, host names, process IDs, application latency, and network latency.
Based on the output of the Tetration appliance, the IT administrator can now automate the creation of policies in the data center. The Tetration platform also allows the administrator to experiment with new policies and use what-if scenarios to understand whether the new chosen policy has adverse effects on the current operation of the network. The application map displays every flow in the network.
Finally, the data that is collected by Tetration is exported via different formats such as JavaScript Object Notation (JSON), Extensible Markup Language (XML), and YAML Ain’t Markup Language (YAML) and imported by other engines such as Cisco ACI to enforce policies on the network [1].
Cisco Workload Optimizer
One of the software products that benefits enterprises as they run their applications on HyperFlex is the Cisco Workload Optimizer (CWOM). This product is based on the Turbonomic platform. CWOM is an automation tool that monitors the performance consumption of applications and matches the resources with the needs of the application. If applications are starving for resources, CWOM suggests manual adjustments to the user or does the adjustments automatically. In the same manner, if an application is oversubscribed, CWOM suggests or automates freeing up resources.
The CWOM is an agentless technology, meaning there are no software agents installed on hosts. CWOM detects all the elements in the network and draws an application map showing the interdependencies between the elements.
CWOM tracks, reports, and views the behavior of elements such as CPU, memory, storage, input/output operations per second (IOPS), latency, and database transaction unit (DTU) between regions. The DTU, for example, is a blended measure of CPU, memory, and input/output (I/O) reads and writes given to a database that benchmarks the level of performance of that database to decide whether it needs more or fewer resources.
CWOM works for private, hybrid, and multicloud workloads. It is more of a demand supply model for both performance and cost. An application, a VM, and a container demand a set of resources that the respective infrastructure supplies. CWOM first determines if the supply meets the demand, is below the demand, or exceeds the demand. CWOM then decides what to do with the resources. If a VM, for example, is experiencing low performance, CWOM evaluates whether this is because of lack of CPU, lack of memory, or lack of IOPS. Based on this evaluation, CWOM might suggest giving the VM more resources on the host it is running on or simply suggest moving the VM to another host with more resources.
Also, CWOM has a supply-demand-cost model based on where the application is running: in a private data center or on the cloud. Because each resource has a cost—whether a CAPEX/OPEX cost that you can enter or a resource cost published on AWS or Azure—CWOM calculates the different component costs and makes a recommendation to the user. Some applications have a better cost running on AWS, others on Azure, and others in your private data center. When calculating the cost of running an application on AWS, for example, multiple components come into play, such as the cost of compute, cost of storage, cost of IP addresses for a virtual private cloud, and cost of load balancers. CWOM gives a full view of the cost and compares the cost of running the same application on the cloud and on-premise so the user can make the best decision.
CWOM integrates with other tools from Cisco, such as the Cisco Tetration [2]. Feedback from Cisco Tetration is passed to CWOM via REST APIs. Based on this information, CWOM has an actual view of the interaction between different applications running in the network. If, for example, two VMs are chatty and possibly congesting the network, CWOM suggests that the user move the VMs to closer hosts or maybe to the same host to alleviate the network congestion and lower the latency of the east-west traffic inside the data center.
In principal, the CWOM (Turbonomic) platform allows you to define different physical or virtual entities. In this example, the entities are the physical servers inside a Unified Computing System (UCS) domain, virtual machines taken from vCenter, or flows between virtual machines taken from Cisco Tetration. CWOM creates an application map and interdependencies between all of these components. CWOM defines elements called dPods, which are hardware resources that are closely related inside the data center. As an example, the dPod groups all the UCS server blades or rack servers in a HyperFlex cluster in the same UCS domain. CWOM also defines vPods, which represent a collection of endpoints such as VMs with interdependencies as taken from Cisco Tetration.
CWOM also gives an exact view of how resources—such as CPU, memory, storage, and network utilization—are consumed between endpoints. CWOM detects, for example, that the network utilization for a particular flow has reached 100% between two endpoints, causing congestion.
As such, CWOM gives recommendations for better placements of the VMs, such as moving a VM from one host to another to drop down the utilization. If it is configured to do so, CWOM does the VM placement automatically, considering all of the other components and resource utilization in the application map.
CWOM works below the VM level to understand the infrastructure resources and optimize the supply-demand model of resources to lower the cost of usage.
Another product that digs deeper into the applications to monitor the transactions between the end user and the application and between applications is called Cisco AppDynamics, which is discussed next.
Cisco AppDynamics
AppDynamics [3] provides an application and business monitoring platform that allows enterprises to monitor their business applications and transactions and make sure they are delivering the best performance. In case of misbehaving applications or slow business processes, AppDynamics identifies the root cause of the problem down to the software code level. Cisco AppDynamics works in conjunction with the HyperFlex platform, Cisco Tetration, CWOM, and other cloud orchestration products such as Cisco CloudCenter to optimize the use of the applications and business transactions in private, hybrid, and cloud data centers.
As explained, CWOM optimizes the workload in a HyperFlex environment by monitoring and efficiently allocating resources to an application such as CPU, memory, storage, and network. AppDynamics takes the monitoring process even further and deeper into the application. AppDynamics looks at every transaction that is occurring in distributed applications and evaluates the health of not only the application but also the “transactions” in terms of different metrics such as response time. The transactions are monitored in the context of the business. Examples of business transactions are logging into a retail website, placing products into a shopping cart, researching a product from within the site, checking out, and making payment transactions. Each such transaction triggers a set of interactions between different endpoints. These endpoints could be the user talking to a web server or the web server talking to the app server that is talking to a Structured Query Language (SQL) database. Because problems can occur at any point in the transaction, AppDynamics gives the IT administrator the ability to dig deeper and identify the cause of any potential problem.
Cisco AppDynamics is a powerful tool to be used by IT operations, production support, developers, architects, and basically anyone who is accountable for enhancing the application performance from a technical standpoint as well as a business standpoint.
Cisco AppDynamics is divided into multiple components:
- Application performance management (APM)
- End user monitoring
- Infrastructure visibility
- Business monitoring
Delivering business applications is becoming complex. Business delivery through web applications involves many distributed applications working together, such as distributed web servers in different locations, distributed application servers, and databases. When users complain about a bad web experience, such as being unable to log in or to browse, purchase, or deposit, pinpointing the problem has surpassed the ability of any manual troubleshooting. The AppDynamics APM gives the user the following functionality:
- Automatically discover the application topology map from user to web, application, and database services.
- Discover all end-to-end transactions between the elements of the topology map.
- Visualize and prioritize the business transaction performance.
- Create a performance baseline for transactions and alert the administrator to deviations from the baseline.
- Monitor every transaction, but only capture data from misbehaving transactions to ensure scalability of the solution.
- Integrate with incidents and alerting systems such as ServiceNow, PagerDuty, and Jira.
- Isolate performance issues by digging deeper at the software code level.
- Cover popular programming languages and frameworks, including Java, .NET, Node.js, PHP, Python, C, and C++.
- Monitor applications inside private data centers as well as across multiple clouds.
- Create a war room where different entities can collaborate and troubleshoot while sharing the performance data.
- Solve issues such as uptime and availability, slow response time, memory leaks, thrash, stalls, deadlocks, slow database response, database connection pool areas, end user monitoring, mobile APM, and so on.
AppDynamics gives the user visibility into the different elements of a distributed application all the way from the user to the databases. Performance characteristics of the transactions are displayed, and issues are highlighted where they can be easily resolved.
AppDynamics collects information from the application and displays it using agents and controllers. Agents are plug-ins that monitor the performance and behavior of an application. The agents are deployed inside devices, applications, hosts, and more. The agents monitor the transactions down to the code level and tag the code requests and headers to be able to trace a transaction from beginning to end. The agents send the information to a controller, which helps monitor, troubleshoot, and analyze the transaction details.
In addition to APM, Cisco AppDynamics supports end user monitoring, infrastructure visibility, and business monitoring.
With end user monitoring, every user interaction is monitored. AppDynamics monitors all network requests and collects errors, crashes, and page load details, among other things. This allows the system to track the user experience such as response time and benchmark it against healthy user interactions. Infrastructure visibility is done by monitoring the health of servers, databases, network service devices such as load balancers, and so on. Last, AppDynamics provides business monitoring by correlating application performance and end user experience with business outcome. Business iQ tracks business transactions such as accessing a website home page and login, adding an item to a cart, and so on. It identifies the performance metrics of such transactions to understand how the performance of such transactions affects the business.
Looking Ahead
Part IV of this book covered the Cisco HyperFlex product line and the different tools and products that offer a high level of automation in deploying and managing the hyperconverged environment. Many HCI implementations are on the market today, and all claim high performance, advanced storage functionality, and cloud-like simplicity. However, when you dig through the details, you will discover that not all implementations are created equal.
Part V lists some of the known implementations on the market and covers in more detail implementations from VMware and Nutanix. Part V also discusses some aspects of OpenStack as they relate to compute and storage so the reader gets comfortable with the terminology when it’s discussed in the context of hyperconvergence.
References
[1] https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_Tetration_Deployment_Part_1_of_2.html
[2] https://www.cisco.com/c/en/us/products/data-center-analytics/tetration-analytics/index.html