Probing Procedures

In DevOps, the idea is to quickly and continually provide new software features, bug fixes, and desired modifications to the customer. The focus is on continual small changes to the app as opposed to large, monolithic updates:

• Continual App Processing  One DevOps layer involves software revision control (see Chapter 27, “Controlling Versions with Git”) that quickly integrates app changes into the main software branch (continuous integration). In addition, these changes undergo automated testing to avoid breaking the app when the branch merges (continuous testing). With the help of the two previous components, software is delivered to the customer on a continual basis (continuous delivery).
• Controlling the App Environment  To support this continuous app processing layer, it is critical in DevOps that the development and production environments match. This includes equivalent hardware, device drivers, operating system versions, software libraries, and so on. The requirement provides a software development environment that creates production code free from bugs and complications due to mismatched environments.

  In addition, environment changes must be controlled and tested in a manner similar to how software revisions are orchestrated. For example, if a controlled update to the Linux kernel in the development environment introduces an unforeseen app bug, the environment can be rolled back to its previous version until the bug is addressed. Environment updates are typically done in small chunks, much like app revisions are handled, so that each environment modification can be properly and discretely managed. Tested new environments are added to a registry where older environments are also maintained in case a rollback to an earlier environment is needed.

• Defining the App Environment  In DevOps, the development and production environments (infrastructure) have predefined specifications, such as what hardware to employ, the essential operating system, and any needed software packages as well as critical code libraries. The non-hardware specifications are typically implemented into the environment via automated code (configuration management).

  Besides an environment's configuration, security measures, such as firewall ACLs (Chapter 18, “Overseeing Linux Firewalls”) and authentication policies (Chapter 16, “Looking at Access and Authentication Methods”), must also be orchestrated. This too is implemented into the environment via automated code (policy as code).

  Configuration management and policy as code fall under the umbrella term infrastructure as code. A benefit of using infrastructure as code is that the environments are repeatable. Also, they can be versioned, which is needed to implement revision control in the app environments for both policies and configuration. This DevOps component is covered more in depth later in this chapter.

• Deploying the App Environment  The app and its development environment are often moved to a production status (production environment) in a continual manner. The “continual manner” can be hourly, daily, weekly, or whatever meets the app's business requirements.

  The benefit of employing infrastructure as code techniques is that the process of deploying the app and its environment can be easily automated (infrastructure automation). Red Hat's Ansible product is one tool used for this DevOps deployment.

• Monitoring the App Environment  When the app is operating in its production environment, it needs to be monitored and logged. Software metrics, infrastructure resource usage, and performance statistics are just a few of the items to monitor and log. The goal is to track the app environment and ensure that it is meeting predetermined conditions (environment health). Often this monitoring is automated as well.

  As business needs change, the logged data can be valuable for future desired environment states. It provides threshold and performance measurements that make much easier decisions as to what app or environment infrastructure modifications are needed.

  In addition, monitoring can provide alerts to potential failures or resource depletion events. If a particular preset limit is crossed, the monitoring software can issue an alert or even handle the event itself using predefined event rules.

Orchestration is the key to agile DevOps. Besides the methodology, various orchestration tools provide additional speed needed to succeed in the continual software delivery business setting.

Analyzing Attributes

Virtualization, and more specifically containers (covered in Chapter 28, “Understanding Cloud and Virtualization Concepts”), greatly assist in the DevOps process. Containers in DevOps provide the following:

• Static Environment  Containers provide a predetermined app environment (also called a container image) that does not change through time (immutable). The container is created with preset library and operating system versions and security settings. All these settings are recorded. No software updates (such as via a sudo apt-get dist-upgrade command) are issued within the image.
• Version Control  After the software development process and prior to moving a modified app container image into production, the container and its recorded configuration are registered with a version control system. The version control system can contain previous app container images, including the one currently in production. Some companies use a manual version control system implemented with backups.
• Replace Not Update  After registration, the app container is ready to move into production. Instead of the production app container image being updated to match the development image, the production container is stopped. The development app container image then replaces the production container and starts as the production environment. Thus, an environment switch occurs.
• High Availability  Replication is the process of creating multiple copies of the production app container image and running them. This allows you to stop old currently unused production app containers and replace them with the new production app containers, which provides continual uptime for your app users. With containers and replication, the old method of shutting down your production environment to update it during a time period in the wee hours of the morning is gone.

To accomplish these tasks with container images, you need orchestration. Orchestration principles provide tools, utilities, and guidance for implementing app container images in a fast-paced environment, such as one that uses DevOps.

Coding the Infrastructure

Much like a physical data center, a container infrastructure needs to be managed and controlled. In orchestration, the container's configuration is treated in a manner similar to how software revisions are treated:

• Determine the infrastructure.  Along with the app requirements, the environment on which the app is executed must be preplanned. This activity is a mutual one between software development and tech ops. In this mutual activity, the container's operating system, libraries, services, security configuration, and any other supporting software or networking utilities are chosen.

  The determined infrastructure is frozen to provide a nonchanging (immutable) environment for development and then production. Typically this app container infrastructure is only accessible via the app's or developer's API. Remote access services such as OpenSSH are not provided in order to protect its immutable environment.

• Document the infrastructure.  The preset app container infrastructure is typically documented through an orchestration tool. The configuration management and policy as code settings (covered earlier in this chapter) are loaded into the utility's infrastructure as code portal, in a process called automated configuration management. The data is later used to deploy and replicate the app containers through build automation.
• Provide revision control.  The infrastructure as code information is not just documented—it is also inserted into an orchestration tool registry, providing version control. Every time a change occurs in the container image infrastructure, its modifications are tracked.
• Troubleshoot the infrastructure.  If an app container is deployed into production and problems occur, tech ops, software developers, or both handle the troubleshooting process. One item to check is the production container's documented configuration (as well as its revisions) to determine if any infrastructure items may be to blame.

  This provides an easier method for tracking down that pesky new software library that caused all the problems. Various orchestration tools allow a quick determination of modified infrastructure components and quicker problem resolution.

Handling the application's infrastructure in this manner increases the agility of your app container deployment. It also improves the time to production speed. The basic life cycle of an orchestrated app container image is shown in Figure 30.1.

Notice that at the end of the container's life cycle (typically when the new app container image is ready), the container image is removed. However, the old app container image's configuration should be stored within a version control system (or backup storage) and thus redeployed if any problems occur with the new app container image.

Automating the Infrastructure

With automated configuration management, not only can you troubleshoot the infrastructure more easily and roll the environment back to an earlier version, the deployment is automated. Often more than one app container is needed for both performance and balance loading. Your environment may require hundreds of running production app containers.

Manually configuring this infrastructure is tedious and certainly not fast or cost effective. With orchestration tools and automated configuration management, you can easily replicate the production app container and don't even have to be involved in the process (build automation). You simply let your orchestration tool know that you need X number of production app container images running at any one time.

Here are a few popular automation utilities in use today:

• Ansible: Owned by Red Hat, its main selling point is that remotely controlled servers don't need to run a separate agent software package. Ansible uses OpenSSH and Python to communicate using JSON-based protocols to remote servers. The configuration file is a standard text file, which can be stored in an encrypted vault.
• Chef: A Ruby-based package that uses Ruby-based “recipes” for defining server configurations; can run in either a client-server mode or a stand-alone mode.
• Puppet: Uses its own language to define system configurations for remote servers, thus requiring little to no programming knowledge to configure. It uses a client-server architecture, requiring remotely controlled servers to run a client application in the background. Currently is available in both commercial and open source versions.
• SaltStack: Owned by VMWare, this is a Python-based configuration management tool that stores server configuration data in a YAML data structures.
• Terraform: Owned by HashiCorp, this uses its own declarative language for storing server configurations, which you can define using the standard JSON data format. It has the ability to graph all resources, allowing operators to view infrastructure dependencies.

Comparing Agent and Agentless

Orchestration monitoring, logging, and reporting tools let you track app containers' health (how well they are performing, and if each one is still alive). Concerns over how these tools may adversely affect an app container's health gave rise to the agent versus agentless dispute.

Agent monitoring tools are orchestration utilities that require software (an agent) to be installed in the app container being monitored. These agents collect the data and transmit it to another location, such as a monitor server. The monitor server manages the information, provides analysis reporting, and also sends alerts for events, such as a container crashing.

Agentless monitoring tools are also orchestration utilities. In this case, an agent is not installed in the app container being monitored. Instead, the tool uses preexisting and/or embedded software in the container or the container's external environment to conduct its monitoring activity.

Whether to use an agent-based or an agentless orchestration utility is hotly debated. Some people feel that an agent is detrimental to an app container's performance, whereas others see only minor effects. Some tech ops insist that agentless tools are inflexible; others believe installing and maintaining an agent in their containers is an unnecessary hassle. Whatever side you choose, realize that most companies use a combination of agent and agentless orchestration tools.

Investigating the Inventory

Orchestration monitoring utilities can automatically deal with an app container's untimely demise. When an app container shuts down, this triggers an event and the desired state is no longer met. A desired state is a predetermined setting that declares how many containers should be deployed and running.

For example, imagine that your software application needs to have 10 production app containers running to efficiently handle the workload. If one of those containers crashes, the container inventory now switches to 9. This triggers an event in the monitoring utility that the desired state is no longer being met.

Many orchestration utilities employ self-healing. With self-healing, if the desired state is not currently being achieved, the orchestration tool can automatically deploy additional production app containers. In the previous example, this means that the orchestration tool would immediately start up an additional production app container using the container's stored configuration settings (build automation). No human involvement is needed.

When a new production app container is first deployed, the self-healing orchestration property will cause containers to be deployed automatically until the desired state is met. That's handy.

Embracing Kubernetes

Originally designed and used by Google, Kubernetes is an open source orchestration system that is considered by many to be the de facto standard. Not only is Kubernetes very popular and free, it is also highly scalable, fault tolerant, and (relatively) easy to learn.

This system contains years of Google's orchestration experience, and because it is open source, additional community-desired features have been added. This is one reason so many companies have adopted its use for container orchestration.

Each Kubernetes managed service or application has the following primary components:

• Cluster service: Uses a YAML file to deploy and manage app pods
• Pod: Contains one or more running app containers
• Worker: Pod host system that uses a kubelet (agent) to communicate with cluster services
• YAML file: Contains a particular app container's automated configuration management and desired state settings

This distributed component configuration allows high scalability and great flexibility. It also works very well for continuous software delivery desired by companies employing the DevOps model.

Inspecting Docker Swarm

Docker, the popular app container management utility, created its own orchestration system, called Docker Swarm (also called Swarm). A group of Docker containers is referred to as a cluster, which appears to a user as a single container. To orchestrate a Docker cluster, you can employ Swarm.

With the Swarm system, you can monitor the cluster's health and return the cluster to the desired state should a container within the cluster fail. You can also deploy additional Docker containers if the desired app performance is not currently being met. Swarm is typically faster than Kubernetes when it comes to deploying additional containers.

Though not as popular as the Kubernetes orchestration system, Docker Swarm has its place. It is often used by those who are new to orchestration and already familiar with Docker tools.

Surveying Mesos

Apache Mesos is not a container orchestration system. Instead, Apache Mesos, created at the University of California, Berkeley, is a distributed systems kernel. It is similar to the Linux kernel, except that it operates at a higher construct level. One of its features is the ability to create containers. The bottom line is that Apache Mesos combined with another product, Marathon, does provide a type of container orchestration system framework. You could loosely compare Mesos with Marathon to Docker with Swarm.

Mesos with Marathon provides high availability and health monitoring integration and can support both Mesos and Docker containers. This orchestration framework has a solid history for large container deployment environments.

To find out more about Mesos with Marathon, don't use search engine terms like Mesos orchestration. Instead, go straight to the source at https://mesosphere.github.io/marathon.

1. Which of the following can use orchestration? (Choose all that apply.)
   1. Software development
   2. Music production
   3. Server monitoring
   4. DevOps
   5. Containers
2. Connie, the tech ops lead, provides the development team with a development environment that currently matches the production environment. She insists that besides the app modifications, the environment should not change. What attribute should this development environment have to meet Connie's requirement?
   1. Self-replicating
   2. Immutable
   3. Kubernetes
   4. Infrastructure as code
   5. Self-healing
3. Bill, the software development team leader, understands Connie's (tech ops lead) desire for a static and matching development environment. However, since they are employing containers, Bill suggests a better improvement. Which of the following should he suggest?
   1. Modify the production container so that it matches the development container when the app is ready for production.
   2. Move the app from the development container to the production container when the app is ready for production.
   3. Stop any software updates on the development container from occurring before the app is ready for production.
   4. Replace the production container with the development container when the app is ready for production.
   5. Remove any updates on the development container that are not on the production container when the app is ready for production.
4. Which of the following orchestrated container attributes best provides high availability to an app user?
   1. Immutability
   2. Version control
   3. Replication
   4. Automation
   5. Documentation
5. In DevOps and container orchestration, non-hardware items such as the operating system and libraries and security policies are documented within the orchestration tool, implemented into the desired environment, and are called what?
   1. Marathon
   2. Build automation
   3. A development environment
   4. A container
   5. Infrastructure as code
6. Ms. Ward needs to move an app container into production. She employs an orchestration tool, which uses the predefined infrastructure as code to deploy and replicate the needed production container images. What orchestration concept did Ms. Ward use?
   1. Monitoring
   2. Build automation
   3. Replication
   4. Version control
   5. Docker Swarm
7. Which of the following is a benefit of automated configuration management in container orchestration? (Choose all that apply.)
   1. Containers can be deployed automatically.
   2. Containers can be replicated automatically.
   3. Troubleshooting infrastructure issues is easier.
   4. Continuous software delivery is enabled.
   5. Infrastructure modifications are tracked.
8. Mr. Abbot is recording requirements for the new orchestration tool. He needs the tool to automatically deploy and replicate any containers that have crashed. What is this called?
   1. Self-healing
   2. Build automation
   3. Continuous integration
   4. Infrastructure as code
   5. Pod builds
9. Which of the following are items that may be collected, used, or watched by an orchestration monitoring tool? (Choose all that apply.)
   1. Version control errors
   2. App container performance
   3. App metrics
   4. App container health
   5. Default states
10. Connie has picked an agentless orchestration monitoring utility for her app containers. Which of the following reasons might she have used in reaching this decision? (Choose all that apply.)
    1. Connie wanted monitoring software installed on each app container.
    2. She did not want monitoring software installed on each app container.
    3. Connie was concerned about container performance being adversely affected by the monitoring software.
    4. She was not concerned about container performance being adversely affected by the monitoring software.
    5. Connie wanted to use what most companies use for orchestration monitoring.