The BOSH Director

Deployments are managed by the BOSH Director, a centralized management and orchestration server. To deploy a BOSH release, the release is first uploaded to a BOSH Director along with the appropriate stemcell, and the deployment manifest is set. Upon deployment, the Director then causes the instance group machines (often VMs) requested in the manifest to be allocated and stored, forming a deployment. All interaction with the BOSH Director is through the BOSH CLI.

This is an overview of the process to create a deployment:

1. Create a deployment manifest referencing one or more releases.

2. Through the BOSH CLI, upload stemcells and releases used by the manifest to the BOSH Director.

3. Direct BOSH to the location of the deployment manifest.

4. Initiate a deployment by using the bosh deploy command to form a deployment managed by the Director.

A user (or CI/CD pipeline) continues to manage a BOSH deployment through changes to the deployment manifest. In addition to defining the initial deployment, the deployment manifest contains all required information for tracking, managing, and updating the release software layered onto the deployment’s machines. After a deployment has been installed, the BOSH Director keeps track of all the associated machines and any attached persistent disks used by the deployment. If the deployment manifest is changed and redeployed, old machines are replaced and updated, but persistent disks are retained and reattached to the newer machines. Updating an existing deployment is effectively the same procedure as for that of the original deployment:

1. Update deployment manifest with necessary changes.

2. Upload any new stemcells or releases.

3. Direct BOSH to the location of the deployment manifest.

4. Initiate a deployment by using the bosh deploy command to apply changes to the deployment.

The Director creates actionable tasks:

• By translating commands sent by the Platform Operator through the BOSH CLI

• From scheduled processes like backups or snapshots

• By self-healing through reconciling expected machine state with actual machine state

The Director adds these tasks to the Task Queue. Worker processes then take tasks from the Task Queue to act on them.

BOSH Agent

An agent is included on every machine that BOSH deploys. The agent is a single process that runs continuously. It listens for instructions from the Director via messages (currently sent via NATS). The agent then executes instructions in response to the messages it receives from the Director.

The agent receives job specifications from the Director and uses them to assign a role or instance group to the machine. These specifications include which packages to install and how jobs should be configured to create the instance group VM. In essence, it is the agent that converts undifferentiated stemcells into the specialized components known as instance groups.

Errand

A BOSH errand is a short-lived BOSH job, defined in a deployment manifest, that the Platform Operator can run several times after a deploy has finished. Example errands include the CATS that you should run after every new Cloud Foundry deployment, and the Cloud Foundry service broker broker-registrar and broker-deregistrar errands that simplify registering service brokers in a Cloud Foundry deployment.

The Command Line Interface

The CLI is the primary operator interface to BOSH. An operator uses the CLI to interact with the BOSH Director. The BOSH Director then translates those commands and performs actions against the underlying cloud infrastructure. The CLI is typically installed on a machine or VM that can directly communicate with the Director’s API; for example, an operator’s laptop or a jump box in a data center. The BOSH CLI is what you use to run BOSH commands. For assistance with BOSH commands, you can run $ bosh help --all to view the help. We discuss BOSH CLI commands further in “Basic BOSH Commands”.

The Cloud Provider Interface

A CPI is an API that the Director uses to interact with an IaaS provider to create and manage networks, VMs, and disks. A CPI abstracts the cloud infrastructure from the rest of BOSH, allowing for portability of BOSH releases across different IaaS layers. You can read more about the CPI in “The Cloud Provider Interface”.

Health Monitor

The Health Monitor uses status and life cycle events received from BOSH Agents to monitor the health of machines. If the Health Monitor detects a problem with a machine, it can send an alert through a notification plug-in, or trigger the Resurrector. The Health Monitor works in the following way:

1. The Health Monitor pings the Director for a list of agents.

2. The Health Monitor listens for pings (currently via NATS) from worker agents.

3. The Health Monitor uses the CPI to contact nonresponding agents.

4. If no response is retrieved, it follows retry logic (it instructs the Director to run the bosh cck command).

Resurrector

If enabled, the Resurrector plug-in automatically re-creates machines identified by the Health Monitor as missing or unresponsive. It uses the same Director API that CLI uses.

Message Bus (NATS)

The Director and the agents communicate through a lightweight publish-subscribe messaging system called NATS. These messages have two purposes:

• To perform the provisioning of instructions to the VMs

• To inform the Health Monitor about changes in the health of monitored processes

You can also use NATS for cross-component communication.

Creating a New VM

To better understand how the BOSH components interact, let’s explore what happens when a new machine or VM is created. Figure 13-2 depicts the process, and is followed by a description of each of the numbered stages in the figure.

Figure 13-2. BOSH VM creation

1. Through the BOSH CLI, the Platform Operator takes an action such as deploying Cloud Foundry for the first time or scaling up an existing deployment. These specific actions require creating a new machine.

2. The BOSH CLI passes the deployment manifest to the Director. The Director then determines that one or more new machines are needed.

3. The Director uses the CPI to instruct the IaaS to create disks and machines. The CPI communicates with the IaaS and requests that it create a new machine using the defined stemcell image.

4. The IaaS provides the Director with information (IP addresses and IDs) that the BOSH agent requires to configure the VM.

5. The Director updates the BOSH registry with the configuration information for the machine.

6. The BOSH agent is started. The agent is responsible for all machine configuration. It requests the configuration information for the machine from the BOSH registry.

7. The registry responds with IP addresses and IDs.

8. The agent then uses the IP addresses and IDs to configure the machine. As the machine starts, an init script starts the BOSH agent. An init (short for initialization) process is the first process started when the system boots. The BOSH agent remains continually running on the machine. Ubuntu-based stemcells use Linux utilities to monitor the BOSH agent using a runsv service containing a run script that takes over the init jobs and manages the BOSH agent process. Bootstrapping the BOSH agent deals with all the standard Linux concerns such as creating the vcap user, starting the SSH daemon, and starting the Monit daemon. The BOSH agent also deals with creating disks and networking.

As part of the VM configuration in step 8, BOSH will create disks and set up networking for the VM.

Disk Creation

It is possible to define two different types of storage solutions: ephemeral and persistent disks. These are both in addition to the root machine disk. The distinction between ephemeral and persistent provides the flexibility to use more cost-effective storage for temporary storage requirements and persistent storage for long-lived data.

Within the machine, the BOSH agent checks for the presence of the root disk. By default, all stemcells have 3 GB of root disk (2.5 GB available after system use). The BOSH agent partitions a root filesystem on the root disk.

If an instance group is using an ephemeral disk, two partitions are created: one for swap and one for a tmp space under /var/vcap/data. The creation of the disk depends on the IaaS and CPI implementation. For vSphere, the Director creates an ephemeral disk and attaches it to the machine. For AWS and OpenStack, it is the IaaS that creates the disk based on the instance type.

If there is no ephemeral disk, the BOSH agent uses the unused root disk space to create the swap and tmp partitions. Although the root partition is usually 3 GB, the real disk size is often greater, depending on the instance type, so often some unused space will remain.

Persistent disks are created after the machine has been created. If a persistent disk has been specified in the instance group, the CPI communicates with the IaaS to create a persistent disk and then attaches the persistent disk to the required machine. The CPI communicates with the BOSH agent to inform it about the new disk. The BOSH agent then looks for that disk, mounts it, and creates the /var/vcap/store for persistent data.

Let’s consider three different Cloud Foundry jobs.

Instance groups using an ephemeral disk have a VM with two mounted filesystems; for example:

/ mounted on the local VM storage /dev/sda1
/var/vcap/data mounted on ephemeral disk  /dev/sdb2

Instance groups that specify a persistent disk will have a third filesystem; for example:

/var/vcap/store mounted on /dev/sdc1

You can also mount additional directories. One of the Cloud Foundry jobs is a blobstore for an Amazon S3 bucket in AWS. The Cloud Foundry Cloud Controller VM has an additional remote directory:

/var/vcap/nfs residing on the blobstore server under /var/vcap/store.

Networking Definition

There are three options for setting up networking for the VM:

1. Dynamic

2. Manual

3. VIP

If you have a choice, the recommended approach, as of this writing, is to use manual networks.

The BOSH CLI v2

There is a new v2 BOSH CLI written in Go. It is designed to be significantly faster and more stable than the original BOSH CLI that was written in Ruby. Consistency across different developer environments including Mac OS, Windows, and Linux is of primary importance.

All of the original CLI behavior remains, and most of the commands are the same, but the user experience (UX) has changed slightly, making a few of the older commands redundant. In addition, commands have become hyphenated; for example, upload release becomes upload-release.

The UX also enforces consistency. For example, you must always specify -e <my-environment> and -d <my-deployment>. For example, to set the deployment manifest:

$ bosh -e <my-director-env> -d <my-deployment> deploy <deployment-name.yml> 

Specifying the deployment manifest in this manner means that when we invoke other commands such as BOSH -e <env> cck, you do not need to download manifests; you need only reference the deployment name. The only time you need a deployment manifest is when you deploy. This is a significant enhancement to overall usability. The -d argument is per-deployment context. Commands that are outside of a specific deployment context, such as the generic cloud-config, do not require the -d argument. Here’s an example:

$ bosh -e <env> update-cloud-config ./cloud.yml

Another key difference between the v1 and v2 CLIs is that the new CLI combines bosh-init functionality with the CLI, as shown here:

$ bosh create-env ...

You can find more information on the BOSH v2 2.0 CLI at the BOSH website.

Basic BOSH Commands

Knowledge of some basic BOSH commands is essential for rudimentary debugging. As discussed in “BOSH Top-Level Primitives”, a deployment requires three key items: a stemcell, release, and deployment manifest.

To view a list of available commands, use the following:

$ bosh -h

Before deploying, you will need to upload a release and stemcell by using the following commands:

$ bosh -e <env> upload-release <release-location>
$ bosh -e <env> upload-stemcell <stemcell-location>

A release file and a stemcell location can be a local file or a remote URI. These two commands upload these artifacts to BOSH’s blobstore. In addition to uploading a precompiled release, you can upload a release by uploading a release manifest. When uploading just the manifest, BOSH fetches the release components from a blobstore defined in the release.

Before deploying, you can specify the Director IaaS properties through cloud configuration. To apply the cloud configuration to the BOSH Director, use the update-cloud-config command, as follows:

$ bosh -e <env> update-cloud-config ./cloud.yml

You can retrieve the current cloud configuration from the BOSH Director and save the output via stdout:

$ bosh -e <my-director-env> cloud-config > cloud.yml

The deployment manifest is not uploaded to BOSH; rather, it is named via the -d argument. After you have uploaded your stemcell and release, specify the deployment manifest and deploy via:

$ bosh -e <env> -d <my-dep> deploy <deployment-name>.yml

As mentioned in “Director Task, Queue, and Workers”, a task is a basic unit of work performed by the Director. You can view the currently running tasks by running the following:

$ bosh -e <env> tasks

You can monitor task status throughout its life cycle or retrieve the status of previously run tasks:

$ bosh -e <env> task <task-number>

You can retrieve the logs for any previously run or currently running task:

$ bosh -e <env> task <task-number> --debug

If you require more advanced troubleshooting, you can view all of the machines in the BOSH environment:

$ bosh -e <env> instances

And, you can use SSH to access a specific machine:

$ bosh -e <env> -d <my-dep> ssh <vm-name>

For troubleshooting instance groups, start with this:

$ bosh -e <env> -d <my-dep> cck

CCK (short for Cloud Check) scans for differences between the desired VM state, as stored in the Director database, and the actual state of the VMs. For each difference the scan detects, bosh cck offers possible repair and recreate options.

Summary

This chapter provided a basic overview of the BOSH components and essential BOSH commands. To recap:

• BOSH deployments are managed by the BOSH Director, a centralized management and orchestration server.

• The BOSH Director comprises a number of subcomponents.

• The BOSH Director manages the machines it deploys via a BOSH agent. The agent residing on BOSH-managed machines is provided by the stemcell.

• Knowledge of some basic BOSH commands is essential for deploying BOSH releases and for rudimentary debugging.

This chapter concludes our deep dive into the BOSH release-engineering tool chain. Now that you are familar with some basic BOSH commands, Chapter 14 focuses on the debugging and logging aspects of running Cloud Foundry.