Single-Node versus Distributed BOSH

Single-node BOSH is a complete BOSH environment that runs on a single machine. Although you can deploy BOSH in a distributed fashion, in reality, most deployments (including Cloud Foundry) work fine using a single-node BOSH machine. Even though being on a single machine can be considered a single point of failure, a single-node BOSH removes network segmentation faults, and it can be protected and mitigated by the IaaS layer. BOSH downtime does not cause downtime of Cloud Foundry or the apps running on it. The only real advantage of a multinode (distributed) BOSH is that it is easier to update because a single-node BOSH can then be used to upgrade the distributed BOSH.

BOSH Lite

Not everyone requires a full-blown deployment for their BOSH release. For example, if you just want to play around with Cloud Foundry, you might not want to incur the costs of running anywhere between 10 to 30 VMs on AWS. The solution to running an easy-to-configure, lightweight version of Cloud Foundry, or any other release, is BOSH Lite.

BOSH Lite is a prebuilt Vagrant box that includes the BOSH Director. However, instead of using a traditional IaaS CPI such as AWS, BOSH Lite utilizes a CPI for Garden. This means that instead of the CPI creating new VMs, the Garden CPI uses containers to emulate VMs. This makes BOSH Lite an excellent choice for scenarios such as these:

• General BOSH exploration without investing time and resources to configure an IaaS

• Getting started with BOSH and Cloud Foundry without incurring significant IaaS costs

• Development of BOSH releases

• Testing releases locally or as part of a CI/CD pipeline

BOSH Lite is a great environment for trying out Cloud Foundry, but be mindful that because everything is running on a single VM, it is suitable only for experimentation and not production workloads.

Stemcells

In biology, a stemcell is an undifferentiated (or basic) cell that can differentiate into several types of specialized cells throughout the body. A BOSH stemcell follows this same principle; it is a hardened and versioned base OS image wrapped with minimal IaaS-specific packaging. A common example of stemcell hardening is that SSHD, the OpenSSH daemon program for SSH, has been reconfigured to support only appropriate cyphers, only allow specific users to log in, and enforce time-out of client connections. Another example is that all compilers have been removed from the stemcell to reduce the attack surface. A typical stemcell contains the following:

• A bare-minimum OS skeleton with essential preinstalled common utilities

• Configuration files to securely configure the OS

• A BOSH agent for communication back to the Director

• IaaS-specific packaging and versioning

All machines created by BOSH are created from stemcells, including the Director machine itself and any machine subsequently created by the Director.

Stemcells provide clear separation between the base OS and any later-installed software used to specialize the machine. Stemcells are agnostic to what will be installed on them. They do not contain specific or sensitive information relating to the downstream software used to specialized them, so they can be shared at will with other BOSH users. Stemcells ultimately become transformed into specialized VMs in the cluster through BOSH instantiating and installing additional software on them during a BOSH deployment. This specialized downstream software is referred to as a BOSH release.

This separation of the base OS (stemcells) and specialized downstream software (BOSH release) is a powerful concept. Stemcells capture a specific base OS image; for example, Ubuntu Trusty. They are exactly the same and you can use them across different software deployments and different VM types. Because the same base OS is reused, it becomes extremely easy to apply versioning changes to the OS; for example, OS security fixes. This property of generic stemcells makes it possible for BOSH users to effortlessly switch among different infrastructures without being concerned about potential differences between OS images. It is similar to Java’s JVM aim of “write once, run anywhere” for compiled Java code. Because the same base OS is reused everywhere, it becomes easy to apply versioning changes systematically, to every deployed OS image. This benefit should not be underestimated; the ability to rapidly and confidently patch all machines within a distributed system is an extremely powerful capability.¹

Stemcells are distributed as tarballs, and the Cloud Foundry BOSH team is responsible for producing and maintaining an official set of stemcells. You can view and download the most recent and currently supported stemcells at bosh.io.

Releases

BOSH deploys software that has been packaged into a self-contained BOSH release. A BOSH release is your software, including all configuration and dependencies required to build and run your software in a reproducible way. A single BOSH release contains one or more pieces of software that are designed to work in unison. Therefore, BOSH becomes an excellent choice for deploying both an individual component and an entire distributed system.

A release is the software layer placed on top of a stemcell to transform it into a specialized component. Conceptually, a release is the packaging of your software (code) that you want to deploy. Releases are IaaS-agnostic, which allows for increased portability of your software across different IaaS environments. The key point here is that your BOSH releases and the subsequent deployment of those releases are not locked into any IaaS layer. This is because IaaS differences are abstracted away by the CPI layer and the Director’s Cloud Configuration (discussed further in “Cloud Configuration”). This decoupling is powerful for your infrastructure strategy. With the rise of viable alternatives to vSphere and AWS, there is a genuine desire for portability across different IaaS offerings. For example, using only a small team, Pivotal moved its hosted version of Cloud Foundry, known as Pivotal Web Services, from AWS to GCP in a matter of weeks with zero app downtime. This move was possible only because the Cloud Foundry software release is not tied into the underlying IaaS layer.

Releases are made up of one or more release jobs, and you can co-locate different releases during deployment. What you choose to actually deploy is not dictated by releases; you have the freedom to select which release jobs you would like to deploy and on which machines those jobs should reside. The anatomy of a release is discussed in “Anatomy of a BOSH Release”. For now, it is important to know that the release jobs you deploy to a VM form a specific component known as an instance group. Examples of instance group components are a Diego Cell and Cloud Foundry’s Cloud Controller. As implied by the name, an instance group can run several instances of the same component; for example, you typically run several Cells within a Cloud Foundry deployment.

As discussed in Chapter 5, Cloud Foundry is encapsulated by the cf-deployment BOSH release, located at cf-deployment.

BOSH is designed for deploying distributed systems such as Cloud Foundry, but you can use it equally to deploy smaller individual components such as etcd or redis. Because it can deploy almost any other software that is able to be run on the supported infrastructure, it has been used extensively within the Cloud Foundry ecosystem to deploy a wide variety of services such as databases, message brokers, and caching technologies. It is worth noting that many popular services such as RabbitMQ, MySQL, PostgreSQL, and Redis already have existing BOSH releases. Community service brokers are located in the Cloudfoundry-community GitHub repo.

This book does not delve into the detail of creating a new BOSH release; however, it is valuable to become familiar with release primitives and how releases are composed in order to understand how cf-deployment is structured and deployed. Understanding BOSH releases will also come in handy if you want to create additional BOSH releases for services that back your apps running on Cloud Foundry. Chapter 11 covers BOSH releases in detail.

Deployments

BOSH deploys software to the IaaS layer using a deployment manifest, one or more stemcells, and one or more releases. This is known in BOSH terminology as a BOSH deployment, which consists of a collection of one or more machines (VMs), as depicted in Figure 10-3. Machines are built from stemcells and then layered and configured with specified components from one or more BOSH releases. A BOSH deployment requires the following:

• The appropriate stemcell(s) for the IaaS of choice

• The releases (software) to deploy

• An IaaS environment (or infrastructure) to which to deploy the release

• A deployment manifest describing the deployment

Figure 10-3. A BOSH deployment

BOSH creates the deployment using ephemeral resources and persistent disks. The deployment is stable because BOSH can keep your software running by re-creating machines that fail, or restarting failed processes. State stored on the persistent disk (e.g., database data files) can survive when BOSH re-creates a VM because persistent disks can be reattached. As just discussed, deployments are portable across different kinds of cloud infrastructure with minimal changes to the deployment manifest.

The anatomy of a deployment is discussed in Chapter 12.

Cloud Configuration

Previously, deployment manifests included all of the IaaS-specific resource configurations required for that deployment. To keep BOSH deployments IaaS-agnostic, you now can configure IaaS resource configuration (networks, resource pools, disk pools, compilation) in one location at the Director level. Each BOSH deployment can then reference the predefined IaaS resources by name. You can define the IaaS configuration in an additional manifest such as iaas.yml, as follows:

compilation:
  workers: 6
  network: my-network
  az: z1
  reuse_compilation_vms: true
  vm_type: m3.medium
  vm_extensions:
  - 100GB_ephemeral_disk

azs:
- name: z1
  cloud_properties:
    availability_zone: us-west-1a

networks:
- name: my-network
  type: manual
  subnets:
  - az: z1
    gateway: 10.0.16.1
    range: 10.0.16.0/20
    reserved:
    - 10.0.16.2-10.0.16.3
    - 10.0.31.255
    static:
    - 10.0.31.190-10.0.31.254
    cloud_properties:
      subnet: subnet-XXXXXXXX
      security_groups:
      - sg-XXXXXXXX

vm_types:
- name: m3.medium
  cloud_properties:
    instance_type: m3.medium
    ephemeral_disk:
      size: 1024
      type: gp2

vm_extensions:
- name: 100GB_ephemeral_disk
  cloud_properties:
    ephemeral_disk:
      size: 102400
      type: gp2
- name: router-lb
  cloud_properties:
    elbs:
    - stack-bbl-CFRouter-XXXXXXXX
    security_groups:
    - sg-XXXXXXXA
    - sg-XXXXXXXB
- name: ssh-proxy-lb
  cloud_properties:
    elbs:
    - stack-bbl-CFSSHPro-XXXXXXXX
    security_groups:
    - sg-XXXXXXXA
    - sg-XXXXXXXB

disk_types:
- name: 5GB
  disk_size: 5120
  cloud_properties:
    type: gp2
    encrypted: true

The cloud configuration construct provides the flexibility to stipulate IaaS-specific information once; for example, a small, medium, and large vm-type, which can then be used by the various different BOSH deployments. The preceding manifest example specifies information about the following:

• Compilation VM

• AZs

• Networks

• vm_types

• vm_extensions

• disk_types

The manifest specifies that the compilation VM resides on an AZ named z1 and a network named my-network. The properties of z1 and my-network are specified in the respective AZs and networks hashes. The compilation vm_type is specified as m3.medium with an 100GB_ephemeral_disk vm_extension. Again, the details of m3.medium and 100GB_ephemeral_disk are specified in the respective vm_types and vm_extensions hashes. The vm_extensions hash contains additional hashes that describe the required load balancers, named router-lb and ssh-proxy-lb. Finally, you can specify any other IaaS component in this manifest such as disk_types. If that description feels complex, don’t worry, we will walk through the specifics of each component in a moment.

Because the entire IaaS configuration is now encapsulated in a separate cloud configuration manifest, the deployment manifest is not only simpler, it’s significantly smaller. In the subsections that follow, we examine each section of the cloud configuration.

networks:
- name: my-network
  type: manual
  subnets:
  - az: z1
    gateway: 10.0.16.1
    range: 10.0.16.0/20
    reserved:
    - 10.0.16.2-10.0.16.3
    - 10.0.31.255
    static:
    - 10.0.31.190-10.0.31.254
    cloud_properties:
      subnet: subnet-XXXXXXXX
      security_groups:
      - sg-XXXXXXXX

/dev/sda1 with / mounted
/dev/sdb2 with /var/vcap/data mounted

/dev/sdc1        1011416    8136    934688   1% /var/vcap/store

df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda1        2886304 1293408   1426564  48% /
none                   4       0         4   0% /sys/fs/cgroup
udev              498216       4    498212   1% /dev
tmpfs             101760     648    101112   1% /run
none                5120       0      5120   0% /run/lock
none              508784       0    508784   0% /run/shm
none              102400       0    102400   0% /run/user
/dev/sdb2        9182512  478144   8214872   6% /var/vcap/data
tmpfs               1024      16      1008   2% /var/vcap/data/sys/run
/dev/loop0        122835    1583    117321   2% /tmp

df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda1        2886304 1293640   1426332  48% /
none                   4       0         4   0% /sys/fs/cgroup
udev              498216       4    498212   1% /dev
tmpfs             101760     656    101104   1% /run
none                5120       0      5120   0% /run/lock
none              508784       0    508784   0% /run/shm
none              102400       0    102400   0% /run/user
/dev/sdb2        9182512   57524   8635492   1% /var/vcap/data
tmpfs               1024      12      1012   2% /var/vcap/data/sys/run
/dev/loop0        122835    1550    117354   2% /tmp
/dev/sdc1        1011416    8136    934688   1% /var/vcap/store

bosh_3x78y23hg@25e1aec0-36aa-424b-a70e-eff65b7b5490:~$ df -k
Filesystem                   1K-blocks    Used Available Use% Mounted on
/dev/sda1                      2886304 1296128   1423844  48% /
none                                 4       0         4   0% /sys/fs/cgroup
udev                            498216       4    498212   1% /dev
tmpfs                           101760     648    101112   1% /run
none                              5120       0      5120   0% /run/lock
none                            508784       0    508784   0% /run/shm
none                            102400       0    102400   0% /run/user
/dev/sdb2                      3057452  360844   2521584  13% /var/vcap/data
tmpfs                             1024      12      1012   2% /var/vcap/data/sys
                                                              /run
/dev/loop0                      122835    1550    117354   2% /tmp
10.129.48.12:/var/vcap/store 103079936 2251776  95568896  3% /var/vcap/nfs

compilation:
  workers: 6
  network: my-network
  az: z1
  reuse_compilation_vms: true
  vm_type: m3.medium
  vm_extensions:
  - 100GB_ephemeral_disk

BOSH Links

BOSH links give the Platform Operator a means to configure a deployment involving multiple VMs, where at least one instance group knows about another; for example, a web server that depends on a database. Traditionally, operators had to assign static IPs or DNS names to one job and pass it via properties to the other. This configuration was error-prone and unnecessary. It was also difficult to automate for the case of on-demand deployments.

Links provide a solution to this problem and abstract away manual versus dynamic (DNS-based) networking from the instance groups. Additionally, you can use links to share other non-networking configurations (job properties) between instance groups.

From the operator’s perspective, introduction of links removes tedious cross-referencing of IPs and other properties between different BOSH releases and deployments.

To take advantage of links in releases, extra metadata needs to be specified by using the consumes and provides directives. Each release job that needs information about another release job must specify consumes with the name of the link type it consumes. Each release job that can satisfy link type must specify provides. BOSH also has a concept of explicit linking of release jobs. If a link type is provided by only one job within a deployment, all release jobs in that deployment that consume links of that type will be implicitly connected to that provider. You can find more information about BOSH links at https://bosh.io/docs/links.

Orphaned Disks

Orphaned disks provide an additional safety feature because losing persistent data is never good. If you require a larger disk size, you can modify this in the cloud configuration manifest (or select a larger instance from the disks listed in the cloud configuration); BOSH will detach the old disk and reattach the new disk, migrating data across to the new disk in the process. The challenge here is that if you rename your instance group (from nameA to nameB), BOSH sees this as a new instance group, as opposed to an upgraded instance group, and will delete the old instance group (and old persistent disks) without moving the data on the old disk. To address this valid but often unintended behavior, when deleting a disk, BOSH keeps old disks around for five days and will garbage-collect them retroactively. Therefore, if you accidentally delete a deployment, you can recover your persistent disk after the fact.

Addons

Addons provide the ability for you to add additional releases to BOSH deployments. This feature is extremely valuable for adding new capabilities to an existing or future deployment. Addons are especially powerful because they provide the ability for the operator to define Director-wide policies. For example, additional releases that you might want to “add on” could include a corporate security module or an IPSec BOSH release to encrypt communication between all deployed VMs. As a positive side effect, addons also remove some of the clutter from deployment manifests.

Your BOSH environment might consist of multiple release jobs (e.g., Cloud Foundry and additional service brokers). There might be requirements to add an additional release job to all deployed VMs. Instead of modifying the deployment manifest to add additional release templates and then redeploying, BOSH employs the concept of runtime-config in the form of BOSH addons. Addons are a new release job that can be co-located with existing instance groups. When a new bosh deploy is invoked, BOSH deploys the addons to the defined deployment.