Action Abstraction

Because each microservice is scoped to a specific Diego component, each service is free to express its work using its own abstraction. This design is incredibly powerful because bounded abstractions offer an unprecedented degree of flexibility. Abstraction levels move from course high-level abstractions to fine-grained implementations as work moves through the system. For example, work can begin its journey at the Cloud Controller as an app, but ultimately all work ends up as a scheduled process running within a created container. This low-level implementation of a scheduled process is too specific to be hardwired into every Diego component. If it were hardwired throughout, the distributed system would become incredibly complex for end users and brittle to ongoing change. The abstraction boundaries provides two key benefits:

• The freedom of a plug-and-play model

• A higher-level concern for Diego clients

Plug-and-play offers the freedom to replace key parts of the system when required, without refactoring the core services. A great example of this is a pluggable container implementation. Processes can be run via a Docker image, a droplet plus a stack, or in a Windows container. The required container implementation (both image format and container backend) can be plugged into Diego as required without refactoring the other components.

Moreover, clients of Diego have high-level concerns. Clients should not need to be concerned with underlying implementation details such as how containers (or, more specifically, process isolation) are created in order to run their applications. Clients operate at a much higher level of abstraction, imperatively requesting “run my application.” They are not required to care about any of the underlying implementation details. This is one of the core benefits of utilizing a platform to drive application velocity. The more of the undifferentiated heavy lifting you allow the platform to undertake, the faster your business code will progress.

Composable Actions

At the highest level of abstraction, the work performed by a Task or LRP is expressed in terms of composable actions, exposed via Diego’s public API. As described earlier, composable actions are the actual actions that run within a container; for example, a RunAction that runs a process in the container, or a DownloadAction that fetches and extracts an archive into the container.

Conceptually each composable action implements a specific instruction. A set of composable actions can then be used to describe an explicit imperative action, such as “stage my application” or “run my application.”² Composable actions are, by and large, hidden from Cloud Foundry users. Cloud Foundry users generally interact only with the Cloud Controller. The Cloud Controller (via the CC-Bridge) then interacts with Diego through Diego’s composable actions. However, even though as a Platform Operator you do not interact with composable actions directly, it is essential that you understand the available composable actions when it comes to debugging Cloud Foundry. For example, the UploadAction might fail due to misconfigured blobstore credentials, or a TimeoutAction might fail due to a network partition.

Composable actions include the following:

1. RunAction runs a process in the container.

2. DownloadAction fetches an archive (.tgz or .zip) and extracts it into the container.

3. UploadAction uploads a single file, in the container, to a URL via POST.

4. ParallelAction runs multiple actions in parallel.

5. CodependentAction runs multiple actions in parallel and will terminate all codependent actions after any single action exits.

6. SerialAction runs multiple actions in order.

7. EmitProgressAction wraps another action and logs messages when the wrapped action begins and ends.

8. TimeoutAction fails if the wrapped action does not exit within a time interval.

9. TryAction runs the wrapped action but ignores errors generated by the wrapped action.

Because composable actions are a high-level Diego abstraction, they describe generic activity, not how the activity is actually achieved. For example, the UploadAction describes uploading a single file to a URL; it does not specify that the URL should be the Cloud Controller’s blobstore. Diego, as a generic execution environment, does not care about the URL; Cloud Foundry, as a client of Diego, is the entity responsible for defining that concern. This concept ties back to the action abstraction discussed previously, allowing Diego to remain as an independently deployable subsystem, free from specific end-user concerns.

So, how do composable actions relate to the aforementioned Cloud Foundry Tasks and LRPs? Consider the steps involved when the Cloud Controller issues a run command for an already staged application. To bridge the two abstractions, there is a Cloud Foundry-to-Diego bridge component known as the Cloud Controller Bridge (CC-Bridge). This essential function is discussed at length in “The CC-Bridge”. For now, it is sufficient to know that the CC-Bridge knows how to take the various resources (e.g., a droplet, metadata ,and blobstore location) that the Cloud Controller provides, coupled with a desired application message. Then, using these composable actions, the CC-Bridge directs Diego to build a sequence of composable actions to run the droplet within the container, injecting the necessary information provided by the Cloud Controller. For example, the specific composable action sequence for a run command will be:

• DownLoadAction to download the droplet from the CC blobstore into a specified location inside the container.

• DownLoadAction to download the set of static plugin (AppLifeCycle) binaries from a file server into a specified location within the container.

• RunAction to run the start command in the container, with the correct parameters. RunAction ensures that the container runs the code from the droplet using the helper (AppLifeCycle) binaries that correctly instantiate the container environment.

Applications are broken into a set of Tasks such as “stage an app” and “run an app.” All Diego Tasks will finally result in a tree of composable actions to be run within a container.

CAPI

Diego Tasks and LRPs are submitted to Diego via a Diego client. In Cloud Foundry’s case, the Diego client is the CAPI, exposed by the Cloud Controller. Cloud Foundry users interact with the Cloud Controller through the CAPI. The Cloud Controller then interacts with Diego’s BBS via the CC-Bridge, a Cloud Foundry-to-Diego translation layer. This interaction is depicted in Figure 6-3.

Figure 6-3. API interaction from the platform user through to Diego

The Cloud Controller provides REST API endpoints for Cloud Foundry users to interact with Cloud Foundry for commands including the following:

• Pushing, staging, running, and updating

• Pushing and running discrete one-off tasks

• Creating users, Orgs, Spaces, Routes, Domains and Services, and so on

• Retrieving application logs

The Cloud Controller (discussed in “The Cloud Controller”) is concerned with imperatively dictating policy to Diego, stating “this is what the user desires; Diego, make it so!”; for example, “run two instances of this application.” Diego is responsible for orchestrating and executing the required workload. It deals with orchestration through a more autonomous subsystem at the backend. For example, Diego deals with the orchestration of the Cells used to fulfill a workload request through an auction process governed by the Auctioneer.

This design means that the Cloud Controller is not coupled to the execution machines (now known as Cells) that run your workload. The Cloud Controller does not talk to Diego directly; instead, it talks only to the translation component, the CC-Bridge, which translates the Cloud Controller’s app-specific messages to the more generic Diego language of Tasks and LRPs. The CC-Bridge is discussed in-depth in “The CC-Bridge”. As just discussed, this abstraction allows the Cloud Foundry user to think in terms of apps and tasks, while allowing each Diego service to express its work in a meaningful abstraction that makes sense to that service.

Staging Workflow

To better understand the Cloud Controller interaction with Diego, we will explore what happens during a staging request. Exploring staging introduces you to two Diego components:

• The BBS: Diego’s database that exposes the Diego API.

• Cells: the execution machines responsible for running applications in containers.

These two components are discussed at length later in this chapter. Understanding the staging process provides you with a clear picture of how Cloud Foundry interprets the $ cf push and translates it into a running ActualLRP instance. Figure 6-4 provides an overview of the process.

Figure 6-4. Interaction between Cloud Foundry’s Cloud Controller components and Diego, while staging and running an application

The numbered list that follows corresponds to the callout numbers in Figure 6-4 and provides an explanation of each stage:

1. A developer/Platform Operator uses the Cloud Foundry command-line tool to issue a cf push command.

2. The Cloud Foundry command-line tool instructs the Cloud Controller to create a record for the application and sends over the application metadata (e.g., the app name, number of instances, and the required buildpack, if specified).

3. The Cloud Controller stores the application metadata in the CCDB.

4. The Cloud Foundry command-line tool uploads the application files (such as a .jar file) to the Cloud Controller.

5. The Cloud Controller stores the raw application files in the Cloud Controller blobstore.

6. The Cloud Foundry command-line tool issues an app start command (unless a no-start argument was specified).

7. Because the app has not already been staged, the Cloud Controller, through the CC-Bridge, instructs Diego to stage the application.

8. Diego, through an auction process, chooses a Cell for staging and sends the staging task to the Cell.

9. The staging Cell downloads the required life-cycle binaries that are hosted on the Diego file server, and then uses the instructions in the buildpack to run the staging task in order to stage the application.

10. The staging Cell streams the output of the staging process (to loggregator) so that the developer can troubleshoot application staging problems.

11. The staging Cell packages the resulting staged application into a tarball (.tar file) called a droplet and uploads it to the Cloud Controller blobstore.

12. Diego reports to the Cloud Controller that staging is complete. In addition, it returns metadata about the application back to the CCDB.

13. The Cloud Controller (via CC-Bridge) issues a run AI command to Diego to run the staged application.

14. Diego, through an auction process, chooses a Cell to run an LRP instance as an ActualLRP.

15. The running Cell downloads the application droplet directly from the Cloud Controller blobstore (the ActualLRP has a Cloud Controller URL for the asset).

16. The running Cell downloads the Application Life-Cycle Binaries hosted by the Diego file server and uses these binaries to create an appropriate container and then starts the ActualLRP instance.

17. Diego reports the status of the application to the Cloud Controller, which periodically receives the count of running instances and any crash events.

18. The Loggregator log/metric stream goes straight from the Cell to the Loggregator system (not via the CC-Bridge). The application logs can then be obtained from the Loggregator through the CF CLI.

The CC-Bridge

The CC-Bridge (Figure 6-5) is a special component comprised of four microservices. It is a translation layer designed to interact both with the Cloud Controller and Diego’s API, which is exposed by Diego’s BBS. The BBS is discussed shortly in “The BBS”.

Figure 6-5. The CC-Bridge

The components on the CC-Bridge are essential for establishing domain freshness. Domain freshness means two things:

1. The actual state reflects the desired state.

2. The desired state is always understood.

Domain freshness is established through a combination of self-healing and closed feedback loops.

CC-Bridge services translate the Cloud Controller’s domain-specific requests of stage and run applications into Diego’s generic language of LRP and Tasks. In other words, Diego does not explicitly know about applications or staging tasks; instead, it just knows about a Task or LRP that it has been requested to execute. The CC-Bridge services include the following:

Stager

The Stager handles staging requests.

CC-Uploader

A file server to serve static assets to the Cloud Controller’s blobstore.

Nsync

This service is responsible for handling domain freshness from the Cloud Controller to Diego.

TPS

This service is responsible for handling domain freshness from Diego to the Cloud Controller.

Nsync and TPS

Nsync and TPS are the two components responsible for handling domain freshness, matching desired and actual state between Diego and the Cloud Controller. They are effectively two sides of the same coin:

• The Nsync primarily retrieves information from the Cloud Controller. It is the component responsible for constructing the DesiredLRP that corresponds with the application request originating from the Cloud Controller.

• The TPS primarily provides feedback information from Diego to the Cloud Controller.

Both components react to events via their listener process, and will periodically check state validity via their respective bulker/watcher processes.

Figure 6-6 shows the high-level component interaction between the Cloud Controller, Diego’s Nsync, and Converger components right through to the Cell. The figure also illustrates the convergence from a DesiredLRP stored in the BBS to an ActualLRP running on a Cell.

Figure 6-6. High-level process interaction involving domain freshness, convergence, and ActualLRP placement into a Cell’s container

The Nsync is responsible for keeping Diego “in sync” with the Cloud Controller. It splits its responsibilities between two independent processes: a bulker and a listener. Let’s look at each of them:

Nsync-Listener

The Nsync-Listener is a service that responds to DesiredLRP requests from the Cloud Controller. It actively listens for desired app requests and, upon receiving a request, either creates or updates the DesiredLRP via a record in the BBS database. This is the initial mechanism for dictating desired state from the Cloud Controller to Diego’s BBS.

Nsync-Bulker

The Nsync-Bulker is focused on maintaining the system’s desired state, periodically polling the Cloud Controller and Diego’s BBS for all DesiredLRPs to ensure that the DesiredLRP state known to Diego is up to date. This component provides a closed feedback loop, ensuring that any change of desired state from the Cloud Controller is reflected on to Diego.

The process status reporter (TPS) is responsible for reporting Diego’s status; it is Diego’s “hall monitor.” It splits its responsibilities between two independent processes: listener and watcher submodules. Here’s what each one does:

TPS-Listener

The TPS-Listener provides the Cloud Controller with information about currently running ActualLRP instances. It responds to the Cloud Foundry CLI requests for cf apps and cf app <may-app-name>.

TPS-Watcher

The TPS-Watcher monitors ActualLRP activity for crashes and reports them to the Cloud Controller.

Logging and Traffic Routing

To conclude our review of the Cloud Foundry layer of user-facing components, let look at logging.

Diego uses support for streaming logs from applications to Cloud Foundry’s Loggregator system and provides support for routing traffic to applications via the routing subsystem. With the combined subsystems—Diego, Loggregator, and the routing subsystem—we have everything we need to do the following:

• Run any number of applications as a single user

• Route traffic to the LRPs

• Stream logs from the LRPs

The Loggregator aggregates and continually streams log and event data. Diego uses the Loggregator’s Metron agent to provide real-time streaming of logs for all Tasks and LRPs in addition to the streaming of logs and metrics for all Diego components. The routing system routes incoming application traffic to ActualLRPs running within Garden containers on Diego Cells. (The Loggregator and routing subsystem were discussed in Chapter 3.)

The BBS

The BBS manages Diego’s database by maintaining an up-to-date cache of the state of the Diego cluster including a picture-in-time of all DesiredLRPs, running ActualLRP instances, and in-flight Tasks. Figure 6-7 provides an overview of the CC–Bridge-to-BBS interaction.

Figure 6-7. The BBS interaction with the Cloud Controller via the CC-Bridge

The BBS provides Diego’s internal components and external clients with a consistent API to carry out the following:

1. Query and update the system’s shared state (the state machine)

2. Trigger operations that execute the placement of Tasks and LRPs

3. View the datastore underpinning the state machine

For example, certain operations will cause effects. Consider creating a new LRP in the system. To achieve this, the CC-Bridge communicates to the BBS API endpoints. These endpoints will save any required state in the BBS database. If you specify a DesiredLRP with N instances, the BBS will automatically trigger the required actions for creating N ActualLRPs. This is the starting point of the state machine. Because understanding the state machine is important for debugging the platform, we cover it in more detail in “The Diego State Machine and Workload Life Cycles”.

Communication to the BBS is achieved through Google protocol buffers (protobufs).

Diego Cell Components

Cells are where applications run. The term application is a high-level construct; Cells are concerned with running desired Tasks and LRPs. Cells are comprised of a number of subcomponents (Rep/Executor/Garden; see Figure 6-8) that deal with running and maintaining Tasks and LRPs. One Cell typically equates to a single VM, as governed by the CPI in use. You can scale-out Cells both for load and resilience concerns.

Figure 6-8. The Cell processes gradient from the Rep to the Executor through to Garden and its containers

There is a specificity gradient across the Rep, Executor, and Garden. The Rep is concerned with Tasks and LRPs, and knows the details about their life cycles. The Executor knows nothing about Tasks and LRPs but merely knows how to manage a collection of containers and run the composable actions in these containers. Garden, in turn, knows nothing about actions and simply provides a concrete implementation of a platform-specific containerization technology that can run arbitrary commands in containers.

Rep

The Rep is the Cell’s API endpoint. It represents the Cell and mediates all communication between the BBS and Brain—the Rep is the only Cell component that communicates with the BBS. This single point of communication is important to understand; by using the Rep for all communication back to the Brain and BBS, the other Cell components can remain implementation independent. The Rep is also free to be reused across different types of Cells. The Rep is not concerned with specific container implementations; it knows only that the Brain wants something run. This means that specific container technology implementations can be updated or swapped out and replaced at will, without forcing additional changes within the wider distributed system. The power of this plug-and-play ability should not be underestimated. It is an essential capability for upgrading the system with zero-downtime deployments.

Specifically, the Rep does the following (see also Figure 6-9):

• Participates in the Brain’s Auctioneer auctions to bid for Tasks and LRPs. It bids for work based on criteria such as its capacity to handle work, and then subsequently tries to accept what is assigned.

• Schedules Tasks and LRPs by asking its in-process Executor to create a container to run generic action recipes in the newly created container.

• Repeatedly maintains the presence of the Cell in the BBS. Should the Cell fail catastrophically, the BBS will invoke the Brain’s Converger to automatically move the missing instances to another Cell with availability.

• Ensures that the set of Tasks and ActualLRPs stored in the BBS are synchronized with the active containers that are present and running on the Cell, thus completing the essential feedback loop between the Cell and the BBS.

Figure 6-9. The BBS, auction, and Cell feedback loop

The Cell Rep is responsible for getting the status of a container periodically from Garden via its in-process Executor and reporting that status to the BBS database. There is only one Rep running on every Diego Cell.

The Diego Brain

We have already discussed the Brain’s Converger process. The Brain (Figure 6-10) is also responsible for running the Auctioneer process. Auctioning is the key component of Diego’s scheduling capability. There are two components in Diego that participate in auctions:

The Auctioneer

Responsible for holding auctions whenever a Task or LRP needs to be scheduled

The Cell’s Rep

Represents a Cell in the auction by making bids for work and, if picked as the winner, running the Task or LRP

Figure 6-10. The Cell–Brain interaction

When holding an auction, the Auctioneer communicates with the Cell Reps via HTTP. The auction process decides where Tasks and ActualLRP instances are run (remember that a client can dictate that one LRP requires several ActualLRP instances for availability). The Auctioneer maintains a lock in the BBS such that only one Auctioneer may handle auctions at any given time. The BBS determines which Auctioneer is active from a lock record (effectively the active Auctioneer holding the lock). When the BBS is at a point at which it wants to assign a payload to a Cell for execution, the BBS directs the Brain’s Auctioneer by requesting, “here is a task with a payload, find a Cell to run it.” The Auctioneer asks all of the Reps what they are currently running and what their current capacity is. Reps proactively bid for work and the Auctioneer uses the Reps’ responses to make a placement decision.

A classic optimization problem in distributed systems is that there is a small lag between the time the system realizes it is required to make a decision (e.g., task placement) and the time when it takes action on that decision. During this lag the input criteria that the original decision was based upon might have changed. The system needs to take account of this to optimize its work-placement decisions.

Consequently, there are currently two auction actions for LRPs: LRPStartAuction and LRPStopAuction. Let’s look at each:

LRPStartAuctions

These occur when LRPs need to be assigned somewhere to run. Essentially, one of Diego’s Auctioneers is saying, “We need another instance of this app. Who wants it?”

LRPStopAuctions

These occur when there are too many LRPs instances running for a particular application. In this case, the Auctioneer is saying, “We have too many instances of this app at index X. Who wants to remove one?"

The Cell that wins the auction either starts or stops the requested LRP.

The Access VM

The access VM contains the file server and the SSH proxy services.

The Application Life Cycle

The application life cycle is as follows:

1. When a client expresses the desire to run an application, the request results in Diego’s Nsync process creating an ActualLRP record.

2. An ActualLRP has a state field (a process globally unique identifier [GUID] with an index) recorded in the BBS. The ActualLRP begins its life in an UNCLAIMED state, resulting in the BBS passing it over to the Auctioneer process.

3. Work collectively is batched up and then distributed by the Auctioneer. When the Auctioneer has a batch of work that requires allocating, it looks up all the running Cells through service discovery and individually asks each Cell (Rep) for a current snapshot of its state, including how much capacity the Cell has to receive extra work. Auction activity is centrally controlled. It is like a team leader assigning tasks to team members based on their desire for the work and their capacity to perform it. The batch of work is broken up and distributed appropriately across the Cells.

4. An auction is performed and the ActualLRP is placed on a Cell. The Cell’s Rep immediately transfers the ActualLRP state to CLAIMED. If placement is successful, the LRP is now RUNNING and the Rep now owns this record. If the ActualLRP cannot be placed (if a Cell is told to do work but cannot run that work for some reason), it reenters the auction process in an UNCLAIMED state. The Cell rejects the work and the Auctioneer can retry the work later.

5. The Cell’s response on whether it has the capacity to do the work should not block actually starting the work. Therefore, the “perform” request from the Auctioneer instructs the Cell to try to undertake the work. The Cell then attempts to reserve capacity within the Executor’s state management, identifying the extra resource required for the work.

6. The Cell quickly reports success or failure back to the Auctioneer. Upon success, the Cell then reserves this resource (as quickly as possible) so as not to advertise reserved resources during future auctions. The Executor is now aware that it has the responsibility for the reserved work and reports this back to the Rep, which knows the specifics of the Diego state machine for Tasks and LRPs.

7. Based on the current state that the Executor is reporting to the Rep, the Rep is then going to make decisions about how to progress workloads through their life cycles. In general, if anything is in reserved state, the Rep states to the Executor: "start running that workload.” At that point, the Executor takes the container specification and creates a container in Garden.

8. The Auctioneer is responsible for the placement of workloads across the entire cluster of Cells. The Converger is responsible for making sure the desired and actual workloads are reconciled across the entire cluster of Cells. If the ActualLRP crashes, it is placed in a CRASHED state and the Rep moves the state ownership back to the BBS because the ActualLRP is no longer running. When the Rep undertakes its own local convergence cycle, trying to converge the actual running states in its Garden containers with its representation within the BBS, the Rep will discover the ActualLRP CRASHED state. The Rep then looks for the residual state that might still reside from its management of that ActualLRP. Even if the container itself is gone, the Executor might still have the container represented in a virtual state. The virtual state’s “COMPLETED” state information might provide a clue as to why the ActualLRP died (e.g., it may have been placed in failure mode). The Rep then reports to the BBS that the ActualLRP has crashed. The BBS will attempt three consecutive restarts and then the restart policy will begin to back off exponentially, attempting subsequent restarts after a delayed period.

9. The Brain’s Converger runs periodically looking for “CRASHED” ActualLRPs. Based on the number of times it has crashed (which is also retained in the BBS), the Converger will pass the LRP back to the Auctioneer to resume the ActualLRP. The Converger deals with the mass of unclaimed LRPs, moving them to ActualLRPs “CLAIMED” and “RUNNING”. The Converger maps the state (held in the BBS) on to the desired LRPs. If the BBS desires five instances but the Rep only reports on four records, the Converger will make the fifth record in the BBS to kickstart the placement.

10. There is a spectrum of responsibilities that extends from the Converger to the BBS. Convergence requires a persistence-level convergence, including the required cleanup process. There is also a business-model convergence in which we can strip away any persistence artifacts and deal with the concepts we are managing—and Diego ensures that the models of these concepts are in harmony. The persistence layer always happens in the BBS, but it is triggered by the Converger running a convergence loop.

Task Life Cycle

Tasks in Diego also undergo a life cycle. This life cycle is encoded in the Task’s state as follows:

PENDING

When a task is first created, it enters the PENDING state.

CLAIMED

When successfully allocated to a Diego Cell, the Task enters the CLAIMED state and the Task’s Cell_id is populated.

RUNNING

The Task enters the RUNNING state when the Cell begins to create the container and run the defined Task action.

COMPLETED

Upon Task completion, the Cell annotates the TaskResponse (failed, failure_reason, result), and the Task enters the COMPLETED state.

Upon Task completion, it is up to the consumer of Diego to acknowledge and resolve the completed Task, either via a completion callback or by deleting the Task. To discover if a Task is completed, the Diego consumer must either register a completion_callback_url or periodically poll the API to fetch the Task in question. When the Task is being resolved, it first enters the RESOLVING state and is ultimately removed from Diego. Diego will automatically reap Tasks that remain unresolved after two minutes.

The Route-Emitter

The Route-Emitter is responsible for registering and unregistering the ActualLRPs routes with the GoRouter. It monitors DesiredLRP state and ActualLRP state via the information stored in the BBS. When a change is detected, the Route-Emitter emits route registration/unregistration messages to the router. It also periodically emits the entire routing table to the GoRouter. You can read more about routing in Chapter 7.

Consul

Consul is a highly available and distributed service discovery and key-value store. Diego uses it currently for two reasons:

• It provides dynamic service registration and load balancing via internal DNS resolution.

• It provides a consistent key-value store for maintenance of distributed locks and component presence. For example, the active Auctioneer holds a distributed lock to ensure that other Auctioneers do not compete for work. The Cells Rep maintains a global presence in Consul so that Consul can maintain a correct view of the world. The Converger also maintains a global lock in Consul.

To provide DNS, Consul uses a cluster of services. For services that require DNS resolution, a Consul agent is co-located with the hosting Diego component. The consul-agent job adds 127.0.0.1 as the first entry in the nameserver list. The consul-agent that is co-located on the Diego component VM serves DNS for consul-registered services on 127.0.0.1:53. When Consul tries to resolve an entry, the Consul domain checks 127.0.0.1 first. This reduces the number of component hops involved in DNS resolution. Consul allows for effective intercomponent communication.

Other services that expect external DNS resolution also need a reference to the external DNS server to be present in /etc/resolv.conf.

Like all RAFT stores, if Consul loses quorum, it may require manual intervention. Therefore, a three-Consul-node cluster is required, preferably spanning three AZs. If you restart a node, when it comes back up, it will begin talking to its peers and replay the RAFT log to get up to date and synchronized with all the database history. It is imperative to ensure that a node is fully back up and has rejoined the cluster prior to taking a second node down; otherwise, when the second node goes offline, you might lose quorum. BOSH deploys Consul via a rolling upgrade to ensure that each node is fully available prior to bringing up the next.

Application Life-Cycle Binaries

Diego aims to be platform agnostic. All platform-specific concerns are delegated to two types of components:

• the Garden backend

• the Application Life-Cycle Binaries

The process of staging and running an application is complex. These concerns are encapsulated in a set of binaries known collectively as the Application Life-Cycle Binaries. There are different Application Life-Cycle Binaries depending on the container image (see also Figure 6-12):

• Buildpack-Application Life Cycle implements a traditional buildpack-based life cycle.

• Docker-Application Life Cycle implements a Docker-based OCI-compatible life cycle.

• Windows-Application Life Cycle implements a life cycle for .NET applications on Windows.

Figure 6-12. The Application Life-Cycle Binaries

Each of the aforementioned Application Life Cycles provides a set of binaries that manage a specific application type. For the Buildpack-Application Life Cycle, there are three binaries:

• The Builder stages a Cloud Foundry application. The CC-Bridge runs the Builder as a Task on every staging request. The Builder performs static analysis on the application code and performs any required preprocessing before the application is first run.

• The Launcher runs a Cloud Foundry application. The CC-Bridge sets the Launcher as the Action on the app’s DesiredLRP. The Launcher executes the user’s start command with the correct system context (working directory, environment variables, etc.).

• The Healthcheck performs a status check of the running ActualLRP from within the container. The CC-Bridge sets the Healthcheck as the Monitor action on the app’s DesiredLRP.

The Stager Task produced by the CC-Bridge downloads the appropriate Application Life-Cycle Binaries and runs them to invoke life cycle scripts such as stage, start, and health-check in the ActualLRP.

This is a pluggable module for running OS-specific components. Because the life cycle is OS specific, the OS is an explicitly specified field required by the LRP. For example, the current Linux setting references the cflinuxfs2 rootfs. In addition to the rootfs, the only other OS-specific component that is explicitly specified is the backend container type.

In addition to the Linux life cycle, Diego also supports a Windows life cycle and a Docker life cycle. The Docker life cycle (to understand how to stage a Docker image) is based on metadata from the Cloud Controller, based on what type of app we want to run. As part of the information for “stage these app bits,” there will be some indication of what branching the Stager is required to undertake. The binaries have tried to ensure that, as much as possible, these specifications are shared.