Apache Mesos
Apache Mesos is an open-source cluster manager and scheduling framework for pooling and sharing resources like CPU, memory, and storage across multiple nodes. Instead of having a specific node for a web server or database server, Mesos provides a pool of machines for running multiple services simultaneously.
In Figure 5.4 we see an Apache Mesos Master with a standby node for high availability. Zookeeper is used for cluster state and leader election, and frameworks are used for scheduling work in the cluster.
Core Components
An Apache Mesos cluster consists of three major components: masters, agents, and frameworks.
Masters
A master daemon manages one or more agent daemons. It’s responsible for tracking each agent’s available resources, tracking active applications, and delegating tasks to agents.
Agents
Agent daemons run on each cluster node. Agents cluster nodes to provide the physical resources that the master tracks, pools, and distributes to each application/framework.
Frameworks
Frameworks are applications that run on an Apache Mesos cluster. Each framework is split into tasks, which are individual work units that run on Apache Mesos agents. Unlike many clustering solutions, Mesos enables frameworks to control much of the delegation of tasks. When Apache Mesos allocates resources to a framework, it makes an offer to the framework; if the offer doesn’t satisfy the framework’s constraints, the framework can reject it. Frameworks can be configured to accept or reject resources based on certain constraints.
Frameworks themselves consist of two components: schedulers and executors. Schedulers register with the master and handle resource offers. Executors are processes on the agent nodes that run tasks.
Mesosphere DCOS
Mesosphere Data Center Operating System (DCOS) is a commercial product from Mesosphere, Inc. (http://mesosphere). DCOS is based on Apache Mesos and includes enterprise-grade security in addition to many other core system services, like Mesos-DNS, a Command Line Interface (CLI), an API, and more.
Marathon
Marathon is an open-source framework from Mesosphere’s DCOS, and is used for long-running tasks. The microservices we will deploy are generally long running, and we will need to ensure they continue running until we decide we need to stop them for some reason, like reducing the count or updating them with a new version. Marathon can be used to ensure a defined number of our services remain running in the cluster, even if a node on which one of our service instances is running goes away—provided there are enough resources, of course. In addition to scheduling containers, marathon can schedule and run commands in the cluster. For example, we could bootstrap a node.js application that pulls down the necessary artifacts and runs. Marathon comes with a UI and an API that can be used to schedule tasks.
We will actually use Marathon to deploy and run our sample application in a Microsoft Azure Container Service.
Chronos
Chronos is an open-source framework from Mesosphere’s DCOS, and is used as a replacement for running cron jobs in a cluster. It’s a distributed, fault-tolerant scheduler that supports custom Mesos executors as well as default command executors. Chronos does a lot more than cron, and can schedule jobs that run inside Docker containers, use repeating interval notation, and can even trigger jobs by the completion of other jobs. Chronos includes a UI and API that can be used to schedule tasks.
Almost all large distributed applications include a growing number of scheduled jobs based on time or interval.