Apache YARN is a framework that has two primary responsibilities, when it comes to Hadoop:
- Management of the cluster resources
- Scheduling of Jobs and monitoring them for failures/completion.
The following fundamentals are important to understand the concepts of YARN:
- Resource Manager: Resource Manager is a global instance whose primary responsibility is to be the authority to arbitrate resources among all the applications in the system. The Resource Manager consists of two components:
- Scheduler: Applications get executed on various nodes of the cluster and the Scheduler’s responsibility is to make sure that appropriate resources are allocated to these running applications and are subject to familiar constraints of capacities, queues, and so on. The Scheduler, as the name suggests, is nothing but a scheduler and does not perform any monitoring or status-tracking for the application. It is also not responsible for restarting or handling in any of the failed tasks. The Scheduler performs its scheduling function based on the resource requirements of the applications; it does so based on the abstract notion of a resource Container, which incorporates elements such as memory, cpu, disk, and network.
- Applications Manager: The Applications Manager's main responsibility is to negotiate job submissions from the client with the set of containers that can execute the job. It can restart the container in case of failures. The per-application ApplicationMaster is responsible for negotiating resource containers from the Scheduler, tracking their status and monitoring for progress.
- Node Manager: Node Manager is an agent that runs on each machine of the cluster and is primarily responsible for monitoring container resource usages (cpu, memory, disk, network) and then reporting it back to the Resource Manager.
- Application Master: Application Master is a per-application library that negotiates resources from the Resource Manager and works with the Node Manager(s) to execute and monitor the tasks.
