Executors

The Executors tab uses ExecutorsListener to collect information about executors for a Spark application. An executor is a distributed agent that is responsible for executing tasks. Executors are instantiated in different ways. For example, they can be instantiated when CoarseGrainedExecutorBackend receives RegisteredExecutor message for Spark Standalone and YARN. The second case is when a Spark job is submitted to Mesos. The Mesos's MesosExecutorBackend gets registered. The third case is when you run your Spark jobs locally, that is, LocalEndpoint is created. An executor typically runs for the entire lifetime of a Spark application, which is called static allocation of executors, although you can also opt in for dynamic allocation. The executor backends exclusively manage all the executors in a computing node or clusters. An executor reports heartbeat and partial metrics for active tasks to the HeartbeatReceiver RPC endpoint on the driver periodically and the results are sent to the driver. They also provide in-memory storage for RDDs that are cached by user programs through block manager. Refer to the following figure for a clearer idea on this:


Figure 12: Spark driver instantiates an executor that is responsible for HeartbeatReceiver's Heartbeat message handler

When an executor starts, it first registers with the driver and communicates directly to execute tasks, as shown in the following figure:


Figure 13: Launching tasks on executor using TaskRunners

You should be able to access the Executors tab at http://localhost:4040/executors.

Figure 14: Executor tab on Spark web UI

As shown in the preceding figure, Executor ID, Address, Status, RDD Blocks, Storage Memory, Disk Used, Cores, Active Tasks, Failed Tasks, Complete Tasks, Total Tasks, Task Time (GC Time), Input, Shuffle Read, Shuffle Write, and Thread Dump about the executor can be seen.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.222.232.128