Configuring the parallelism of a topology

There are a number of components in a Storm topology. The throughput (processing speed) of the topology is decided by the number of instances of each component running in parallel. This is known as the parallelism of a topology. Let's first look at the processes or components responsible for the parallelism feature of the Storm cluster.

The worker process

A Storm topology is executed across multiple nodes in the Storm cluster. Each of the nodes in the cluster can run one or more JVMs called worker processes that are responsible for processing a part of the topology.

A Storm cluster can run multiple topologies at the same time. A worker process is bound to one of these topologies and can execute multiple components of that topology. If multiple topologies are run at the same time, none of them will share any of the workers, thus providing some degree of isolation between topologies.

The executor

Within each worker process, there can be multiple threads that execute parts of the topology. Each of these threads is called an executor. An executor can execute only one of the components, that is, any one spout or bolt in the topology.

Each executor, being a single thread, can only execute tasks assigned to it serially. The number of executors defined for a spout or bolt can be changed dynamically while the topology is running. This means that you can easily control the degree of parallelism for various components in your topology.

Tasks

A task is the most granular unit of task execution in Storm. Each task is an instance of a spout or bolt. While defining a Storm topology, you can specify the number of tasks for each spout and bolt. Once defined, the number of tasks cannot be changed for a component at runtime. Each task can be executed alone or with another task of the same type or another instance of the same spout or bolt.

The following diagram depicts the relationship between the worker process, executors, and tasks. Each of the blocks that contains tasks is an executor, for example, there are two executors for each component, and each component hosts a different number of tasks.

Also, as you can see in the following diagram, there are two executors and eight instances for Task A. The two executors are running in two different workers. If you are not getting enough performance out of this configuration, you can easily change the number of executors to four or eight for Task A to increase performance. The following diagram shows the relationship between various components of a topology:

Tasks

Relationship between executors, tasks, and worker processes

Configuring parallelism at the code level

In Storm, we can achieve the desired level of parallelism for tuning parameters such as the number of worker processes, number of executors, and number of tasks. Storm provides an API to configure these parameters. In this section, the following steps will show how we can configure parallelism at the code level:

  1. Set the number of worker processes.

    We can set the number of worker processes at the code level using the setNumWorkers method of the backtype.storm.Config class. The following is the code snippet that shows these settings in practice:

    Config conf = new Config();
conf.setNumWorkers(3);

    In the preceding code, we have configured the number of workers to three. Storm will run the three workers for the LearningStormSingleNodeTopology topology.

  2. Set the number of executors.

    We can set the number of executors at the code level by passing the parallelism_hint argument in the setSpout(args,args,parallelism_hint) or setBolt(args,args,parallelism_hint) method of the backtype.storm.topology.TopologyBuilder class. The following is the code snippet to show these settings in practice:

    TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("LearningStormSpout", new LearningStormSpout(), 2);
builder.setBolt("LearningStormBolt", new LearningStormBolt(), 4);

    In the preceding code, we have set the parallelism_hint parameter to 2 for LearningStormSpout and 4 for LearningStormBolt. At the time of execution, Storm will assign two executors for LearningStormSpout and four executors for LearningStormBolt.

  3. Set the number of tasks.

    We can configure the number of tasks that can execute inside the executors. The following is the code snippet to show these settings in practice:

    builder.setSpout("LearningStormSpout", new LearningStormSpout(), 2).setNumTasks(4);

    In the preceding code, we have configured the two executors and four tasks of LearningStormSpout. For LearningStormSpout, Storm will assign two tasks per executor. By default, Storm will run one task per executor if the user does not set the number of tasks at the code level.

Distributing worker processes, executors, and tasks in the sample topology

Let's assume the number of worker processes set for the sample topology is three, the number of executors for LearningStormSpout is three, and the number of executors for LearningStormBolt is three. Also, we have configured the number of tasks for LearningStormBolt as six, which means each executor will run two tasks. Then, the following diagram shows how the sample topology would look in the operation:

Distributing worker processes, executors, and tasks in the sample topology

The Hello-World topology distribution

The total parallelism of the topology can be calculated with the total parallelism = number of spout tasks + number of bolt tasks formula.

If the total parallelism of the topology is not a multiple of the number of workers, Storm will distribute the tasks as evenly as possible.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.191.62.122