Deploying a topology on a remote Storm cluster

In this section, we will focus on how we can deploy topologies on a remote Storm cluster. We will start with the installation of a Storm client on the client machine, which can be different from the machines in the Storm cluster because submitting and deploying topologies on a remote Storm cluster requires a Storm client.

The following are the steps that need to be performed to set up a Storm client:

  1. Download the latest stable Storm release from https://storm.incubator.apache.org/downloads.html.
  2. Once you have downloaded the latest version, copy and unzip it to the client machine. Now, we set the STORM_HOME environment variable to make the installation easier.
  3. Go to the $STORM_HOME/conf directory at the client node and add the following line to the storm.yaml file:
    nimbus.host: "nimbus.host.ip"
  4. Also, now place the copy of the storm.yaml file located at $STORM_HOME/conf in the ~/.storm folder on the client machine.

Once the installation of the Storm client is done, we are good to go to deploy a topology on the remote machine. To demonstrate how we can deploy a topology on the remote cluster, we will use the sample topology developed in Chapter 1, Setting Up Storm on a Single Machine. The following are the commands that need to be executed to deploy a topology on the remote cluster.

Go to the $STORM_HOME directory on the client machine and run the following command:

bin/storm jar jarName.jar [TopologyMainClass] [Args]

Deploying the sample topology on the remote cluster

This section will explain how we can deploy the sample topology created in Chapter 1, Setting Up Storm on a Single Machine, on the Storm cluster by performing the following steps:

  1. Execute the following command on the Storm client machine to deploy the sample topology on the remote Storm cluster. The client will then submit this topology across the network to the Nimbus, which will then distribute it to the supervisors.
    bin/storm jar $STORM_PROJECT_HOME/target/storm-example-0.0.1-SNAPSHOT-jar-with-dependencies.jar com.learningstorm.storm_example.LearningStormSingleNodeTopology LearningStormClusterTopology

    The output of the preceding command is as follows:

    18   [main] INFO  backtype.storm.StormSubmitter  - Uploading topology jar ../storm-example/target/storm-example-0.0.1-SNAPSHOT-jar-with-dependencies.jar to assigned location: /tmp/storm-data/nimbus/inbox/stormjar-aa96e582-1676-4654-a995-15a4e88b6a50.jar
28   [main] INFO  backtype.storm.StormSubmitter  - Successfully uploaded topology jar to assigned location: /tmp/storm-data/nimbus/inbox/stormjar-aa96e582-1676-4654-a995-15a4e88b6a50.jar
29   [main] INFO  backtype.storm.StormSubmitter  - Submitting topology test-ack in distributed mode with conf {"topology.workers":3}
196  [main] INFO  backtype.storm.StormSubmitter  - Finished submitting topology: LearningStormClusterTopology

    The preceding console output shows that the LearningStormClusterTopology topology is submitted on the remote cluster, and three worker processes are executed.

  2. Run the jps commands on the supervisor machines to view the worker process:
    1. Run the jps command on the first supervisor machine:
      jps

      The preceding command's output is as follows:

      24347 worker
       23940 supervisor
       24593 Jps
24349 worker

      Two worker processes are assigned to the first supervisor machine.

    2. Run the jps command on the second supervisor machine:
      jps

      The preceding command's output is as follows:

      24344 worker
      23941 supervisor
      24543 Jps

      One worker process is assigned to the second supervisor machine.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.145.173.199