VM modes

Local VM

You can install a virtual machine support (e.g., VirtualBox) on your local computer and then create a Linux VM into which you can install HBase and run it in either standalone or pseudodistributed mode.

Public cloud

This is similar to the previous option, except that you create the Linux VM (instance) on a public cloud such as Amazon EC2. You then install the HBase software and run it in standalone or pseudodistributed mode. You can also provision multiple virtual nodes and run HBase in a fully distributed mode. Be careful with running a production HBase cluster in a public cloud, as some specific configuration might be required.

Local virtual distribution

Install a virtual machine support (e.g., VirtualBox) on your local computer and instead of installing software manually, simply download the Cloudera QuickStart VM, which includes a fully functional HBase installation in pseudodistributed mode (no manual installation required).

Linux environment

If you don’t have a computer available on which to install a Linux distribution, you can run one on your local environment by running it in a local VM. The following applications will help you to run a Linux environment in a local VM:

• VirtualBox

• VMWare

• KVM

Hadoop distribution

This book covers HBase application design and development, but to be fully implemented, some use cases might require other external applications, such as Solr or Flume. Installing, configuring, and running those tools is outside the scope of this book. If you want to implement those integration examples, you will need to have a VM with an entire Hadoop distribution already installed. A Hadoop distribution is a bundle, usually packaged by a Hadoop vendor, including most if not all of the Hadoop-related applications. This is usually easier to install, as all the applications are configured to work together and they usually come with some setup scripts to prepare them all.

For simplicity and because it already contains all the applications we need (HDFS, HBase, Solr, Flume, etc.), you can use the Cloudera QuickStart VM. We used version 5.4.2 of this VM to build all those use cases, and all the examples are tested with it. If you are not familiar with Linux or HBase and Hadoop installation, it might be easier for you to follow the same path and use the QuickStart VM.

All of the examples in the book are built for HBase 1.0.1. If you choose to use another Hadoop distribution in a virtual environment or if you decide to install HBase locally, you’ll need to make sure that the distribution you choose comes with the same HBase version. While the examples might work with more recent versions, most of them will fail with older ones.

If you choose to use a Hadoop distribution, we still recommend reading the following sections so that you can become familiar with HBase, its constraints, and its installation.

In order to make the most of HBase’s capabilities, all of the examples in this book are tested against a local Linux standalone HBase server and the Cloudera QuickStart VM. If you have access to a Hadoop test cluster (physical or virtual), you can also use it. That will allow you to choose the environment with which you are the most comfortable to run your tests.

Memory

Whether you are working in your local environment or on a VM, you need enough memory for provide to HBase and the other services to run. A standalone HBase environment running locally will require a minimum of 1 GB memory. However, if possible, we recommend allocating more memory, as doing so will allow you to run the tests on bigger datasets and to have a better view of the results and the behaviors of the application. If you are testing the examples directly in a Hadoop distribution running in a virtual environment, you should allocate a minimum of 4 GB to the VM. Here again, giving more memory to the environment will allow you to have a better response time from your applications and to run on bigger datasets. That’s why we recommend allocating at least 8 GB to your virtual machine.

Disk space

The HBase installation file takes less than 100 MB (and the examples require just a few megabytes), so you can run all of the examples with less than 1 GB of available disk space. However, if you want to see all of the features in the examples pushed further, you’ll need to create bigger datasets—and that requires more space for you to test. Therefore, we recommend allowing for at least 10 GB of available disk space.

If you want to run in a virtual environment, you will need at least 4 GB of available disk space to download the VM and another 4 GB to extract it. The Cloudera QuickStart VM is configured to allow its virtual disk to grow up to 64 GB. Creating big datasets and running HBase for a long period will create a lot of data in your virtual environment. As a result, you will easily use up to 50 GB of disk space. Although it’s possible to start with less, we recommend that you to have this space available before you start.

Pros

• Requires less than 1 GB of disk space and only a few gigabytes of memory

• Very fast to stop and start

Cons

• Does not allow running of complex use cases where other tools are required

• Needs to run on a Linux-like environment

Pros

• Allows you to easily run HBase on non-Linux systems

• Avoid conflicts with already-installed applications

Cons

• Requires at least 3 GB of memory (2 GB for the operating system and 1 GB for HBase)

• Slower to start compared to the local mode

Pros

• Allows you to easily run HBase and other related applications (Solr, HDFS, YARN, etc.)

• Avoid conflicts with already installed applications

Cons

• Requires more memory than the two other modes (4 GB minimum, with 8+ GB recommended)

• Slowest mode to start/restart

help

The help command lists all of the available HBase shell commands. Because the output of the help command in the book as it is quite lengthy, we won’t repeat it here; however, over the next few chapters, we will issue most of the commands reported by help.

create

The create command helps to create a new HBase table. The following command creates a table called t1 with a single column family called f1, all with the default parameters:

hbase(main):002:0> create 't1', 'f1'
0 row(s) in 1.2860 seconds

=> Hbase::Table - t1

list

The list command shows the newly created table:

hbase(main):003:0> list
TABLE
t1
1 row(s) in 0.0100 seconds

=> ["t1"]
hbase(main):004:0>

list only reports user tables. User tables are nonsystem tables (i.e., they are tables that are explicitly created by the user).

If everything has been installed and is running correctly, you should have been able to run those three commands with outputs very similar to what has been reported.

Download from command line

The first thing we need to do is download the source repository. As we explained in the previous section, examples are hosted on a Git repository. If you prefer to use Eclipse for all those operations, refer to “Download and build using Eclipse”.

The following Git command will download a copy of the examples into a local folder:

git clone https://github.com/ArchitectingHBase/examples.git ~/ahae

This will create a directory called architecting-hbase-applications-examples, which looks like this:

#:~/ahae$ ll
total 32K
drwxr-xr-x 2 jmspaggiari jmspaggiari  4096 Sep 11 21:09 conf
-rw-r--r-- 1 jmspaggiari jmspaggiari 11324 Sep 11 21:09 LICENSE
-rwxr-xr-x 1 jmspaggiari jmspaggiari  5938 Sep 11 21:09 pom.xml
-rw-r--r-- 1 jmspaggiari jmspaggiari    38 Sep 11 21:09 README.md
drwxr-xr-x 3 jmspaggiari jmspaggiari  4096 Sep 11 21:09 src

If you are using Eclipse from the VM, you can now import the project into Eclipse using the Import command and the “Maven/Existing maven projects” entry.

Build from command line

Now that you have the sources available, you can build them using Maven with the following command:

#:~/ahae$ mvn package

This will download all of the required dependencies, compile the sources into binaries, and package them. You should see something like this in your console:

#:~/ahae$ mvn package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building architecting-hbase-applications-examples 1
[INFO] ------------------------------------------------------------------------
Downloading: http://onejar-maven-plugin.googlecode.com/svn/mavenrepo/org/apache/
maven/plugins/maven-resources-plugin/2.3/maven-resources-plugin-2.3.pom
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/
maven-resources-plugin/2.3/maven-resources-plugin-2.3.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/
maven-resources-plugin/2.3/maven-resources-plugin-2.3.pom (5 KB at 18.1 KB/sec)
Downloading: http://onejar-maven-plugin.googlecode.com/svn/mavenrepo/org/apache/
maven/plugins/maven-resources-plugin/2.3/maven-resources-plugin-2.3.jar
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/
maven-resources-plugin/2.3/maven-resources-plugin-2.3.jar
.
.
.
[INFO] Building jar: /home/cloudera/ahae/target/ahae.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51.574s
[INFO] Finished at: Thu Sep 11 21:16:10 EDT 2014
[INFO] Final Memory: 40M/916M
[INFO] ------------------------------------------------------------------------

The first time you run the build command, depending on the speed of your Internet connection and the dependencies to be downloaded, this operation can take a few minutes. Dependencies will be stored in a local Maven cache folder and will not have to be downloaded again unless they change. When completed, you should find the examples packaged under the target directory with the name ahae-1.jar. This is the file we will use to run all of our examples.

Download and build using Eclipse

If you want to use Eclipse to look at the examples, you will need to make sure that you have the m2e Eclipse Maven plugin and the required EGit connectors installed. Using Eclipse will allow you to run the examples in the step-by-step mode to inspect the different variables and results. This is the way we recommend to run the examples.

Import the examples into the Eclipse Java Browsing perspective by selecting the Import option from the File menu. Eclipse will prompt you with a list of all the different kinds of projects it can import. Under the Maven section, select “Check out Maven Projects from SCM” and press Next. This option will be available only if you have correctly installed the m2e plugin or if your Eclipse version comes with the plugin already installed.

As illustrated in Figure 5-1, select “git” in the SCM URL drop-down menu, and enter the Git repository URL in the text area: https://github.com/ArchitectingHBase/examples. If the “git” option is not available, make sure you have the correct connector installed. You can use the “m2e Marketplace” link in the same window to install the connector. When all of this is completed, simply press Finish to get the project imported. This will close the window, download all the dependencies defined into the project, and import the source code into your Eclipse workshop into a project called architecting-hbase-applications-examples. This operation can take a few minutes, so now is a good time to grab a coffee or go for a walk—by the time you get back, the project should be available in your workspace.

Figure 5-1. Eclipse MVN import window

We are also going to use Maven to build the application using Eclipse. To do so, right-click on the newly imported project, select the “Run As” option, and then “Maven Build…”. This will open a window to configure the way you want to build this project. The only part you need to modify is the “Goals” text field where you need to enter “package”. Once you’ve completed these steps, simply click on the “Apply” button, and then the “Run” button. In the “Console” view, you will see Maven building and packaging the examples for you. The output should be the same as in “Build from command line”.

To simplify all of the command lines, create a symbolic link to the project folder in your home directory:

ln -s ~/workspace/architecting-hbase-applications-examples ~/ahae

From command line

When running the examples from the command line, you need to make sure that all HBase binaries and configuration files are in the classpath. HBase provides you its classpath when you issue the HBase classpath command. In addition to the example packaged binaries, those are the paths you will need to give to Java to correctly run the examples.

The following command will run the TestInstallation example from the com.architecting.ch05 package:

#:~/ahae$ java -classpath ~/ahae/target/ahae.jar:`hbase classpath`
                                         com.architecting.ch05.TestInstallation

TestInstallation is a simple piece of Java code that will try to call HBase to determine whether it is running (see Example 5-1).

public class TestInstallation {
  private static final Log LOG = LogFactory.getLog(TestInstallation.class);

  public static void main(String[] args) {
    Configuration conf = HBaseConfiguration.create();
    try {
      LOG.info("Testing HBase connection...");
      HBaseAdmin.checkHBaseAvailable(conf);
      LOG.info("HBase is running correctly...");
    } catch (MasterNotRunningException e) {
      LOG.error("Unable to find a running HBase instance", e);
    } catch (ZooKeeperConnectionException e) {
      LOG.error("Unable to connect to ZooKeeper", e);
    } catch (ServiceException e) {
      LOG.error("HBase service unavailable", e);
    } catch (IOException e) {
      LOG.error("Error when trying to get HBase status", e);
    }
  }
}

This command calls the hbase classpath command to get the current HBase classpath, then it appends the example’s JAR and runs one of the programs in the JAR. This command will produce a lot of output from the different HBase classes. What you need to look at is the output from the example itself, which should look like the following snippet (note that some lines have been removed or shortened for the sake of brevity; also, you can export the classpath variable into your running environment to avoid having to specify it for each of the Java commands you will run):

export CLASSPATH= ~/ahae/target/ahae.jar:`hbase classpath`

2014-09-14 INFO [main] ch05.TestInstallation: Testing HBase connection... 
2014-09-14 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop
    library for your platform... using builtin-java classes where applicable
2014-09-14 INFO [main] zk.RecoverableZooKeeper: Process identifier=hconnection-
    0x783c342b connecting to ZooKeeper ensemble=localhost:2181
2014-09-14 INFO [main] zk.ZooKeeper: Client environment:zookeeper.
    version=3.4.6-1565, built on 02/20/2014 09:09 GMT
2014-09-14 INFO [main] zk.ZooKeeper: Client environment:host.name=t430s
2014-09-14 INFO [main] zk.ZooKeeper: Client environment:java.version=1.7.0_60
2014-09-14 INFO [main] zk.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2014-09-14 INFO [main] zk.ZooKeeper: Client environment:java.compiler=<NA>
2014-09-14 INFO [main] zk.ZooKeeper: Client environment:os.name=Linux
2014-09-14 INFO [main] zk.ZooKeeper: Initiating client connection,
    connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-
    0x783c342b, quorum=localhost:2181, baseZNode=/hbase
2014-09-14 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Socket
    connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2014-09-14 INFO [main] client.HConnectionManager$HConnectionImplementation:
    Closing master protocol: MasterService
2014-09-14 INFO [main] client.HConnectionManager$HConnectionImplementation:
    Closing zookeeper sessionid=0x1487467e621000a
2014-09-14 INFO [main] zk.ZooKeeper: Session: 0x1487467e621000a closed
2014-09-14 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2014-09-14 INFO [main] ch05.TestInstallation: HBase is running correctly... 

and are two lines printed by the example.

If you don’t see the “HBase is running correctly…” line, you’ll need to make sure your HBase server is running correctly and examples have been built as expected.

From Eclipse

To run an example from Eclipse, you simply need to right-click on it, select “Run As” and then “Java Application”. Output of the same TestInstallation example run from Eclipse should be shorter and should include the HBase outputted logs. You should see the following lines printed in the console:

2014-09-14 18:45:34 INFO ch05.TestInstallation: Testing HBase connection...
...
2014-09-14 18:45:35 INFO ch05.TestInstallation: HBase is running correctly...