The output is as follows:

In the preceding code, you can see that the command adduser has been used to create or add a user. The Linux command passwd has been used to provide a password to our new user pysparkbook.

A new user is created. You might be thinking why new user. Why cant we install Hadoop in existing user. The reason behind that is, we want to provide a dedicated user for all the big data frameworks. In following lines of code we are going to create a user “pysparkbook”.

Output:

In the preceding code, you can see that the command adduser has been used to create or add a user. The command passwd has been used to provide a password for our new user pysparkbook to the sudo.

We will create two directories. The binaries directory under the home directory will be used to download software, and the allPySpark directory under the root (/) directory will be used to install big data frameworks:

Hadoop, Hive, Spark and many big data frameworks use Java to run on. That’s why we are first going to install Java. We are going to use OpenJDK for this purpose; we’ll install the eighth version of OpenJDK. We can install Java on CentOS by using the yum installer, as follows:

After installation of any software, it is a good idea to check the installation to ensure that everything is fine.

The output is as follows:

Java has been installed. Now we have to look for the environment variable JAVA_HOME, which will be used by all the distributed frameworks. After installation, JAVA_HOME can be found by using jrunscript as follows:

Here is the output:

Use this command to create a passwordless login :

Here is the output:

Here is the output:

Here is the output:

We are going to download Hadoop from the Apache website. As noted previously, we will download all the software into the binaries directory. We’ll use the wget command to download Hadoop:

Here is the output:

Our installation directory is allPySpark. The downloaded software is hadoop-2.6.5.tar.gz, which is a compressed directory. So at first we have to decompress it by using the tar command as follows:

We have to make some changes in the Hadoop environment file . This file is found in the Hadoop configuration directory. In our case, the Hadoop configuration directory is /allPySpark/hadoop/etc/hadoop/. Use the following line of code to add JAVA_HOME to the hadoop-env.sh file:

After opening the Hadoop environment file, add the following line:

In this step, we are concerned with three properties files :

These properties files are found in the Hadoop configuration directory. In the preceding chapter, we discussed HDFS. You learned that HDFS has two components: NameNode and DataNode. You also learned that HDFS uses data replication for fault-tolerance. In our hdfs-site.xml file, we are going to set the NameNode directory by using the dfs.name.dir parameter, the DataNode directory by using the dfs.data.dir parameter, and the replication factor by using the dfs.replication parameter.

After updating hdfs-site.xml, we are going to update core-site.xml. In core-site.xml, we are going to update only one property, fs.default.name. This property is used to determine the host, port, and other details of the file system. Add the following lines to core-site.xml:

Finally, we are going to modify mapred-site.xml. We also are going to modify mapreduce.framework.name, which will decide which runtime framework has to be used. The possible values are local, classic, or yarn. Add the following to mapred-site.xml:

Next, we’ll add the following lines to the .bashrc file . Open the .bashrc file:

Then add the following lines:

We have updated some property files. We are supposed to run in NameNode format so that all the changes are reflected in our framework. Use the following command to run NameNode format:

Here is the output:

Hadoop has been installed, and now we can start it. We can find the Hadoop starting script in /allPySpark/hadoop/sbin/. Although this script has been deprecated, we’ll use it for this example:

We know that the jps command will show all the Java processes running on the machine. Here is the command:

If everything is fine, we will see the process running as shown here:

Congratulations! We have finally installed Hadoop on our systems.

Use this command to extract the .tgz file :

Now we have to move the extracted Spark directory under the /allPySpark location . Use this command:

The Spark environment file possesses all the environment variables required to run Spark. We are going to set the following environment variables in an environment file:

First we have to copy the spark-env.sh.template file to spark-env.sh. The Spark environment file, spark-env.sh, is found inside spark/conf (the configuration directory location). Here is the command:

In the .bashrc file, we have to add the Spark bin directory. Use the following command:

We can start the PySpark shell by using the pyspark script. Discussion of the pyspark script will continue in the next recipe.

We have completed one more successful installation. Other installations are required to move through this book, but first let’s focus on the PySpark shell.

We can download Hive from the Apache Hive website. We download the Hive tar.gz file by using the wget command as follows:

Here is the output:

We have downloaded apache-hive-2.0.1-bin.tar.gz, a .tar.gz. So now we have to extract it. We can extract it by using the tar command as follows:

Hive is dispatched with the embedded Derby database for metastores. The Derby database is memory-less. Hence it is better to provide a definite location for it. We can provide that location in hive-site.xml. For that, we have to move hive-default.xml.template to hive-site.xml.

You can add the following lines to the end of hive-site.xml or you can change javax.jdo.option.ConnectionURL in the hive-site.xml file:

After that, we have to add HADOOP_HOME to the hive_env.sh file, as follows:

Open the .bashrc file . This file will stay in the home directory:

Now we have to create data warehouse directories. This data warehouse directory is used by Hive to place the data files.

Sometimes it is necessary to initiate a schema. You might be thinking, a schema of what? We know that Hive stores metadata of tables in a relational database. For the time being, we are going to use a Derby database as the metastore database for Hive. And then, in upcoming recipes, we are going to connect our Hive to an external PostgreSQL. In Ubuntu, Hive installation works without this command. But in CentOS I found it indispensable to run. Without the following command, Hive throws errors:

Now that Hive has been installed, we should check our work. Start the Hive shell by using the following command:

Then we will find that the Hive shell has been opened as follows:

PostgreSQL can be installed using the yum installer. Here is the command:

[sudo] password for pysparkbook:

To use PostgreSQL, we first need to use the initdb utility to initialize the database. If we don’t initialize the database, we cannot use it. At the time of database initialization, we can also specify the data file of the database. After installing PostgreSQL, we have to initialize it. The database can be initialized using following command:

Here is the output:

Here is the output:

We need a JDBC connector so that the Hive process can connect to the external PostgreSQL. We can get a JDBC connector by using the following command:

After getting the JDBC connector, we have to put it in the Hive lib directory:

Use this command to connect to PostgreSQL:

In this step, we are going to create a PostgreSQL user, pysparkBookUser. Then we are going to create a database named pymetastore. Our database is going to hold all the tables related to the Hive metastore.

First, create the user:

Here is the output:

Next, create the database:

Here is the output:

The c PostgreSQL command stands for connect. We have created our database pymetastore. Now we are going to connect to this database by using our c command:

You are now connected to the pymetastore database. You can see more PostgreSQL commands at www.postgresql.org/docs/9.0/static/app-psql.html .

Hive possess its own PostgreSQL scripts to populate tables for the metastore. The i command reads commands from the PostgreSQL script and executes those commands. The following command runs the hive-txn-schema-2.0.0.postgres.sql script, which will create all the tables required for the Hive metastore:

Here is the output:

The following commands will grant some permissions:

Remember that in order to update pg_hba.conf, you are supposed to be the root user. So first become the root user. Then open the pg_hba.conf file:

Then change all the peers and indent them to trust:

You can find more details about this change at http://stackoverflow.com/questions/2942485/psql-fatal-ident-authentication-failed-for-user-postgres .

Come out of root user.

Next, we’ll test that we are easily able to enter our database as our created user:

Here is the output:

We can modify the Hive-related configuration in the configuration file hive-site.xml. We have to modify the following properties:

Either modify these properties or add the following lines at the end of the Hive property file to get the required result:

We have connected Hive to an external relational database management system. So now it is time to start Hive and check that everything is fine. First, start Hive:

Our activities will be reflected in PostgreSQL. Let’s create a database and a table inside the database. We’ll create the database apress and the table apressBooks via the following commands:

Here is the output:

Here is the output:

Here is the output:

The significant work needed to connect Hive to an external database is done. In the following recipe, we are going to install Apache Mesos.

Use this command to copy the Hive property file:

Use this command to start PySpark:

Use this command to obtain Apache Mesos:

The output is as follows:

To extract Mesos, use this command:

To install Maven, we first need to install the repo:

It is time to install the dependencies required to install Maven:

Now we’re ready to download Maven:

Here is the output:

As with other software, we have to extract and move the Maven directory:

[sudo] password for pysparkbook:

Here is the output:

You may get errors as follows:

If you do get these errors, you need to perform the following installations and upgrade pytz:

Use this command to install Mesos:

After successful installation of Mesos, we can start the master by using the following command:

Use this command to start the slave on the same machine:

In an upcoming chapter, you will see how to start the PySpark shell on Mesos.

HBase also looks for JAVA_HOME. So we’ll update the HBase environment file with JAVA_HOME:

Even HBase can put its data on a local machine, in the case of a single-machine installation. But to make things clearer about the workings of HBase, we are going to create a directory on HDFS for HBase. First, we have to start HDFS. Use the following commands to create a directory in HDFS:

Let’s start with updating the property file, and then we will update the .bashrc file. In the HBase property file, we are going to update hbase:rootdir. The HBase property file stays in the HBase configuration directory. For us, the HBase configuration directory is /allPySpark/hbase/conf.

Here is the output: