Operating Systems

Accumulo is regularly run and tested on several versions of Linux:

• Red Hat Enterprise Linux

• CentOS 6

• Ubuntu 12 and above

Development platforms include Linux and Mac OS X.

Kernel Tweaks

A few low-level kernel settings that can dramatically impact the responsiveness of the cluster might need to be tuned for scaling up the per-machine resources allowed when more than 100 machines are in a cluster.

echo 1 > /proc/sys/vm/swappiness

echo "vm.swappiness = 1" >> /etc/sysctl.conf

accumulo  nofile  soft  65536
accumulo  nofile  hard  65536

Native Libraries

Because Java garbage collection can cause pauses that make it difficult to determine the status of a process, Accumulo employs its own memory management for newly written entries. This requires the use of binary libraries compiled for the specific architecture on which Accumulo is deployed. In older versions of Accumulo, binary libraries for the Linux x86-64 architecture were provided in the distributed files. If you are deploying to another architecture, or using version 1.6, these libraries must be built after installation.

If the binary libraries are not available, Accumulo will fall back on a pure-Java implementation, but at the cost of decreased performance and stability.

User Accounts

Many distributions of Hadoop configure a mapred and a hdfs user. Accumulo can be configured to use its own accumulo account.

If Accumulo will be installed from RPM or Debian packages, the package scripts can create the accumulo user account.

Linux Filesystem

Accumulo stores data in HDFS, which in turn stores blocks of data in the underlying Linux filesystem. Popular Linux filesystems include ext3, ext4, and XFS.

Accumulo 1.4 and earlier versions required the ability to write to a local directory to store write-ahead logs. A directory must be created for this purpose and must be writeable by the accumulo user. It can improve performance to put write-ahead logs on separate disks from disks storing HDFS data. These directories are specified in the accumulo-site.xml file described in “Server Configuration Files”.

Accumulo 1.5 and later versions store these files in HDFS so no additional directories need to be created.

System Services

Accumulo relies on several system services to operate properly:

Domain Name Service (DNS)

Hadoop requires that domain names of machines be resolvable from domain to IP address and from IP address to domain name.

Network Time Protocol (NTP)

When an Accumulo table is configured to use a TimeType of milliseconds (MILLIS), Accumulo’s tablet servers rely on system time for applying timestamps to mutations that do not otherwise have a timestamp provided. With many machines in a cluster, some machines are bound to have clocks that are off. Running NTP daemons can help keep clocks closer in sync and avoid situations in which assigned timestamps jump forward as tablets are migrated from one server to the next. Tablet servers ensure that the timestamps they assign never decrease for any given tablet.

Secure Shell (SSH)

Accumulo ships with scripts that use SSH to start and stop processes on all machines in a cluster from a single node. This is not required, however, if another means of starting and stopping processes is used, such as init.d scripts.

To keep your Accumulo data secure, you also need to ensure that these services are secure just like any other system built on top of Hadoop. For the scope of this book, we cover a few important details in “Security”.

Software Dependencies

Accumulo depends on several software packages. First, Accumulo is written in Java. Java versions 1.6 and 1.7 have been tested and are known to work. The Sun/Oracle JDK is used more often in production, although OpenJDK is often used for development.

maxClientCnxns=250

Tarball Distribution Install

Accumulo can be downloaded as a Gzipped TAR file from a mirror at http://accumulo.apache.org/downloads/.

Extract the tarball to the desired location and ensure that the files are owned by the user that Accumulo processes will run as. As of Accumulo1.6, native libraries must be built from source. Follow the instructions in “Building native libraries” for building these and proceed to “Configuration”.

Installing on Cloudera’s CDH

Cloudera provides a popular commercial distribution of Hadoop called CDH (the Cloudera Distribution for Hadoop) that ships Accumulo 1.6 as part of CDH version 5. Accumulo is packaged as an optional parcel in the Cloudera Manager that can be installed as part of the general installation process (Figure 11-1).

Figure 11-1. Selecting the Accumulo parcel during CDH5 install

Selecting Accumulo from the list of additional parcels and clicking Continue will begin the installation process (Figure 11-2).

Figure 11-2. Installing the Accumulo parcel

Accumulo is now an available service. Next we need to add the service to our cluster, which will configure and start up Accumulo. From the main Cloudera Manager page, select the Add Service option. This will show a list of services to be installed (Figure 11-3). Selecting Accumulo will install Accumulo and all its dependent services, including HDFS and ZooKeeper.

Figure 11-3. Adding the Accumulo service to the cluster

The Add Service Wizard will prompt you to assign roles to various machines in the cluster. These include which machines will run the master process, tablet server processes, etc. (Figure 11-4).

Figure 11-4. Assigning Accumulo roles to machines

Next the wizard will prompt for any configuration options that should be set before starting the service (Figure 11-5). This is a good opportunity to set things like the Instance Secret. Other options have good starting defaults.

Figure 11-5. Setting configuration options before installing the Accumulo service

Finally, the wizard will start Accumulo and dependent services, if any are not already started (Figure 11-6).

Figure 11-6. Services are started by the Add Service Wizard

After installation Accumulo will show up as a service on the main view of a cluster in the Cloudera Manager (Figure 11-7).

Figure 11-7. Accumulo in the overall Cloudera cluster view

You can see Accumulo-specific details by clicking the Accumulo 1.6 link, which will display the Accumulo Service details page (Figure 11-8). There is also a link to the Accumulo monitor.

Figure 11-8. Viewing the Accumulo service in Cloudera Manager

Further configuration of the Accumulo service can be done via the Configuration tab at the top of the service details page (Figure 11-9).

Figure 11-9. Additional Accumulo configuration

After installation, the Accumulo files are located in /opt/cloudera/parcels/ACCUMULO. The commands in the bin/ directory will be part of the PATH.

Configuration files are in /etc/accumulo/conf.

Custom JARs should be placed in /opt/cloudera/parcels/ACCUMULO/lib/accumulo/lib/ext/ if you’re not using HDFS to distribute files, as detailed in “Using HDFS”.

Installing on Hortonworks’ HDP

The Hortonworks Data Platform (HDP) is a distribution of Hadoop and other scalable data processing technologies that emphasizes open source contribution. HDP 2.1 ships with Accumulo 1.5.1 as an optional RPM.

On each machine to participate in the accumulo cluster, run:

# yum install accumulo

This will install the Accumulo JARs and configuration files customized for use with HDP:

Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package accumulo.x86_64 0:1.5.1.2.1.5.0-695.el6 will be installed
--> Finished Dependency Resolution
...
Total download size: 11 M
Installed size: 13 M
Is this ok [y/N]: y
Downloading Packages:
...
accumulo-1.5.1.2.1.5.0-695.el6.x86_64.rpm
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : accumulo-1.5.1.2.1.5.0-695.el6.x86_64
  Verifying  : accumulo-1.5.1.2.1.5.0-695.el6.x86_64
Installed:
  accumulo.x86_64 0:1.5.1.2.1.5.0-695.el6
Complete!

The RPM will install Accumulo in /usr/lib/accumulo, and put configuration files in /etc/accumulo/conf. Configuration files appropriate for the memory available on servers can be copied from /etc/accumulo/conf/examples. For example, to copy the files that configure Accumulo to use 3 GB of RAM we would do:

# cp /etc/accumulo/conf/examples/3GB/standalone/* /etc/accumulo/conf/

The accumulo-site.xml and accumulo-env.sh files can be adjusted according to the instructions in “Configuration”, but the directory references are already set up according to the HDP layout.

Next we need to make the directory in HDFS that our configuration files expect:

# hdfs dfs -mkdir -p /usr/accumulo/data

HDP instructions recommend changing permissions on the data directory to grant access to all users. This should be restricted to only the accumulo user for production environments:

# hdfs dfs -chmod -R 777 /usr/accumulo/data

Next we’ll change the ownership of the Accumulo data directory:

# sudo -u hdfs hdfs dfs -chown -R accumulo:hdfs /usr/accumulo/data

We can now initialize Accumulo. We need to do this only once and from one machine:

# /usr/lib/accumulo/bin/accumulo init
[util.Initialize] INFO : Hadoop Filesystem is hdfs://sandbox.hortonworks.com:8020
[util.Initialize] INFO : Accumulo data dir is /user/accumulo/data
[util.Initialize] INFO : Zookeeper server is localhost:2181
[util.Initialize] INFO : Checking if Zookeeper is available. If this hangs, then
    you need to make sure zookeeper is running

Instance name : hdp
Enter initial password for root (this may not be applicable for your security
    setup): ******
Confirm initial password for root: ******
[Configuration.deprecation] INFO : dfs.replication.min is deprecated. Instead,
    use dfs.namenode.replication.min
[Configuration.deprecation] INFO : dfs.block.size is deprecated. Instead, use
    dfs.blocksize
[master.Master] INFO : Loaded class :
    org.apache.accumulo.server.security.handler.ZKAuthorizor
[master.Master] INFO : Loaded class :
    org.apache.accumulo.server.security.handler.ZKAuthenticator
[master.Master] INFO : Loaded class :
    org.apache.accumulo.server.security.handler.ZKPermHandler
[security.AuditedSecurityOperation] INFO : Initialized root user with username:
    root at the request of user !SYSTEM

Accumulo can be started with the start-all.sh script. See Chapter 12 for more details on this process:

[root@sandbox conf]# /usr/lib/accumulo/bin/start-all.sh
Starting monitor on localhost
Starting tablet servers .... done
Starting tablet server on localhost
[server.Accumulo] INFO : Attempting to talk to zookeeper
[server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk
    to HDFS
[server.Accumulo] INFO : Connected to HDFS
Starting master on localhost
Starting garbage collector on localhost
Starting tracer on localhost

[root@sandbox conf]#

The default RPM may not ship with native libraries built. See “Building native libraries” for more information on building native libraries, which will provide better performance.

Installing on MapR

MapR is a distribution of Hadoop that includes a completely redesigned, proprietary, distributed filesystem, called MapR-FS, that takes the place of HDFS. The MapR distribution includes additional features for security and enterprise integration such as NFS compatibility.

Accumulo 1.6 can be installed on MapR version 3.1 or 3.0 using the following steps (also see these instructions):

# wget http://mirror.cc.columbia.edu/pub/software/apache/accumulo/1.6.1/
  accumulo-1.6.1-bin.tar.gz
# tar -xzf accumulo-1.6.1-bin.tar.gz

# mkdir /opt/accumulo
# mv accumulo-1.6.1 /opt/accumulo/

Create a volume for storing Accumulo data. This volume will take the place of the /accumulo directory in HDFS and has additional capabilities, such as snapshots, mirroring, and quotas:

# maprcli volume create -name project.accumulo -path /accumulo

We’ll disable the MapR filesystem’s compression because Accumulo compresses data by default:

# hadoop mfs -setcompression off /accumulo

We’ll create a configuration directory containing mostly links to the original MapR configuration files, with the exception of core-site.xml, so we can alter it without affecting other services:

[accumulo-1.6.1]# mkdir hadoop
[accumulo-1.6.1]# mkdir hadoop/hadoop-0.20.2
[accumulo-1.6.1]# cd  hadoop/hadoop-0.20.2
[hadoop-0.20.2]# ln -s /opt/mapr/hadoop/hadoop-0.20.2/* .
[hadoop-0.20.2]# rm conf
rm: remove symbolic link `conf'? y
[hadoop-0.20.2]# mkdir conf
[hadoop-0.20.2]# cd conf
[conf]# ln -s /opt/mapr/hadoop/hadoop-0.20.2/conf/* .
[conf]# cp core-site.xml t
[conf]# mv t core-site.xml
mv: overwrite `core-site.xml'? y

Adding the following properties to core-site.xml will tell the MapR filesystem to disable read/write caching, because Accumulo does its own caching:

<property>
  <name>fs.mapr.readbuffering</name>
  <value>false</value>
</property>
<property>
  <name>fs.mapr.aggregate.writes</name>
  <value>false</value>
</property>

Edit the following lines of /opt/mapr/conf/warden.conf to allow Accumulo to use up to 2 GB of memory:

service.command.os.heapsize.max=2750   (from 750)
service.command.os.heapsize.min=2256 (from 256)

Next we’ll configure Accumulo. Follow the instructions in “Server Configuration Files” for copying configuration files and “Building native libraries” on building native libraries. Then return here for MapR-specific configuration.

Edit accumulo-env.sh to point to the proper Hadoop and ZooKeeper directories:

test -z "$HADOOP_PREFIX" && 
export HADOOP_PREFIX=/opt/accumulo/accumulo-1.6.1/hadoop/hadoop-0.20.2/

test -z "$ZOOKEEPER_HOME" && 
export ZOOKEEPER_HOME=/opt/accumulo/accumulo-1.6.1/hadoop/hadoop-0.20.2/lib

Edit accumulo-site.xml with following properties:

<property>
  <name>instance.zookeeper.host</name>
  <value>maprdemo:5181</value>
  <description>comma separated list of zookeeper servers</description>
</property>

<property>
  <name>tserver.port.client</name>
  <value>9996</value>
</property>

<property>
  <name>master.walog.closer.implementation</name>
  <value>org.apache.accumulo.server.master.recovery.MapRLogCloser</value>
</property>

<property>
  <name>tserver.wal.blocksize</name>
  <value>562M</value>
</property>

Now Accumulo can be initialized as described in “Initialization”.

Running via Amazon Web Services

Amazon Web Services (AWS) provide a set of scripts for spinning up an Accumulo cluster that runs on virtual machines in Amazon’s EC2 (Elastic Compute Cloud).

This involves first setting up a ZooKeeper cluster using Apache Whirr, and then using the Elastic MapReduce command-line tools.

To setup a ZooKeeper cluster using Apache Whirr, use the following commands:

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr

$ bin/whirr launch-cluster --cluster-name=zookeeper
--instance-templates='1 zookeeper'
--provider=aws-ec2
--identity=$AWS_ACCESS_KEY_ID
--credential=$AWS_SECRET_ACCESS_KEY

The Security Groups must be configured to allow the Accumulo machines to connect to ZooKeeper processes on these machines. See the AWS documentation on EC2 security groups for Linux instances.

Obtain the EMR CLI tools:

 $ elastic-mapreduce --create --alive --name "Accumulo" --bootstrap-action
 s3://elasticmapreduce/samples/accumulo/accumulo-install.sh
 --args "IP,DBNAME,PASSWORD" --bootstrap-name "install Accumulo"
 --enable-debugging --log-uri s3://BUCKETNAME/accumulo-logs/
 --instance-type m1.large --instance-count 4 --key-pair KEY

Use your own values for IP, DBNAME, PASSWORD, BUCKETNAME, and KEY:

IP

The IP address of one of the ZooKeeper nodes.

DBNAME

The name that will be used as the Accumulo instance name.

PASSWORD

The password to use for the Accumulo root account.

BUCKETNAME

The name of an S3 bucket that will be used store Accumulo logs.

KEY

The name of an EC2 SSH key-pair.

The --instance-type parameter describes the type of Amazon virtual machine to be used. m1.large machines work well because they have local storage disks and a modest amount of memory and CPU. m1.xlarge instance types have also been used to build Accumulo clusters in EC2 but may not be completely configured for these scripts.

Once the script starts running, you will see a line of output similar to:

Created job flow j-1XIYVOM2PGH3R

This identifier can be used to obtain the address of the Accumulo master node:

$ elastic-mapreduce --list j-1XIYVOM2PGH3R

j-1XIYVOM2PGH3R     Waiting  ec2-23-22-183-67.compute-1.amazonaws.com
  Accumulo
  PENDING        Setup Hadoop Debugging

$ ssh [email protected]

The Accumulo instance is running at this point.

Building from Source

Accumulo is distributed under the Apache open source license, so the source code can be downloaded and modified to suit a particular need. Any modifications to the source code, or to use options different from those that were used to create the binary distributions, will require building Accumulo from source.

Several tools make this process easier. Specifically, the Java SDK and Maven build tool should be installed. Java JDK version 1.6 or 1.7 and Maven version 3.0.4 will work.

To build from source, first download the source packages from a mirror listed on the Apache Accumulo website.

Source code is found in the file ending in src.tar.gz. Once downloaded it can be unpacked via:

tar -xzf accumulo-1.6.x-src.tar.gz

This will create a directory containing all the source files.

cd accumulo-1.6.x

mvn package -P assemble

mvn -Dhadoop.version=0.23.5 package -P assemble

mvn -Dhadoop.profile=1 -Dhadoop.version=1.1.0 package -P assemble

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64/

[centos@centos accumulo-1.6.0]$ bin/build_native_library.sh
g++ -m64 -g -fPIC -shared -O2 -fno-omit-frame-pointer -fno-strict-aliasing -Wall
-I/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/include
-I/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/include/linux -Ijavah
-o libaccumulo.so nativeMap/org_apache_accumulo_tserver_NativeMap.cc
Successfully installed native library
[centos@centos accumulo-1.6.0]$

cd server/src/main/c++
make

[centos@centos accumulo-1.6.0]$ ls lib/native/
libaccumulo.so

File Permissions

Accumulo stores primary data in HDFS, including its write-ahead logs. In HDFS the user that runs Accumulo processes must have the ability to read and write to files and directories under the directory specified as instance.dfs.dir in accumulo-site.xml, which defaults to /accumulo. The user must also be able to create this directory, which means writing to the HDFS root directory if the default directory is unchanged. It is recommended to create a directory that the accumulo user has permission to write to (e.g., /user/accumulo), and to make the instance.dfs.dir a subdirectory of this directory (e.g., /user/accumulo/accumulo).

For example:

hdfs dfs -mkdir /user/accumulo/accumulo

Accumulo needs to be able to write application logfiles for debugging and monitoring. If you use the accumulo user to start Accumulo processes, these directories should be writable by the accumulo user.

For a Debian-based system these logs are in the /var/log and /var/lib directories:

sudo mkdir /var/log/accumulo
sudo mkdir /var/lib/accumulo
sudo chown -R accumulo:accumulo /var/log/accumulo
sudo chown -R accumulo:accumulo /var/lib/accumulo

Server Configuration Files

Accumulo ships with some examples based on various memory configurations. To start with these example files, copy the files into Accumulo’s conf directory (such as /etc/accumulo/conf/):

cd /etc/accumulo/conf
sudo cp -r examples/3GB/native-standalone/* .

Accumulo needs to know how to talk to HDFS and ZooKeeper in order to start up. Two files, accumulo-env.sh and accumulo-site.xml, control most of Accumulo’s startup settings. These two files should be copied to each machine that will run Accumulo processes and should be kept in sync if anything changes.

Heterogeneous clusters made of machines with differing hardware can have configuration files that differ in their memory settings, for example. However, properties such as the instance secret and hostnames of ZooKeeper must be the same.

Client Configuration

In addition to inheriting settings from accumulo-site.xml, Accumulo clients can be configured via a Java properties file in the home directory of the user that Accumulo client processes run as. Typically this will be accumulo. This properties file allows each Accumulo client to be configured separately, rather than having them all inherit the same settings from server configuration files or settings stored in ZooKeeper.

The properties file is located at $HOME/.accumulo/config. Each line of the properties file can specify one property and one value, separated by a tab character. Lines beginning with an octothorp (#) are ignored.

For example:

# turn on SSL
instance.rpc.ssl.enabled true

We’ll look at an example using this configuration file when we discuss SSL in “Configuring SSL”.

Deploying JARs

Accumulo processes are started from JAR files distributed as part of the installation. These live in $ACCUMULO_HOME/lib.

In addition, any Java classes that customize Accumulo, such as iterators and constraints must be distributed to servers across the cluster so that tablet servers and clients can load them.

There are several ways to get the Accumulo JARs and custom JARs onto servers.

[user@host accumulo-examples]$ mvn compile package
...
[INFO]
[INFO] --- maven-jar-plugin:2.3.2:jar (default-jar) @ accumulo-examples ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.954s
[INFO] Finished at: Mon Sep 01 13:49:33 EDT 2014
[INFO] Final Memory: 24M/310M
[INFO] ------------------------------------------------------------------------

cp accumulo-examples-0.0.1-SNAPSHOT.jar lib/ext/

[centos@centos accumulo-1.6.0]$ bin/accumulo shell -u root
Password: ******

Shell - Apache Accumulo Interactive Shell
-
- version: 1.6.0
- instance name: test
-
- type 'help' for a list of available commands
-
root@test test> createtable test
root@test test> constraint -a -t test 
          com.accumulobook.advanced.ValidHeightWeightConstraint
Added constraint com.accumulobook.advanced.ValidHeightWeightConstraint
          to table test with number 2
root@test test>

<property>
  <name>general.vfs.classpaths</name>
  <value></value>
</property>

[centos@centos ~]$ hdfs dfs -mkdir /accumulo/ext
[centos@centos ~]$ hdfs dfs -put accumulo-examples-0.0.1-SNAPSHOT.jar 
                   /accumulo/ext/

<property>
  <name>general.vfs.classpaths</name>
  <value>hdfs://centos:8020/accumulo/ext/.*.jar</value>
</property>

root@test> constraint -a -t test 
    com.accumulobook.advanced.ValidHeightWeightConstraint
Added constraint com.accumulobook.advanced.ValidHeightWeightConstraint to table
    test with number 2
root@test>

Setting Up Automatic Failover

Essential to running a large cluster is Accumulo’s ability to tolerate certain types of failure. When certain processes fail, their workload is automatically reassigned to remaining worker nodes or backup processes on other machines. In general, setting up automatic failover for Accumulo processes is simply a matter of running an instance of a process on more than one server.

Tablet servers

The master process ensures that any tablets that were being served by a failed machine are reassigned to remaining tablet servers, which perform any recovery necessary by reading from write-ahead logs. For more on this process see “Recovery”.

Masters

A master process must be running in order for tablet server failover to happen. If no master is running, most client operations can proceed, but if any tablet server fails while the master is down, some tablets will be unavailable until a master process is started.

To avoid a situation in which tablets can become unavailable, multiple masters processes can be run to ensure that at least one master is running at all times. For this purpose, the $ACCUMULO_HOME/conf/masters file should contain the hostnames of the machines on which master processes are run.

The master processes use ZooKeeper to coordinate electing an active master and to elect a new active master in the event that the active master fails.

Garbage collectors

The garbage collector process is not critical to client operations but must run to ensure aged-off, deleted, and redundant data is removed from HDFS.

Initialization

Before Accumulo is started for the first time, an Accumulo instance must be initialized. This can be done on a machine that can connect to both ZooKeeper and HDFS, via the command:

accumulo init

This command should be run under the user account under which later Accumulo processes will be run, such as the accumulo user.

Be sure to verify that the init script is using the correct values for ZooKeeper servers and the Hadoop filesystem:

INFO: Hadoop Filesystem is hdfs://[your-namenode]:8020
INFO: ZooKeeper server is [your-zookeeper]:2181

If an error occurs, such as “java.io.IOException: No FileSystem for scheme: hdfs,” check that the path to the Hadoop HDFS JARs are included in the general.classpaths setting in the accumulo-site.xml file.

This script will create an entry in ZooKeeper that will be used to coordinate all configuration information for this Accumulo instance. In addition, the script will create a directory called /accumulo in HDFS, in which all table data will be stored.

If the accumulo user cannot write to the root directory of HDFS, an error will be thrown. Ensure that the accumulo user can create the /accumulo directory in HDFS, or create it beforehand and grant ownership to accumulo. Alternatively, configure Accumulo to use a different directory that the accumulo user has permission to write to by changing the instance.dfs.dir property in accumulo-site.xml as described in “File Permissions”.

After initialization, there will be three tables in Accumulo: accumulo.root, accumulo.metadata, and the trace table. See Chapter 10 for more information on how Accumulo makes use of those tables.

hadoop fs -rmr /accumulo

hadoop fs -mv /accumulo /new path

Networking

As a distributed application, Accumulo relies heavily on the network that connects servers to one another. Like Apache Hadoop, Accumulo does not require exotic networking hardware, and it is designed to operate well on commodity-class networking components such as Gigabit Ethernet and 10 Gigabit Ethernet. Modern Hadoop configuration recommendations include considering 10 Gigabit Ethernet for reduced latency.

Limits

The largest clusters begin to be bottlenecked not by any component of Accumulo but by the underlying subsystems on which it runs. In particular, a single HDFS NameNode becomes a bottleneck in terms of the number of update operations that the entire cluster can perform over time. This limit can be observed by looking at the time that the Accumulo garbage collector takes to complete one pass. If the garbage collector is taking over five minutes to run, the NameNode is likely a bottleneck.

Accumulo 1.6 introduces the ability to run Accumulo over multiple NameNodes to overcome this limitation. See “Using Multiple HDFS Volumes”.

Metadata Table

Accumulo’s metadata table (called accumulo.metadata in Accumulo 1.6) is a special table designed to store the current location and other information about each tablet of every other table. As such it plays an important role in the operation of every Accumulo application.

By default, the metadata table is configured to be scalable and to provide good performance for even large clusters. Understanding how the configuration of the metadata table affects performance and scalability can help you fine-tune your cluster. In large Accumulo clusters the metadata table can be split and hosted by additional tablet servers in order to scale with the number of clients performing metadata lookups.

The default configuration for the metadata table is different from that of other tables. It is tuned for high performance and availability, and to take advantage of its relatively small size.

To improve query performance, the size of compressed file blocks is reduced from 100 K to 32 K, the data block cache is enabled so that the frequent reads by clients are serviced very quickly from data cached in memory, and the major compaction ratio is set to 1.

To increase availability and decrease the possibility of data loss, the file replication is increased to 5 (or the maximum replication defined in HDFS if that is less than 5, or the minimum replication defined in HDFS if that is greater than 5).

The split threshold is decreased from 1 GB to 64 MB because the metadata table is so much smaller than data tables, and because we want the metadata tablets spread onto multiple tablet servers for better read and write throughput.

Two locality groups are configured, so that columns that are accessed together frequently can be read more efficiently.

The metadata table’s design is not a limiting factor in the scalability of Accumulo. Going by the following simple calculation in the Bigtable paper, the metadata table can address more data than can be stored in HDFS:

Each METADATA row stores approximately 1KB of data in memory. With a modest limit of 128 MB METADATA tablets, our three-level location scheme is sufficient to address 2³⁴ tablets (or 2⁶¹ bytes in 128 MB tablets).

Tablet Sizing

Having fewer, larger tablets can reduce the overhead of managing a large-scale cluster. This can be achieved by increasing the split threshold for splitting one tablet into two. Tablets that are tens of gigabytes in size are not unreasonable.

To increase the tablet split threshold, change the table.split.threshold in the shell:

user@accumulo myTable> config -t myTable -s table.split.threshold=20GB

File Sizing

For the same reason that larger tablet sizes can reduce overhead, it can be useful to increase the block size in HDFS to a value closer to the size of tablets. This reduces the amount of information the NameNode has to manage for each file, allowing the NameNode to manage more overall files.

To increase the block size for a table, set table.file.blocksize:

user@accumulo> config -t mytable -s table.file.blocksize=1GB

Using Multiple HDFS Volumes

Accumulo version 1.6 and later can store files using multiple HDFS volumes, potentially on multiple NameNodes. There are several options for doing this. One is to configure Accumulo to run over two or more separate HDFS instances, each with a NameNode and a set of DataNodes, and DataNodes each store data for only one NameNode (Figure 11-10). In this case, DataNodes are not shared between NameNodes and they operate without any knowledge of one another. Accumulo simply keeps track of which files live in which HDFS instance.

Accumulo does this by using full path names to all the files under management, including the hostname of the NameNode of the HDFS cluster in which each file lives. After Accumulo is informed of the list of NameNodes to use in the configuration file, no other configuration is necessary. Accumulo will automatically distribute files across HDFS clusters evenly.

Figure 11-10. Accumulo on multiple HDFS clusters

Another option is to use NameNode federation, in which a set of DataNodes are shared between two or more NameNodes (Figure 11-11). Federation can make it easier to keep the data in HDFS balanced because each NameNode can see the information on every DataNode and place data based on the load of all the DataNodes.

Figure 11-11. Accumulo on an HDFS cluster using NameNode federation

In either of these cases, the configuration of Accumulo is the same. A list of NameNodes to use is specified in the accumulo-site.xml file under the instance.volumes property:

 <property>
    <name>instance.volumes</name>
    <value>hdfs://namenode1:9001/accumulo,hdfs://namenode2:9001/accumulo</value>
 </property>

Accumulo instances that utilize multiple NameNodes are capable of scaling to extremely large sizes, beyond a few thousand nodes to ten thousand or more. With modern hard drives each server could have up to 30 TB raw, 10 TB after replication, and a cluster of 10,000 servers could store up to 100 PB. Accumulo provides a single unified view of all of this data, and lookups remain fast because of the ordering of the keys in each table.

 <property>
    <name>instance.volumes.replacements</name>
    <value>hdfs://namenodeA:9001/accumulo hdfs://namenode1:9001/accumulo</value>
 </property>

 <property>
   <name>instance.volumes.replacements</name>
   <value>hdfs://namenodeA:9001/accumulo hdfs://namenode1:9001/accumulo,
          hdfs://namenodeB:9001/accumulo hdfs://namenode2:9001/accumulo</value>
 </property>

Column Visibilities and Accumulo Clients

Accumulo will authenticate a user according to the user’s credentials (such as a password), and authorize that user to read data according to the column visibilities present within that data and the authorizations granted to the user. All other means of accessing Accumulo table data must be restricted.

Supporting Software Security

Because Accumulo stores data in HDFS, access to these files must be restricted. This includes access to both the RFiles, which store long-term data, and Accumulo’s write-ahead logs, which store recently written data. Accumulo should be the only application allowed to access these files in HDFS.

Similarly, HDFS stores blocks of files in an underlying Linux filesystem. Users who have access to blocks of HDFS data stored in the Linux filesystem would also bypass data-level protections. Access to the file directories on which HDFS data is stored should be limited to the HDFS daemon user.

Unnecessary services should be turned off.

The accumulo-site.xml file should not be readable except by the accumulo user, because it contains the instance secret and the trace user’s password. A separate conf/ directory with files readable by other users can be created for client use, with an accumulo-site.xml file that does not contain those two properties.

Network Security

IPTables or other firewall implementations can be used to help restrict access to TCP ports.

Accumulo uses the port numbers listed in Table 11-2 by default. These should be reachable by Accumulo clients as well as by one another.

Accumulo tablet servers must be able to communicate with HDFS DataNodes and the NameNode.

Only trusted client applications should be allowed to connect to ZooKeeper and Accumulo tablet servers.

 instance.rpc.ssl.clientAuth true
 instance.rpc.ssl.enabled true
 rpc.javax.net.ssl.keyStore path_to_keystore
 rpc.javax.net.ssl.keyStorePassword keystore_password

 rpc.javax.net.ssl.trustStore path_to_truststore
 rpc.javax.net.ssl.trustStorePassword truststore_password

monitor.ssl.keyStore
monitor.ssl.keyStorePassword
monitor.ssl.trustStore
monitor.ssl.trustStorePassword

[centos@centos ~]$ openssl genrsa -des3 -out root.key 4096
Generating RSA private key, 4096 bit long modulus
...............................++
..............................................................................
      ......++
e is 65537 (0x10001)
Enter pass phrase for root.key:
Verifying - Enter pass phrase for root.key:

[centos@centos ~]$ ls
root.key

[centos@centos ~]$ openssl req -x509 -new -key root.key  -days 365 
                         -out root.pem
Enter pass phrase for root.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:DC
Locality Name (eg, city) [Default City]:Washington
Organization Name (eg, company) [Default Company Ltd]:Accumulo Corp
Organizational Unit Name (eg, section) []:Developers
Common Name (eg, your name or your server's hostname) []:Cert Auth
Email Address []:[email protected]
[centos@centos ~]$ ls
root.key  root.pem

[centos@centos ~]$ openssl x509 -outform der -in root.pem -out root.der

[centos@centos ~]$ keytool -import -alias root-key -keystore truststore.jks 
                         -file root.der
Enter keystore password:
Re-enter new password:
Owner: [email protected], CN=Cert Auth, OU=Developers,
          O=Accumulo Corp, L=Seattle, ST=Washington, C=US
Issuer: [email protected], CN=Cert Auth, OU=Developers,
          O=Accumulo Corp, L=Seattle, ST=Washington, C=US
Serial number: abd68bf897fcf631
Valid from: Sun Sep 14 15:57:19 GMT-05:00 2014 until: Mon Sep 14 15:57:19
          GMT-05:00 2015
...
Trust this certificate? [no]:  yes
Certificate was added to keystore
[centos@centos ~]$ ls
root.der  root.key  root.pem  truststore.jks

[centos@centos ssl]$ openssl genrsa -out server.key 4096
Generating RSA private key, 4096 bit long modulus
..............................................................................
      ..........................................................................
      ..................++
...........................................................................++
e is 65537 (0x10001)

[centos@centos ssl]$ openssl req -new -key server.key -out server.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[centos@centos ssl]$ ls
root.key  root.pem  server.csr  server.key  truststore.jks

[centos@centos ssl]$ openssl x509 -req -in server.csr -CA root.pem 
                       -CAkey root.key -CAcreateserial -out server.pem -days 365
Signature ok
subject=/C=XX/L=Default City/O=Default Company Ltd
Getting CA Private Key
Enter pass phrase for root.key:
[centos@centos ssl]$
[centos@centos ssl]$ ls
root.key  root.pem  root.srl  server.csr  server.key  server.pem  truststore.jks

[centos@centos ssl]$ openssl pkcs12 -export -in server.pem -inkey server.key 
                       -certfile server.pem -name 'server-key' -out server.p12
Enter Export Password:
Verifying - Enter Export Password:
[centos@centos ssl]$ ls
root.key  root.pem  root.srl  server.csr  server.key  server.p12  server.pem
      truststore.jks

[centos@centos ssl]$ keytool -importkeystore -srckeystore server.p12 
                       -srcstoretype pkcs12 -destkeystore server.jks 
                       -deststoretype JKS
Enter destination keystore password:
Re-enter new password:
Enter source keystore password:
Entry for alias server-key successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed
          or cancelled

[centos@centos ssl]$ rm server.p12

Encryption of Data at Rest

Accumulo controls access to data for client programs that are configured to pass user authorizations as part of scan operations. Cryptography can be used to secure data from those with physical access to storage components in which data is stored.

As of Accumulo version 1.6.0, data stored on disk can be encrypted via pluggable modules, with the exception of data stored in HDFS as part of the tablet server recovery process. An example implementation ships with Accumulo, although it stores the master key for all encryption keys in HDFS along with encrypted files.

To configure Accumulo to use encryption when storing files, add the following properties to accumulo-site.xml:

<property>
  <name>crypto.module.class</name>
  <value>org.apache.accumulo.core.security.crypto.DefaultCryptoModule</value>
</property>

<property>
  <name>crypto.cipher.suite</name>
  <value>AES/CFB/NoPadding</value>
</property>

<property>
  <name>crypto.cipher.algorithm.name</name>
  <value>AES</value>
</property>

<property>
  <name>crypto.cipher.key.length</name>
  <value>128</value>
</property>

<property>
  <name>crypto.secure.rng</name>
  <value>SHA1PRNG</value>
</property>

<property>
  <name>crypto.secure.rng.provider</name>
  <value>SUN</value>
</property>

<property>
  <name>crypto.secret.key.encryption.strategy.class</name>
  <value>org.apache.accumulo.core.security.crypto.
               CachingHDFSSecretKeyEncryptionStrategy</value>
</property>

<property>
    <name>crypto.default.key.strategy.cipher.suite</name>
    <value>AES/ECB/NoPadding</value>
</property>

Kerberized Hadoop

Apache Hadoop can be deployed using Kerberos to control the processes that are allowed to participate in the cluster. To use Accumulo with a kerberized HDFS instance, you must create an Accumulo principal:

kadmin.local -q "addprinc -randkey accumulo/[hostname]"

Principals can then be exported to a keytab file. There can be a separate keytab file for each server, or all principals can be globbed into a single keytab file as follows:

kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"

The keytab file for a server must be stored locally on that server, owned by the accumulo user with file permissions 400. A suggested location for the keytab file is the $ACCUMULO_HOME/conf directory. The absolute local path to the the keytab file on each server must be specified in the accumulo-site.xml file, as well as the principal. The placeholder _HOST can be used for the hostname, but the realm must be specified:

<property>
  <name>general.kerberos.keytab</name>
  <value>$ACCUMULO_CONF_DIR/accumulo.keytab</value>
</property>

<property>
  <name>general.kerberos.principal</name>
  <value>accumulo/_HOST@[realm]</value>
</property>

Application Permissions

Accumulo has the concept of a user permission, but more often these are associated with a particular application that may provide access to multiple users. Accumulo clients can do their own authentication of multiple users and also look up any associated authorization tokens, which they then faithfully pass to Accumulo tablet servers when doing scans.

Before any user can read any data, however, an account must be created, authorization tokens assigned, and access to tables granted. Administrators can work with application developers to determine the right level of access for the account and how to determine the set of authorization tokens to grant to the account.

To create an account in the shell, run the createuser command:

root@accumulo> createuser myapp
Enter new password for 'myapp': *****
Please confirm new password for 'myapp': *****

To allow this account to read a particular table, run:

root@accumulo> grant Table.READ -t mytable -u myapp
root@accumulo> System permissions:

Table permissions (!METADATA): Table.READ
Table permissions (mytable): Table.READ

To grant authorizations to an account, run:

root@accumulo> setauths -u myapp -s myauth
root@accumulo> getauths -u myapp
myauth

To see what permissions a user account has, run:

root@accumulo> userpermissions -u myapp

Once this has been done, an Accumulo client identified by the myapp account can connect to Accumulo, passing in the password specified, and perform scans against the mytable table and pass in the myauth authorization token . If a client tries to read from another table, or tries to write to mytable, or tries to pass in a different authorization token, it will receive an Authorization exception.

A list of available permissions can be seen via the systempermissions and tablepermissions commands.

See “Permissions” for more on permissions.