Setting up a Fully Distributed HBase Cluster

Wrote by Post Comment

This post will discuss on how to setup a fully distributed hbase cluster. Here we will not run zookeeper as a separate server, but will be using the zookeeper which is embedded in hbase itself. And our setup will consist of 1 master node, and 2 slave nodes.



Prerequisites

  • Update /etc/hostname file to include hadoop.master, hadoop.slave1, hadoop.slave2 respectively, as the hostnames of the machines.
  • Download hbase 1.2.1
  • Setup a fully distributed Hadoop cluster [1].
  • Start the hadoop server.

Configure HBase

Fisrt create a directory for hbase in the hadoop file system. For that navigate to <hadoop_home>/bin and execute:
hadoop fs -mkdir /hbase

Do the following confgiurations in <hbase_home>/conf/hbase-site.xml. Note that, here the host and port of "hbase.rootdir" should be the same host and port as hadoop's fs.default.name, we gave at the prerequisites step.

<property>
    <name>hbase.rootdir</name>
    <value>hdfs://hadoop.master:9000/hbase</value>
</property>
<!-- Note that above should be the same host and port as hadoop's fs.default.name-->

<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2222</value>
    <description>Property from ZooKeeper's config zoo.cfg.The port at which the clients will connect.</description>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>hadoop.slave1,hadoop.slave2</value>
</property>
<property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/home/wso2/Desktop/hbase/localDirs/zookeeper-dataDir</value>
</property>


Here, hbase.rootdir should be on namenode. In our case, master is the namenode. Zookeeper quorums should be the slave nodes. This tells which nodes should run the zookeerper. It is preferred to have odd number of nodes for zookeeper.

Add the hostname of slave nodes, to  <hbase_home>/conf/regionservers, in all nodes, except master/namenode
hadoop.slave1
hadoop.slave2

Finally, set the following jvm properties in <hbase_home>/conf/hbase-env.sh file. The property HBASE_MANAGES_ZK  is to indicate that hbase is managing the zookeeper, and no external zookeeper server is running.

# To use built in zookeeper
export HBASE_MANAGES_ZK=true

# set java class path
export JAVA_HOME=your_java_home

# Add hadoop-conf directory to hbase class path:
export HBASE_CLASSPATH=$HBASE_CLASSPATH:<hadoop_home>/etc/hadoop

Now all the configurations are complete. Now we can start the server by navigating to <hbase_home>/bin directory and executing the following:
./start-hbase.sh

Once the hbase server is up, you can navigate to its master web UI from http://hadoop.master:16010/

References

[1] http://supunsetunga.blogspot.com/2016/08/setting-up-fully-distributed-hadoop.html

Tags:

Share:

0 comments