[hbase]-로컬 환경 - Failed to become active master 해결

테스트 용도로 hbase 1.x.x(hbase 1.0, hbase 1.2, hbase1.2.2)를 설치하고, hbase.rootdir를 로컬 디렉토리로 정하고(로컬 디렉토리로 정하지 않아도 에러가 발생했다), hbase-site.xml에 정의한 후, standalone으로 실행할 때, 다음과 같은 에러가 발생할 수 있다. 

2016-08-02 20:49:27,597 FATAL [Samuelui-MacBook-Pro:59298.activeMasterManager] master.HMaster: Failed to become active master

java.net.ConnectException: Call From Samuelui-MacBook-Pro.local/1.1.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)

at org.apache.hadoop.ipc.Client.call(Client.java:1415)

at org.apache.hadoop.ipc.Client.call(Client.java:1364)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

at com.sun.proxy.$Proxy16.setSafeMode(Unknown Source)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at com.sun.proxy.$Proxy16.setSafeMode(Unknown Source)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:602)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)

at com.sun.proxy.$Proxy17.setSafeMode(Unknown Source)

at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2264)

at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:986)

at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:970)

at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:525)

at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:971)

at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:424)

at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:153)

at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)

at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:638)

at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:184)

at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1729)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)

at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)

at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)

at org.apache.hadoop.ipc.Client.call(Client.java:1382)

... 29 more

2016-08-03 10:53:19,227 DEBUG [main-SendThread(fe80:0:0:0:0:0:0:1%1:2181)] zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown input

java.net.SocketException: Socket is not connected

        at sun.nio.ch.Net.translateToSocketException(Net.java:145)

        at sun.nio.ch.Net.translateException(Net.java:179)

        at sun.nio.ch.Net.translateException(Net.java:185)

        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:404)

        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200)

        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)

Caused by: java.nio.channels.NotYetConnectedException

        at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:782)

        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)

        ... 3 more

2016-08-02 16:07:11,339 WARN  [RS:0;172.26.116.174:50808-SendThread(fe80:0:0:0:0:0:0:1%1:2181)] zookeeper.ClientCnxn: Session 0x1564a13ad290003 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

에러가 발생하는 코드는 다음과 같다.

https://github.com/apache/hbase/blob/branch-1.0/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java

        MonitoredTask status = TaskMonitor.get().createStatus("Master startup");

        status.setDescription("Master startup");

        try {

          if (activeMasterManager.blockUntilBecomingActiveMaster(timeout, status)) {

            finishActiveMasterInitialization(status);

          }

        } catch (Throwable t) {

          status.setStatus("Failed to become active: " + t.getMessage());

          LOG.fatal("Failed to become active master", t);

          // HBASE-5680: Likely hadoop23 vs hadoop 20.x/1.x incompatibility

          if (t instanceof NoClassDefFoundError &&

              t.getMessage().contains("org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction")) {

            // improved error message for this special case

            abort("HBase is having a problem with its Hadoop jars.  You may need to "

              + "recompile HBase against Hadoop version "

              +  org.apache.hadoop.util.VersionInfo.getVersion()

              + " or change your hadoop jars to start properly", t);

          } else {

            abort("Unhandled exception. Starting shutdown.", t);

          }

        } finally {

          status.cleanup();

        }

문제를 해결하기 위해 네트워크를 확인했다.

1) ssh 확인

22 번 포트로 ssh 키로 잘 접속되는지 확인.

ssh localhost 

2) 방화벽 확인

맥의 Perference의 Sharing icon를 선택해서 방화벽을 오픈되는지 확인

telnet localhost 123123

3) hdfs 또는 file 확인

hbase-site.xml 파일에서 다음으로 작성. 로그 상 9000번 포트로 접근하는 걸 봐서는 hdfs로 접근하는 것 같아서, file:///을 사용해서 변경하니 잘 작동했다.

 <property>

    <name>hbase.rootdir</name>

    <value>/tmp/hbase/data</value>

  </property>

        

==> 

 <property>

    <name>hbase.rootdir</name>

    <value>file:///tmp/hbase/data</value>

  </property>