hbase集羣的啓動,注意幾個問題

1.hbase的改的會影響器他的組件的使用, 

  故而, 在修改 hadoop的任何組件後, 必定要記得其它的組件也能受到影響,  

一下是我在將hadoop的集羣改了以後 , 再次運行hbase的時候, 就會發生異常, 緣由是在鏈接namenode的時候,發生鏈接不到, 就是由於我改了以前的配置,將端口號改了,

沒有去將配置文件在hbase中進行更新, 具體問以下:

##我今天起得hbase的集羣時候,發生了

1.http://node3:60010/master-status 

HTTP ERROR 503

Problem accessing /master-status. Reason:

    Master not ready

 

---

Powered by Jetty://

2.http://node3:60030/rs-status

The RegionServer is initializing!

 

在hbase-hadoop-master-node3.log中, 報異常是:

2016-10-12 09:45:04,616 DEBUG [main-EventThread] master.ActiveMasterManager: A master is now available

2016-10-12 09:59:10,484 WARN [master:node3:60000] retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode. Not retrying because failovers (15) exceeded maximum allowed (15)
java.net.ConnectException: Call From node3/192.168.8.13 to node2:8020 failed on connection exception: java.net.ConnectException: 拒絕鏈接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy12.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:561)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2146)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:983)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:426)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:851)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:435)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:790)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:607)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: 拒絕鏈接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
... 25 more
2016-10-12 10:02:04,280 WARN [master:node3:60000] retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode. Not retrying because failovers (15) exceeded maximum allowed (15)
java.net.ConnectException: Call From node3/192.168.8.13 to node1:8020 failed on connection exception: java.net.ConnectException: 拒絕鏈接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy12.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:561)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2146)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:983)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:967)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:432)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:851)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:435)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:790)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:607)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: 拒絕鏈接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
... 21 more
2016-10-12 10:02:04,282 FATAL [master:node3:60000] master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call From node3/192.168.8.13 to node1:8020 failed on connection exception: java.net.ConnectException: 拒絕鏈接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy12.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:561)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2146)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:983)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:967)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:432)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:851)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:435)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:790)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:607)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: 拒絕鏈接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
... 21 more

2016-10-12 10:02:04,352 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:192)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2799)

 

而:hbase-hadoop-regionserver-node3.log中沒有錯誤, 可是有以下:

016-10-12 09:55:51,150 INFO [main] regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver60020
2016-10-12 10:00:45,947 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:05:45,947 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:10:45,948 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:15:45,948 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:20:45,948 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:25:45,981 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2016-10-12 10:30:45,947 DEBUG [LruStats #0] hfile.LruBlockCache: Total=3.26 MB, free=393.42 MB, max=396.68 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN

 

這是爲啥, 我找了好久 , 記得之前是以下緣由引發的, 

1.經過hostname設置爲regionserver端連接時指定的hostname便可 , 這個就是域名和主機不一致致使的,  (hbase分佈式部署問題之org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master) 

--->很快我就排除了, 由於個人hadoop的集羣啓動正常,故這個排除

2.時間不一致引發的,相似這我的提到(時間同步上有問題，查了一下RegionServer和Master兩臺機的時間，果真不一致。因爲以前把主節點的時間同步而未同步子節點的時間 ,)

我在各臺機子上都就行查看,用:date;hwclock -r 進行查看 , 沒有問題,(如何 一步到位修改linux系統時間  date -s "20091112 18:30:50" &&hwclock --systohc)

3.hbase 未成功啓動  一、關閉防火牆：service iptables stop  ,我有排除了, 緣由是個人hadoop集羣是啓動的, 沒有這個問題

 

最後, 我無奈去繼續找, 去看看我以前配置hbase的配置, 是我以前有其餘問題, 由於我早一個多月前試了試hbase,是沒有問題,還有簡單編程的嘗試的,因而

 

查看搭建：分佈式的hbase 的文檔, 

1.上傳hbase安裝包

2.解壓

3.配置hbase集羣，要修改3個文件（首先zk集羣已經安裝好了）

         注意：要把hadoop的hdfs-site.xml和core-site.xml 放到hbase/conf下

3.1修改hbase-env.sh

         export JAVA_HOME=/usr/java/jdk1.7.0_55

         //告訴hbase使用外部的zk

         export HBASE_MANAGES_ZK=false

        

         vim hbase-site.xml

         <configuration>

<!-- 指定hbase在HDFS上存儲的路徑,其中：ns1是在hadoop的core-site.xml中配置的 -->

        <property>

                <name>hbase.rootdir</name>

                <value>hdfs://ns1/hbase</value>

        </property>

                   <!-- 指定hbase是分佈式的 -->

        <property>

                <name>hbase.cluster.distributed</name>

                <value>true</value>

        </property>

                   <!-- 指定zk的地址，多個用「,」分割 -->

        <property>

                <name>hbase.zookeeper.quorum</name>

                <value>itcast04:2181,itcast05:2181,itcast06:2181</value>

        </property>

                  

         </configuration>

        

         vim regionservers

         itcast03

         itcast04

         itcast05

         itcast06

        

         3.2拷貝hbase到其餘節點

                   scp -r /itcast/hbase-0.96.2-hadoop2/ itcast02:/itcast/

                   scp -r /itcast/hbase-0.96.2-hadoop2/ itcast03:/itcast/

                   scp -r /itcast/hbase-0.96.2-hadoop2/ itcast04:/itcast/

                   scp -r /itcast/hbase-0.96.2-hadoop2/ itcast05:/itcast/

                   scp -r /itcast/hbase-0.96.2-hadoop2/ itcast06:/itcast/

4.將配置好的HBase拷貝到每個節點並同步時間。

 

5.啓動全部的hbase

         分別啓動zk

                   ./zkServer.sh start

         啓動hbase集羣

                   start-dfs.sh

         啓動hbase，在主節點上運行：

                   start-hbase.sh

6.經過瀏覽器訪問hbase管理頁面

         192.168.1.201:60010

7.爲保證集羣的可靠性，要啓動多個HMaster

         hbase-daemon.sh  start master

 

到第三部的時候,我就發現了問題, 爲啥, 緣由是我以前改過那個hadoop的dfs.namenode.rpc-address.myserver.nn1的端口 <value>node1:8020</value>, 

改成了

<property>
<name>dfs.namenode.rpc-address.myserver.nn2</name>
<value>node2:9000</value>
</property>

 

,這時候, 

Call From node3/192.168.8.13 to node1:8020 failed on connection exception: java.net.ConnectException: 拒絕鏈接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

 

故,問題就在這裏

 

配置問題的問題