一次HBase問題的解決過程（Status: INCONSISTENT）
==版本信息==
HBase：2.7.1css
Storm：1.0.1html
RocketMQ：3.4.6（阿里版）java
==問題描述==
2018年9月3號晚上23點左右，例行巡檢系統運行情況時，node
發現Storm消費RocketMQ入HBase的過程當中，RocketMQ的TPS極低，甚至爲0了。web
（Storm從RocketMQ消費消息自後，讀寫HBase）shell
==問題定位==
查看了Storm的日誌，發現有以下異常信息：apache
2018-09-03 20:29:43.696 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] java.net.SocketTimeoutException: callTimeout=1200000, callDuration=1536845: row '97390002332830' on table 'DBN_ANE' at region=DBN_ANE,97,1507795766296.69724c5bd4e82bae4a3ec8a95c8f9c65., hostname=dscn33,16020,1529215379398, seqNum=17296637 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:862) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:828) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at com.dscn.topology.BasicBolt.execute(BasicBolt.java:125) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$fn__5030$tuple_action_fn__5032.invoke(executor.clj:729) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$mk_task_receiver$fn__4951.invoke(executor.clj:461) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.disruptor$clojure_handler$reify__4465.onEvent(disruptor.clj:40) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:482) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:460) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$fn__5030$fn__5043$fn__5096.invoke(executor.clj:848) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at clojure.lang.AFn.run(AFn.java:22) 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at java.lang.Thread.run(Thread.java:745) 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] Caused by: java.io.IOException: HRegionInfo was null in DBN_ANE, row=keyvalues={DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:seqnumDuringOpen/1535968873207/Put/vlen=8/seqid=0, DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:server/1535968873207/Put/vlen=12/seqid=0, DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:serverstartcode/1535968873207/Put/vlen=8/seqid=0} 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289) 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181) 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152) 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1136) 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957) 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83) 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75) 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] ... 14 more
而後查看HBase集羣日誌，裏面也出現錯誤，日誌以下：json
2018-09-03 17:58:57,603 WARN [MemStoreFlusher.0] regionserver.HStore: Failed flushing store file, retrying num=0 java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2018-09-03 17:58:57,606 INFO [regionserver/dscn33/10.11.2.31:16020.logRoller] wal.FSHLog: Slow sync cost: 16261 ms, current pipeline: [10.11.2.31:50010] 2018-09-03 17:58:57,606 ERROR [regionserver/dscn33/10.11.2.31:16020.logRoller] wal.FSHLog: Failed close of WAL writer hdfs://ns/hbase/WALs/dscn33,16020,1529215379398/dscn33%2C16020%2C1529215379398.default.1535968646203, unflushedEntries=30 org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1615) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:833) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:699) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ... 1 more Caused by: java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
2018-09-03 17:58:41,358 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region DBN_JBD,30270769371,1515408107652.fca3d8ee774898302a377535e3f0ccac. org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
檢查HBase集羣狀態（hbase hbck），發現異常，定位爲HBase集羣問題：api
[root@dscn12 ~]# hbase hbck SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadmin/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] HBaseFsck command line options: 2018-09-04 00:01:54,217 INFO [main] util.HBaseFsck: Launching hbck 2018-09-04 00:01:54,456 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x57576994 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181 2018-09-04 00:01:54,471 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2018-09-04 00:01:54,471 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=dscn12 2018-09-04 00:01:54,471 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_73 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_73/jre 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/home/hadmin/hbase/conf:/usr/java/default/lib/tools.jar:/home/hadmin/hbase:/home/hadmin/hbase/lib/activation-1.1.jar:/home/hadmin/hbase/lib/antisamy-1.4.3.jar:/home/hadmin/hbase/lib/aopalliance-1.0.jar:/home/hadmin/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadmin/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadmin/hbase/lib/api-asn1-api-1.0.0-M20.jar:/home/hadmin/hbase/lib/api-util-1.0.0-M20.jar:/home/hadmin/hbase/lib/asm-3.1.jar:/home/hadmin/hbase/lib/avro-1.7.4.jar:/home/hadmin/hbase/lib/batik-css-1.7.jar:/home/hadmin/hbase/lib/batik-ext-1.7.jar:/home/hadmin/hbase/lib/batik-util-1.7.jar:/home/hadmin/hbase/lib/bsh-core-2.0b4.jar:/home/hadmin/hbase/lib/commons-beanutils-1.7.0.jar:/home/hadmin/hbase/lib/commons-beanutils-core-1.7.0.jar:/home/hadmin/hbase/lib/commons-cli-1.2.jar:/home/hadmin/hbase/lib/commons-codec-1.9.jar:/home/hadmin/hbase/lib/commons-collections-3.2.2.jar:/home/hadmin/hbase/lib/commons-compress-1.4.1.jar:/home/hadmin/hbase/lib/commons-configuration-1.6.jar:/home/hadmin/hbase/lib/commons-daemon-1.0.13.jar:/home/hadmin/hbase/lib/commons-digester-1.8.jar:/home/hadmin/hbase/lib/commons-el-1.0.jar:/home/hadmin/hbase/lib/commons-fileupload-1.2.jar:/home/hadmin/hbase/lib/commons-httpclient-3.1.jar:/home/hadmin/hbase/lib/commons-io-2.4.jar:/home/hadmin/hbase/lib/commons-lang-2.6.jar:/home/hadmin/hbase/lib/commons-logging-1.2.jar:/home/hadmin/hbase/lib/commons-math-2.2.jar:/home/hadmin/hbase/lib/commons-math3-3.1.1.jar:/home/hadmin/hbase/lib/commons-net-3.1.jar:/home/hadmin/hbase/lib/disruptor-3.3.0.jar:/home/hadmin/hbase/lib/esapi-2.1.0.jar:/home/hadmin/hbase/lib/findbugs-annotations-1.3.9-1.jar:/home/hadmin/hbase/lib/guava-12.0.1.jar:/home/hadmin/hbase/lib/guice-3.0.jar:/home/hadmin/hbase/lib/guice-servlet-3.0.jar:/home/hadmin/hbase/lib/hadoop-annotations-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-auth-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-client-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-hdfs-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-api-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-client-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-server-common-2.5.1.jar:/home/hadmin/hbase/lib/hbase-annotations-1.2.1.jar:/home/hadmin/hbase/lib/hbase-annotations-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-client-1.2.1.jar:/home/hadmin/hbase/lib/hbase-common-1.2.1.jar:/home/hadmin/hbase/lib/hbase-common-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-examples-1.2.1.jar:/home/hadmin/hbase/lib/hbase-external-blockcache-1.2.1.jar:/home/hadmin/hbase/lib/hbase-hadoop2-compat-1.2.1.jar:/home/hadmin/hbase/lib/hbase-hadoop-compat-1.2.1.jar:/home/hadmin/hbase/lib/hbase-it-1.2.1.jar:/home/hadmin/hbase/lib/hbase-it-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-prefix-tree-1.2.1.jar:/home/hadmin/hbase/lib/hbase-procedure-1.2.1.jar:/home/hadmin/hbase/lib/hbase-protocol-1.2.1.jar:/home/hadmin/hbase/lib/hbase-resource-bundle-1.2.1.jar:/home/hadmin/hbase/lib/hbase-rest-1.2.1.jar:/home/hadmin/hbase/lib/hbase-server-1.2.1.jar:/home/hadmin/hbase/lib/hbase-server-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-shell-1.2.1.jar:/home/hadmin/hbase/lib/hbase-thrift-1.2.1.jar:/home/hadmin/hbase/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hbase/lib/httpclient-4.2.5.jar:/home/hadmin/hbase/lib/httpcore-4.4.1.jar:/home/hadmin/hbase/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hbase/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hbase/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hbase/lib/jackson-xc-1.9.13.jar:/home/hadmin/hbase/lib/jamon-runtime-2.4.1.jar:/home/hadmin/hbase/lib/jasper-compiler-5.5.23.jar:/home/hadmin/hbase/lib/jasper-runtime-5.5.23.jar:/home/hadmin/hbase/lib/javax.inject-1.jar:/home/hadmin/hbase/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hbase/lib/jaxb-api-2.2.2.jar:/home/hadmin/hbase/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hbase/lib/jcodings-1.0.8.jar:/home/hadmin/hbase/lib/jersey-client-1.9.jar:/home/hadmin/hbase/lib/jersey-core-1.9.jar:/home/hadmin/hbase/lib/jersey-guice-1.9.jar:/home/hadmin/hbase/lib/jersey-json-1.9.jar:/home/hadmin/hbase/lib/jersey-server-1.9.jar:/home/hadmin/hbase/lib/jets3t-0.9.0.jar:/home/hadmin/hbase/lib/jettison-1.3.3.jar:/home/hadmin/hbase/lib/jetty-6.1.26.jar:/home/hadmin/hbase/lib/jetty-sslengine-6.1.26.jar:/home/hadmin/hbase/lib/jetty-util-6.1.26.jar:/home/hadmin/hbase/lib/joni-2.1.2.jar:/home/hadmin/hbase/lib/jruby-complete-1.6.8.jar:/home/hadmin/hbase/lib/jsch-0.1.42.jar:/home/hadmin/hbase/lib/jsp-2.1-6.1.14.jar:/home/hadmin/hbase/lib/jsp-api-2.1-6.1.14.jar:/home/hadmin/hbase/lib/jsr305-1.3.9.jar:/home/hadmin/hbase/lib/junit-4.12.jar:/home/hadmin/hbase/lib/leveldbjni-all-1.8.jar:/home/hadmin/hbase/lib/libthrift-0.9.3.jar:/home/hadmin/hbase/lib/log4j-1.2.17.jar:/home/hadmin/hbase/lib/metrics-core-2.2.0.jar:/home/hadmin/hbase/lib/nekohtml-1.9.12.jar:/home/hadmin/hbase/lib/netty-all-4.0.23.Final.jar:/home/hadmin/hbase/lib/paranamer-2.3.jar:/home/hadmin/hbase/lib/protobuf-java-2.5.0.jar:/home/hadmin/hbase/lib/servlet-api-2.5-6.1.14.jar:/home/hadmin/hbase/lib/servlet-api-2.5.jar:/home/hadmin/hbase/lib/slf4j-api-1.7.7.jar:/home/hadmin/hbase/lib/slf4j-log4j12-1.7.5.jar:/home/hadmin/hbase/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hbase/lib/spymemcached-2.11.6.jar:/home/hadmin/hbase/lib/xalan-2.7.0.jar:/home/hadmin/hbase/lib/xml-apis-1.3.03.jar:/home/hadmin/hbase/lib/xml-apis-ext-1.3.04.jar:/home/hadmin/hbase/lib/xmlenc-0.52.jar:/home/hadmin/hbase/lib/xom-1.2.5.jar:/home/hadmin/hbase/lib/xz-1.0.jar:/home/hadmin/hbase/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/etc/hadoop:/home/hadmin/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadmin/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-nfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.1.jar:.::/home/hadmin/hadoop/contrib/capacity-scheduler/*.jar:/home/hadmin/hadoop/bin 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadmin/hadoop/lib/native 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-229.el7.x86_64 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/root 2018-09-04 00:01:54,474 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x575769940x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase 2018-09-04 00:01:54,499 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error) 2018-09-04 00:01:54,506 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session 2018-09-04 00:01:54,564 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9a, negotiated timeout = 40000 Version: 1.2.1 Number of live region servers: 25 Number of dead region servers: 0 Master: dscn11,16000,1529215352324 Number of backup masters: 1 Average load: 305.88 Number of requests: 2507 Number of regions: 7647 Number of regions in transition: 0 2018-09-04 00:01:56,473 INFO [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 2 2018-09-04 00:01:57,276 INFO [main] util.HBaseFsck: getHTableDescriptors == tableNames => [DBN_BSHTIndex, DBN_TTKDIndex, SJYB_UPS, SJYB_ANE, DBN_SUR, DBN_CAEIndex, SJYB_BSHT, DBN_STOIndex, DBN_YTO, DBN_TTKD, DBN_FEDEX, SJYB_ZJS, DBN_ANEIndex, DBN_TNTIndex, SJYB_TNT, SJYB_DHL, DBN_ZJS, DBN_KJIndex, BOX, DBN_DEPPONIndex, DBN_ZGYZIndex, DBN_YUNDA, hbase:namespace, DBN_TNT, DBN_FEDEXIndex, DBN_YTOIndex, DBN_PJIndex, SJYB_EMS, SJYB_KJ, SJYB_ZGYZ, DBN_JBD, DBN_DEPPON, SJYB_KYE, SJYB_SUR, SJYB_PJ, SJYB_UC, DBN_SURIndex, DBN_EMS, DBN_STO, SJYB_TTKD, SJYB_SF, SJYB_JBD, SJYB_APEX, DBN_ZTOIndex, DBN_APEX, DBN_PJ, DBN_UPS, DBN_ZJSIndex, DBN_UPSIndex, DBN_UC, DBN_GTO, DBN_BSHT, DBN_EMSIndex, DBN_APEXIndex, SJYB_CAE, DBN_ZTO, DBN_CAE, DBN_SF, DBN_UCIndex, DBN_SFIndex, SJYB_YUNDA, SJYB_FEDEX, SJYB_STO, DBN_JBDIndex, DBN_ANE, DBN_DHLIndex, DBN_KYE, DBN_KJ, SJYB_GTO, DBN_ZGYZ, SJYB_YTO, SJYB_DEPPON, DBN_YUNDAIndex, SJYB_ZTO, DBN_DHL] 2018-09-04 00:01:57,280 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x51e8e6e6 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181 2018-09-04 00:01:57,280 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x51e8e6e60x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase 2018-09-04 00:01:57,281 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error) 2018-09-04 00:01:57,281 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session 2018-09-04 00:01:57,283 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9c, negotiated timeout = 40000 2018-09-04 00:01:57,296 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 2018-09-04 00:01:57,296 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9c 2018-09-04 00:01:57,297 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9c closed 2018-09-04 00:01:57,298 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down Number of Tables: 75 2018-09-04 00:01:57,308 INFO [main] util.HBaseFsck: Loading region directories from HDFS .............................................................................................................................................................................................................................................................................................................. 2018-09-04 00:01:59,904 INFO [main] util.HBaseFsck: Loading region information from HDFS ............................................................................ 2018-09-04 00:02:02,282 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7792d851 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181 2018-09-04 00:02:02,282 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x7792d8510x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase 2018-09-04 00:02:02,286 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error) 2018-09-04 00:02:02,287 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session 2018-09-04 00:02:02,289 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda1, negotiated timeout = 40000 2018-09-04 00:02:02,303 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181 2018-09-04 00:02:02,303 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase 2018-09-04 00:02:02,304 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error) 2018-09-04 00:02:02,305 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session 2018-09-04 00:02:02,306 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda2, negotiated timeout = 40000 2018-09-04 00:02:02,352 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda2 closed 2018-09-04 00:02:02,352 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-09-04 00:02:02,352 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bda1 2018-09-04 00:02:02,354 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda1 closed 2018-09-04 00:02:02,354 INFO [main] util.HBaseFsck: Checking and fixing region consistency 2018-09-04 00:02:02,354 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/95a27c1ae2df5db14fa66d13966a8daf, deployed => dscn33,16020,1535968743109;DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109 ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/e051ccd37678ff95fdd7652106b1fd5b, deployed => dscn33,16020,1535968743109;DBN_ANE,97,1535968864242.e051ccd37678ff95fdd7652106b1fd5b., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109 2018-09-04 00:02:02,733 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially. ERROR: There is a hole in the region chain between 97 and 97581111303371. You need to create a new .regioninfo and region dir in hdfs to plug the hole. ERROR: Found inconsistency in table DBN_ANE 2018-09-04 00:02:02,876 INFO [main] util.HBaseFsck: Computing mapping of all store files ................................................................................................................................................................................................................................................................................................................................................................................... 2018-09-04 00:02:10,477 INFO [main] util.HBaseFsck: Validating mapping using HDFS state 2018-09-04 00:02:10,662 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181 2018-09-04 00:02:10,662 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase 2018-09-04 00:02:10,664 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn11/10.11.2.4:2181. Will not attempt to authenticate using SASL (unknown error) 2018-09-04 00:02:10,665 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Socket connection established to dscn11/10.11.2.4:2181, initiating session 2018-09-04 00:02:10,666 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn11/10.11.2.4:2181, sessionid = 0x656ba3c1b1c04f, negotiated timeout = 40000 2018-09-04 00:02:10,714 INFO [main] zookeeper.ZooKeeper: Session: 0x656ba3c1b1c04f closed 2018-09-04 00:02:10,714 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-09-04 00:02:10,714 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181 2018-09-04 00:02:10,715 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase 2018-09-04 00:02:10,717 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error) 2018-09-04 00:02:10,717 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session 2018-09-04 00:02:10,719 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda7, negotiated timeout = 40000 2018-09-04 00:02:10,752 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda7 closed Summary: Table DBN_ANE is okay. Number of regions: 147 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_CAE is okay. Number of regions: 100 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_DHL is okay. Number of regions: 100 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496 Table DBN_EMS is okay. Number of regions: 226 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,15342212024962018-09-04 00:02:10,754 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down Table DBN_GTO is okay. Number of regions: 51 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn35,16020,1534221202496 Table DBN_JBD is okay. Number of regions: 238 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_KYE is okay. Number of regions: 51 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_STO is okay. Number of regions: 440 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_SUR is okay. Number of regions: 100 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_TNT is okay. Number of regions: 100 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_UPS is okay. Number of regions: 100 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_YTO is okay. Number of regions: 638 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_ZJS is okay. Number of regions: 102 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_ZTO is okay. Number of regions: 867 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_ANE is okay. Number of regions: 43 Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_CAE is okay. Number of regions: 21 Deployed on: dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_DHL is okay. Number of regions: 21 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_EMS is okay. Number of regions: 90 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_GTO is okay. Number of regions: 21 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496 Table SJYB_JBD is okay. Number of regions: 46 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496 Table SJYB_KYE is okay. Number of regions: 21 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn31,16020,1529215376767 dscn35,16020,1534221202496 Table SJYB_STO is okay. Number of regions: 111 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_SUR is okay. Number of regions: 21 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn35,16020,1534221202496 Table SJYB_TNT is okay. Number of regions: 21 Deployed on: dscn11,16020,1535969281236 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn26,16020,1535018986330 dscn31,16020,1529215376767 dscn34,16020,1535675408409 Table SJYB_UPS is okay. Number of regions: 21 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn29,16020,1535540080040 dscn31,16020,1529215376767 Table SJYB_YTO is okay. Number of regions: 111 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_ZJS is okay. Number of regions: 21 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_ZTO is okay. Number of regions: 137 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_TNTIndex is okay. Number of regions: 1 Deployed on: dscn34,16020,1535675408409 Table SJYB_KJ is okay. Number of regions: 21 Deployed on: dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn34,16020,1535675408409 Table SJYB_PJ is okay. Number of regions: 33 Deployed on: dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496 Table SJYB_SF is okay. Number of regions: 92 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_UC is okay. Number of regions: 36 Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 Table DBN_DEPPONIndex is okay. Number of regions: 1 Deployed on: dscn20,16020,1529215363444 Table DBN_SURIndex is okay. Number of regions: 1 Deployed on: dscn14,16020,1535088098764 Table DBN_BSHTIndex is okay. Number of regions: 2 Deployed on: dscn31,16020,1529215376767 Table DBN_APEX is okay. Number of regions: 200 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_BSHT is okay. Number of regions: 439 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_TTKD is okay. Number of regions: 267 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_ZGYZ is okay. Number of regions: 302 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table hbase:meta is okay. Number of regions: 1 Deployed on: dscn35,16020,1534221202496 Table DBN_ANEIndex is okay. Number of regions: 1 Deployed on: dscn13,16020,1529215824290 Table DBN_DEPPON is okay. Number of regions: 145 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_DHLIndex is okay. Number of regions: 1 Deployed on: dscn26,16020,1535018986330 Table DBN_SFIndex is okay. Number of regions: 2 Deployed on: dscn17,16020,1534932216753 Table DBN_YTOIndex is okay. Number of regions: 2 Deployed on: dscn30,16020,1533181074642 Table DBN_FEDEX is okay. Number of regions: 100 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_YUNDA is okay. Number of regions: 499 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_APEX is okay. Number of regions: 21 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn32,16020,1535798049804 Table SJYB_BSHT is okay. Number of regions: 118 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_TTKD is okay. Number of regions: 86 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table SJYB_ZGYZ is okay. Number of regions: 38 Deployed on: dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 Table DBN_CAEIndex is okay. Number of regions: 1 Deployed on: dscn20,16020,1529215363444 Table BOX is okay. Number of regions: 141 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_JBDIndex is okay. Number of regions: 2 Deployed on: dscn14,16020,1535088098764 Table DBN_TTKDIndex is okay. Number of regions: 1 Deployed on: dscn20,16020,1529215363444 Table hbase:namespace is okay. Number of regions: 1 Deployed on: dscn12,16020,1533367433799 Table DBN_KJIndex is okay. Number of regions: 1 Deployed on: dscn21,16020,1529215364453 Table DBN_ZJSIndex is okay. Number of regions: 1 Deployed on: dscn18,16020,1535022020180 Table DBN_UCIndex is okay. Number of regions: 1 Deployed on: dscn14,16020,1535088098764 Table DBN_PJIndex is okay. Number of regions: 1 Deployed on: dscn18,16020,1535022020180 Table DBN_FEDEXIndex is okay. Number of regions: 2 Deployed on: dscn26,16020,1535018986330 Table DBN_ZGYZIndex is okay. Number of regions: 1 Deployed on: dscn32,16020,1535798049804 Table DBN_ZTOIndex is okay. Number of regions: 2 Deployed on: dscn23,16020,1529215366710 dscn33,16020,1535968743109 Table SJYB_DEPPON is okay. Number of regions: 33 Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496 Table DBN_APEXIndex is okay. Number of regions: 1 Deployed on: dscn16,16020,1534933465007 Table DBN_EMSIndex is okay. Number of regions: 2 Deployed on: dscn15,16020,1529371666615 Table DBN_YUNDAIndex is okay. Number of regions: 2 Deployed on: dscn13,16020,1529215824290 Table SJYB_FEDEX is okay. Number of regions: 21 Deployed on: dscn11,16020,1535969281236 dscn13,16020,1529215824290 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 Table DBN_STOIndex is okay. Number of regions: 2 Deployed on: dscn24,16020,1535104303466 Table SJYB_YUNDA is okay. Number of regions: 102 Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_KJ is okay. Number of regions: 119 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_PJ is okay. Number of regions: 164 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_SF is okay. Number of regions: 457 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_UC is okay. Number of regions: 210 Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496 Table DBN_UPSIndex is okay. Number of regions: 2 Deployed on: dscn15,16020,1529371666615 5 inconsistencies detected. Status: INCONSISTENT 2018-09-04 00:02:10,770 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 2018-09-04 00:02:10,771 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9a 2018-09-04 00:02:10,772 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9a closed 2018-09-04 00:02:10,773 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down [root@dscn12 ~]#
==緣由分析==
從hbase hbck檢查結果中能夠看到如下關鍵信息ruby
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
ERROR: There is a hole in the region chain between 97 and 97581111303371. You need to create a new .regioninfo and region dir in hdfs to plug the hole.
錯誤提示了hbase meta數據存在問題，缺失的rowkey範圍邊界，也就是hole（Region之間有空隙，正常的Region之間是無交叉、無縫隙的）。
致使HBase整個集羣狀態出現問題，影響Storm的讀寫處理。
==解決過程==
之前有位大神給了在線hbck修復的運維命令
前提：HDFS fsck確保hbase根目錄下文件沒有損壞丟失，若是有，則先進行corrupt block移除。 切記：必定要在全部Region都上線以後再修復，不然修復以後可能出現重複Region。 步驟1. hbase hbck 檢查輸出因此ERROR信息，每一個ERROR都會說明錯誤信息。 步驟2. hbase hbck -fixTableOrphans 先修復tableinfo缺失問題，根據內存cache或者hdfs table 目錄結構，從新生成tableinfo文件。 步驟3. hbase hbck -fixHdfsOrphans 修復regioninfo缺失問題，根據region目錄下的hfile從新生成regioninfo文件。 步驟4. hbase hbck -fixHdfsOverlaps 修復region重疊問題，merge重疊的region爲一個region目錄，並重新生成一個regioninfo。 步驟5. hbase hbck -fixHdfsHoles 修復region缺失，利用缺失的rowkey範圍邊界，生成新的region目錄以及regioninfo填補這個空洞。 步驟6. hbase hbck -fixMeta 修復meta表信息，利用regioninfo信息，從新生成對應meta row填寫到meta表中，併爲其填寫默認的分配regionserver。 步驟7. hbase hbck -fixAssignments 把這些offline的region觸發上線，當region開始從新open 上線的時候，會被從新分配到真實的RegionServer上 , 並更新meta表上對應的行信息。
我嘗試執行了兩個命令：
hbase hbck -fixHdfsHoles
hbase hbck -fixAssignments -fixMete
從新運行hbase hbck以後，集羣恢復了正常。
==經驗總結==
hbase hbck檢查發現集羣不健康的時候，
嘗試確診不健康的緣由（有hole？regioninfo缺失？或是tableinfo缺失？）
選擇適合的在線修復命令修復HBase，
修復以後，再次執行hbase hbck命令，
當檢查結果爲下面這種狀態時，hbase爲健康狀態：
（HBase集羣不須要重啓，由於是在線修復命令）
--END--
相關文章
- 1. 一次解決問題的通過
- 2. HBase一次慢查詢請求的問題排查與解決過程
- 3. 一次問題解決Socket問題過程總結
- 4. 記一次多線程問題的解決過程
- 5. 第一次編寫bootsect時的問題解決過程
- 6. 記一次SQLServer2019安裝和卸載問題的解決過程
- 7. 記一次很是痛苦的解決問題過程
- 8. Android 記一次解決問題的過程
- 9. 一次ORA-39083,ORA-01917的問題解決過程
- 10. 一次線上問題的排查解決過程
- 更多相關文章...
- • Redis悲觀鎖解決高併發搶紅包的問題 - 紅包項目實戰
- • Redis樂觀鎖解決高併發搶紅包的問題 - 紅包項目實戰
- • PHP Ajax 跨域問題最佳解決方案
- • IntelliJ IDEA中SpringBoot properties文件不能自動提示問題解決
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
