HBase問題修復

轉載自HBase問題修復

[hbase版本1.1.2]

【第一次檢查】

執行命令： hbase hbck -details "default:test_tony" > 20171227_hbck_test_tony 2>&1 &

查看執行日誌 20171227_hbck_test_tony 發現3種錯誤：[1] First region should start with an empty key、[2] Region not deployed on any region server、[3] a hole in the region chain

日誌摘要以下 [hbase@kmr-core1-001 ~]$ less 20171227_hbck_test_tony ... ... ERROR: (region test_tony,P_4013488,1512319359517.c6892d77b1ea148f3e0642d9fdce68af.) First region should start with an empty key.  You need to  create a new region and regioninfo in HDFS to plug the hole. 'ksai:export_import_table_test': There is a hole in the region chain between P_802083 and P_901927D.  You need to create a new .regioninfo and region dir in hdfs to plug the hole. ERROR: There is a hole in the region chain between P_A011FE2 and P_B00FAE.  You need to create a new .regioninfo and region dir in hdfs to plug the hole. ---- Table 'test_tony': overlap groups There are 0 overlap groups with 0 overlapping regions ERROR: Found inconsistency in table test_tony 2017-12-27 16:27:47,925 INFO  [main] util.HBaseFsck: Computing mapping of all store files ... ... ERROR: Region { meta => test_tony,P_2009AE6,1511517508282.1dc960c2cb30a897ec50bbb656e9faa9., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/1dc960c2cb30a897ec50bbb656e9faa9, deployed => , replicaId => 0 } not deployed on any region server. ERROR: Region { meta => test_tony,P_F007E:,1511757663778.5534b6665b97db8d8a7851955ccd8dc5., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/5534b6665b97db8d8a7851955ccd8dc5, deployed => , replicaId => 0 } not deployed on any region server. ERROR: Region { meta => test_tony,P_802083,1513409109894.57ff7ad1a791fb0f8c70f751108d55dd., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/57ff7ad1a791fb0f8c70f751108d55dd, deployed => , replicaId => 0 } not deployed on any region server. ERROR: Region { meta => test_tony,,1511517508282.d0a9bbf43b36b122b7c7f4256f9cdba4., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/d0a9bbf43b36b122b7c7f4256f9cdba4, deployed => , replicaId => 0 } not deployed on any region server. ERROR: Region { meta => test_tony,P_A011FE2,1511537471096.f5777d857532db7ac592e3b621c0372e., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/f5777d857532db7ac592e3b621c0372e, deployed => , replicaId => 0 } not deployed on any region server. ... ... 2017-12-27 16:27:49,637 INFO  [main] util.HBaseFsck: Finishing hbck Summary: Table hbase:meta is okay.     Number of regions: 1     Deployed on:  kmr-5b9c18fc-gn-7b3518df-core-1-005.ksc.com,16020,1514187996430 Table test_tony is inconsistent.     Number of regions: 8     Deployed on:  kmr-5b9c18fc-gn-7b3518df-core-1-001.ksc.com,16020,1514188220947 kmr-5b9c18fc-gn-7b3518df-core-1-003.ksc.com,16020,1514187978942 kmr-5b9c18fc-gn-7b3518df-core-1-004.ksc.com,16020,1514187984473 kmr-5b9c18fc-gn-7b3518df-core-1-005.ksc.com,16020,1514187996430 kmr-5b9c18fc-gn-7b3518df-core-1-006.ksc.com,16020,1514188010078 kmr-5b9c18fc-gn-7b3518df-core-1-008.ksc.com,16020,1514188032392 9 inconsistencies detected. Status: INCONSISTENT

阻塞在了轉移一個region到其餘region server的過程當中，可能的緣由是源region未分配給其餘region server，可能的解決辦法是：在保證這個hfile存在的前提下，手動強制assign該region

【第一次修復】

注意：修復此表以前先停掉它

hbase(main):022:0> disable 'default:test_tony' 0 row(s) in 4.9250 seconds

再執行命令： hbase hbck -repair "default:test_tony" > 20171227_tried_repaired_test_tony 2>&1 &

查看執行日誌 20171227_tried_repaired_test_tony 發現上述的三種錯誤中的兩種已修復，可是 Region not deployed on any region server錯誤還沒能修復

日誌摘要以下 ERROR: Region { meta => test_tony,P_2009AE6,1511517508282.1dc960c2cb30a897ec50bbb656e9faa9., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/1dc960c2cb30a897ec50bbb656e9faa9, deployed => , replicaId => 0 } not deployed on any region server. ERROR: Region { meta => test_tony,P_F007E:,1511757663778.5534b6665b97db8d8a7851955ccd8dc5., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/5534b6665b97db8d8a7851955ccd8dc5, deployed => , replicaId => 0 } not deployed on any region server. Trying to fix unassigned region... Trying to fix unassigned region... ERROR: Region { meta => test_tony,,1511517508282.d0a9bbf43b36b122b7c7f4256f9cdba4., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/d0a9bbf43b36b122b7c7f4256f9cdba4, deployed => , replicaId => 0 } not deployed on any region server. Trying to fix unassigned region... ERROR: Region { meta => test_tony,P_802083,1513409109894.57ff7ad1a791fb0f8c70f751108d55dd., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/57ff7ad1a791fb0f8c70f751108d55dd, deployed => , replicaId => 0 } not deployed on any region server. Trying to fix unassigned region... ERROR: Region { meta => test_tony,P_A011FE2,1511537471096.f5777d857532db7ac592e3b621c0372e., hdfs => hdfs://hdfs-ha/apps/hbase/data/data/default/test_tony/f5777d857532db7ac592e3b621c0372e, deployed => , replicaId => 0 } not deployed on any region server. Trying to fix unassigned region...

【第二次修復】

執行命令： hbase hbck -fixMeta -fixAssignments "default:test_tony" > 20171227_2nd_tried_repaired_test_tony 2>&1 &

查看執行日誌 20171227_2nd_tried_repaired_test_tony 發現上面的三種錯誤中[1]和[3]修好了，可是[2]還沒修復，並且新發現了錯誤 [4] Region failed to move out of transition within timeout XXXXXXms

日誌摘要以下 2017-12-27 17:07:19,032 WARN  [hbasefsck-pool1-t42] util.HBaseFsck: Unable to complete check or repair the region 'test_tony,P_802083,1513409109894.57ff7ad1a791fb0f8c70f751108d55dd.'. java.io.IOException: Region {ENCODED => 57ff7ad1a791fb0f8c70f751108d55dd, NAME => 'test_tony,P_802083,1513409109894.57ff7ad1a791fb0f8c70f751108d55dd.', STARTKEY => 'P_802083', ENDKEY => 'P_901927D'} failed to move out of transition within timeout 120000ms         at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:149)         at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:2114)         at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:2315)         at org.apache.hadoop.hbase.util.HBaseFsck.access$1100(HBaseFsck.java:197)         at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1887)         at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1875)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745) 2017-12-27 17:05:21,028 INFO  [hbasefsck-pool1-t28] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => f5777d857532db7ac592e3b621c0372e, NAME => 'test_tony,P_A011FE2,1511537471096.f5777d857532db7ac592e3b621c0372e.', STARTKEY => 'P_A011FE2', ENDKEY => 'P_B00FAE'} 2017-12-27 17:05:22,028 INFO  [hbasefsck-pool1-t26] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => d0a9bbf43b36b122b7c7f4256f9cdba4, NAME => 'test_tony,,1511517508282.d0a9bbf43b36b122b7c7f4256f9cdba4.', STARTKEY => '', ENDKEY => 'P_2009AE6'}

出現錯誤[4]以及Region still in transition(RIT)的緣由是，該region被原來的Region Server unassigned了，可是尚未被assigned到一個新的RS上，處於無主狀態。

【第三次修復】

在linux命令行執行 hbase hbck -fixAssignments 'default:test_tony' > 20171227_toFix-_an_empty_key 2>&1 &

查看執行日誌 20171227_toFix-_an_empty_key ，成功修復！ 

Summary: Table hbase:meta is okay.     Number of regions: 1     Deployed on:  kmr-5b9c18fc-gn-7b3518df-core-1-005.ksc.com,16020,1514187996430 Table test_tony is okay.     Number of regions: 0     Deployed on:  0 inconsistencies detected. Status: OK

再校驗一下數據是否是好了

hbase hbck -fixAssignments 'default:test_tony' > 20171227_After_fixAssgnmentOf_should_end_with_an_empty_key_FORtest_tony 2>&1 &

查看執行日誌 20171227_After_fixAssgnmentOf_should_end_with_an_empty_key_FORtest_tony 發現Status: OK

成功修復之後，再從新啓用表

hbase(main):028:0> enable 'default:test_tony' 0 row(s) in 2.3100 seconds

至此，修復完成。

做者：Tony_仔 來源：CSDN 原文：http://www.javashuo.com/article/p-rzcggimr-ec.html 版權聲明：本文爲博主原創文章，轉載請附上博文連接！