hadoop namenode啓動失敗

hadoop version=3.1.2java

生產環境中，一臺namenode節點忽然掛掉了，，從新啓動失敗，日誌以下：node

Info=-64%3A1391355681%3A1545175191847%3ACID-9160c87b-3ab7-4372-98a1-536a59dd36ef&inProgressOk=true' to transaction ID 159168296
2019-03-05 14:38:06,460 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'http://xxx:8480/getJournal?jid=GD-AI&segmentTxId=162853718&storageInfo=-64%3A1391355681%3A1545175191847%3ACID-9160c87b-3ab7-4372-98a1-536a59dd36ef&inProgressOk=true' to transaction ID 159168296
2019-03-05 14:38:06,487 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: There appears to be a gap in the edit log.  We expected txid 159168296, but got txid 162853718.
        at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:238)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:160)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:890)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1097)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
2019-03-05 14:38:06,490 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@6950ed69{/,null,UNAVAILABLE}{/hdfs}
2019-03-05 14:38:06,494 INFO org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@5f20155b{HTTP/1.1,[http/1.1]}{xxx:50070}
2019-03-05 14:38:06,494 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@4722ef0c{/static,file:///data1/hadoop/hadoop-3.1.2/share/hadoop/hdfs/webapps/static/,UNAVAILABLE}
2019-03-05 14:38:06,494 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@5b38c1ec{/logs,file:///data1/hadoop/hadoop-3.1.2/logs/,UNAVAILABLE}
2019-03-05 14:38:06,495 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2019-03-05 14:38:06,496 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2019-03-05 14:38:06,496 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2019-03-05 14:38:06,496 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: There appears to be a gap in the edit log.  We expected txid 159168296, but got txid 162853718.
        at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:238)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:160)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:890)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1097)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
2019-03-05 14:38:06,497 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.io.IOException: There appears to be a gap in the edit log.  We expected txid 159168296, but got txid 162853718.
2019-03-05 14:38:06,499 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************c

從報錯來看，，是獲取edit log日誌出錯。說白點，就是namenode元數據破壞了，須要修復。web

解決：apache

（1）、在出錯的機器執行以下命令，一路按c或者y安全

# hadoop namenode -recover

（2）、若是第一種沒有解決，那麼按以下的方法來解決bash

解決步驟與命令：
1）確保另一個active的nn是正常的且不要去關閉，若是此前提不保證，則尋找另外解決方法，底下忽略；
2）而後檢查active-nn的元數據目錄下的fsimage是不是最新的，可根據當前機器時間來大體判斷，如不然須要進入安全模式後savenamespace，操做以下：
du -sh /hadoop/journal/ 這個的大小也要肯定下，太大則很會很慢app

su hdfs
export HADOOP_CLIENT_OPTS="-D transwarp.maintenance.only.mode=true"
hdfs dfsadmin –safemode get
hdfs dfsadmin -safemode enter
hdfs dfsadmin –saveNamespace
hdfs dfsadmin -safemode leaveeclipse

3）而後把hive1(當前active) 的disk1的這current兩個最新的標紅色fsimge以及對應的md5的文件，scp到hive2（啓動失敗的nn）的disk1 的一樣目錄current目錄下，固然通常dfs.namenode.name.dir配置的是兩塊磁盤，另一個也須要scp過去，最後須要注意複製過去文件的權限，須要修改權限chown hdfs:hdfs xx文件webapp

重啓此前失敗的standby-namenode;oop

借鑑：http://support.transwarp.cn/t/namenode/2242