解決hadoop集羣中datanode啓動後自動關閉的問題

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /var/lib/hadoop-0.20/cache/hdfs/dfs/data: namenode namespaceID = 240012870; datanode namespaceID = 1462711424 .

　　問題：Namenode上namespaceID與datanode上namespaceID不一致。

　　問題產生緣由：每次namenode format會從新建立一個namenodeId,而tmp/dfs/data下包含了上次format下的id,namenode format清空了namenode下的數據,可是沒有清空datanode下的數據,因此形成namenode節點上的namespaceID與datanode節點上的namespaceID不一致。啓動失敗。

　　第一種解決方法：即:

　　(1)停掉集羣服務

　　(2)在出問題的datanode節點上刪除data目錄，data目錄便是在hdfs-site.xml文件中配置的dfs.data.dir目錄，本機器上那個是/var/lib/hadoop-0.20/cache/hdfs/dfs/data/ (注：咱們當時在全部的datanode和namenode節點上均執行了該步驟。以防刪掉後不成功，能夠先把data目錄保存一個副本).

　　(3)格式化namenode.

　　(4)從新啓動集羣。

　　問題解決。

        這種方法帶來的一個反作用便是，hdfs上的全部數據丟失。若是hdfs上存放有重要數據的時候，不建議採用該方法，能夠嘗試提供的網址中的第二種方法。

下面給出兩種解決辦法，我使用的是第二種。

Workaround 1: Start from scratch

I can testify that the following steps solve this error, but the side effects won't make you happy (me neither). The crude workaround I have found is to:

1.     stop the cluster

2.     delete the data directory on the problematic datanode: the directory is specified by dfs.data.dir in conf/hdfs-site.xml; if you followed this tutorial, the relevant directory is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data

3.     reformat the namenode (NOTE: all HDFS data is lost during this process!)

4.     restart the cluster

When deleting all the HDFS data and starting from scratch does not sound like a good idea (it might be ok during the initial setup/testing), you might give the second approach a try.

Workaround 2: Updating namespaceID of problematic datanodes

Big thanks to Jared Stehler for the following suggestion. I have not tested it myself yet, but feel free to try it out and send me your feedback. This workaround is "minimally invasive" as you only have to edit one file on the problematic datanodes:

1.     stop the datanode

2.     edit the value of namespaceID in <dfs.data.dir>/current/VERSION to match the value of the current namenode

3.     restart the datanode

If you followed the instructions in my tutorials, the full path of the relevant file is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data/current/VERSION (background: dfs.data.dir is by default set to ${hadoop.tmp.dir}/dfs/data, and we set hadoop.tmp.dir to /usr/local/hadoop-datastore/hadoop-hadoop).

If you wonder how the contents of VERSION look like, here's one of mine:

#contents of <dfs.data.dir>/current/VERSION

namespaceID=393514426

storageID=DS-1706792599-10.10.10.1-50010-1204306713481

cTime=1215607609074

storageType=DATA_NODE

layoutVersion=-13

緣由:每次namenode format會從新建立一個namenodeId,而tmp/dfs/data下包含了上次format下的id,namenode format清空了namenode下的數據,可是沒有晴空datanode下的數據,致使啓動時失敗,所要作的就是每次fotmat前,清空tmp一下 的全部目錄.