Hadoop - Mac OSX下配置和啓動hadoop以及常見錯誤解決

0. 安裝JDK

參考網上教程在OSX下安裝jdk

 

1. 下載及安裝hadoop

a) 下載地址:

http://hadoop.apache.org

 

b) 配置ssh環境

在terminal裏面輸入: ssh localhost

若是有錯誤提示信息，表示當前用戶沒有權限。這個多半是系統爲安全考慮，默認設置的。

更改設置以下：進入system preference --> sharing --> 勾選remote login，並設置allow access for all users。

再次輸入「ssh localhost"，再輸入密碼並確認以後，能夠看到ssh成功。

 

 

c) ssh免登錄配置

命令行輸入:

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa  

ssh-keygen表示生成祕鑰；-t表示祕鑰類型；-P用於提供密語；-f指定生成的祕鑰文件。

這個命令在」~/.ssh/「文件夾下建立兩個文件id_dsa和id_dsa.pub，是ssh的一對兒私鑰和公鑰。

接下來，將公鑰追加到受權的key中去，輸入：

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

********************************************************************************

免密碼登陸localhost

1. ssh-keygen -t rsa Press enter for each line 提示輸入直接按回車就好

2. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

3. chmod og-wx ~/.ssh/authorized_keys

測試 ssh localhost若是仍然提示要輸入密碼，那麼能夠vim ~/.ssh/config文件，添加如下代碼。

Host localhost
   AddKeysToAgent yes
   UseKeychain yes
   IdentityFile ~/.ssh/id_rsa

測試 ssh localhost，再也不提示須要輸入密碼。

 

********************************************************************************

d) 設置環境變量

在實際啓動Hadoop以前，有三個文件須要進行配置。

但在這以前，咱們須要在咱們的bash_profile中配置以下幾個配置

 

命令行輸入: 

open ~/.bash_profile;

 

# hadoop
export HADOOP_HOME=/Users/YourUserName/Documents/Dev/hadoop-2.7.3
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

 

e) 配置hadoop-env.sh

在${HADOOP_HOME}/etc/hadoop目錄下，找到hadoop-env.sh，打開編輯確認以下設置是否正確：

export JAVA_HOME=${JAVA_HOME}

export HADOOP_HEAPSIZE=2000（去掉註釋）

export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"（去掉註釋）

 

f) 配置core-site.xml——指定了NameNode的主機名與端口

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->  
  
<configuration>  
    <property>  
        <name>/Users/YourUserName/Documents/Dev/hadoop-2.7.3/hadoop-${user.name}</name>  
        <value>hdfs://localhost:9000</value>  
        <description>A base for other temporary directories.</description>  
    </property>  
    <property>  
        <name>fs.default.name</name>  
        <value>hdfs://localhost:8020</value>  
    </property>  
</configuration>   

 

g) 配置hdfs-site.xml——指定了HDFS的默認參數副本數，由於僅運行在一個節點上，因此這裏的副本數爲1

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>  
    <property>  
        <name>dfs.replication</name>  
        <value>1</value>  
    </property>  
</configuration>   

 

h) 配置mapred-site.xml——指定了JobTracker的主機名與端口

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>  
    <property>  
        <name>mapred.job.tracker</name>  
        <value>hdfs://localhost:9001</value>  
    </property>  
    <property>  
        <name>mapred.tasktracker.map.tasks.maximum</name>  
        <value>2</value>  
    </property>  
    <property>  
        <name>mapred.tasktracker.reduce.tasks.maximum</name>  
        <value>2</value>  
    </property>  
</configuration>  

 

i) 安裝HDFS

通過以上的配置，就能夠進行HDFS的安裝了

命令行輸入:

cd $HADOOP_HOME/bin

hadoop namenode -format 

若是出現下圖, 說明你的HDFS已經安裝成功了

 

j) 啓動Hadoop

cd ${HADOOP_HOME}/sbin

start-dfs.sh
start-yarn.sh

 

k) 驗證hadoop

若是在啓動過程當中沒有發生任何錯誤

啓動完成以後,在命令行輸入: jps

若是結果以下:

3761 DataNode
4100 Jps
3878 SecondaryNameNode
3673 NameNode
4074 NodeManager
3323 ResourceManager

以上幾個節點都打印出來,那麼恭喜你,你已經成功安裝和啓動hadoop了!

最後,咱們能夠在瀏覽器經過http的方式進行驗證

瀏覽器輸入:

http://localhost:8088/

結果以下:

瀏覽器輸入:

http://localhost:50070/

結果以下:

 

 

2. 常見錯誤解決

hadoop namenode不能啓動

org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /tmp/hadoop-javoft/dfs/name is in an inconsistent state: storage di rectory does not exist or is not accessible.

緣由在於core-site.xml你必須覆蓋hadoop.tmp.dir爲你本身的hadoop目錄... hadoop.tmp.dir /home/javoft/Documents/hadoop/hadoop-${user.name}