我先在CentOS 7.4的服務器的/usr/
目錄新建了一個hadoop目錄,而後進入新建的hadoop目錄,下載hadoop。html
命令:
wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz
node
到hadoop下載頁面選擇合適的hadoop版本進行下載,我使用的是2.8.5,截止如今,由於hbase最高支持2.8.X的版本,使用Hadoop2.9.X會有問題。linux
把下載好的壓縮包解壓出來apache
命令:
tar xvzf hadoop-2.8.5.tar.gz
vim
配置/etc/profile的JAVA_HOME服務器
配置${HADOOP_HOME}/etc/hadoop/core-site.xml
less
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration>
配置${HADOOP_HOME}/etc/hadoop/hdfs-site.xml
這個配置是配置HDFS的備份數的,不配置的話默認是備份3分,由於這裏是測試,因此1份就夠了。通常用默認配置,土豪能夠調3+。ssh
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
如今經過ssh localhost
看看需不須要密碼訪問。若是發現須要密碼訪問,那麼試用一下命令配置ssh-keygen
:分佈式
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 0600 ~/.ssh/authorized_keys
${HADOOP_HOME}/bin/hdfs namenode -format
${HADOOP_HOME}/sbin/start-dfs.sh
${HADOOP_HOME}/bin/hdfs dfs -mkdir /user ${HADOOP_HOME}/bin/hdfs dfs -mkdir /user/<username>
${HADOOP_HOME}/bin/hdfs dfs -put etc/hadoop input
${HADOOP_HOME}/bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar grep input output 'dfs[a-z.]+'
跑完會輸出這個:工具
最後中止運行:
${HADOOP_HOME}/sbin/stop-dfs.sh
先配置MapReduce的管理工具爲yarn:
cp ${HADOOP_HOME}/etc/hadoop/mapred-site.xml.template ${HADOOP_HOME}/etc/hadoop/mapred-site.xml vim ${HADOOP_HOME}/etc/hadoop/mapred-site.xml
mapred-site.xml配置以下:
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
配置yarn-site.xml:
$ vim ${HADOOP_HOME}/etc/hadoop/yarn-site.xml <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
啓動yarn:
${HADOOP_HOME}/sbin/start-yarn.sh jps 32934 NodeManager 32999 Jps 32857 ResourceManager