bigdata learning unit one--Hadoop environment setting

 

 

一、配置ssh，使集羣服務器之間的通信，再也不每次都輸入密碼進行認證。

二、

[root@hc--uatbeta2 hadoop]# start-all.sh
Starting namenodes on [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2018-08-07 06:56:20,552 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
[root@hc--uatbeta2 hadoop]#

解決：把全部環節變量都加上   vi hadoop-env.sh

export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

--------

[root@hc--uatbeta2 hadoop]# vi core-site.xml
[root@hc--uatbeta2 hadoop]# vi mapred-site.xml
[root@hc--uatbeta2 hadoop]# vi yarn-site.xml

vi hdfs-site.xml

----將hostname 改成IP-------  改成IP也不行，得修改hosts：vi /etc/hosts

二、格式化namenode,

#CD /usr/hadoop/hadoop-3.0.0
# ./bin/hdfs namenode -format

這個報錯：

出現未知的主機名的問題，異常信息以下所示：

 [shirdrn@localhost bin]$ hadoop namenode -format
11/06/22 07:33:31 INFO namenode.NameNode: STARTUP_MSG: 

解決方案仍是修改hosts；

10.167.202.135:50070  

10.167.202.135:8088

10.167.202.135:50070  不能訪問的話

vi  /etc/selinux/config 

修改
 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
 
爲
 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled

　　

設置默認訪問端口

mapred-site.xml 添加下面兩個
<property>
  <name>mapred.job.tracker.http.address</name>
  <value>0.0.0.0:50030</value>
</property>
<property>
  <name>mapred.task.tracker.http.address</name>
  <value>0.0.0.0:50060</value>
</property>

hdfs-site.xml 添加下面配置
<property>
  <name>dfs.http.address</name>
  <value>0.0.0.0:50070</value>
</property>
而後中止全部進程，刪除name、data文件夾下數據從新格式化，從新啓動後訪問正常

　　

 測試hdfs 是否能夠使用：

一、上傳文件：hadoop fs -put /root/服務器地址.java hdfs://localhost:9000/    localhost：9000 配置在hadoop-env.sh

二、查看文件：hadoop fs -ls /

三、刪除文件：hadoop  fs  -rm hdfs://10.167.202.135:9000/服務器地址.java