HDFS的安裝和部署

HDFS的安裝和部署

1.準備工做

準備3臺機器，設置好hosts 一臺做爲Namenode，cc-staging-session2命名爲master， 兩臺做爲dataNode，cc-staging-front命名爲slave1, cc-staging-imcenter 命名爲slave2 #3臺機器都建立hadoop用戶 useradd hadoop passwd hadoop # 安裝JDK，並設置JAVA_HOME和PATH #下載安裝jdk1.7 http://www.oracle.com/technetwork/java/javase/downloads/index.html tar zxvf jdk-7u21-linux-x64.gz -C /usr/local/ #/etc/profile增長環境變量 pathmunge /usr/local/jdk1.7.0_21/bin export JAVA_HOME=/usr/local/jdk1.7.0_21/ export JRE_HOME=/usr/local/jdk1.7.0_21/jre export  CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar

2.下載安裝hadoop

#下載hadoop 下載地址https://ccp.cloudera.com/display/SUPPORT/CDH3+Downloadable+Tarballs wget  http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u6.tar.gz wget  http://archive.cloudera.com/cdh/3/hbase-0.90.6-cdh3u6.tar.gz wget  http://archive.cloudera.com/cdh/3/hive-0.7.1-cdh3u6.tar.gz #在3太機器上建立相同的目錄路徑, name目錄只存放在master上，且權限爲755，不然會致使後面的格式化失敗 mkdir -p  /hadoop/{install,name,data1, data2,tmp} #解壓安裝包到/hadoop/install下 tar zxvf hadoop-0.20.2-cdh3u6.tar.gz -C /hadoop/install/ #修改屬主爲hadoop chown -R hadoop.hadoop /hadoop

3.設置hadoop帳戶的ssh信任關係

#在master機器上操做 su – hadoop ssh-keygen ssh-copy-id -i .ssh/id_rsa.pub  hadoop@cc-staging-front ssh-copy-id -i .ssh/id_rsa.pub  hadoop@cc-staging-imcenter ssh-copy-id -i .ssh/id_rsa.pub  hadoop@cc-staging-session2 #測試一下,都能成功登陸就行 ssh hadoop@master ssh hadoop@slave1 ssh hadoop@slave2

4.編輯HDFS配置文件,因此節點都有保持一致

cd /hadoop/install/hadoop-0.20.2-cdh3u6/conf #core-site.xml核心配置 <configuration> <property>       <name>fs.default.name</name>       <value>hdfs://master:9000</value>  </property> </configuration> #hdfs-site.xml：站點多項參數配置 <configuration> <property>       <name>dfs.replication</name>      <value>2</value> </property> <property>      <name>dfs.name.dir</name>      <value>/hadoop/name</value> </property> <property>      <name>dfs.data.dir</name>       <value>/hadoop/data1,/hadoop/data2</value> </property> <property>      <name>dfs.tmp.dir</name>      <value>/hadoop/tmp</value> </property> </configuration> #在hadoop-env.sh中配置JAVA_HOME變量 export JAVA_HOME=/usr/local/jdk1.7.0_21/

5.初始化namenode節點

#在master上操做,格式化Image文件的存儲空間,必需是大寫的Y su - hadoop cd  /hadoop/install/hadoop-0.20.2-cdh3u6/bin/ [hadoop@cc-staging-session2 bin]$ ./hadoop  namenode -format 13/04/27 01:46:40 INFO namenode.NameNode:  STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG:   host = cc-staging-session2/127.0.0.1 STARTUP_MSG:   args = [-format] STARTUP_MSG:   version = 0.20.2-cdh3u6 STARTUP_MSG:   build = git://ubuntu-slave01/var/lib/jenkins/workspace/CDH3u6-Full-RC/build/cdh3/hadoop20/0.20.2-cdh3u6/source  -r efb405d2aa54039bdf39e0733cd0bb9423a1eb0a; compiled by 'jenkins' on Wed Mar  20 11:45:36 PDT 2013 ************************************************************/ Re-format filesystem in  /hadoop/name ? (Y or N) Y 13/04/27 01:46:42 INFO util.GSet: VM  type       = 64-bit 13/04/27 01:46:42 INFO util.GSet: 2% max  memory = 17.77875 MB 13/04/27 01:46:42 INFO util.GSet:  capacity      = 2^21 = 2097152 entries 13/04/27 01:46:42 INFO util.GSet:  recommended=2097152, actual=2097152 13/04/27 01:46:42 INFO  namenode.FSNamesystem: fsOwner=hadoop (auth:SIMPLE) 13/04/27 01:46:42 INFO  namenode.FSNamesystem: supergroup=supergroup 13/04/27 01:46:42 INFO  namenode.FSNamesystem: isPermissionEnabled=true 13/04/27 01:46:42 INFO  namenode.FSNamesystem: dfs.block.invalidate.limit=1000 13/04/27 01:46:42 INFO  namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0  min(s), accessTokenLifetime=0 min(s) 13/04/27 01:46:43 INFO common.Storage:  Image file of size 112 saved in 0 seconds. 13/04/27 01:46:43 INFO common.Storage: Storage directory /hadoop/name has been successfully  formatted. 13/04/27 01:46:43 INFO namenode.NameNode:  SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at  cc-staging-session2/127.0.0.1 ************************************************************/ #啓動namenode和datanode cd  /hadoop/install/hadoop-0.20.2-cdh3u6/bin/ ./hadoop-daemon.sh start namenode #在/hadoop/install/hadoop-0.20.2-cdh3u6/bin/下有不少命令，  * start-all.sh 啓動全部的Hadoop守護，包括namenode,  datanode，jobtracker，tasktrack，secondarynamenode。  * stop-all.sh 中止全部的Hadoop。  * start-mapred.sh 啓動Map/Reduce守護，包括Jobtracker和Tasktrack。  * stop-mapred.sh 中止Map/Reduce守護  * start-dfs.sh 啓動Hadoop DFS守護，Namenode和Datanode。  * stop-dfs.sh 中止DFS守護 #在slave1和slave2上啓動datanode cd  /hadoop/install/hadoop-0.20.2-cdh3u6/bin/ ./hadoop-daemon.sh start datanode #能夠在各個節點上運行jps命令查看是否啓動成功 [hadoop@cc-staging-session2 bin]$  jps 11926 NameNode 12566 Jps 12233 SecondaryNameNode 12066 DataNode #數據節點必需在硬盤上否則會報錯 [hadoop@cc-staging-front bin]$ jps 14582 DataNode 14637 Jps [hadoop@cc-staging-imcenter bin]$ jps 23355 DataNode 23419 Jps #

6.簡單的測試

#在任意一個節點建立目錄： ./hadoop dfs -mkdir test #能夠在全部數據節點上查看到目錄： ./hadoop dfs -ls Found 2 items drwxr-xr-x   - hadoop supergroup          0 2013-04-27 02:32  /user/hadoop/test #拷貝文件，即把本地的文件存放到HDFS中 ./hadoop dfs -put /etc/services test #刪除文件 ./hadoop dfs -rm test/services