Hadoop搭建_02

配置Hadoop

上傳hadoop使用hadoop用戶

 

解壓縮

[hadoop@master installer]$ tar xzf hadoop-1.2.1.tar.gz

 

[hadoop@master installer]$ ll

總計 62428

drwxr-xr-x 15 hadoophadoop     4096 2013-07-23 hadoop-1.2.1

-rw-r--r--  1 hadoophadoop 63851630 12-01 13:20 hadoop-1.2.1.tar.gz

[hadoop@master installer]$

建立軟鏈接

[hadoop@master installer]$ mv hadoop-1.2.1 ..

[hadoop@master installer]$ cd ..

[hadoop@master ~]$ ln -s hadoop-1.2.1/ hadoop

[hadoop@master ~]$ ll

總計 8

lrwxrwxrwx  1hadoophadoop   13 12-01 13:22 hadoop -> hadoop-1.2.1/

drwxr-xr-x 15 hadoophadoop 4096 2013-07-23 hadoop-1.2.1

drwxrwxr-x  2hadoophadoop 4096 12-01 13:22 installer

[hadoop@master ~]$

 

配置環境變量

[hadoop@master ~]$ vim .bashrc

# .bashrc

 

# Source global definitions

if [ -f /etc/bashrc ]; then

        . /etc/bashrc

fi

 

# User specific aliases and functions

#Hadoop1.0

export JAVA_HOME=/usr/java/jdk1.6.0_27

export HADOOP1_HOME=/home/hadoop/hadoop

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP1_HOME/bin

export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib

 

拷貝到slave1和slave2

[hadoop@master ~]$ scp .bashrc slave1:~

.bashrc                                                                  100%  308     0.3KB/s   00:00   

[hadoop@master ~]$ scp .bashrc slave2:~

The authenticity of host 'slave2 (192.168.1.102)' can't be established.

RSA key fingerprint is 61:e5:be:d1:92:41:b4:22:4e:88:ff:b1:b1:a1:64:bb.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave2,192.168.1.102' (RSA) to the list of known hosts.

.bashrc                                                                  100%  308     0.3KB/s   00:00   

[hadoop@master ~]$

 

配置hadoop文件

[hadoop@master ~]$ cdhadoop

[hadoop@masterhadoop]$ cdconf

[hadoop@masterconf]$ vim hadoop-env.sh

 

 

[hadoop@masterconf]$ vim core-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://master:9000</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop/tmp</value>

</property>

</configuration>

 

 

 

[hadoop@masterconf]$ vim hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

<property>

<name>dfs.data.dir</name>

<value>/data/hadoop</value>

</property>

<property>

<name>dfs.replication</name>

<value>2</value>

</property>

</configuration>

 

[hadoop@masterconf]$ vim mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>master:9001</value>

</property>

</configuration>

~                     

 

 

[hadoop@masterconf]$ vim masters

master

[hadoop@masterconf]$ vim slaves

slave1

slave2

 

 

在slave節點建立數據目錄

要使用root用戶

[hadoop@slave1 ~]$ su - root

口令：

[root@slave1 ~]# mkdir -p /data/hadoop

[root@slave1 ~]# chownhadoop.hadoop /data/hadoop/

 

 

[root@slave2 ~]# mkdir -p /data/hadoop

[root@slave2 ~]# chownhadoop.hadoop /data/hadoop/

[root@slave2 ~]#

 

拷貝

[hadoop@master ~]$ scp -r hadoop-1.2.1/ slave1:~

 

在slave1 建立軟鏈接

[hadoop@slave1 ~]$ ln -s hadoop-1.2.1/ hadoop

 

[hadoop@master ~]$ scp -r hadoop-1.2.1/ slave2:~

 

[hadoop@slave2 ~]$ ln -s hadoop-1.2.1/ hadoop

[hadoop@slave2 ~]$ ll

總計 8

lrwxrwxrwx  1hadoophadoop   13 12-01 13:51 hadoop -> hadoop-1.2.1/

drwxr-xr-x 11 hadoophadoop 4096 12-01 13:51 hadoop-1.2.1

drwxrwxr-x  2hadoophadoop 4096 12-01 13:06 installer

 

 

格式化

[hadoop@master ~]$ hadoopnamenode -format

14/12/01 13:49:36 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = master/192.168.1.100

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 1.2.1

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG:   java = 1.6.0_27

************************************************************/

14/12/01 13:49:37 INFO util.GSet: Computing capacity for map BlocksMap

14/12/01 13:49:37 INFO util.GSet: VM type       = 32-bit

14/12/01 13:49:37 INFO util.GSet: 2.0% max memory = 101384192

14/12/01 13:49:37 INFO util.GSet: capacity      = 2^19 = 524288 entries

14/12/01 13:49:37 INFO util.GSet: recommended=524288, actual=524288

14/12/01 13:49:37 INFO namenode.FSNamesystem: fsOwner=hadoop

14/12/01 13:49:37 INFO namenode.FSNamesystem: supergroup=supergroup

14/12/01 13:49:37 INFO namenode.FSNamesystem: isPermissionEnabled=true

14/12/01 13:49:37 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

14/12/01 13:49:37 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

14/12/01 13:49:37 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0

14/12/01 13:49:37 INFO namenode.NameNode: Caching file names occuring more than 10 times

14/12/01 13:49:37 INFO common.Storage: Image file /home/hadoop/tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds.

14/12/01 13:49:38 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/hadoop/tmp/dfs/name/current/edits

14/12/01 13:49:38 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/hadoop/tmp/dfs/name/current/edits

14/12/01 13:49:38 INFO common.Storage: Storage directory /home/hadoop/tmp/dfs/name has been successfully formatted.

14/12/01 13:49:38 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at master/192.168.1.100

************************************************************/

 

 

啓動集羣

[hadoop@master ~]$ start-all.sh

startingnamenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-namenode-master.out

slave2: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-slave2.out

slave1: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-slave1.out

The authenticity of host 'master (192.168.1.100)' can't be established.

RSA key fingerprint is 61:e5:be:d1:92:41:b4:22:4e:88:ff:b1:b1:a1:64:bb.

Are you sure you want to continue connecting (yes/no)? yes

master: Warning: Permanently added 'master,192.168.1.100' (RSA) to the list of known hosts.

master: starting secondarynamenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-master.out

startingjobtracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-master.out

slave1: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-slave1.out

slave2: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-slave2.out

[hadoop@master ~]$ jps

15276 NameNode

15630 Jps

15447 SecondaryNameNode

15519 JobTracker

 

 

[hadoop@slave1 ~]$ jps

15216 DataNode

15390 Jps

15312 TaskTracker

[hadoop@slave1 ~]$

 

 

[hadoop@slave2 ~]$ jps

15244 TaskTracker

15322 Jps

15149 DataNode

[hadoop@slave2 ~]$

 

 

 

 

驗證

建立input目錄而後

[hadoop@master input]$ echo "hello world">test2.txt

[hadoop@master input]$ echo "hello hadoop" > test1.txt

[hadoop@master input]$ ls

test1.txt  test2.txt

 

 

hadoop@master input]$hadoopfs -ls

ls: Cannot access .: No such file or directory.

[hadoop@master input]$ hadoopfs -mkdir input

 [hadoop@master input]$ hadoopfs -ls

Found 1 items

drwxr-xr-x   - hadoopsupergroup          0 2014-12-01 13:59 /user/hadoop/input

 

 

[hadoop@master input]$ hadoopfs -put test1.txt input

[hadoop@master input]$ hadoopfs -ls input

Found 1 items

-rw-r--r--   2 hadoopsupergroup         13 2014-12-01 14:00 /user/hadoop/input/test1.txt

 

 

 

測試MapReduce

[hadoop@master input]$ cd

[hadoop@master ~]$ cdhadoop

[hadoop@masterhadoop]$ hadoop jar hadoop-examples-1.2.1.jar wordcount input output

 

14/12/01 14:08:17 INFO input.FileInputFormat: Total input paths to process : 2

14/12/01 14:08:17 INFO util.NativeCodeLoader: Loaded the native-hadoop library

14/12/01 14:08:17 WARN snappy.LoadSnappy: Snappy native library not loaded

14/12/01 14:08:17 INFO mapred.JobClient: Running job: job_201412011351_0001

14/12/01 14:08:18 INFO mapred.JobClient:  map 0% reduce 0%

14/12/01 14:08:34 INFO mapred.JobClient:  map 50% reduce 0%

14/12/01 14:08:37 INFO mapred.JobClient:  map 100% reduce 0%

14/12/01 14:08:43 INFO mapred.JobClient:  map 100% reduce 100%

14/12/01 14:08:44 INFO mapred.JobClient: Job complete: job_201412011351_0001

14/12/01 14:08:44 INFO mapred.JobClient: Counters: 29

14/12/01 14:08:44 INFO mapred.JobClient:   Job Counters

14/12/01 14:08:44 INFO mapred.JobClient:     Launched reduce tasks=1

14/12/01 14:08:44 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=30756

14/12/01 14:08:44 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0

14/12/01 14:08:44 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0

14/12/01 14:08:44 INFO mapred.JobClient:     Launched map tasks=2

14/12/01 14:08:44 INFO mapred.JobClient:     Data-local map tasks=2

14/12/01 14:08:44 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=8829

14/12/01 14:08:44 INFO mapred.JobClient:   File Output Format Counters

14/12/01 14:08:45 INFO mapred.JobClient:     Bytes Written=25

14/12/01 14:08:45 INFO mapred.JobClient:   FileSystemCounters

14/12/01 14:08:45 INFO mapred.JobClient:     FILE_BYTES_READ=55

14/12/01 14:08:45 INFO mapred.JobClient:     HDFS_BYTES_READ=247

14/12/01 14:08:45 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=165602

14/12/01 14:08:45 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=25

14/12/01 14:08:45 INFO mapred.JobClient:   File Input Format Counters

14/12/01 14:08:45 INFO mapred.JobClient:     Bytes Read=25

14/12/01 14:08:45 INFO mapred.JobClient:   Map-Reduce Framework

14/12/01 14:08:45 INFO mapred.JobClient:     Map output materialized bytes=61

14/12/01 14:08:45 INFO mapred.JobClient:     Map input records=2

14/12/01 14:08:45 INFO mapred.JobClient:     Reduce shuffle bytes=61

14/12/01 14:08:45 INFO mapred.JobClient:     Spilled Records=8

14/12/01 14:08:45 INFO mapred.JobClient:     Map output bytes=41

14/12/01 14:08:45 INFO mapred.JobClient:     CPU time spent (ms)=6040

14/12/01 14:08:45 INFO mapred.JobClient:     Total committed heap usage (bytes)=247537664

14/12/01 14:08:45 INFO mapred.JobClient:     Combine input records=4

14/12/01 14:08:45 INFO mapred.JobClient:     SPLIT_RAW_BYTES=222

14/12/01 14:08:45 INFO mapred.JobClient:     Reduce input records=4

14/12/01 14:08:45 INFO mapred.JobClient:     Reduce input groups=3

14/12/01 14:08:45 INFO mapred.JobClient:     Combine output records=4

14/12/01 14:08:45 INFO mapred.JobClient:     Physical memory (bytes) snapshot=327409664

14/12/01 14:08:45 INFO mapred.JobClient:     Reduce output records=3

14/12/01 14:08:45 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=1123352576

14/12/01 14:08:45 INFO mapred.JobClient:     Map output records=4

[hadoop@masterhadoop]$ hadoopfs -ls

Found 2 items

drwxr-xr-x   - hadoopsupergroup          0 2014-12-01 14:06 /user/hadoop/input

drwxr-xr-x   - hadoopsupergroup          0 2014-12-01 14:08 /user/hadoop/output

 

[hadoop@masterhadoop]$ hadoopfs -ls output

Found 3 items

-rw-r--r--   2 hadoopsupergroup          0 2014-12-01 14:08 /user/hadoop/output/_SUCCESS

drwxr-xr-x   - hadoopsupergroup          0 2014-12-01 14:08 /user/hadoop/output/_logs

-rw-r--r--   2 hadoopsupergroup         25 2014-12-01 14:08 /user/hadoop/output/part-r-00000

 

[hadoop@masterhadoop]$ hadoopfs -cat output/part-r-00000

hadoop     1

hello          2

world        1

[hadoop@masterhadoop]$

 

 

數據其實就在slave1和slave2 咱們指定的位置

[hadoop@slave2 ~]$ cd /data/hadoop/

[hadoop@slave2 hadoop]$ ls

blocksBeingWritten  current  detach  in_use.lock  storage  tmp

[hadoop@slave2 hadoop]$ ls -lR

.:

總計 20

drwxrwxr-x 2 hadoophadoop 4096 12-01 14:11 blocksBeingWritten

drwxrwxr-x 2 hadoophadoop 4096 12-01 14:11 current

drwxrwxr-x 2 hadoophadoop 4096 12-01 13:54 detach

-rw-rw-r-- 1 hadoophadoop    0 12-01 13:54 in_use.lock

-rw-rw-r-- 1 hadoophadoop  157 12-01 13:54 storage

drwxrwxr-x 2 hadoophadoop 4096 12-01 13:54 tmp

 

./blocksBeingWritten:

總計 0

 

./current:

總計 116

-rw-rw-r-- 1 hadoophadoop    25 12-01 14:11 blk_1479755060734035889

-rw-rw-r-- 1 hadoophadoop    11 12-01 14:11 blk_1479755060734035889_1012.meta

-rw-rw-r-- 1 hadoophadoop    13 12-01 14:03 blk_-3339809334521550603

-rw-rw-r-- 1 hadoophadoop    11 12-01 14:03 blk_-3339809334521550603_1002.meta

-rw-rw-r-- 1 hadoophadoop 16409 12-01 14:11 blk_-5133472030735814708

-rw-rw-r-- 1 hadoophadoop   139 12-01 14:11 blk_-5133472030735814708_1013.meta

-rw-rw-r-- 1 hadoophadoop     4 12-01 13:54 blk_-6305447893317321698

-rw-rw-r-- 1 hadoophadoop    11 12-01 13:54 blk_-6305447893317321698_1001.meta

-rw-rw-r-- 1 hadoophadoop 47530 12-01 14:11 blk_-6888368855936251295

-rw-rw-r-- 1 hadoophadoop   379 12-01 14:11 blk_-6888368855936251295_1011.meta

-rw-rw-r-- 1 hadoophadoop    12 12-01 14:09 blk_-7643878679389011546

-rw-rw-r-- 1 hadoophadoop    11 12-01 14:09 blk_-7643878679389011546_1003.meta

-rw-rw-r-- 1 hadoophadoop   678 12-01 14:11 dncp_block_verification.log.curr

-rw-rw-r-- 1 hadoophadoop   157 12-01 13:54 VERSION

 

./detach:

總計 0

 

./tmp:

總計 0

這裏的文件都是二進制，看不了的。

在slave2上也是這樣。

這是由於在前面配置了

 

[hadoop@masterconf]$ vim hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

<property>

<name>dfs.data.dir</name>

<value>/data/hadoop</value>

</property>

<property>

<name>dfs.replication</name>

<value>2</value>

</property>

</configuration>