基於Hadoop2.5.0的集羣搭建

時間 2019-11-30

標籤基於 hadoop2.5.0 hadoop 集羣搭建欄目 Hadoop 简体版

原文原文鏈接

http://download.csdn.net/download/yameing/8011891php

1、規劃

1. 準備安裝包

JDK:http://download.oracle.com/otn-pub/java/jdk/7u67-b01/jdk-7u67-linux-x64.tar.gzhtml

Hadoop:http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.5.0/hadoop-2.5.0.tar.gzjava

Hive:http://apache.fayea.com/apache-mirror/hive/hive-0.13.1/apache-hive-0.13.1-bin.tar.gznode

ZK:http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gzpython

HBase:http://apache.fayea.com/apache-mirror/hbase/hbase-0.98.5/hbase-0.98.5-hadoop2-bin.tar.gzmysql

MySql:http://ftp.nchu.edu.tw/Unix/Database/MySQL/Downloads/MySQL-5.6/mysql-5.6.12-linux-glibc2.5-x86_64.tar.gzlinux

MysqlConnector:http://ftp.nchu.edu.tw/Unix/Database/MySQL/Downloads/Connector-J/mysql-connector-java-5.1.25.zip算法

Sqoop：complete based on sqoop-1.4.5 and current hadoop versionsql

http://mirror.bit.edu.cn/apache/sqoop/1.4.5/sqoop-1.4.5.tar.gzshell

2. 環境規劃

類型	名稱	配置	IP	安裝內容
Hadoop集羣主節點	mycluster1	16核32G2T	192.168.2.92	Hadoop
Hadoop集羣主節點	mycluster2	16核32G6T	192.168.2.88
Hadoop集羣從節點	mycluster3	4核8G250G	192.168.1.84
	mycluster4	4核8G250G	192.168.1.85
	mycluster5	4核8G250G	192.168.1.86
	mycluster6	4核8G250G	192.168.1.87
	mycluster7	4核8G250G	192.168.1.88
	mycluster8	4核8G250G	192.168.1.89
	mycluster9	4核8G250G	192.168.1.90
	mycluster10	4核8G250G	192.168.1.91
分佈式應用	mycluster11	4核8G250G	192.168.1.92	Hive Sqoop MySQL

2、安裝

1. 環境配置

a) 基本配置

1. 配置各機器的機器名

vi /etc/sysconfig/network

vi /etc/hosts

hostname mycluster*

2. 全部節點關閉防火牆

service iptables stop

3. 將全部機器名配置到各機器中

vi /etc/hosts

#127.0.0.1 localhost localhost.localdomain mycluster5

#::1 localhost localhost.localdomain mycluster5

#這裏註釋掉關於localhost的配置，詳情查看遇到的問題

#由於zookeeper要求配置localhost，因此這裏關於本地地址的配置改成以下：

127.0.0.1 localhost localhost.localdomain

::1 localhost localhost.localdomain

192.168.2.92 mycluster1

192.168.2.88 mycluster2

192.168.1.84 mycluster3

192.168.1.85 mycluster4

192.168.1.86 mycluster5

192.168.1.87 mycluster6

192.168.1.88 mycluster7

192.168.1.89 mycluster8

192.168.1.90 mycluster9

192.168.1.91 mycluster10

192.168.1.92 mycluster11

4. 保證各機器間時間差不超過2分鐘

date

date -s "2014-09-05 23:38:00"

ntpdate time.windows.com

clock -w

查看

修改

若連通互聯網，可同步微軟

寫入BIOS

b) 打通SSH

1. 在各機器建立mycluster用戶。之後的命令都在mycluster下執行。

groupadd mycluster

useradd -g mycluster -G root -d /home/mycluster mycluster

passwd qcpass@lh

2. 在各Slave建立ssh目錄。

mkdir /home/mycluster/.ssh

chmod 700 /home/mycluster/.ssh

目錄權限必須是700，不然沒法ssh登陸

3. 登陸Master，生成SSH公鑰、私鑰，複製公鑰到各Slave。

ssh-keygen -t rsa

cd /home/mycluster/.ssh

cp id_rsa.pub authorized_keys

scp authorized_keys mycluster@mycluster*:/home/mycluster/.ssh

c) 安裝JDK1.7

1. 登陸root用戶安裝JDK到/usr/java目錄下。

tar -zxvf jdk-7u67-linux-x64.gz

ln -s jdk1.7.0_67 jdk

2. 配置環境變量。

vi /etc/profile

vi .bashrc

全部用戶可見的方式

當前用戶可見的方式

export JAVA_HOME=/home/mycluster/jdk

export CLASSPATH=.

export PATH=$JAVA_HOME/bin:$PATH

source /etc/profile

env | grep JAVA_HOME

生效

驗證

2. Hadoop2.5.0安裝

a) 安裝與配置

tar zxvf hadoop-2.5.0.tar.gz

cd hadoop-2.5.0/etc/hadoop/

vi hadoop-env.sh

export JAVA_HOME=/home/mycluster/jdk

vi core-site.xml

<name>fs.defaultFS</name>

</property>

<name>fs.trash.interval</name>

</property>

vi hdfs-site.xml

<name>dfs.namenode.name.dir</name>

<value>/home/mycluster/data/dfs_namenode_name_dir</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/home/mycluster/data/dfs_datanode_data_dir</value>

</property>

<name>dfs.replication</name>

</property>

<!-- 抽查了部分規劃中的slave節點，發現其中最大的一塊存儲都是195G，且僅使用了1%，爲/home目錄所持有

-->

vi mapred-site.xml (yarn必須小寫)

<name>mapreduce.framework.name</name>

</property>

vi yarn-site.xml

<name>yarn.resourcemanager.hostname</name>

<value>mycluster1</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

vi slaves

mycluster3

mycluster4

mycluster5

mycluster6

mycluster7

mycluster8

mycluster9

mycluster10

3. 從Master複製Hadoop目錄到各Slave。

scp -r /home/mycluster/hadoop-2.5.0 mycluster@mycluster3:/home/mycluster

b) 啓動與測試

1. 登陸Master，配置Hadoop環境變量。

vi /home/mycluster/.bash_profile

export HADOOP_HOME=/home/mycluster/hadoop-2.5.0

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

source /home/mycluster/.bash_profile

env | grep HADOOP_HOME

2. 格式化HDFS，啓動Hadoop，測試。

hadoop namenode -format

start-dfs.sh

start-yarn.sh

jps

hadoop jar hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar pi 2 10000

3. 編寫自定義MR程序測試。

(暫不提供)

3. 安裝MySQL

a) 安裝與配置

這裏安裝的是MySQL綠色版，好處是全過程可控，固然圖方即可以安裝RPM。

1. 安裝tar.gz

tar zxvf mysql-5.6.12-linux-glibc2.5-i686.tar.gz

mv mysql-5.6.12-linux-glibc2.5-i686 /usr/local/mysql

2. 建立組、用戶，受權

groupadd mycluster

useradd -g mycluster -G root -d /home/mycluster mycluster

passwd qcpass@lh

cd /usr/local/mysql

chown -R mycluster .

chgrp -R mycluster .

scripts/mysql_install_db --user=mycluster

chown -R root .

chown -R mycluster data

chmod u+x data/ibdata1

mv mycluster11.err mycluster11.err_

3. 配置文件

mv /etc/my.cnf /etc/my.cnf_

cp support-files/my-default.cnf /etc/my.cnf

vi /etc/my.cnf

避免之前安裝過MySQL

[mysqld]

basedir=/usr/local/mysql

datadir=/usr/local/mysql/data

character-set-server=utf8

lower_case_table_names=1

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

b) 啓動與測試

1. 啓動

mv /etc/init.d/mysql /etc/init.d/mysql_

cp support-files/mysql.server /etc/init.d/mysql

service mysql start

chkconfig --add mysql

避免之前安裝過MySQL

當即啓動

開機啓動

2. 修改密碼

vi /mycluster/.bash_profile

export PATH=/usr/local/mysql/bin:$PATH

source /mycluster/.bash_profile

mysql -u root -p

mysql> set password = password('root');

root密碼爲空

修改密碼爲root

4. 安裝Hive

a) 安裝與配置

1. 解壓。

tar zxvf apache-hive-0.13.1-bin.tar.gz

echo 'export HIVE_HOME=/home/mycluster/apache-hive-0.13.1-bin' >> /home/mycluster/.bashrc

echo 'export PATH=$HIVE_HOME/bin:$PATH' >> /home/mycluster/.bashrc

2. 在HDFS中建立Hive目錄。

hadoop fs -mkdir /tmp

hadoop fs -mkdir /user/hive/warehouse

hadoop fs -chmod g+w /tmp

hadoop fs -chmod g+w /user/hive/warehouse

3. 建立MySQL數據庫。

create database hive character set latin1;

4. 配置文件。

cd apache-hive-0.13.1-bin/conf

cp hive-default.xml.template hive-site.xml

vi hive-site.xml

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://localhost:3306/hive</value>

</property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

</property>

<name>javax.jdo.option.ConnectionUserName</name>

</property>

<name>javax.jdo.option.ConnectionPassword</name>

</property>

</configuration>

cp mysql-connector-java-5.1.25-bin.jar /home/mycluster/apache-hive-0.13.1-bin/lib/

5. 配置環境變量。

vi /home/hadoop/.bash_profile

export HIVE_HOME=/home/hadoop/hive-0.9.0

export PATH=$HIVE_HOME/bin:$PATH

source /home/hadoop/.bash_profile

b) 啓動與測試

(幾種啓動方式，暫缺)

5. 安裝Sqoop

a) 安裝與配置

1. 安裝tar.gz

tar -xvf sqoop-1.4.5.bin__hadoop-2.5.0.tar.gz

ln -s sqoop-1.4.5.bin__hadoop-2.5.0 sqoop

export SQOOP_HOME=/home/mycluster/sqoop

export PATH=$SQOOP_HOME/bin:$PATH

2. 添加jar

根據須要，添加mysql connector、oracle connector

scp mysql-connector-java-5.1.25-bin.jar mycluster@mycluster11:/home/mycluster/sqoop/lib

scp ojdbc14.jar mycluster@mycluster11:/home/mycluster/sqoop/lib

3. 配置文件

cd /home/mycluster/sqoop/conf

cp sqoop-env-template.sh sqoop-env.sh

vi sqoop-env.sh

export HADOOP_COMMON_HOME=/home/mycluster/hadoop-2.5.0

export HADOOP_MAPRED_HOME=/home/mycluster/hadoop-2.5.0/share/hadoop/mapreduce

export HIVE_HOME=/home/mycluster/apache-hive-0.13.1-bin

b) 啓動與測試

sqoop list-databases --connect jdbc:mysql://localhost:3306/ --username root --password root

6. 安裝ZooKeeper3.4.6

a) 安裝與配置

1. 安裝與配置

tar -zxvf zookeeper-3.4.6.tar.gz

mkdir /home/mycluster/zookeeper-3.4.6/zookeeperdir/logs

cp zookeeper-3.4.6/conf/zoo_sample.cfg zookeeper-3.4.6/conf/zoo.cfg

vi zookeeper-3.4.6/conf/zoo.cfg

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/home/mycluster/zookeeper-3.4.6/zookeeperdir/zookeeper-data

dataLogDir=/home/mycluster/zookeeper-3.4.6/zookeeperdir/logs

clientPort=2181

server.1=mycluster1:2888:3888

server.2=mycluster3:2888:3888

server.3=mycluster4:2888:3888

vi .bashrc

export ZOOKEEPER_HOME=/home/mycluster/zookeeper-3.4.6

export PATH=$ZOOKEEPER_HOME/bin:$PATH

2. 複製ZK目錄到各主機。

scp -r /home/mycluster/zookeeper-3.4.6 mycluster@mycluster3:/home/mycluster

scp -r /home/mycluster/zookeeper-3.4.6 mycluster@mycluster4:/home/mycluster

3. 設置myid

[mycluster@mycluster1 ~]$ echo "1" > /home/mycluster/zookeeper-3.4.6/zookeeperdir/zookeeper-data/myid

[mycluster@mycluster3 ~]$ echo "2" > /home/mycluster/zookeeper-3.4.6/zookeeperdir/zookeeper-data/myid

[mycluster@mycluster4 ~]$ echo "3" > /home/mycluster/zookeeper-3.4.6/zookeeperdir/zookeeper-data/myid

b) 啓動與測試

1. 登陸各機器啓動ZK。

[mycluster@mycluster1 ~]$ zkServer.sh start

[mycluster@mycluster3 ~]$ zkServer.sh start

[mycluster@mycluster4 ~]$ zkServer.sh start

2. 查看啓動狀態。

因爲ZooKeeper集羣啓動的時候，每一個結點都試圖去鏈接集羣中的其它結點，先啓動的確定連不上後面還沒啓動的，因此日誌前面部分的鏈接異常是能夠忽略的。經過後面部分能夠看到，集羣在選出一個Leader後，最後穩定了。

[mycluster@mycluster1 ~]$ zkServer.sh status

JMX enabled by default

Using config: /home/mycluster/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: follower

[mycluster@mycluster3 ~]$ zkServer.sh status

JMX enabled by default

Using config: /home/mycluster/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: leader

[mycluster@mycluster4 ~]$ zkServer.sh status

JMX enabled by default

Using config: /home/mycluster/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: follower

3. 客戶端測試。

[mycluster@mycluster1 ~]$ zkCli.sh -server mycluster1:2181

[zk: mycluster1:2181(CONNECTED) 0] ls /

[zookeeper]

7. 安裝HBase(未實現)

3、調優(進行中... ...)

1. Hadoop調優

a) HA & Federation

·HA：解決單點故障

·Federation：擴大集羣容量和提升集羣性能

本集羣暫不考慮Federation，由於集羣暫時不會達到很是大的規模。

HA配置：

vi hdfs-site.xml

<name>dfs.nameservices</name>

<value>mycluster</value>

<description>提供服務的NS邏輯名稱，與core-site.xml裏的對應</description>

</property>

<name>dfs.ha.namenodes.mycluster</name>

<value>namenode1,redhat22688</value>

<description>列出該邏輯名稱下的NameNode邏輯名稱</description>

</property>

<name>dfs.namenode.rpc-address.mycluster.namenode1</name>

<value>mycluster1:9000</value>

<description>指定NameNode的RPC位置</description>

</property>

<name>dfs.namenode.http-address.mycluster.namenode1</name>

<value>mycluster1:50070</value>

<description>指定NameNode的Web Server位置</description>

</property>

<name>dfs.namenode.rpc-address.mycluster.redhat22688</name>

<value>redhat22688:9000</value>

<description>指定NameNode的RPC位置</description>

</property>

<name>dfs.namenode.http-address.mycluster.redhat22688</name>

<value>redhat22688:50070</value>

<description>指定NameNode的Web Server位置</description>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://mycluster3:8485;mycluster4:8485;mycluster5:8485/mycluster</value>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/home/mycluster/data/haqjm/dfs_journalnode_edits_dir</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

<description>指定HA作隔離的方法，缺省是ssh，可設爲shell，稍後詳述</description>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/mycluster/.ssh/id_rsa</value>

</property>

<name>dfs.client.failover.proxy.provider.mycluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

vi core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://mycluster</value>

</property>

# 啓動對應機器上的JN（hdfs-site中配置的）

[mycluster@mycluster3 ~]$ hadoop-2.5.0/sbin/hadoop-daemon.sh start journalnode

[mycluster@mycluster4 ~]$ hadoop-2.5.0/sbin/hadoop-daemon.sh start journalnode

[mycluster@mycluster5 ~]$ hadoop-2.5.0/sbin/hadoop-daemon.sh start journalnode

# 格式化一個NN，並啓動

[mycluster@mycluster1 ~]$ hadoop namenode -format

[mycluster@mycluster1 ~]$ hadoop-daemon.sh start namenode

# 格式化另外一個NN，並啓動

[mycluster@mycluster1 ~]$ scp -r data mycluster@redhat22688:/home/mycluster/

[mycluster@redhat22688 ~]$ hadoop namenode -bootstrapStandby

[mycluster@redhat22688 ~]$ hadoop-daemon.sh start namenode

# 這時候，使用瀏覽器訪問http://116.228.171.104:50070/ 和 http://116.228.171.119:50070/ 。

# 若是可以看到兩個頁面，證實NameNode啓動成功了。這時，兩個NameNode的狀態都是standby。

# 或者使用如下命令

[mycluster@mycluster1 ~]$ hdfs haadmin -getServiceState namenode1

# 轉化active

[mycluster@mycluster1 ~]$ hdfs haadmin -transitionToActive namenode1

# 啓動全部DN

[mycluster@mycluster1 ~]$ hadoop-daemons.sh start datanode

啓用故障自動恢復：

vi hdfs-site.xml

<name>dfs.ha.automatic-failover.enabled</name>

<description>或者false</description>

</property>

vi core-site.xml

<name>ha.zookeeper.quorum</name>

<value>mycluster1:2181,mycluster3:2181,mycluster4:2181</value>

<description>指定用於HA的ZooKeeper集羣機器列表</description>

</property>

<name>ha.zookeeper.session-timeout.ms</name>

<description>指定ZooKeeper超時間隔，單位毫秒</description>

</property>

# 在其中一個NN上執行：

[mycluster@mycluster1 ~]$ hdfs zkfc -formatZK

4、遇到的問題

一、參考文檔

Hadoop :http://hadoop.apache.org/docs/r2.5.1/

Hive :http://hive.apache.org/

ZK :http://zookeeper.apache.org/

Sqoop :http://sqoop.apache.org/docs/1.4.5/index.html

二、Hadoop及各組件版本

三、SSH端口不是默認端口22

若是ssh端口不是默認的22，在etc/hadoop/hadoop-env.sh裏改下。如：

export HADOOP_SSH_OPTS="-p 18921"

四、不一樣節點SSH端口不同

對於hadoop來講，SSH並不是很重要的內容，hadoop中僅僅使用其啓動/關閉集羣，因此Hadoop目前不支持不一樣節點配置不一樣的ssh端口。

方案一：手動一個個節點啓動，能夠不用ssh

方案二：本身寫ssh啓動腳本

方案三：修改ssh配置

方案四：端口轉發（這種作法還不如直接直接使用方案三）

五、Address 192.168.2.92 maps to mycluster1, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!

修改hosts文件，使192.168.2.92與mycluster1能惟一對應起來。

六、WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

系統中的glibc的版本和libhadoop.so須要的版本不一致致使的：

[mycluster@mycluster1 ~]$ ls -l /lib/libc.so.*

lrwxrwxrwx 1 root root 11 Apr 18 2012 /lib/libc.so.6 -> libc-2.5.so

[mycluster@mycluster1 ~]$ file /lib/libc-2.5.so

/lib/libc-2.5.so:ELF 32-bit LSBshared object, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, not stripped

[mycluster@mycluster1 ~]$ file hadoop-2.5.0/lib/native/libhdfs.so.0.0.0

hadoop-2.5.0/lib/native/libhdfs.so.0.0.0:ELF 64-bit LSBshared object, AMD x86-64, version 1 (SYSV), not stripped

解決方案：

一、從新編譯hadoop

二、升級gcc

此警告影響的範圍:

一、壓縮算法

七、執行MR程序時的通訊失敗一：MR_AM啓動Task時網絡失敗

[mycluster@mycluster1 ~]$ hadoop-2.5.0/bin/hadoop jar hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar pi 2 2

Number of Maps = 2

Samples per Map = 2

14/09/19 16:47:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Wrote input for Map #0

Wrote input for Map #1

Starting Job

14/09/19 16:47:47 INFO client.RMProxy: Connecting to ResourceManager at mycluster1/192.168.2.92:8032

14/09/19 16:47:47 INFO input.FileInputFormat: Total input paths to process : 2

14/09/19 16:47:47 INFO mapreduce.JobSubmitter: number of splits:2

14/09/19 16:47:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1411112681877_0004

14/09/19 16:47:48 INFO impl.YarnClientImpl: Submitted application application_1411112681877_0004

14/09/19 16:47:48 INFO mapreduce.Job: The url to track the job: http://mycluster1:8088/proxy/application_1411112681877_0004/

14/09/19 16:47:48 INFO mapreduce.Job: Running job: job_1411112681877_0004

14/09/19 16:48:09 INFO mapreduce.Job: Job job_1411112681877_0004 running in uber mode : false

14/09/19 16:48:09 INFO mapreduce.Job: map 0% reduce 0%

#這裏應該是MR_AM啓動Task（詳細信息查看日誌）

14/09/19 16:48:09 INFO mapreduce.Job: Job job_1411112681877_0004 failed with state FAILED due to: Application application_1411112681877_0004 failed 2 times due to Error launching appattempt_1411112681877_0004_000002. Got exception: java.net.ConnectException: Call From mycluster1/192.168.2.92 to localhost:59163 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

... 9 more

. Failing the application.

14/09/19 16:48:09 INFO mapreduce.Job: Counters: 0

Job Finished in 22.193 seconds

# Job異常退出，無結果文件，致使如下錯誤（這個可有可無）

java.io.FileNotFoundException: File does not exist: hdfs://192.168.2.92:9100/user/mycluster/QuasiMonteCarlo_1411116465638_1171059364/out/reduce-out

解決方案：

註釋掉hosts文件中，關於localhost的配置

八、MySQL驅動包版本

(參考：http://dev.mysql.com/doc/connector-j/en/connector-j-versions.html)

九、配置NFS

服務器端：

rpm -qa | grep nfs

yum install nfs-utils rpcbind #非centos6可能不是這名字

mkdir /home/mycluster_nfs

vi /etc/exports

# 將NFS Server 的/home/mycluster_nfs/ 共享給192.168.2.88/92，權限讀寫。

/home/mycluster_nfs 192.168.2.88(rw)

/home/mycluster_nfs 192.168.2.92(rw)

service rpcbind start

service nfs start

exportfs

showmount -e #默認查看本身共享的服務，前提是要DNS能解析本身，否則容易報錯

showmount -a #顯示已經與客戶端鏈接上的目錄信息

chmod 777 -R /home/mycluster_nfs/

客戶端：

showmount -e mycluster11 #查詢NFS的共享狀態

mkdir /home/mycluster_nfs

mount mycluster11:/home/mycluster_nfs /home/mycluster_nfs

十、zkService.sh status報錯

報錯信息：

[mycluster@mycluster4 ~]$ zkServer.sh status

JMX enabled by default

Using config: /home/mycluster/zookeeper-3.4.6/bin/../conf/zoo.cfg

Error contacting service. It is probably not running.

網上找到三種狀況：

1. 沒有裝nc ：yum install nc

2.修改zkService.sh

打開zkServer.sh，找到

STAT=`echo stat | nc localhost $(grep clientPort "$ZOOCFG" | sed -e 's/.*=//') 2> /dev/null| grep Mode`

這行，加上或去掉-q 1（數字1而非字母l）便可。

3./etc/hosts裏面沒有配置localhost

十一、編譯Sqoop

Complit sqoop 1.4.5 for hadoop 2.5.0

--編譯前準備：看了一下 README.txt文件，須要如下軟件包：

Additionally, building the documentation requires these tools:

* asciidoc

* make

* python 2.5+

* xmlto

* tar

* gzip

yum -y install ant

yum -y install asciidoc

yum -y install make

yum -y install xmlto

yum -y install tar

yum -y install gzip

-- python本身去安裝

-----------------------------------------------------------------------------------------------------------------------------

--第一步：解壓 sqoop-1.4.5.tar.gz 文件到 /opt/software目錄下(在該目錄下將生成 sqoop-1.4.5 文件夾)

cd /opt/software

tar -xvf sqoop-1.4.5.tar.gz

-----------------------------------------------------------------------------------------------------------------------------

--第二步：cd 到 sqoop-1.4.5 文件夾, 修改build.xml文件中指定的hadoop版本爲2.5.0

cd /opt/software/sqoop-1.4.5

vi build.xml

<then>

</then>

</elseif>

--第三步：運行ant package

[root@funshion-hadoop194 sqoop-1.4.5]# ant package

...

[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

BUILD FAILED

/opt/software/sqoop-1.4.5/build.xml:1282: impossible to resolve dependencies:

resolve failed - see output for details

Total time: 27 seconds

[ivy:resolve] com.google.protobuf#protobuf-java;2.5.0 by [com.google.protobuf#protobuf-java;2.5.0] in [hadoop200]

---------------------------------------------------------------------

| | modules || artifacts |

| conf | number| search|dwnlded|evicted|| number|dwnlded|

---------------------------------------------------------------------

| hadoop200 | 154 | 59 | 58 | 37 || 120 | 48 |

---------------------------------------------------------------------

[ivy:resolve]

[ivy:resolve] :: problems summary ::

[ivy:resolve] :::: WARNINGS

[ivy:resolve] [FAILED ] org.mortbay.jetty#jetty;6.1.26!jetty.zip: (0ms)

[ivy:resolve] ==== fs: tried

[ivy:resolve] /root/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip

[ivy:resolve] ==== apache-snapshot: tried

[ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip

[ivy:resolve] ==== datanucleus: tried

[ivy:resolve] http://www.datanucleus.org/downloads/maven2/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip

[ivy:resolve] ==== cloudera-releases: tried

[ivy:resolve] https://repository.cloudera.com/content/repositories/releases/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip

[ivy:resolve] ==== cloudera-staging: tried

[ivy:resolve] https://repository.cloudera.com/content/repositories/staging/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip

[ivy:resolve] ==== maven2: tried

[ivy:resolve] http://repo1.maven.org/maven2/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip

[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::

[ivy:resolve] :: FAILED DOWNLOADS ::

[ivy:resolve] :: ^ see resolution messages for details ^ ::

[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::

[ivy:resolve] :: org.mortbay.jetty#jetty;6.1.26!jetty.zip

[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::

[ivy:resolve]

[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

[ivy:resolve] io.netty#netty;3.4.0.Final by [io.netty#netty;3.6.2.Final] in [hadoop200test]

[ivy:resolve] asm#asm;[3.0, 4.0) by [asm#asm;3.1] in [hadoop200test]

[ivy:resolve] asm#asm;3.1 by [asm#asm;3.2] in [hadoop200test]

[ivy:resolve] com.google.protobuf#protobuf-java;2.5.0 by [com.google.protobuf#protobuf-java;2.5.0] in [hadoop200test]

---------------------------------------------------------------------

| | modules || artifacts |

| conf | number| search|dwnlded|evicted|| number|dwnlded|

---------------------------------------------------------------------

| hadoop200test | 156 | 0 | 0 | 38 || 121 | 0 |

---------------------------------------------------------------------

--錯誤1（如上）解決方法：單獨下載 jetty-6.1.26.zip 文件到 /root/.m2/repository/org/mortbay/jetty/jetty/6.1.26/目錄下，解決。

------------------------------------------------------------------------------

[ivy:resolve] com.google.protobuf#protobuf-java;2.5.0 by [com.google.protobuf#protobuf-java;2.5.0] in [hadoop200test]

---------------------------------------------------------------------

| | modules || artifacts |

| conf | number| search|dwnlded|evicted|| number|dwnlded|

---------------------------------------------------------------------

| hadoop200test | 156 | 2 | 2 | 38 || 121 | 2 |

---------------------------------------------------------------------

ivy-retrieve-hadoop-test:

[ivy:retrieve] :: retrieving :: com.cloudera.sqoop#sqoop [sync]

[ivy:retrieve] confs: [hadoop200test]

[ivy:retrieve] 121 artifacts copied, 0 already retrieved (113206kB/376ms)

compile-test:

[mkdir] Created dir: /opt/software/sqoop-1.4.5/build/test/classes

[mkdir] Created dir: /opt/software/sqoop-1.4.5/build/test/extraconf

[javac] Compiling 169 source files to /opt/software/sqoop-1.4.5/build/test/classes

[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.6

[javac] /opt/software/sqoop-1.4.5/src/test/org/apache/sqoop/TestExportUsingProcedure.java:244: error: method repeat in class StringUtils cannot be applied to given types;

[javac] sql.append(StringUtils.repeat("?", ", ",

[javac] ^

[javac] required: String,int

[javac] found: String,String,int

[javac] reason: actual and formal argument lists differ in length

[javac] Note: Some input files use or override a deprecated API.

[javac] Note: Recompile with -Xlint:deprecation for details.

[javac] Note: Some input files use unchecked or unsafe operations.

[javac] Note: Recompile with -Xlint:unchecked for details.

[javac] 1 error

[javac] 1 warning

BUILD FAILED