Sqoop2 環境搭建

 

原文地址：http://www.cnblogs.com/luogankun/p/4209017.html

正在準備作Spark SQL external data source與關係型數據庫交互的部分，參考下Sqoop2是如何操做關係型數據庫的。

 

下載地址：http://archive.cloudera.com/cdh5/cdh/5/

下載並安裝：

cd /home/spark/app/
wget http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.3-cdh5.0.0.tar.gz
tar -zxvf sqoop2-1.99.3-cdh5.0.0.tar.gz
cd sqoop2-1.99.3-cdh5.0.0

注：因爲本地採用的hadoop是2.3.0-cdh5.0.0版本，故本案例中使用的是cdh5.0.0對應的sqoop版本；

 

添加sqoop2到系統環境變量中：

export SQOOP2_HOME=/home/spark/app/sqoop2-1.99.3-cdh5.0.0
export CATALINA_BASE=$SQOOP2_HOME/server
export PATH=.:$SQOOP2_HOME/bin:$PATH

拷貝mysql驅動包到$SQOOP2_HOME/server/lib下

cp mysql-connector-java-5.1.10-bin.jar $SQOOP2_HOME/server/lib/ 

修改配置文件：

$SQOOP2_HOME/server/conf/sqoop.properties

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/spark/app/hadoop-2.3.0-cdh5.0.0/etc/hadoop

$SQOOP2_HOME/server/conf/catalina.properties

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/common/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/common/lib/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/hdfs/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/hdfs/lib/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/mapreduce/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/mapreduce/lib/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/tools/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/tools/lib/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/yarn/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/yarn/lib/*.jar

若是想修改tomcat的端口號等信息能夠在$SQOOP2_HOME/server/conf/server.xml中進行設置；

啓停sqoop-server:

$SQOOP2_HOME/bin/sqoop.sh server start
$SQOOP2_HOME/bin/sqoop.sh server stop
或者
$SQOOP2_HOME/bin/sqoop2-server start
$SQOOP2_HOME/bin/sqoop2-server stop

 

驗證是否啓動成功：

方式一：jps查看進程: Bootstrap 
方式二： http://hadoop000:12000/sqoop/version

方式三：wget -qO - hadoop000:12000/sqoop/version

啓動sqoop客戶端：

$SQOOP2_HOME/bin/sqoop.sh client
或者
$SQOOP2_HOME/bin/sqoop2-shell

 

爲客戶端配置服務器：

sqoop:000> set server --host hadoop000 --port 12000 --webapp sqoop

查看服務器端信息：

sqoop:000> show server --all

 ZOOM 雲視頻會議網站：http://www.zoomonline.cn/