Sqoop2的安裝與使用

1.下載安裝

下載地址: mirrors.hust.edu.cn/apache/sqoo…

解壓: tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz

2.配置依賴:

在目錄server/conf建立文件catalina.properties.

文件內容:

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/hadoop3/share/hadoop/common/*.jar,/usr/hadoop3/share/hadoop/common/lib/*.jar,/usr/hadoop3/share/hadoop/hdfs/*.jar,/usr/hadoop3/share/hadoop/hdfs/lib/*.jar,/usr/hadoop3/share/hadoop/mapreduce/*.jar,/usr/hadoop3/share/hadoop/mapreduce/lib/*.jar,/usr/hadoop3/share/hadoop/tools/lib/*.jar,/usr/hadoop3/share/hadoop/yarn/*.jar,/usr/hadoop3/share/hadoop/yarn/lib/*.jar,/usr/hadoop3/share/hadoop/httpfs/tomcat/lib/*.jar

配置環境變量:

#必定要配置,不然會找不到jar

$HADOOP_HOME/share/hadoop/common

$HADOOP_HOME/share/hadoop/hdfs

$HADOOP_HOME/share/hadoop/mapreduce

$HADOOP_HOME/share/hadoop/yarn

export SQOOP_HOME=/usr/sqoop

export LOGDIR=$SQOOP_HOME/logs/

export CATALINA_BASE=/usr/sqoop/server

export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/extra

export PATH=$SQOOP_HOME/bin:$PATH

conf/sqoop.properties配置文件修改:

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/hadoop3/etc/hadoop/

conf/sqoop.properties 使用mysql持久化數據遷移信息

org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.mysql.MySqlRepositoryHandler

org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED

org.apache.sqoop.repository.jdbc.maximum.connections=10

org.apache.sqoop.repository.jdbc.url=jdbc:mysql://192.168.172.103:3306/sqoop2

org.apache.sqoop.repository.jdbc.driver=com.mysql.jdbc.Driver

org.apache.sqoop.repository.jdbc.user=root

org.apache.sqoop.repository.jdbc.password=123456

hadop要開啓history

<property>

<name>mapreduce.jobhistory.address</name>

<value>ngmm-hadoop-master:10020</value>

</property>

<property>

<name>yarn.app.mapreduce.am.env</name>

<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

</property>

<property>

<name>mapreduce.map.env</name>

<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

</property>

<property>

<name>mapreduce.reduce.env</name>

<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

</property>

3.下載jdbc驅動:

下載 central.maven.org/maven2/mysq… 到server/lib下

4.啓動服務

./sqoop.sh server start

./sqoop.sh client

verify

檢查數據:show version -all

sqoop2-tool upgrade

sqoop2-tool verify

日誌

set option --name verbose --value true

建立任務

create job -f "mysql-link" -t "hdfs-link」

執行任務

start job -n job1