超詳細！CDH版-Hadoop_Hive_Impala單機環境搭建純離線安裝

CDH-Hadoop_Hive_Impala單機環境搭建純離線安裝

環境介紹及安裝包下載地址

Centos6.5

版本選擇：cdh5.14.0-src.tar.gz  CDH5 – 5.14.0

Hadoop安裝包下載地址

http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.14.0.tar.gz

Hive安裝包下載地址

http://archive.cloudera.com/cdh5/cdh/5/hive-1.1.0-cdh5.14.0.tar.gz

impala安裝包下載地址

http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.14.0/RPMS/x86_64/impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.14.0/RPMS/x86_64/impala-catalog-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.14.0/RPMS/x86_64/impala-debuginfo-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.14.0/RPMS/x86_64/impala-server-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.14.0/RPMS/x86_64/impala-shell-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.14.0/RPMS/x86_64/impala-state-store-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm  

http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.14.0/RPMS/x86_64/impala-udf-devel-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

 

參考文檔

https://docs.cloudera.com/documentation/enterprise/5-12-x/topics/cdh_qs_cdh5_pseudo.html#topic_3

https://docs.cloudera.com/documentation/index.html

1.安裝JDK

上傳rpm包

rpm -ivh jdk-8u221-linux-x64.rpm

 

默認安裝在/usr/java中

路徑爲/usr/java/jdk1.8.0_221-amd64

 

配置環境變量

vim /etc/profile

#添加以下內容

JAVA_HOME=/usr/java/jdk1.8.0_221-amd64

JRE_HOME=/usr/java/jdk1.8.0_221-amd64/jre

PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

CLASSPATH=:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

export JAVA_HOME JRE_HOME PATH CLASSPATH

 

[root@sunny conf]# java -version

java version "1.8.0_221"

Java(TM) SE Runtime Environment (build 1.8.0_221-b11)

Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

2.安裝Hadoop

2.1指定Hadoop安裝目錄/data/CDH，進行解壓

cd /data/CDH

tar -zvxf /data/install_pkg_CDH/hadoop-2.6.0-cdh5.14.0.tar.gz

 

hadoop核心配置文件位置：

/data/CDH/hadoop-2.6.0-cdh5.14.0/etc/hadoop/

 

2.2配置Hadoop環境變量

vi /etc/profile

 

#在末尾加入配置：

JAVA_HOME=/usr/java/jdk1.8.0_221-amd64

JRE_HOME=/usr/java/jdk1.8.0_221-amd64/jre

PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

CLASSPATH=:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

export JAVA_HOME JRE_HOME PATH CLASSPATH

export HADOOP_HOME=/data/CDH/hadoop-2.6.0-cdh5.14.0

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

export HADOOP_OPTS="-Djava.library.path=HADOOP_HOME/lib"

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

 

source /etc/profile #使得配置生效

 

2.3先在hadoop安裝目錄下建立數據目錄(用於配置: core-site.xml)

cd /data/CDH/hadoop-2.6.0-cdh5.14.0

mkdir data

 

2.4配置core-site.xml

vi   /data/CDH/hadoop-2.6.0-cdh5.14.0/etc/hadoop/core-site.xml

 

#添加以下配置：

<property>

    <name>fs.defaultFS</name>

    <!-- 這裏填的是你本身的ip，端口默認-->

    <value>hdfs://172.20.71.66:9000</value>

</property>

<property>

    <name>hadoop.tmp.dir</name>

    <!-- 這裏填的是你自定義的hadoop工做的目錄，端口默認-->

    <value>/data/CDH/hadoop-2.6.0-cdh5.14.0/data</value>

</property>

 

<property>

    <name>hadoop.native.lib</name>

    <value>false</value>

    <description>Should native hadoop libraries, if present, be used.

    </description>

</property>

 

2.5配置hdfs-site.xml

vi      /data/CDH/hadoop-2.6.0-cdh5.14.0/etc/hadoop/hdfs-site.xml

 

#添加以下配置：

<property>

         <name>dfs.replication</name>

         <value>1</value>

</property>

<property>

　　　　<name>dfs.namenode.secondary.http-address</name>

　　　　<value>172.20.71.66:50090</value>

</property>

<property>

　　　　<name>dfs.permissions.enabled</name>

　　　　<value>false</value>

</property>

 

2.6配置mapres-site.xml

cd     /data/CDH/hadoop-2.6.0-cdh5.14.0/etc/hadoop

cp mapred-site.xml.template  ./mapred-site.xml  #複製默認文件配置命名爲mapred-site.xml

vi      /data/CDH/hadoop-2.6.0-cdh5.14.0/etc/hadoop/mapred-site.xml

 

#增長以下配置:

<property>

    <name>mapreduce.framework.name</name>

    <value>yarn</value>

</property>

 

2.7配置yarn-site.xml

vi      /data/CDH/hadoop-2.6.0-cdh5.14.0/etc/hadoop/yarn-site.xml 

 

#添加以下配置：

<property>

    <name>yarn.resourcemanager.hostname</name>

    <value>172.20.71.66</value>

</property>

<property>

    <name>yarn.nodemanager.aux-services</name>

    <value>mapreduce_shuffle</value>

</property>

 

2.8配置hadoop-env.sh

vi      /data/CDH/hadoop-2.6.0-cdh5.14.0/etc/hadoop/hadoop-env.sh

export JAVA_HOME=${JAVA_HOME}

更改成：

export JAVA_HOME=/usr/java/jdk1.8.0_221-amd64

 

2.9啓動hadoop

配置好以後切換到sbin目錄下

cd /data/CDH/hadoop-2.6.0-cdh5.14.0/sbin/ 

hadoop namenode –format #格式化hadoop文件格式

 

錯誤提示：hostname與/etc/hosts中的不對應。

執行hostname命令，查看主機名。好比爲testuser

執行vi /etc/hosts

修改

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1      localhost localhost.localdomain localhost6 localhost6.localdomain6

爲

127.0.0.1 localhost localhost.localdomain testuser  localhost4.localdomain4

::1      localhost localhost.localdomain localhost6 localhost6.localdomain6

 

再次格式化： hadoop namenode -format

成功後啓動全部的

./start-all.sh

中間須要輸入屢次當前服務器密碼，按照提示輸入便可

而後再輸入命令 jps 查看服務啓動狀況

出現下圖標記的五個進程，即表示啓動成功！

 

2.10Hadoop頁面訪問

http://172.20.71.66:50070   #把172.20.71.66 換爲你的服務器IP便可

 

3.安裝Hive

3.1指定Hive安裝目錄/data/CDH,而且解壓

cd /data/CDH

tar -zxvf /data/install_pkg_CDH/hive-1.1.0-cdh5.14.0.tar.gz

 

3.2配置環境變量

vim /etc/profile

 

#添加以下內容

export HIVE_HOME=/data/CDH/hive-1.1.0-cdh5.14.0/

export PATH=$HIVE_HOME/bin:$PATH

export HIVE_CONF_DIR=$HIVE_HOME/conf

export HIVE_AUX_JARS_PATH=$HIVE_HOME/lib

 

~ source /etc/profile

 

3.3配置hive-env.sh

cd /data/CDH/hive-1.1.0-cdh5.14.0/conf

cp  hive-env.sh.template hive-env.sh

vi hive_env.sh

 

#在文件末尾添加以下內容

export JAVA_HOME=/usr/java/jdk1.8.0_221-amd64

export HADOOP_HOME=/data/CDH/hadoop-2.6.0-cdh5.14.0

export HIVE_HOME=/data/CDH/hive-1.1.0-cdh5.14.0

export HIVE_CONF_DIR=/data/CDH/hive-1.1.0-cdh5.14.0/conf

export HIVE_AUX_JARS_PATH=/data/CDH/hive-1.1.0-cdh5.14.0/lib

 

3.4配置文件hive-site.xml

cd /data/CDH/hive-1.1.0-cdh5.14.0/conf

vi hive-site.xml

#添加以下內容

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

  <property>

    <name>javax.jdo.option.ConnectionURL</name>

    <value>jdbc:derby:;databaseName=metastore_db;create=true</value>

  </property>

    <property>

    <name>javax.jdo.option.ConnectionDriverName</name>

    <value>org.apache.derby.jdbc.EmbeddedDriver</value>

  </property>

    <property>

    <name>javax.jdo.option.ConnectionUserName</name>

    <value>APP</value>

  </property>

    <property>

    <name>javax.jdo.option.ConnectionPassword</name>

    <value>mine</value>

  </property>

</configuration>

3.5建立HDFS目錄(這裏好像非必須的，也可不操做)

cd /data/CDH/hive-1.1.0-cdh5.14.0

mkdir hive_hdfs

cd hive_hdfs

hdfs dfs -mkdir -p warehouse #使用這條命令的前提是hadoop已經安裝好了

hdfs dfs -mkdir -p tmp

hdfs dfs -mkdir -p log

hdfs dfs -chmod -R 777 warehouse

hdfs dfs -chmod -R 777 tmp

hdfs dfs -chmod -R 777 log

 

3.6初始化hive (建議在安裝目錄的bin目錄裏執行)

cd  /data/CDH/hive-1.1.0-cdh5.14.0/bin

schematool -dbType derby –initSchema

#若是須要反覆執行schematool -dbType derby –initSchema ，能夠手動刪除或重命名metastore_db目錄。

 

 

 

3.7啓動hive

hive --service metastore &    #可在任意目錄輸入

hive --service hiveserver2 &    #可在任意目錄輸入

3.8使用hive

hive  #可在任意目錄輸入

3.9安裝hive時的異常處理

3.9.1 java.lang.ClassNotFoundException: org.htrace.Trace

從maven倉庫中下載htrace-core.jar包拷貝到$HIVE_HOME/lib目錄下

<!-- https://mvnrepository.com/artifact/org.htrace/htrace-core -->

<dependency>

    <groupId>org.htrace</groupId>

    <artifactId>htrace-core</artifactId>

    <version>3.0</version>

</dependency>

 

cp /data/install_pkg_apache/htrace-core-3.0.jar /data/CDH/hive-1.1.0-cdh5.14.0/lib

cd /data/CDH/hive-1.1.0-cdh5.14.0/bin

 

3.9.2 [ERROR] Terminal initialization failed; falling back to unsupported java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected

處理方案：從maven倉庫中下載 jline-2.12.jar(上傳到當前服務器)

cd /data/CDH/hadoop-2.6.0-cdh5.14.0/share/hadoop/yarn/lib

mv jline-2.11.jar jline-2.11.jar_bak

 

cd /data/CDH/hive-1.1.0-cdh5.14.0/lib

cp jline-2.12.jar /data/CDH/hadoop-2.6.0-cdh5.14.0/share/hadoop/yarn/lib

 

從新啓動 hadoop:

cd /data/CDH/hadoop-2.6.0-cdh5.14.0/sbin

sh stop-all.sh

sh start-all.sh

 

再次執行：　schematool -dbType derby -initSchema

錯誤還在！！！

 

處理方案Plus：

vi /etc/profile

export HADOOP_USER_CLASSPATH_FIRST=true

再次執行：　schematool -dbType derby -initSchema　　(ok)

 

接着開始啓動hive:

./hive --service metastore &  

./hive --service hiveserver2 &

 

3.9.3 java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

參考博客地址： https://www.58jb.com/html/89.html

在Hive2.0.0之前的版本，須要在一個Hive的主節點上啓動兩個服務：metastore 、hiveserver2 才能夠登錄到Hive的命令行下。（配置hive-site.xml再啓動便可）

 

hive --service metastore &

hive --service hiveserver2 &

 

3.10Hive基本操做測試

hive  #進入hive

#執行hql語句

show databases;

create database test;

use test;

create table user_test(name string) row format delimited fields terminated by '\001';

 

insert into table user_test values('zs');

select * from user_test;

 

4.安裝Impala

4.1磁盤擴容(服務器配置足夠的跳過便可)

須要來個小插曲，我這是本身搭建的centos環境，服務器配置不夠，須要加空間。

參考博客： http://blog.itpub.net/117319/viewspace-2125079/

windows操做進入cdm命令行：

C:\Users\kingdee\VirtualBox VMs\centos6\centos6.vdi

VBoxManage.exe modifymedium C:\Users\kingdee\VirtualBox VMs\centos6\centos6.vdi resize --20480  #這裏運行有問題 由於文件名稱存在空格

 

C:

cd Users\kingdee\VirtualBox VMs\centos6

F:\VirtualBox\VBoxManage.exe modifymedium centos6.vdi --resize 20480  #這樣執行就能夠啦

 

Centos操做：

fdisk -l

fdisk /dev/sda

n

p

3

回車 (使用默認)

+10G (分配的內存)

w (保存)

 

 

 

查看並當即刷新分區信息

cat /proc/partitions

reboot #重啓

mkfs -t ext3 /dev/sda3

格式分後再掛載到映射目錄

mount /dev/sda3 /home  #取消掛載命令umount /home

將 impala安裝在 /home目錄下

cd /home

mkdir data

mkdir CDH

mkdir impala_rpm

cd /home/data/CDH/impala_rpm

 

4.2安裝impala（ps:全部安裝包都放到/home/data/CDH/impala_rpm_install_pkg 目錄下 ）

cd    /home/data/CDH/impala_rpm

rpm -ivh ../impala_rpm_install_pkg/impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

 

出現以下錯誤：

warning: ../impala_rpm_install_pkg/impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY

error: Failed dependencies:

        bigtop-utils >= 0.7 is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        hadoop is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        hadoop-hdfs is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        hadoop-yarn is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        hadoop-mapreduce is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        hbase is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        hive >= 0.12.0+cdh5.1.0 is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        zookeeper is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        hadoop-libhdfs is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        avro-libs is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        parquet is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        sentry >= 1.3.0+cdh5.1.0 is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        sentry is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        /lib/lsb/init-functions is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

        libhdfs.so.0.0.0()(64bit) is needed by impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64

 

emmmm… 得先安裝 bigtop-util 和 sentry

4.3安裝bigtop-util 和 sentry

bigtop-util 和 sentry安裝包下載地址：

http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.14.0/RPMS/noarch/

也能夠直接點擊下面這兩個連接下載：

http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.14.0/RPMS/noarch/bigtop-utils-0.7.0+cdh5.14.0+0-1.cdh5.14.0.p0.47.el7.noarch.rpm

http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.14.0/RPMS/noarch/sentry-1.5.1+cdh5.14.0+432-1.cdh5.14.0.p0.47.el7.noarch.rpm

 

執行以下命令：

sudo rpm -ivh  ../impala_rpm_install_pkg/bigtop-utils-0.7.0+cdh5.14.0+0-1.cdh5.14.0.p0.47.el7.noarch.rpm

sudo rpm -ivh  ../impala_rpm_install_pkg/sentry-1.5.1+cdh5.14.0+432-1.cdh5.14.0.p0.47.el7.noarch.rpm --nodeps  # --nodeps 不強制依賴

sudo rpm -ivh  ../impala_rpm_install_pkg/impala-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm --nodeps

sudo rpm -ivh  ../impala_rpm_install_pkg/impala-server-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

sudo rpm -ivh  ../impala_rpm_install_pkg/impala-state-store-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

sudo rpm -ivh  ../impala_rpm_install_pkg/impala-catalog-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

sudo rpm -ivh  ../impala_rpm_install_pkg/impala-shell-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

 

執行到impala-shell得時候，須要先安裝：python-setuptools

 

4.4安裝python-setuptools

python-setuptools的下載地址：

      http://mirror.centos.org/altarch/7/os/aarch64/Packages/python-setuptools-0.9.8-7.el7.noarch.rpm

執行以下命令：

sudo rpm -ivh  ../impala_rpm_install_pkg/python-setuptools-0.9.8-7.el7.noarch.rpm --nodeps

 

而後咱們繼續上次的impala-shell安裝：

sudo rpm -ivh  ../impala_rpm_install_pkg/impala-shell-2.11.0+cdh5.14.0+0-1.cdh5.14.0.p0.50.el6.x86_64.rpm

歐克了

4.5配置bigtop-utils

vi /etc/default/bigtop-utils

export JAVA_HOME=/usr/java/jdk1.8.0_221-amd64

4.6配置conf

cd  /etc/impala/conf  #複製hadoop與hive的配置文件到impala的目錄下

cp /data/CDH/hadoop-2.6.0-cdh5.14.0/etc/hadoop/core-site.xml ./

cp /data/CDH/hadoop-2.6.0-cdh5.14.0/etc/hadoop/hdfs-site.xml ./

cp /data/CDH/hive-1.1.0-cdh5.14.0/conf/hive-site.xml ./

 

4.7配置hdfs-site.xml

vi hdfs-site.xml

#添加以下內容：

 <property>

         <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>

         <value>true</value>

</property>

<property>

         <name>dfs.client.file-block-storage-locations.timeout.millis</name>

         <value>10000</value>

</property>

 

4.8處理jar包

4.8.1咱們先來學習一下linux的軟連接

建立軟連接

ln  -s  [源文件或目錄]  [目標文件或目錄]

 

例如：

當前路徑建立test 引向/var/www/test 文件夾

ln –s  /var/www/test  test

 

刪除軟連接

和刪除普通的文件是同樣的，刪除都是使用rm來進行操做

 rm –rf 軟連接名稱（請注意不要在後面加」/」，rm –rf 後面加不加」/」 的區別，可自行去百度下啊）

例如：

刪除test

rm –rf test

 

修改軟連接

ln –snf  [新的源文件或目錄]  [目標文件或目錄]

這將會修改原有的連接地址爲新的地址

例如：

建立一個軟連接

ln –s  /var/www/test   /var/test

修改指向的新路徑

ln –snf  /var/www/test1   /var/test

4.8.2處理impala jar包的軟鏈接

(如下內容能夠複製，記得替換版本號哦，例如個人包版本是 2.6.0-cdh5.14.0, 須要按照你具體的安裝版本替換哦)

Hadoop相關jar包

sudo rm -rf /usr/lib/impala/lib/hadoop-*.jar

sudo ln -s $HADOOP_HOME/share/hadoop/common/lib/hadoop-annotations-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-annotations.jar

sudo ln -s $HADOOP_HOME/share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-auth.jar

sudo ln -s $HADOOP_HOME/share/hadoop/tools/lib/hadoop-aws-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-aws.jar

sudo ln -s $HADOOP_HOME/share/hadoop/common/hadoop-common-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-common.jar

sudo ln -s $HADOOP_HOME/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-hdfs.jar

sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-common.jar

sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-core.jar

sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar

sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar

sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-api-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-api.jar

sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-client-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-client.jar

sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-common.jar

sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar

sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-server-common.jar

sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jar

sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar

sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jar

sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce1/hadoop-core-2.6.0-mr1-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-core.jar

sudo ln -s $HADOOP_HOME/share/hadoop/tools/lib/hadoop-azure-datalake-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-azure-datalake.jar

 

#從新創建軟鏈接：

sudo ln -snf $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-common-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-common.jar

sudo ln -snf $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-yarn-server-common.jar

sudo ln -snf $HADOOP_HOME/share/hadoop/common/lib/avro-1.7.6-cdh5.14.0.jar /usr/lib/impala/lib/avro.jar

sudo ln -snf $HADOOP_HOME/share/hadoop/common/lib/zookeeper-3.4.5-cdh5.14.0.jar /usr/lib/impala/lib/zookeeper.jar

hive相關jar包

sudo rm -rf /usr/lib/impala/lib/hive-*.jar

sudo ln -s $HIVE_HOME/lib/hive-ant-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-ant.jar

sudo ln -s $HIVE_HOME/lib/hive-beeline-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-beeline.jar

sudo ln -s $HIVE_HOME/lib/hive-common-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-common.jar

sudo ln -s $HIVE_HOME/lib/hive-exec-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-exec.jar

sudo ln -s $HIVE_HOME/lib/hive-hbase-handler-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-hbase-handler.jar

sudo ln -s $HIVE_HOME/lib/hive-metastore-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-metastore.jar

sudo ln -s $HIVE_HOME/lib/hive-serde-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-serde.jar

sudo ln -s $HIVE_HOME/lib/hive-service-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-service.jar

sudo ln -s $HIVE_HOME/lib/hive-shims-common-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-shims-common.jar

sudo ln -s $HIVE_HOME/lib/hive-shims-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-shims.jar

sudo ln -s $HIVE_HOME/lib/hive-shims-scheduler-1.1.0-cdh5.14.0.jar /usr/lib/impala/lib/hive-shims-scheduler.jar

sudo ln -snf $HIVE_HOME/lib/parquet-hadoop-bundle-1.5.0-cdh5.14.0.jar /usr/lib/impala/lib/parquet-hadoop-bundle.jar

其它jar包

sudo ln -snf $HIVE_HOME/lib/hbase-annotations-1.2.0-cdh5.14.0.jar /usr/lib/impala/lib/hbase-annotations.jar

sudo ln -snf $HIVE_HOME/lib/hbase-common-1.2.0-cdh5.14.0.jar /usr/lib/impala/lib/hbase-common.jar

sudo ln -snf $HIVE_HOME/lib/hbase-protocol-1.2.0-cdh5.14.0.jar /usr/lib/impala/lib/hbase-protocol.jar

sudo ln -snf /data/CDH/impala-2.11.0-cdh5.14.0/thirdparty/hbase-1.2.0-cdh5.14.0/lib/hbase-client-1.2.0-cdh5.14.0.jar /usr/lib/impala/lib/hbase-client.jar

sudo ln -snf /data/CDH/impala-2.11.0-cdh5.14.0/thirdparty/hadoop-2.6.0-cdh5.14.0/lib/native/libhadoop.so /usr/lib/impala/lib/libhadoop.so

sudo ln -snf /data/CDH/impala-2.11.0-cdh5.14.0/thirdparty/hadoop-2.6.0-cdh5.14.0/lib/native/libhadoop.so.1.0.0 /usr/lib/impala/lib/libhadoop.so.1.0.0

sudo ln -snf /data/CDH/impala-2.11.0-cdh5.14.0/thirdparty/hadoop-2.6.0-cdh5.14.0/lib/native/libhdfs.so /usr/lib/impala/lib/libhdfs.so

sudo ln -snf /data/CDH/impala-2.11.0-cdh5.14.0/thirdparty/hadoop-2.6.0-cdh5.14.0/lib/native/libhdfs.so.0.0.0 /usr/lib/impala/lib/libhdfs.so.0.0.0

 

額外補充

若有遺漏，還有一些jar包軟連接未生效，能夠全局搜索當前服務器上是否存在此jar,若是不存在就去maven上下載一個，而後放到某個目錄，軟鏈接過去便可。

eg：發現還有個 hadoop-core.jar軟鏈接無效。

find  /  -name       「hadoop-core*」;  #根據此命令找到相關包的位置，而後ln –s xx xx 便可

4.9啓動impala

service impala-state-store start

service impala-catalog start

service impala-server start

 

在執行service impala-state-store start 報錯

[root@sunny conf]# service impala-state-store start

/etc/init.d/impala-state-store: line 34: /lib/lsb/init-functions: No such file or directory

/etc/init.d/impala-state-store: line 104: log_success_msg: command not found

 

須要安裝 redhat-lsb

下載地址：http://rpmfind.net/linux/rpm2html/search.php?query=redhat-lsb-core(x86-64)

選擇：redhat-lsb-core-4.1-27.el7.centos.1.x86_64.rpm 下載

 

 

 

下載好後執行命令：

rpm -ivh redhat-lsb-core-4.1-27.el7.centos.1.x86_64.rpm

 

繼續啓動impala

service impala-state-store start

service impala-catalog start

service impala-server start #全部datanode節點啓動impalad

 

全部服務啓動成功後能夠開始訪問impala

4.10使用impala

進入impala:

impala-shell #任意路徑輸入便可，基本操做同hive

訪問impalad的管理界面

http://172.20.71.66:25000/

 

訪問statestored的管理界面

http://172.20.71.66:25010/

 

訪問catalog的管理界面

http://172.20.71.66:25020

impala日誌文件：  /var/log/impala 

impala配置文件：/etc/default/impala

/etc/impala/conf  （core-site.xml, hdfs-site.xml, hive-site.xml）

4.11 安裝impala異常處理

4.11.1 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

處理方案：

參考：http://blog.csdn.net/l1028386804/article/details/51566262

進入所安裝的Hive的conf目錄，找到hive-site.xml，（若沒修改，則是hive-default.xml.template）。

<property>

  <name>hive.metastore.schema.verification</name>

  <value>true</value>

   <description>

   Enforce metastore schema version consistency.

   True: Verify that version information stored in metastore matches with one from Hive jars.  Also disable automatic

         schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures

         proper metastore schema migration. (Default)

   False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.

   </description>

</property>

改成

<property>

  <name>hive.metastore.schema.verification</name>

  <value>false</value>

   <description>

   Enforce metastore schema version consistency.

   True: Verify that version information stored in metastore matches with one from Hive jars.  Also disable automatic

         schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures

         proper metastore schema migration. (Default)

   False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.

   </description>

</property>

 

而後重啓hive

hive --service metastore &  

hive --service hiveserver2 &

 

4.11.2 Invalid short-circuit reads configuration:- Impala cannot read or execute the parent directory of dfs.domain.socket.path

處理方案：建立配置文件中 /etc/impala/conf/hdfs-site.xml,對應目錄便可

 

 

 

cd /var/run

mkdir hdfs-sockets

cd hdfs-sockets

mkdir dn

4.11.3impala insert數據報錯：NoClassDefFoundError: org/apache/hadoop/fs/adl/AdlFileSystem

[localhost.localdomain:21000] > insert into user_test values('zs');

Query: insert into user_test values('zs')

Query submitted at: 2020-12-17 02:39:24 (Coordinator: http://sunny:25000)

ERROR: AnalysisException: Failed to load metadata for table: test.user_test. Running 'invalidate metadata test.user_test' may resolve this problem.

CAUSED BY: NoClassDefFoundError: org/apache/hadoop/fs/adl/AdlFileSystem

CAUSED BY: TableLoadingException: Failed to load metadata for table: test.user_test. Running 'invalidate metadata test.user_test' may resolve this problem.

CAUSED BY: NoClassDefFoundError: org/apache/hadoop/fs/adl/AdlFileSystem

 

處理方案：

impala沒有引用hadoop-azure-datalake.jar ,補充引用

sudo ln -s /data/CDH/hadoop-2.6.0-cdh5.14.0/share/hadoop/tools/lib/hadoop-azure-datalake-2.6.0-cdh5.14.0.jar /usr/lib/impala/lib/hadoop-azure-datalake.jar

 

hadoop core-site.xml 配置解析：http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml

 

到此 hadoop hive impala 安裝結束啦，完結 撒花~