Apache Hive2.1.0安裝筆記

Hive2.x已經足夠穩定了，前面也安裝過hive0.x和Hive1.x的版本，今天咱們來看下hive2.x如何安裝使用。

環境：

centos7.1

Hadoop2.7.3

JDK8

Hive2.1.0

1，首先須要下載hive最新的穩定版本的包，並保證的你Hadoop集羣已是可以正常運行的

http://ftp.kddilabs.jp/infosystems/apache/hive/

2，解壓到指定目錄

首先進入conf目錄把全部帶template後綴的文件，給移除後綴，只有hive-default.xml移除後綴後，須要修更名爲hive-site.xml。

3，配置hive的log

vi conf/hive-log4j2.properties 

配置下面的2個參數：
property.hive.log.dir = /home/search/hive/logs 
property.hive.log.file = hive.log

4，配置使用MySQL做爲元數據存儲

關於安裝mysql以及分配權限的請參考散仙以前的文章：http://qindongliang.iteye.com/blog/2337865

vi hive-site.xml

配置下面的幾項參數
javax.jdo.option.ConnectionURL= jdbc:mysql://192.168.10.40:3306/hive?createDatabaseIfNotExist=true&characterEncoding=utf-8
javax.jdo.option.ConnectionUserName=root
javax.jdo.option.ConnectionPassword=pwd
javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
hive.metastore.warehouse.dir=hdfs://192.168.10.38:8020//user/hive/warehouse

其餘的凡是包含 ${system:java.io.tmpdir}變量的統一替代成絕對路徑，目錄能夠在
hive的根目錄下建一個tmp目錄，統一放進去

最後切記添加mysql的jdbc驅動包到hive/lib的目錄下面

說明下上面的jdbc的url裏面驅動字符串設置爲數據庫編碼爲utf-8此外&符號須要轉義

jdbc:mysql://192.168.10.40:3306/hive?createDatabaseIfNotExist=true&characterEncoding=utf-8

此外默認hive讀取Hbase的lib若是沒有安裝hbase則會啓動不起來: 須要下載hbase而後配置對應的HBASE_HOME，文末會給出全部的環境變量

5，在hive2.x以後，須要先初始化schema以下：

$HIVE_HOME/bin/schematool -initSchema -dbType mysql

注意不執行這個，直接執行hive會報錯：

Caused by: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql))
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3364)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3336)
        at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3590)

執行成功打印結果以下：

[search@es1 ~]$ $HIVE_HOME/bin/schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/search/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/search/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:mysql://192.168.10.40:3306/hive?createDatabaseIfNotExist=true&characterEncoding=utf-8
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:       root
Starting metastore schema initialization to 2.1.0
Initialization script hive-schema-2.1.0.mysql.sql
Initialization script completed
schemaTool completed

6，測試集羣是否正常

在本地磁盤上新建一個文件a，寫入內容以下

1,a
2,b
3,c
4,a
5,a
2,a
4,2
1,a
1,a

編寫的create_sql以下：

-- 存在就刪除
  drop table if exists info ; 
-- 建表
CREATE TABLE info(id string, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE;
 -- 加載數據
load data local inpath '/home/search/test_hive/a' into table info;

最後執行腳本，不報錯就表明經過了：

hive -f create_sql

Hive2.x以後不推薦使用MR的方式運行任務了，推薦使用Tez或者Spark引擎運行job，可是mr仍是支持的

執行下面的語句進行測試

hive -e "select count(*) from info"

運行成功，就表明Hive+Hadoop集成成功。

關於Hive On Tez 集成我下篇文章會介紹。

7，一些環境變量以下：

#JDK
export JAVA_HOME=/home/search/jdk1.8.0_102/
export CLASSPATH=.:$JAVA_HOME/lib
export PATH=$JAVA_HOME/bin:$PATH

#Maven
export MAVEN_HOME=/home/search/apache-maven-3.3.9
export CLASSPATH=$CLASSPATH:$MAVEN_HOME/lib
export PATH=$PATH:$MAVEN_HOME/bin

#Ant
export ANT_HOME=/home/search/ant
export CLASSPATH=$CLASSPATH:$ANT_HOME/lib
export PATH=$PATH:$ANT_HOME/bin

#Hadoop
export HADOOP_HOME=/home/search/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export CLASSPATH=.:$CLASSPATH:$HADOOP_COMMON_HOME:$HADOOP_COMMON_HOME/lib:$HADOOP_MAPRED_HOME:$HADOOP_HDFS_HOME:$HADOOP_HDFS_HOME

#Hbase
export HBASE_HOME=/home/search/hbase
export CLASSPATH=$CLASSPATH:$HBASE_HOME/lib
export PATH=$HBASE_HOME/bin:$PATH

#Pig
export PIG_HOME=/home/search/pig
export PIG_CLASSPATH=$PIG_HOME/lib:$HADOOP_HOME/etc/hadoop
export PATH=/ROOT/server/bigdata/pig/bin:$PATH

#Zookeeper
export ZOOKEEPER_HOME=/home/search/zookeeper
export CLASSPATH=.:$ZOOKEEPER_HOME/lib
export PATH=$PATH:$ZOOKEEPER_HOME/bin

#Hive
export HIVE_HOME=/home/search/hive
export HIVE_CONF_DIR=$HIVE_HOME/conf
export CLASSPATH=$CLASSPATH:$HIVE_HOME/lib
export PATH=$PATH:$HIVE_HOME/bin:$HIVE_HOME/conf

#JStorm
export JSTORM_HOME=/home/search/jstorm-2.1.1
export CLASSPATH=$CLASSPATH:$JSTORM_HOME/lib
export PATH=$PATH:$JSTORM_HOME/bin:$PATH

#Scala
export SCALA_HOME=/home/search/scala
export CLASSPATH=.:$SCALA_HOME/lib
export PATH=$PATH:$SCALA_HOME/bin

#Spark
export SPARK_HOME=/ROOT/server/spark
export PATH=$PATH:$SPARK_HOME/bin