Hadoop2.9.1 安裝 Hive2.3.3 on ubuntu 16.04

Hadoop2.9.1 安裝 Hive2.3.3 on ubuntu 16.04

前言

http://hive.apache.org/downloads.html 上有說明，hadoop3.x版本須要hive3.0.0，而hadoop2.x須要hive2.3.3。由於個人hadoop是2.9，因此選擇下載hive2.3.3。Hive是hadoop的工具，因此只須要安裝在NameNode上，不須要安裝在DataNode上。

Hadoop 安裝請移步

http://www.javashuo.com/article/p-fixfbmga-bs.html

安裝 Hive2.3.3

一、下載 Hive

# 下載地址：
wget http://mirrors.shu.edu.cn/apache/hive/stable-2/apache-hive-2.3.3-bin.tar.gz

# 解壓Hive2.3.3
tar xvf apache-hive-2.3.3-bin.tar.gz -C /usr/local/
chown -R hadoop.hadoop  /usr/local/apache-hive-2.3.3-bin/

二、配置 Hive2.3.3 環境變量

cat << EOF >> ~/.bashrc
HIVE_HOME=/usr/local/apache-hive-2.3.3-bin
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin:$HIVE_HOME/conf
export JAVA_HOME JRE_HOME CLASS_PATH PATH HADOOP_HOME HIVE_HOME
EOF
source /etc/profile

三、配置 hive-env.sh

cat < EOF > /usr/local/apache-hive-2.3.3-bin/hive-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export HIVE_HOME=/usr/local/apache-hive-2.3.3-bin

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=$HIVE_HOME/conf

# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=$HIVE_HOME/lib/*
EOF

四、配置 hive-site.xml

Hive環境模式有3中，本例中採用了內嵌模式
cp hive-default.xml.template hive-site.xml

hive-site.xml 文件內容不須要修改，主要配置

<property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
</property>
..
<property>
    <name>hive.exec.scratchdir</name>
    <value>/tmp/hive</value>
    <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>
</property>

注：

• hive.metastore.warehouse.dir

  該參數指定了Hive的數據存儲目錄，默認位置在HDFS上面的 /user/hive/warehouse路徑下

• hive.exec.scratchdir

  該參數指定了Hive的數據臨時文件目錄，默認位置爲HDFS上面的 /tmp/hive路徑下

五、建立HDFS目錄

hive-site.xml 文件中有兩個重要的HDFS路徑/user/hive/warehouse和/tmp/hive，須要在HDFS中建立，並修改權限

hadoop fs -mkdir /user
hadoop fs -mkdir /user/hive
hadoop fs -mkdir /user/hive/warehouse
hadoop fs -mkdir /tmp/hive
hadoop fs -chmod  777 /user/hive/warehouse
hadoop fs -chmod  777 /tmp/hive

六、初始化數據庫

內嵌模式用的是本地數據庫derby

./schematool -initSchema -dbType derby 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:derby:;databaseName=metastore_db;create=true
Metastore Connection Driver :    org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User:       APP
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.derby.sql
Initialization script completed
schemaTool completed

七、啓動hive

進入$HIVE_HOME/bin目錄

cd $HIVE_HOME/bin
./hive

# 測試
hive> show tables;
OK
Time taken: 7.101 seconds

FAQ

Q1: 啓動 Hive 報錯

Logging initialized using configuration in jar:file:/usr/local/apache-hive-2.3.3-bin/lib/hive-common-2.3.3.jar!/hive-log4j2.properties Async: true
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at org.apache.hadoop.fs.Path.initialize(Path.java:254)
        at org.apache.hadoop.fs.Path.<init>(Path.java:212)
        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:659)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582)
        at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:549)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:750)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at java.net.URI.checkPath(URI.java:1823)
        at java.net.URI.<init>(URI.java:745)
        at org.apache.hadoop.fs.Path.initialize(Path.java:251)
        ... 12 more

A1:

在hive-site.xml文件最前面增長以下配置，重啓hive

建立一個緩存目錄：mkdir -p /usr/local/apache-hive-2.3.3-bin/tmpdir

<property>
     <name>system:java.io.tmpdir</name>
     <value>/usr/local/apache-hive-2.3.3-bin/tmpdir</value>
  </property>
  <property>
     <name>system:user.name</name>
     <value>hive</value>
</property>