Hive學習筆記（二）—— 安裝配置

Hive安裝配置及基本操做

1. Hive安裝及配置

(1). 上傳文件到Hadoop102節點，解壓到/opt/moudle

(2). 修改/opt/module/hive/conf目錄下的hive-env.sh.template名稱爲hive-env.sh

[simon@hadoop102 conf]$ cp hive-env.sh.template hive-env.sh

(3). 配置hive-env.sh文件

​ (a). 配置HADOOP_HOME路徑 :

export HADOOP_HOME=/opt/module/hadoop-2.7.2

​ (b).配置HIVE_CONF_DIR路徑 :

export HIVE_CONF_DIR=/opt/module/hive/conf

2. Hadoop集羣的配置

（1）必須啓動HDFS和YARN

[simon@hadoop102 conf]$ start-dfs.sh
[simon@hadoop103 module]$ start-yarn.sh
[simon@hadoop104 hadoop-2.7.2]$ mr-jobhistory-daemon.sh start historyserver

（2）在HDFS上建立/tmp和/user/hive/warehouse兩個目錄並修改他們的同組權限可寫

[simon@hadoop102 hadoop-2.7.2]$ hadoop fs -mkdir /tmp
[simon@hadoop102 hadoop-2.7.2]$ hadoop fs -mkdir -p /user/hive/warehouse

[simon@hadoop102 hadoop-2.7.2]$ hadoop fs -chmod g+w /tmp
[simon@hadoop102 hadoop-2.7.2]$ hadoop fs -chmod g+w /user/hive/warehouse

3. Hive基本操做

（1）啓動hive

[simon@hadoop102 hive]$ bin/hive

（2）查看數據庫

hive> show databases;

（3）打開默認數據庫

hive> use default;

（4）顯示default數據庫中的表

hive> show tables;

（5）建立一張表

hive> create table student(id int,name string);

（6）顯示數據庫中有幾張表

hive> show tables;

（7）查看錶的結構

hive> desc student;
OK
id                      int                                         
name                    string                                      
Time taken: 0.413 seconds, Fetched: 2 row(s)

（8）向表中插入數據

hive> insert into student values(1000,"simon");

（9）查詢表中數據

hive> select * from student;

（10）退出hive

hive> quit;

4. 將本地文件導入Hive案例

需求:

​ 將本地/opt/module/datas/student.txt這個目錄下的數據導入到hive的student(id int, name string)表 中。

hive> create table student(id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED
 BY '\t';
--導入本地數據
hive> load data local inpath '/opt/module/datas/student.txt' into table student;

查看是否插入成功：

hive> select * from student;
OK
1001    zhangshan
1002    lishi
1003    zhaoliu
Time taken: 0.266 seconds, Fetched: 3 row(s)

再打開一個客戶端窗口啓動hive，會產生java.sql.SQLException異常:

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException:
 Unable to instantiate
 org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)

緣由是，Metastore默認存儲在自帶的derby數據庫中，推薦使用MySQL存儲Metastore;

5. 安裝Mysql及配置Hive的元數據存放位置

5.1．查看mysql是否安裝，若是安裝了，卸載mysql

(1). 查看是否安裝了Mysql:

[root@hadoop102 桌面]# rpm -qa|grep mysql
mysql-libs-5.1.73-7.el6.x86_64

(2). 卸載掉已安裝的Mysql：

[root@hadoop102 桌面]# rpm -e --nodeps mysql-libs-5.1.73-7.el6.x86_64

(3). 解壓mysql-libs.zip 文件到當前目錄

[root@hadoop102 software]# unzip mysql-libs.zip
[root@hadoop102 software]# ls
mysql-libs.zip
mysql-libs

(4). 進入到mysql-libs文件夾下

[root@hadoop102 mysql-libs]# ll
總用量 76048
-rw-r--r--. 1 root root 18509960 3月  26 2015 MySQL-client-5.6.24-1.el6.x86_64.rpm
-rw-r--r--. 1 root root  3575135 12月  1 2013 mysql-connector-java-5.1.27.tar.gz
-rw-r--r--. 1 root root 55782196 3月  26 2015 MySQL-server-5.6.24-1.el6.x86_64.rpm

5.2 安裝Mysql服務器

(1).安裝mysql服務端

[root@hadoop102 mysql-libs]# rpm -ivh MySQL-server-5.6.24-1.el6.x86_64.rpm

(2).查看產生的隨機密碼

[root@hadoop102 mysql-libs]# cat /root/.mysql_secret 
# The random password set for the root user at Sat Nov  2 21:44:35 2019 (local time): sMUtG2RWFg3XUr4M

(3).查看mysql的狀態

[root@hadoop102 mysql-libs]# service mysql staus
Usage: mysql  {start|stop|restart|reload|force-reload|status}  [ MySQL server options ]

(4).啓動mysql

[root@hadoop102 mysql-libs]# service mysql start
Starting MySQL..                                           [肯定]

5.3 安裝mysql客戶端

(1).解壓客戶端安裝包

[root@hadoop102 mysql-libs]# rpm -ivh MySQL-client-5.6.24-1.el6.x86_64.rpm

(2). 鏈接mysql

[root@hadoop102 mysql-libs]# mysql -uroot -psMUtG2RWFg3XUr4M

(3).修改密碼：

mysql> SET PASSWORD=PASSWORD('zxc147');

(4).退出mysql

mysql> quit

5.4 mysql的user表中主機配置

配置只要是root用戶+密碼，在任何主機上都能登陸MySQL數據庫。

(1).進入mysql

[root@hadoop102 mysql-libs]# mysql -uroot -pzxc147

(2).顯示數據庫

mysql> show databases;

(3).使用msql數據庫

mysql> use mysql;

(4).展現mysql數據庫中的全部數據表

mysql>show tables;

(5).使用user表

mysql>use user;

(6). 描述表的結構

mysql>desc User;

(7). 查詢user表

mysql> select Host, User,Password from user;
+-----------+------+-------------------------------------------+
| Host      | User | Password                                  |
+-----------+------+-------------------------------------------+
| localhost | root | *3114DECCB22104A2045E0132C007737A52C7E11D |
| hadoop102 | root | *9178AE26D29780ECED756CA0AABE2442BF1CAA41 |
| 127.0.0.1 | root | *9178AE26D29780ECED756CA0AABE2442BF1CAA41 |
| ::1       | root | *9178AE26D29780ECED756CA0AABE2442BF1CAA41 |
+-----------+------+-------------------------------------------+

(8).修改user表，把Host改爲%

mysql> update user set host='%' where host = 'localhost';

(9).刪除root用戶其餘host

mysql> delete from user where host <>'%';

(10).刷新權限信息

mysql> flush privileges;

(11).退出

mysql>quit;

5.5 配置Metastore到MySql

(1).在/opt/module/hive/conf目錄下建立一個文件hive-site.xml

根據官方文檔配置參數，拷貝數據到hive-site.xml文件中

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://hadoop102:3306/metastore?createDatabaseIfNotExist=true</value>
      <description>JDBC connect string for a JDBC metastore</description>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
      <description>Driver class name for a JDBC metastore</description>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>root</value>
      <description>username to use against metastore database</description>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>zxc147</value>
      <description>password to use against metastore database</description>
    </property>
</configuration>

(2). 將/opt/software/mysql-libs目錄下解壓mysql-connector-java-5.1.27.tar.gz驅動包，

拷貝mysql-connector-java-5.1.27-bin.jar到/opt/module/hive/lib/

[root@hadoop102 mysql-libs]# tar -zxf mysql-connector-java-5.1.27.tar.gz -C /opt/module/hive/lib/

(3). 配置完畢後，若是啓動hive異常，能夠從新啓動虛擬機。（重啓後，別忘了啓動hadoop集羣）

6. 多窗口啓動Hive測試

(1). 首先啓動mysql

[root@hadoop102 mysql-libs]# mysql -uroot -pzxc147

(2). 查看有幾個數據庫

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.00 sec)

(2). 再打開多個窗口，分別啓動hive

[simon@hadoop102 hive]$ bin/hive

(3). 啓動hive後，回到MySQL窗口查看數據庫，顯示增長了metastore數據庫

mysql> show databases;

+--------------------+
| Database           |
+--------------------+
| information_schema |
| metastore          |
| mysql              |
| performance_schema |
| test               |
+--------------------+

7. HiveJDBC訪問

hive的訪問：終端訪問 遠程訪問

終端訪問：安裝hive，利用shell腳本訪問 不可併發訪問

遠程訪問：經過JDBC鏈接數據倉庫 支持併發訪問

(1). 啓動hiveserver2服務

[simon@hadoop102 hive]$ bin/hiveserver2

(2). 啓動beeline

[simon@hadoop102 hive]$ bin/beeline 
Beeline version 1.2.1 by Apache Hive
beeline>

(3). 鏈接hiveserver2

beeline> !connect jdbc:hive2://hadoop102:10000  #(回車)
Connecting to jdbc:hive2://hadoop102:10000
Enter username for jdbc:hive2://hadoop102:10000: simon  #(輸入用戶名回車)
Enter password for jdbc:hive2://hadoop102:10000:       #(直接回車)
Connected to: Apache Hive (version 1.2.1)
Driver: Hive JDBC (version 1.2.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ

(4). 使用測試

0: jdbc:hive2://hadoop102:10000> show databases;
+----------------+--+
| database_name  |
+----------------+--+
| default        |
+----------------+--+
1 row selected (1.083 seconds)

8. Hive的經常使用交互命令

[simon@hadoop102 hive]$ bin/hive -help
usage: hive
 -d,--define <key=value>          Variable subsitution to apply to hive
                                  commands. e.g. -d A=B or --define A=B
    --database <databasename>     Specify the database to use
 -e <quoted-query-string>         SQL from command line
 -f <filename>                    SQL from files
 -H,--help                        Print help information
    --hiveconf <property=value>   Use value for given property
    --hivevar <key=value>         Variable subsitution to apply to hive
                                  commands. e.g. --hivevar A=B
 -i <filename>                    Initialization SQL file
 -S,--silent                      Silent mode in interactive shell
 -v,--verbose                     Verbose mode (echo executed SQL to the console)

(1). 「-e」不進入hive的交互窗口執行sql語句

[simon@hadoop102 hive]$ bin/hive -e "select id from student;"

(2)．「-f」執行腳本中sql語句

​ (a). 編寫腳本文件：在/opt/module/datas目錄下建立hivef.sql文件

[simon@hadoop102 datas]$ touch hivef.sql

​ (b). 文件中寫入正確的sql語句

```

select *from student;
```

​ (c). 執行文件中的sql語句

[simon@hadoop102 hive]$ bin/hive -f /opt/module/datas/hivef.sql

​ (d) .執行文件中的sql語句並將結果寫入文件中

[simon@hadoop102 hive]$ bin/hive -f /opt/module/datas/hivef.sql  > /opt/module/datas/hive_result.txt

9.Hive其餘命令操做

(1). 退出hive窗口：

hive(default)>exit;

hive(default)>quit;

在新版的hive中沒區別了，在之前的版本是有的：

exit:先隱性提交數據，再退出；

quit:不提交數據，退出；

(2).在hive cli命令窗口中如何查看hdfs文件系統

hive(default)>dfs -ls /;

(3). 在hive cli命令窗口中如何查看本地文件系統

hive(default)>! ls /opt/module/datas;

(4). 查看在hive中輸入的全部歷史命令

​ （1）進入到當前用戶的根目錄/root或/home/simon

​ （2）查看. hivehistory文件

[simon@hadoop102 ~]$ cat .hivehistory