【hive】——metastore的三種模式

時間 2019-11-19

標籤 hive metastore 三種模式欄目 Hadoop 简体版

原文原文鏈接

Hive中metastore（元數據存儲）的三種方式：java

內嵌Derby方式
Local方式
Remote方式

[一]、內嵌Derby方式node

這個是Hive默認的啓動模式，通常用於單元測試，這種存儲方式有一個缺點：在同一時間只能有一個進程鏈接使用數據庫。mysql

hive-site.xml 中jdbc URL、驅動、用戶名、密碼等的配置信息以下：sql

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:derby:;databaseName=metastore_db;create=true</value>

<description>JDBC connect string for a JDBC metastore</description>

</property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>org.apache.derby.jdbc.EmbeddedDriver</value>

<description>Driver class name for a JDBC metastore</description>

</property>

<name>javax.jdo.option.ConnectionUserName</name>

<description>username to use against metastore database</description>

</property>

<name>javax.jdo.option.ConnectionPassword</name>

<description>password to use against metastore database</description>

</property>

<name>hive.metastore.warehouse.dir</name>

<value>file:///Users/micmiu/tmp/hive/warehouse</value>

<description>unit test data goes in here on your local filesystem</description>

</property>

執行初始化命令：schematool -dbType derby -initSchemashell

查看初始化後的信息： schematool -dbType derby -info數據庫

配置完成後就可在shell中以CLI的方式訪問hive 進行操做驗證。apache

[二]、Local方式單元測試

以本地Mysql數據庫爲例：建立好用戶：hive；database：hive。測試

配置文件 hive-site.xml 中jdbc URL、驅動、用戶名、密碼等屬性值配置以下：spa

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true</value>

<description>JDBC connect string for a JDBC metastore</description>

</property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

<description>Driver class name for a JDBC metastore</description>

</property>

<name>javax.jdo.option.ConnectionUserName</name>

<description>username to use against metastore database</description>

</property>

<name>javax.jdo.option.ConnectionPassword</name>

<value>micmiu</value>

<description>password to use against metastore database</description>

</property>

<name>hive.metastore.warehouse.dir</name>

<value>/user/hive/warehouse</value>

<description>location of default database for the warehouse</description>

</property>

ps：須要把mysql的驅動包copy到目錄 <HIVE_HOME>/lib 中

若是是第一次須要執行初始化命令：schematool -dbType mysql -initSchema

micmiu-mbp:mysql micmiu$ schematool -dbType mysql -initSchema

14/02/17 14:46:08 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive

14/02/17 14:46:08 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize

14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize

14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack

14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node

14/02/17 14:46:08 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

14/02/17 14:46:08 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

Metastore connection URL: jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true

Metastore Connection Driver : com.mysql.jdbc.Driver

Metastore connection User: hive

Starting metastore schema initialization to 0.12.0

Initialization script hive-schema-0.12.0.mysql.sql

Initialization script completed

schemaTool completeted

查看初始化後信息 schematool -dbType mysql -info

初始化後查看mysql中表狀況：show tables;

mysql> show tables;

+---------------------------+

| Tables_in_hive |

+---------------------------+

| BUCKETING_COLS |

| CDS |

| COLUMNS_V2 |

| DATABASE_PARAMS |

| DBS |

| DB_PRIVS |

| DELEGATION_TOKENS |

| GLOBAL_PRIVS |

| IDXS |

| INDEX_PARAMS |

| MASTER_KEYS |

| NUCLEUS_TABLES |

| PARTITIONS |

| PARTITION_EVENTS |

| PARTITION_KEYS |

| PARTITION_KEY_VALS |

| PARTITION_PARAMS |

| PART_COL_PRIVS |

| PART_COL_STATS |

| PART_PRIVS |

| ROLES |

| ROLE_MAP |

| SDS |

| SD_PARAMS |

| SEQUENCE_TABLE |

| SERDES |

| SERDE_PARAMS |

| SKEWED_COL_NAMES |

| SKEWED_COL_VALUE_LOC_MAP |

| SKEWED_STRING_LIST |

| SKEWED_STRING_LIST_VALUES |

| SKEWED_VALUES |

| SORT_COLS |

| TABLE_PARAMS |

| TAB_COL_STATS |

| TBLS |

| TBL_COL_PRIVS |

| TBL_PRIVS |

| TYPES |

| TYPE_FIELDS |

| VERSION |

+---------------------------+

41 rows in set (0.00 sec)

配置完成後就可在shell中以CLI的方式訪問hive 進行操做驗證。

[三]、Remote方式

以Mysql數據庫(192.168.6.77)爲例：建立好用戶：hive；database：hive_meta。Remote方式須要分別配置服務端和客戶端的配置文件：

服務端的 hive-site.xml 中jdbc URL、驅動、用戶名、密碼等屬性值配置以下：

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://192.168.6.77/hive_meta?createDatabaseIfNotExist=true</value>

<description>JDBC connect string for a JDBC metastore</description>

</property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

<description>Driver class name for a JDBC metastore</description>

</property>

<name>javax.jdo.option.ConnectionUserName</name>

<description>username to use against metastore database</description>

</property>

<name>javax.jdo.option.ConnectionPassword</name>

<value>micmiu</value>

<description>password to use against metastore database</description>

</property>

<name>hive.metastore.warehouse.dir</name>

<value>/user/hive/warehouse</value>

<description>base hdfs path :location of default database for the warehouse</description>

</property>

ps：須要把mysql的驅動包copy到目錄 <HIVE_HOME>/lib 中

若是是第一次須要執行初始化命令：schematool -dbType mysql -initSchema

客戶端中配置內容修改以下：

<name>hive.metastore.uris</name>

<value>thrift://192.168.6.77:9083</value>

<description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description>

</property>

<name>hive.metastore.warehouse.dir</name>

<value>/user/hive/warehouse</value>

<description>location of default database for the warehouse</description>

</property>

hive metastore 服務端啓動命令：
hive --service metastore -p <port_num>
若是不加端口默認啓動：hive --service metastore，則默認監聽端口是：9083 ，注意客戶端中的端口配置須要和啓動監聽的端口一致。服務端啓動正常後，客戶端就能夠執行hive操做了。

參考：

https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin