hive整合phoenix

版本：

hbase-0.98.21-hadoop2-bin.tar.gz

phoenix-4.8.0-HBase-0.98-bin.tar.gz

apache-hive-1.2.1-bin.tar.gz

--------------------------------------------------

首先須要phoenix整合hbase

hive整合hbase，此處參照以前的筆記

將phoenix{core,queryserver,4.8.0-HBase-0.98,hive}拷貝到$hive/lib/

根據官網要求修改配置文件

> vim conf/hive-env.sh

> vim conf/hive-site.xml

啓動：

> hive -hiveconf phoenix.zookeeper.quorum=hadoop01:2181

建立內部表

create table phoenix_table (

s1 string,

i1 int,

f1 float,

d1 double

)

STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler'

TBLPROPERTIES (

"phoenix.table.name" = "phoenix_table",

"phoenix.zookeeper.quorum" = "hadoop01",

"phoenix.zookeeper.znode.parent" = "/hbase",

"phoenix.zookeeper.client.port" = "2181",

"phoenix.rowkeys" = "s1, i1",

"phoenix.column.mapping" = "s1:s1, i1:i1, f1:f1, d1:d1",

"phoenix.table.options" = "SALT_BUCKETS=10, DATA_BLOCK_ENCODING='DIFF'"

);

建立成功。查詢phoenix和hbase中都有相應的表生成：phoenix

hbase：

屬性

1. phoenix.table.name

• • phoenix指定表名
  • 默認值:hive同樣的表

1. phoenix.zookeeper.quorum

• • 指定ZK地址
  • 默認值:localhost

1. phoenix.zookeeper.znode.parent

• • 指定HBase在ZK的目錄
  • 默認值:/ hbase

1. phoenix.zookeeper.client.port

• • 指定ZK端口
  • 默認值:2181

1. phoenix.rowkeys

• • 指定phoenix的rowkey，即hbase的rowkey
  • 要求

1. phoenix.column.mapping

• • hive與phoenix之間的列映射。

插入數據

使用hive測試表pokes導入數據

> insert into table phoenix_table select bar,foo,12.3 as fl,22.2 as dl from pokes;

成功、查詢

在phoenix中查詢

還能夠使用phoenix導入數據，看官網的解釋

注意：phoenix4.8認爲加tbale關鍵字爲語法錯誤，其餘版本沒試，不知道官網怎麼沒說明

建立外部表

For external tables Hive works with an existing Phoenix table and manages only Hive metadata. Deleting an external table from Hive only deletes Hive metadata and keeps Phoenix table

首先在phoenix建立表

phoenix> create table PHOENIX_TABLE_EXT(aa varchar not null primary key,bb varchar);

再在hive中建立外部表：

create external table phoenix_table_ext_1 ( aa string, bb string ) STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler' TBLPROPERTIES ( "phoenix.table.name" = "phoenix_table_ext ", "phoenix.zookeeper.quorum" = "hadoop01", "phoenix.zookeeper.znode.parent" = "/hbase", "phoenix.zookeeper.client.port" = "2181", "phoenix.rowkeys" = "aa", "phoenix.column.mapping" = "aa:aa, bb:bb" );

建立成功，插入成功

這些選項能夠設置在hive CLI

性能調優

參數	默認值	描述
phoenix.upsert.batch.size	1000	批量大小插入。
[phoenix-table-name].disable.wal	false	它暫時設置表屬性DISABLE_WAL = true。可用於提升性能
[phoenix-table-name].auto.flush	false	當WAL是disabled 的flush又爲真，則按文件刷進庫

查詢數據

能夠使用HiveQL在phoenix表查詢數據。一個簡單表查詢當hive.fetch.task.conversion=more and hive.exec.parallel=true.就能夠像在Phoenix CLI同樣快。

參數	默認值	描述
hbase.scan.cache	100	爲一個單位請求讀取行大小。
hbase.scan.cacheblock	false	是否緩存塊。
split.by.stats	false	If true, mappers will use table statistics. One mapper per guide post.
[hive-table-name].reducer.count	1	reducer的數量. In tez mode is affected only single-table query. See Limitations
[phoenix-table-name].query.hint		Hint for phoenix query (like NO_INDEX)

遇到的問題：

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.hadoop.hbase.client.Scan.isReversed()Z

最開始我用的hbase-0.96.2-hadoop2版本，不能整合，這個是須要hbase-client-0.98.21-hadoop2.jar包，更換這個jar包就解決了，可是仍是會報下面的錯

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:ERROR 103 (08004): Unable to establish connection.

因而更換了hbase的版本爲0.98.21的 ok了

---------

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.StringIndexOutOfBoundsException: String index out of range: -1

由於字段對應不同

create table phoenix_table_3 (a string,b int) STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler' TBLPROPERTIES ("phoenix.table.name" = "phoenix_table_3","phoenix.zookeeper.quorum" = "hadoop01","phoenix.zookeeper.znode.parent" = "/hbase","phoenix.zookeeper.client.port" = "2181","phoenix.rowkeys" = "a1","phoenix.column.mapping" = "a:a1, b:b1","phoenix.table.options" = "SALT_BUCKETS=10, DATA_BLOCK_ENCODING='DIFF'");

hive表字段與phoenix字段同樣就能夠了

----------

建立成功，插入也能成功，就是hive查詢的時候報錯找不到a1列，由於phoenix是aa列

Failed with exception java.io.IOException:java.lang.RuntimeException: org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703): Undefined column. columnName=A1

create external table phoenix_table_ext (a1 string,b1 string)STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler' TBLPROPERTIES ("phoenix.table.name" = "phoenix_table_ext","phoenix.zookeeper.quorum" = "hadoop01","phoenix.zookeeper.znode.parent" = "/hbase","phoenix.zookeeper.client.port" = "2181","phoenix.rowkeys" = "aa","phoenix.column.mapping" = "a1:aa, b1:bb");

解決辦法：同上hive表字段與phoenix字段同樣就能夠了