021-cdh6.2+kylin2.6.2

這是堅持技術寫做計劃（含翻譯）的第21篇，定個小目標999，每週最少2篇。

本文主要介紹，如何使用大數據神獸Kylin(2.6.2)鏈接cdh6.2。

提示

• 由於cdh6.2使用的是hadoop3，而目前的kylin3.0beta版本只是hadoop2,因此只能安裝kylin2.5+,此處選擇kylin2.6.2-cdh60（cdh6.0版）

安裝kylin

下載kylin2.6.2二進制包

wget http://mirrors.tuna.tsinghua.edu.cn/apache/kylin/apache-kylin-2.6.2/apache-kylin-2.6.2-bin-cdh60.tar.gz
tar zxf apache-kylin-2.6.2-bin-cdh60.tar.gz -C /usr/local/
ln -s /usr/local/apache-kylin-2.6.2-bin-cdh60 /usr/local/kylin
複製代碼

配置kylin環境變量

cat << EOF | sudo tee -a /etc/profile
#設置java環境
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/
export CLASSPATH=.:\$JAVA_HOME/lib:\$JAVA_HOME/jre/lib:\$CLASSPATH
export KYLIN_HOME=/usr/local/kylin
export PATH=\$JAVA_HOME/bin:\$JAVA_HOME/jre/bin:\$PATH
export CDH_HOME=/opt/cloudera/parcels/CDH
export HBASE_HOME=\${CDH_HOME}/lib/hbase
export HBASE_CLASSPATH=\${HBASE_HOME}/lib/hbase-common-2.1.0-cdh6.2.0.jar
EOF
source /etc/profile
複製代碼

若是不加 $HBASE_HOME 會報 hbase-common lib not found

Retrieving hadoop conf dir...
KYLIN_HOME is set to /usr/local/kylin
Retrieving hive dependency...
Retrieving hbase dependency...
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
hbase-common lib not found
複製代碼

在hdfs建立kylin和spark目錄

export HADOOP_USER_NAME=hdfs
複製代碼

不然會報

$KYLIN_HOME/bin/check-env.sh
Retrieving hadoop conf dir...
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
KYLIN_HOME is set to /usr/local/kylin
mkdir: Permission denied: user=root, access=WRITE, inode="/kylin":hdfs:supergroup:drwxr-xr-x
Failed to create hdfs:///kylin/spark-history. Please make sure the user has right to access hdfs:///kylin/spark-history
複製代碼

yum install -y net-tools
複製代碼

不然會報

$KYLIN_HOME/bin/check-env.sh
Retrieving hadoop conf dir...
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
KYLIN_HOME is set to /usr/local/kylin
/usr/local/kylin/bin/check-port-availability.sh: line 27: netstat: command not found
複製代碼

下載spark

$KYLIN_HOME/bin/download-spark.sh
複製代碼

不然會報

$KYLIN_HOME/bin/kylin.sh start
Retrieving hadoop conf dir...
錯誤: 找不到或沒法加載主類 org.apache.hadoop.hbase.util.GetJavaProperty
KYLIN_HOME is set to /usr/local/kylin
Retrieving hive dependency...
Retrieving hbase dependency...
錯誤: 找不到或沒法加載主類 org.apache.hadoop.hbase.util.GetJavaProperty
Retrieving hadoop conf dir...
錯誤: 找不到或沒法加載主類 org.apache.hadoop.hbase.util.GetJavaProperty
Retrieving kafka dependency...
Retrieving Spark dependency...
spark not found, set SPARK_HOME, or run bin/download-spark.sh
複製代碼

若是知己指定了不兼容的spark版本，可能會致使404，參考 Kylin web UI http 404 error

啓動kylin

$KYLIN_HOME/bin/kylin.sh start
複製代碼

若是成功會輸出

A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
Check the log at /usr/local/kylin/logs/kylin.log
Web UI is at http://<hostname>:7070/kylin
複製代碼

瀏覽器打開 http://IP:7070/kylin ，用戶名密碼是 ADMIN/KYLIN 

使用kylin(以官方demo演示)

導入數據

$KYLIN_HOME/bin/sample.sh

Retrieving hadoop conf dir...
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
Loading sample data into HDFS tmp path: /tmp/kylin/sample_cube/data
Going to create sample tables in hive to database DEFAULT by cli
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hive-common-2.1.1-cdh6.2.0.jar!/hive-log4j2.properties Async: false
OK
//....
Sample cube is created successfully in project 'learn_kylin'.

** Restart Kylin Server or click Web UI => System Tab => Reload Metadata to take effect **

複製代碼

從新加載元數據

選擇 learn_kylin

構建Cube

選擇 Model，選擇kylin_sales_model,選擇build

此處選擇起止日期。

若是沒關閉hdfs權限校驗，此處確定會build失敗。能夠經過右側 > 圖標點擊查看進度。

build成功後，回到Insight界面，此時已經成功構建出5張表了。

講解demo表

Kylin的示例是銷售業務分析

• KYLIN_SALES 事實表，存有銷售訂單的詳細信息(賣家，商品分類，訂單金額，商品數量等)
• KYLIN_COUNTRY 維度表，存有國家信息(簡寫，名稱等)
• KYLIN_CATEGORY_GROUPINGS 維度表，存有商品分類的詳細介紹(分類名稱等)
• KYLIN_CAL_DT 維度表，存有時間擴展信息(日期所在年始，月始，周始，年份，月份等)
• KYLIN_ACCOUNT 維度表，存有帳戶信息(帳戶id，賣家等級，買家等級，國家等)

運行查詢語句

執行 select count(1) from kylin_sales 點擊submit，下方會顯示執行結果，以及執行耗時(此處是1.8秒)。kylin會緩存執行結果，再次執行發現變成了0.18秒

執行稍微複雜的SQL語句

select sum(KYLIN_SALES.PRICE) 
as price_sum,KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME 
from KYLIN_SALES inner join KYLIN_CATEGORY_GROUPINGS
on KYLIN_SALES.LEAF_CATEG_ID = KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID and 
KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
group by KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME
order by KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME asc,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME desc
複製代碼

自帶簡單的可視化。

參考資料

• 如何在CDH中部署及使用Kylin
• Kylin web UI http 404 error
• Kylin 2.6.1 on Ambari 2.7.1.0 花式踩坑集錦

招聘小廣告

山東濟南的小夥伴歡迎投簡歷啊 加入咱們 , 一塊兒搞事情。長期招聘，Java程序員，大數據工程師，運維工程師，前端工程師。