分析高動態數據集《譯》

時間  2019-12-05
標籤 分析 動態 數據 简体版
原文   原文鏈接

今天的數據是動態的和被驅動的應用。一個商業應用驅動的的新時代的發展由動態產業如網絡、社會、移動和物聯網等與新數據類型和新的數據模型造成數據集。這些應用程序是迭代的,相關的數據模型一般是半結構化的,非模式化和不斷進化的。半結構化數據模型能夠被複雜/嵌套,非模式化,可以在每一行和不一樣領域持續發展達到頻繁添加和刪除以知足業務需求。apache

本教程向您展現瞭如何查詢本地動態數據集,如JSON和在幾分鐘內查詢來自任何類型的數據派生的洞察力。被使用的數據集的例子是Yelp簽到數據集，它具備如下結構:json

check-in
{
    'type': 'checkin',
    'business_id': (encrypted business id),
    'checkin_info': {
        '0-0': (number of checkins from 00:00 to 01:00 on all Sundays),
        '1-0': (number of checkins from 01:00 to 02:00 on all Sundays),
        ...
        '14-4': (number of checkins from 14:00 to 15:00 on all Thursdays),
        ...
        '23-6': (number of checkins from 23:00 to 00:00 on all Saturdays)
    }, # if there was no checkin for a hour-day block it will not be in the dataset
}

值得重複的評論這個代碼片斷的底部:數組

# if there was no checkin for a hour-day block it will not be in the dataset.

你在checkin_info看到的元素名稱是未知的前期而且每一行可能會有所不一樣。數據,雖然簡單,可是高度動態的數據。分析數據不須要先在相關平面結構表示這個數據集,由於您將在Hadoop使用任何其餘SQL技術。網絡

第1步:首先下載鑽,若是你尚未這樣作,到您的機器中函數

http://drill.apache.org/download/
tar -xvf apache-drill-0.9.0.tar

在本地桌面安裝鑽（嵌入模式）。你不須要Hadoop。oop

♣第二步：開始鑽殼。code

bin/drill-embedded

第三步：開始使用SQL分析數據排序

首先，咱們先看一下數據集：教程

0: jdbc:drill:zk=local> SELECT * FROM dfs.`/home/zj/software/drill/yelp/yelp_academic_dataset_checkin.json` limit 2;
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+------------------------+
|                                                                 checkin_info                                                                                                                                                             |    type    |      business_id       |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+------------------------+
| {"3-4":1,"13-5":1,"6-6":1,"14-5":1,"14-6":1,"14-2":1,"14-3":1,"19-0":1,"11-5":1,"13-2":1,"11-6":2,"11-3":1,"12-6":1,"6-5":1,"5-5":1,"9-2":1,"9-5":1,"9-6":1,"5-2":1,"7-6":1,"7-5":1,"7-4":1,"17-5":1,"8-5":1,"10-2":1,"10-5":1,"10-6":1} | checkin    | JwUE5GmEO-sH1FuwJgKBlQ |
| {"6-6":2,"6-5":1,"7-6":1,"7-5":1,"8-5":2,"10-5":1,"9-3":1,"12-5":1,"15-3":1,"15-5":1,"15-6":1,"16-3":1,"10-0":1,"15-4":1,"10-4":1,"8-2":1}                                                                                               | checkin    | uGykseHzyS5xAMWoN6YUqA |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+------------------------+

請注意：
本文檔將鑽例如輸出的目的。在這種狀況下鑽輸出不對齊。文檔

你直接查詢JSON文件中的數據。定義模式在hive倉庫並非必須的。內部元素的名稱 checkin_info列第一行和第二行之間是不一樣的。

▲▲▲▲▲鑽提供了一個名爲KVGEN的函數(鍵值生成器)在處理複雜的數據時是有用的,其中包含任意地組成的動態圖和未知的元素名稱例如checkin_info。KVGEN轉動態映射到一個數組的鍵值對,其中鍵表明動態元素名稱。

讓咱們在checkin_info元素應用KVGEN生成鍵-值對。

0: jdbc:drill:zk=local> SELECT KVGEN(checkin_info) checkins FROM dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_checkin.json` LIMIT 2;
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                    checkins                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| [{"key":"3-4","value":1},{"key":"13-5","value":1},{"key":"6-6","value":1},{"key":"14-5","value":1},{"key":"14-6","value":1},{"key":"14-2","value":1},{"key":"14-3","value":1},{"key":"19-0","value":1},{"key":"11-5","value":1},{"key":"13-2","value":1},{"key":"11-6","value":2},{"key":"11-3","value":1},{"key":"12-6","value":1},{"key":"6-5","value":1},{"key":"5-5","value":1},{"key":"9-2","value":1},{"key":"9-5","value":1},{"key":"9-6","value":1},{"key":"5-2","value":1},{"key":"7-6","value":1},{"key":"7-5","value":1},{"key":"7-4","value":1},{"key":"17-5","value":1},{"key":"8-5","value":1},{"key":"10-2","value":1},{"key":"10-5","value":1},{"key":"10-6","value":1}] |
| [{"key":"6-6","value":2},{"key":"6-5","value":1},{"key":"7-6","value":1},{"key":"7-5","value":1},{"key":"8-5","value":2},{"key":"10-5","value":1},{"key":"9-3","value":1},{"key":"12-5","value":1},{"key":"15-3","value":1},{"key":"15-5","value":1},{"key":"15-6","value":1},{"key":"16-3","value":1},{"key":"10-0","value":1},{"key":"15-4","value":1},{"key":"10-4","value":1},{"key":"8-2","value":1}]                                                                                                                                                                                                                                                                               |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

鑽提供了另外一個函數來操做複雜的數據稱爲「Flatten」，打破「KVGen」形成的鍵值對列表到單獨的行進一步應用分析功能。

0: jdbc:drill:zk=local> SELECT FLATTEN(KVGEN(checkin_info)) checkins FROM dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_checkin.json` LIMIT 20;
+--------------------------+
|         checkins         |
+--------------------------+
| {"key":"3-4","value":1}  |
| {"key":"13-5","value":1} |
| {"key":"6-6","value":1}  |
| {"key":"14-5","value":1} |
| {"key":"14-6","value":1} |
| {"key":"14-2","value":1} |
| {"key":"14-3","value":1} |
| {"key":"19-0","value":1} |
| {"key":"11-5","value":1} |
| {"key":"13-2","value":1} |
| {"key":"11-6","value":2} |
| {"key":"11-3","value":1} |
| {"key":"12-6","value":1} |
| {"key":"6-5","value":1}  |
| {"key":"5-5","value":1}  |
| {"key":"9-2","value":1}  |
| {"key":"9-5","value":1}  |
| {"key":"9-6","value":1}  |
| {"key":"5-2","value":1}  |
| {"key":"7-6","value":1}  |
+--------------------------+

能夠從數據中快速的得到值，經過在數據集應用KVGEN和FLATTEN函數動態數據集在fly——no須要耗時的模式定義和數據存儲中間格式。

夷爲平地的數據輸出,您可使用標準SQL功能,如過濾器、聚合、排序。 讓咱們看看幾個例子。

▲▲▲▲▲在Yelp數據集獲取簽到記錄的總數

0: jdbc:drill:zk=local> SELECT SUM(checkintbl.checkins.`value`) AS TotalCheckins FROM (
. . . . . . . . . . . >  SELECT FLATTEN(KVGEN(checkin_info)) checkins FROM dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_checkin.json` ) checkintbl
. . . . . . . . . . . >  ;
+---------------+
| TotalCheckins |
+---------------+
| 4713811       |
+---------------+

▲▲▲▲▲獲取專門爲週日午夜的簽到的數量

0: jdbc:drill:zk=local> SELECT SUM(checkintbl.checkins.`value`) AS SundayMidnightCheckins FROM (
. . . . . . . . . . . >  SELECT FLATTEN(KVGEN(checkin_info)) checkins FROM dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_checkin.json` ) checkintbl WHERE checkintbl.checkins.key='23-0';
+------------------------+
| SundayMidnightCheckins |
+------------------------+
| 8575                   |
+------------------------+

▲▲▲▲▲獲取每星期幾簽到的數量

0: jdbc:drill:zk=local> SELECT `right`(checkintbl.checkins.key,1) WeekDay,sum(checkintbl.checkins.`value`) TotalCheckins from (
. . . . . . . . . . . >  select flatten(kvgen(checkin_info)) checkins FROM dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_checkin.json`  ) checkintbl GROUP BY `right`(checkintbl.checkins.key,1) ORDER BY TotalCheckins;
+------------+---------------+
|  WeekDay   | TotalCheckins |
+------------+---------------+
| 1          | 545626        |
| 0          | 555038        |
| 2          | 555747        |
| 3          | 596296        |
| 6          | 735830        |
| 4          | 788073        |
| 5          | 937201        |
+------------+---------------+

▲▲▲▲▲獲取一天中每小時的簽到數量

0: jdbc:drill:zk=local> SELECT SUBSTR(checkintbl.checkins.key,1,strpos(checkintbl.checkins.key,'-')-1) AS HourOfTheDay ,SUM(checkintbl.checkins.`value`) TotalCheckins FROM (
. . . . . . . . . . . >  SELECT FLATTEN(KVGEN(checkin_info)) checkins FROM dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_checkin.json` ) checkintbl GROUP BY SUBSTR(checkintbl.checkins.key,1,strpos(checkintbl.checkins.key,'-')-1) ORDER BY TotalCheckins;
+--------------+---------------+
| HourOfTheDay | TotalCheckins |
+--------------+---------------+
| 3            | 20357         |
| 4            | 21076         |
| 2            | 28116         |
| 5            | 33842         |
| 1            | 45467         |
| 6            | 54174         |
| 0            | 74127         |
| 7            | 96329         |
| 23           | 102009        |
| 8            | 130091        |
| 22           | 140338        |
| 9            | 162913        |
| 21           | 211949        |
| 10           | 220687        |
| 15           | 261384        |
| 14           | 276188        |
| 16           | 292547        |
| 20           | 293783        |
| 13           | 328373        |
| 11           | 338675        |
| 17           | 374186        |
| 19           | 385381        |
| 12           | 399797        |
| 18           | 422022        |
+--------------+---------------+

總結

在本教程中,您將衝浪結構化和半結構化數據沒有任何前期或ETL模式管理。

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程