Hive之執行計劃分析(explain)
- Hive是經過把sql轉換成對應mapreduce程序，而後提交到Hadoop上執行，查看具體的執行計劃能夠經過執行explain sql知曉
- 一條sql會被轉化成由多個階段組成的步驟，每一個步驟有執行順序和依賴關係，能夠稱之爲有向無環圖（DAG:Directed Acyclic Graph）
- 這些步驟可能包含：元數據的操做，文件系統的操做，map/reduce計算等
- 語法格式：
EXPLAIN [EXTENDED|DEPENDENCY|AUTHORIZATION|LOCKS|VECTORIZATION] query
- explain輸出內容包括：
- 抽象語法樹
- 執行計劃不一樣階段的依賴關係
- 各個階段的描述
- extended輸出更加詳細的信息
- denpendency輸出依賴的數據源
- authorization輸出執行sql受權信息
- locks 輸出鎖狀況
- vectorization相關
- Adds detail to the EXPLAIN output showing why Map and Reduce work is not vectorized.
- Syntax: EXPLAIN VECTORIZATION [ONLY] [SUMMARY|OPERATOR|EXPRESSION|DETAIL]
- ONLY option suppresses most non-vectorization elements.
- SUMMARY (default) shows vectorization information for the PLAN (is vectorization enabled) and a summary of Map and Reduce work.
- OPERATOR shows vectorization information for operators. E.g. Filter Vectorization. Includes all information of SUMMARY.
- EXPRESSION shows vectorization information for expressions. E.g. predicateExpression. Includes all information of SUMMARY and OPERATOR.
- DETAIL shows detail-level vectorization information. It includes all information of SUMMARY, OPERATOR, and EXPRESSION.
- 帶上FORMATTED 關鍵子，能夠json格式輸出
- sort order： +表示升序 -表示降序
- 大概瞭解一下相關的執行狀況
# explain默認
0: jdbc:hive2://> explain select * from sort_test sort by id desc limit 10;
+--------------------------------------------------------------------------------------------------+--+
| Explain |
+--------------------------------------------------------------------------------------------------+--+
| STAGE DEPENDENCIES: |
| Stage-1 is a root stage |
| Stage-2 depends on stages: Stage-1 |
| Stage-0 depends on stages: Stage-2 |
| |
| STAGE PLANS: |
| Stage: Stage-1 |
| Map Reduce |
| Map Operator Tree: |
| TableScan |
| alias: sort_test |
| Statistics: Num rows: 8 Data size: 890 Basic stats: COMPLETE Column stats: NONE |
| Select Operator |
| expressions: id (type: int), name (type: string) |
| outputColumnNames: _col0, _col1 |
| Statistics: Num rows: 8 Data size: 890 Basic stats: COMPLETE Column stats: NONE |
| Reduce Output Operator |
| key expressions: _col0 (type: int) |
| sort order: - |
| Statistics: Num rows: 8 Data size: 890 Basic stats: COMPLETE Column stats: NONE |
| value expressions: _col1 (type: string) |
| Reduce Operator Tree: |
| Select Operator |
| expressions: KEY.reducesinkkey0 (type: int), VALUE._col0 (type: string) |
| outputColumnNames: _col0, _col1 |
| Statistics: Num rows: 8 Data size: 890 Basic stats: COMPLETE Column stats: NONE |
| Limit |
| Number of rows: 10 |
| Statistics: Num rows: 8 Data size: 890 Basic stats: COMPLETE Column stats: NONE |
| File Output Operator |
| compressed: false |
| table: |
| input format: org.apache.hadoop.mapred.SequenceFileInputFormat |
| output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat |
| serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe |
| |
| Stage: Stage-2 |
| Map Reduce |
| Map Operator Tree: |
| TableScan |
| Reduce Output Operator |
| key expressions: _col0 (type: int) |
| sort order: - |
| Statistics: Num rows: 8 Data size: 890 Basic stats: COMPLETE Column stats: NONE |
| value expressions: _col1 (type: string) |
| Reduce Operator Tree: |
| Select Operator |
| expressions: KEY.reducesinkkey0 (type: int), VALUE._col0 (type: string) |
| outputColumnNames: _col0, _col1 |
| Statistics: Num rows: 8 Data size: 890 Basic stats: COMPLETE Column stats: NONE |
| Limit |
| Number of rows: 10 |
| Statistics: Num rows: 8 Data size: 890 Basic stats: COMPLETE Column stats: NONE |
| File Output Operator |
| compressed: false |
| Statistics: Num rows: 8 Data size: 890 Basic stats: COMPLETE Column stats: NONE |
| table: |
| input format: org.apache.hadoop.mapred.TextInputFormat |
| output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat |
| serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe |
| |
| Stage: Stage-0 |
| Fetch Operator |
| limit: 10 |
| Processor Tree: |
| ListSink |
| |
+--------------------------------------------------------------------------------------------------+--+
# authorization
0: jdbc:hive2://> explain formatted authorization select * from sort_test sort by id desc limit 10;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| Explain |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| {"CURRENT_USER":"root","OPERATION":"SWITCHDATABASE","INPUTS":["badou@sort_test"],"OUTPUTS":["hdfs://master:9000/tmp/hive/root/fac1e10c-babb-4927-886e-411b3e9190fb/hive_2018-10-18_11-04-47_534_1155924552647075339-1/-mr-10000"]} |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
參考資料
【0】Hive wiki - LanguageManual Explainsql
【1】hive入門學習：explain執行計劃的理解express
相關文章
- 1. mysql之SQL執行計劃分析EXPLAIN
- 2. ☆explain執行計劃解析
- 3. MySQL之Explain(執行計劃)
- 4. MongoDB之——explain 執行計劃
- 5. explain-SQL語句執行計劃分析
- 6. 【深刻MySQL 】Explain執行計劃分析
- 7. MYSQL——explain執行計劃結果分析
- 8. (4) MySQL中EXPLAIN執行計劃分析
- 9. Hive----查詢執行計劃(explain)和分析表數據(ANALYZE)
- 10. explain執行計劃
- 更多相關文章...
- • 屏幕分辨率 統計 - 瀏覽器信息
- • 高併發系統的分析和設計 - 紅包項目實戰
- • 互聯網組織的未來：剖析GitHub員工的任性之源
- • 使用Rxjava計算圓周率
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. 字節跳動21屆秋招運營兩輪面試經驗分享
- 2. Java 3 年，25K 多嗎？
- 3. mysql安裝部署
- 4. web前端開發中父鏈和子鏈方式實現通信
- 5. 3.1.6 spark體系之分佈式計算-scala編程-scala中trait特性
- 6. dataframe2
- 7. ThinkFree在線
- 8. 在線畫圖
- 9. devtools熱部署
- 10. 編譯和鏈接
歡迎關注本站公眾號,獲取更多信息
