hive explain

hive 語句執行順序

大體順序
from... where.... select...group by... having ... order by...

explain查看執行計劃

hive語句和mysql均可以經過explain查看執行計劃，這樣就能夠查看執行順序，好比

explain
    select city,ad_type,device,sum(cnt) as cnt
    from tb_pmp_raw_log_basic_analysis
    where day = '2016-05-28' and type = 0 and media = 'sohu' and (deal_id = '' or deal_id = '-' or deal_id is NULL)    
    group by city,ad_type,device

顯示執行計劃以下

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Map Operator Tree:
          TableScan
            alias: tb_pmp_raw_log_basic_analysis
            Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
            Filter Operator
              predicate: (((deal_id = '') or (deal_id = '-')) or deal_id is null) (type: boolean)
              Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
              Select Operator
                expressions: city (type: string), ad_type (type: string), device (type: string), cnt (type: bigint)
                outputColumnNames: city, ad_type, device, cnt
                Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
                Group By Operator
                  aggregations: sum(cnt)
                  keys: city (type: string), ad_type (type: string), device (type: string)
                  mode: hash
                  outputColumnNames: _col0, _col1, _col2, _col3
                  Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
                  Reduce Output Operator
                    key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string)
                    sort order: +++
                    Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string)
                    Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
                    value expressions: _col3 (type: bigint)
      Reduce Operator Tree:
        Group By Operator
          aggregations: sum(VALUE._col0)
          keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
          mode: mergepartial
          outputColumnNames: _col0, _col1, _col2, _col3
          Statistics: Num rows: 4097678 Data size: 290028976 Basic stats: COMPLETE Column stats: NONE
          Select Operator
            expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: bigint)
            outputColumnNames: _col0, _col1, _col2, _col3
            Statistics: Num rows: 4097678 Data size: 290028976 Basic stats: COMPLETE Column stats: NONE
            File Output Operator
              compressed: false
              Statistics: Num rows: 4097678 Data size: 290028976 Basic stats: COMPLETE Column stats: NONE
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

  Stage: Stage-0
    Fetch Operator
      limit: -1

具體介紹以下 

**stage1的map階段**
        TableScan：from加載表，描述中有行數和大小等
        Filter Operator：where過濾條件篩選數據，描述有具體篩選條件和行數、大小等
        Select Operator：篩選列，描述中有列名、類型，輸出類型、大小等。
        Group By Operator：分組，描述了分組後須要計算的函數，keys描述用於分組的列，outputColumnNames爲輸出的列名，能夠看出列默認使用固定的別名_col0，以及其餘信息
        Reduce Output Operator：map端本地的reduce，進行本地的計算，而後按列映射到對應的reduce
**stage1的reduce階段Reduce Operator Tree**
        Group By Operator：整體分組，並按函數計算。map計算後的結果在reduce端的合併。描述相似。mode: mergepartial是說合並map的計算結果。map端是hash映射分組
        Select Operator：最後過濾列用於輸出結果
        File Output Operator：輸出結果到臨時文件中，描述介紹了壓縮格式、輸出文件格式。
        stage0第二階段沒有，這裏能夠實現limit 100的操做。

總結

1，每一個stage都是一個獨立的MR，複雜的hql語句能夠產生多個stage，能夠經過執行計劃的描述，看看具體步驟是什麼。
2，執行計劃有時預測數據量，不是真實運行，可能不許確