Hive高級聚合GROUPING SETS,ROLLUP以及CUBE

scala> import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.HiveContext

scala> val hcon=new HiveContext(sc)
warning: there was one deprecation warning; re-run with -deprecation for details
hcon: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@dd102ea

scala> hcon.sql("select age,sex,count(1) from gamedw.customers group by age,sex").show
+---+---+--------+
|age|sex|count(1)|
+---+---+--------+
| 56| 0| 7|
| 32| 1| 7|
| 20| 1| 7|
| 50| 1| 7|
| 5| 1| 4|
| 47| 0| 7|
| 85| 1| 7|
|100| 0| 5|
+---+---+--------+

scala> hcon.sql("select age,sex,count(1) from gamedw.customers group by age,sex grouping sets((age,sex),sex,())").show
+----+----+--------+
| age| sex|count(1)|
+----+----+--------+
| 56| 0| 7|
|null| 1| 32|
| 20| 1| 7|
|null|null| 51|
| 32| 1| 7|
| 5| 1| 4|
| 85| 1| 7|
| 47| 0| 7|
| 100| 0| 5|
|null| 0| 19|
| 50| 1| 7|
+----+----+--------+

 

GROUPING SETS

在一個GROUP BY查詢中，根據不一樣的維度組合進行聚合，等價於將不一樣維度的GROUP BY結果集進行UNION ALL,SETS的子句中若是包含()數據集，則表示總體聚合

scala> hcon.sql("select age,sex,count(1) from gamedw.customers group by age,sex grouping sets((age,sex),sex,()) order by age,sex").show
+----+----+--------+
| age| sex|count(1)|
+----+----+--------+
|null|null| 51|
|null| 0| 19|
|null| 1| 32|
| 5| 1| 4|
| 20| 1| 7|
| 32| 1| 7|
| 47| 0| 7|
| 50| 1| 7|
| 56| 0| 7|
| 85| 1| 7|
| 100| 0| 5|
+----+----+--------+

scala> hcon.sql("select age,sex,count(1) from gamedw.customers group by age,sex grouping sets((age,sex),sex,age,()) order by age,sex").show
+----+----+--------+
| age| sex|count(1)|
+----+----+--------+
|null|null| 51|
|null| 0| 19|
|null| 1| 32|
| 5|null| 4|
| 5| 1| 4|
| 20|null| 7|
| 20| 1| 7|
| 32|null| 7|
| 32| 1| 7|
| 47|null| 7|
| 47| 0| 7|
| 50|null| 7|
| 50| 1| 7|
| 56|null| 7|
| 56| 0| 7|
| 85|null| 7|
| 85| 1| 7|
| 100|null| 5|
| 100| 0| 5|
+----+----+--------+

CUBE

根據GROUP BY的維度的全部組合進行聚合。

scala> hcon.sql("select age,sex,count(1) from gamedw.customers group by age,sex with cube order by age,sex").show
+----+----+--------+
| age| sex|count(1)|
+----+----+--------+
|null|null| 51|
|null| 0| 19|
|null| 1| 32|
| 5|null| 4|
| 5| 1| 4|
| 20|null| 7|
| 20| 1| 7|
| 32|null| 7|
| 32| 1| 7|
| 47|null| 7|
| 47| 0| 7|
| 50|null| 7|
| 50| 1| 7|
| 56|null| 7|
| 56| 0| 7|
| 85|null| 7|
| 85| 1| 7|
| 100|null| 5|
| 100| 0| 5|
+----+----+--------+

ROLLUP

是CUBE的子集，以最左側的維度爲主，從該維度進行層級聚合。

scala> hcon.sql("select age,sex,count(1) from gamedw.customers group by age,sex with rollup order by age,sex").show+----+----+--------+ | age| sex|count(1)|+----+----+--------+|null|null| 51|| 5|null| 4|| 5| 1| 4|| 20|null| 7|| 20| 1| 7|| 32|null| 7|| 32| 1| 7|| 47|null| 7|| 47| 0| 7|| 50|null| 7|| 50| 1| 7|| 56|null| 7|| 56| 0| 7|| 85|null| 7|| 85| 1| 7|| 100|null| 5|| 100| 0| 5|+----+----+--------+