Spark中Shuffle的前世今生

1.Shuffle的變遷 Spark 0.8及以前 Hash Based Shuffle Spark 0.8.1 爲Hash Based Shuffle引入File Consolidation機制 Spark 0.9 引入ExternalAppendOnlyMap Spark 1.1 引入Sort Based Shuffle，但默認仍爲Hash Based Shuffle Spark 1.2 默認

>>阅读原文<<