hadoop&spark mapreduce對比 & 框架設計和理解

Hadoop MapReduce:

MapReduce在每次執行的時候都要從磁盤讀數據，計算完畢後都要把數據放到磁盤

spark map reduce:

RDD is everything for dev:

Basic Concepts:

Graph RDD:

Spark Runtime:

schedule:

Depency Type:

Scheduler Optimizations：

Event Flow:

Submit Job:

New Job Instance:

Job In Detail:

executor.launchTask:

Standalone：

Work Flow:

Standalone detail:

Driver application to Clustor:

Worker Exception:

Executor Exception:

Master Exception:

Master HA: