SPARK 源碼分析技術分享 (帶bilibili視頻)

SPARK 源碼分析技術分享 (帶bilibili視頻)

【本站點正在持續更新中......2018-12-09......】

• SPARK 1.6.0-cdh5.15.0
• Hadoop 2.6.0-cdh5.15.0
• spark-scala-maven
• 微信(技術交流) : thinktothings
• SPARK 源碼分析技術分享(視頻彙總套裝視頻): https://www.bilibili.com/video/av37442139/
• SPARK 源碼分析技術分享(視頻彙總在線看):https://blog.csdn.net/thinktothings/article/details/84726769
• SPARK 源碼分析技術分享 (github) : https://github.com/opensourceteams/spark-scala-maven

RDD依賴 Dependency

NarrowDependency

• OneToOneDependency

  • bilibili 視頻: https://www.bilibili.com/video/av37442139/?p=1
  • 詳細說明文檔： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/OneToOneDenpendency.md

  <iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=65822237&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

• RangeDependency

  • bilibili 視頻: https://www.bilibili.com/video/av37442139/?p=2
  • 詳細說明文檔： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/RangeDependency.md

  <iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=65822237&page=2" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

• PruneDependency filter

  • bilibili 視頻: https://www.bilibili.com/video/av37442139/?p=3
  • 詳細說明文檔： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/PruneDependency.md

  <iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=65822237&page=3" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

• PruneDependency RangePartitioner

  • bilibili 視頻: https://www.bilibili.com/video/av37442139/?p=4
  • 詳細說明文檔： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/PruneDependencyRangePartitioner.md

  <iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=65822237&page=4" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

ShuffleDependency

• bilibili 視頻: https://www.bilibili.com/video/av37442139/?p=5
• 詳細說明文檔：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/ShuffleDependency.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=65822237&page=5" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

SparkContext 分析

• SparkContext原理分析(Youtube視頻): https://youtu.be/euIuutjAB4I
• SparkContext源碼分析(Youtube視頻): https://youtu.be/tUH7QnCcwgg
• SparkContext源碼分析(文檔詳解）： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/SparkContext.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442161&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark 通訊原理分析

• Spark通訊原理分析(Youtube視頻): https://youtu.be/3vUVwbEGf1E
• Spark通訊原理分析（文檔詳解）： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/OutBoxAndInBox.md
• bilibili : https://www.bilibili.com/video/av37442199/

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442199&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark Master啓動源碼分析

• Spark Master啓動源碼分析(Youtube視頻): https://youtu.be/74q1nddoaiY​
• Spark Master啓動源碼分析(Bilibili視頻): https://www.bilibili.com/video/av37442271/
• Master啓動源碼分析詳細說明文檔： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/StartMaster.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442271&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

• Spark Master資源調度--worker向master註冊(Youtube視頻): https://youtu.be/74q1nddoaiY​
• Spark Master資源調度--worker向master註冊(Bilibili視頻): https://www.bilibili.com/video/av37442280/
• Spark Master資源調試--worker向master註冊(文檔詳解)： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/MasterScheduler_workerRegisterMaster.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442280&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

• Spark Master資源調度--SparkContext向全部master註冊(Youtube視頻): https://youtu.be/AXxCnCc5Mh0​
• Spark Master資源調度--SparkContext向全部master註冊(Bilibili視頻): https://www.bilibili.com/video/av37442295/
• Spark Master資源調度--SparkContext向全部master註冊(文檔詳解): https://github.com/opensourceteams/spark-scala-maven/blob/master/md/MasterScheduler_SparkContextRegisterMaster.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442295&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark Worker啓動源碼分析

• Spark Worker 啓動源碼分析(Youtube視頻): https://youtu.be/ll_Ae6rP7II​​
• Spark Worker 啓動源碼分析(Bilibili視頻): https://www.bilibili.com/video/av37442247/
• Spark Worker 啓動源碼分析(文檔詳解): https://github.com/opensourceteams/spark-scala-maven/blob/master/md/StartWorker.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442247&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark Executor啓動源碼分析

• Spark Executor啓動源碼分析(Youtube視頻): https://youtu.be/1qg4UMPV3pQ
• Spark Executor啓動源碼分析(Bilibili視頻): https://www.bilibili.com/video/av37442311/
• Spark Executor啓動源碼分析(文檔詳解)： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/CoarseGrainedExecutorBackend_start.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442311&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark 觸發Job提交

• Spark 觸發Job提交(youtube視頻) : https://youtu.be/X49RIqz2AjM
• Spark 觸發Job提交(bilibili視頻) : https://www.bilibili.com/video/av37445008/
• Spark 觸發Job提交(文檔詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/jobSubmitTrigger.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37445008&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark DAG調度器事件循環處理器

• Spark DAG調度器事件循環處理器(Youtube視頻) : https://youtu.be/fT-dpf0KFOA
• Spark DAG調度器事件循環處理器(Bilibili視頻) : https://www.bilibili.com/video/av37445034/
• Spark DAG調度器事件循環處理器(文檔詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/DAGSchedulerEventProcessLoop.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37445034&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark DAGScheduler FinalStage處理(Stage劃分)

• Spark FinalStage處理(Stage劃分)(Youtube視頻) : https://youtu.be/yFJugOV0Fak
• Spark FinalStage處理(Stage劃分)(Bilibili視頻) : https://www.bilibili.com/video/av37445057/
• Spark FinalStage處理(Stage劃分)(文檔詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/FinalStage.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37445057&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark DAGScheduler Stage提交

• Spark Stage提交(Youtube視頻) : https://youtu.be/NI8-_X6mbl4
• Spark Stage提交(Bilibili視頻) : https://www.bilibili.com/video/av37445077/
• Spark Stage提交(文檔詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/SubmitStage.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37445077&page=1" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark DAGScheduler把stage轉換成TaskSet的過程

• Spark DAGScheduler把stage轉換成TaskSet的過程(Bilibili視頻) https://www.bilibili.com/video/av37442139/?p=18
• Spark DAGScheduler把stage轉換成TaskSet的過程(文檔詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/DAGScheduler_stageToTaskSet.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=65893099&page=18" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark TaskSchedulerImpl 任務調度方式(FIFO)

• Spark TaskSchedulerImpl 任務調度方式(FIFO)(文檔詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/SchedulerFIFO.md
• Spark TaskSchedulerImpl 任務調度方式(FIFO)(bilibili視頻) : https://www.bilibili.com/video/av37442139/?p=19

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66005253&page=19" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark TaskSchedulerImpl TaskSet處理

• Spark TaskSchedulerImpl TaskSet處理(文檔詳解)：https://github.com/opensourceteams/spark-scala-maven/blob/master/md/TaskSchedulerTaskSetDeal.md
• Spark TaskSchedulerImpl TaskSet原理分析(bilibili視頻) : https://www.bilibili.com/video/av37442139/?p=20
• Spark TaskSchedulerImpl TaskSet原碼分析(bilibili視頻) : https://www.bilibili.com/video/av37442139/?p=21

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66006637&page=20" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66008946&page=21" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark 源碼分析之ShuffleMapTask處理

• Spark 源碼分析之ShuffleMapTask處理(文檔詳解): https://github.com/opensourceteams/spark-scala-maven/blob/master/md/ShuffleMapTask.md
• Spark 源碼分析之ShuffleMapTask處理原理分析圖解 (bilibili視頻): https://www.bilibili.com/video/av37442139/?p=22
• Spark 源碼分析之ShuffleMapTask處理源碼分析 (bilibili視頻): https://www.bilibili.com/video/av37442139/?p=23
• Spark 源碼分析之ShuffleMapTask處理原理分析圖解 (youtube視頻): https://youtu.be/datHorBipMc
• Spark 源碼分析之ShuffleMapTask處理源碼分析 (youtube視頻): https://youtu.be/cRW_MZ0k5Lw

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66008946&page=22" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66008946&page=23" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark 源碼分析之ResultTask處理

• Spark 源碼分析之ResultTask處理(文檔詳情):https://github.com/opensourceteams/spark-scala-maven/blob/master/md/ResultTask.md
• Spark 源碼分析之ResultTask原理分析圖解(bilibili視頻):https://www.bilibili.com/video/av37442139/?p=24
• Spark 源碼分析之ResultTask處理(bilibili視頻):https://www.bilibili.com/video/av37442139/?p=25

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66008946&page=24" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66008946&page=25" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Hadoop RDD

• HadoopRdd partition的開始位置計算(文檔詳情): https://github.com/opensourceteams/spark-scala-maven/blob/master/md/HadoopRddPartitionDivide.md

• Hadoop RDD 讀取文件(文檔詳情): https://github.com/opensourceteams/spark-scala-maven/blob/master/md/HadoopRDD_2.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66303785&page=26" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66303785&page=27" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=66303785&page=28" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

======================================================================================

Spark 遠程調試

• 詳細說明文檔： https://github.com/opensourceteams/spark-scala-maven/blob/master/md/SparkRemoteDebug.md

相關連接

• HadoopRDD源碼分析說明文檔詳細

Youtub 視頻

• HadoopRdd源碼分析-讀取本地文件需求分析-01
• HadoopRDD源碼分析-文件拆分partition劃分-02
• HadoopRdd源碼分析 本地文件讀取源碼分析 03
• HadoopRdd源碼分析 本地文件讀取源碼分析 04