SPARK 2.4.0 學習筆記分享

SPARK 2.4.0 學習筆記分享

• 【本站點正在持續更新中......2018-12-26......】
• 微博: https://weibo.com/thinktothings
• SPARK 2.4.0 學習筆記分享(bilibili整套視頻): https://www.bilibili.com/video/av38193405/

更多資源

• SPARK 1.6.0-cdh5.15.0 源碼分析: https://github.com/opensourceteams/spark-scala-maven

前置條件

• Hadoop版本: hadoop-2.9.2
• Spark版本: spark-2.4.0-bin-hadoop2.7
• Hive版本: apache-hive-3.1.1-bin
• JDK.1.8.0_191
• scala2.11.12

Spark 環境配置

大數據開發工具介紹

• 大數據開發工具介紹(文檔說明): https://github.com/opensourceteams/spark-scala-maven-2.4.0/blob/master/md/tool.md

Spark 2.4.0 standalone 模式安裝

• Spark 2.4.0 standalone 模式安裝（詳細說明文檔)： https://github.com/opensourceteams/spark-scala-maven-2.4.0/blob/master/md/standaloneInstall.md

Spark 2.4.0 編程指南

快速入門(Quick Start)

• a quick introduction to the Spark API; start here!
• 快速介紹Spark API;從這裏開始

Spark 2.4.0 編程指南--快速入門

• Spark 2.4.0 編程指南--快速入門(文檔說明): https://github.com/opensourceteams/spark-scala-maven-2.4.0/blob/master/md/quick-start.md
• Spark 2.4.0 編程指南--快速入門(bilibili視頻) : https://www.bilibili.com/video/av38193405/?p=2

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=38193405&cid=67137841&page=2" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark SQL, Datasets, and DataFrames

• processing structured data with relational queries (newer API than RDDs)
• 使用關係查詢處理結構化數據（比RDD更新的API）

Spark 2.4.0編程指南--spark sql入門

• Spark 2.4.0編程指南--spark dataSet action(文檔說明): https://github.com/opensourceteams/spark-scala-maven-2.4.0/blob/master/md/sql/spark-sql-dataSet-action.md
• Spark 2.4.0編程指南--spark dataSet action(bilibili視頻) : https://www.bilibili.com/video/av38193405/?p=3

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=38193405&cid=67137841&page=3" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>

Spark 2.4.0編程指南--Spark SQL UDF和UDAF

• Spark 2.4.0編程指南--Spark SQL UDF和UDAF(文檔說明): https://github.com/opensourceteams/spark-scala-maven-2.4.0/blob/master/md/sql/spark-sql-UDF-UDAF.md

Spark 2.4.0 集成Hive 2.3.4

• Spark 2.4.0 集成Hive 2.3.4(文檔說明): https://github.com/opensourceteams/spark-scala-maven-2.4.0/blob/master/md/SparkAndHive.md

Spark 2.4.0編程指南--Spark DataSources

• Spark 2.4.0編程指南--Spark DataSources(文檔說明): https://github.com/opensourceteams/spark-scala-maven-2.4.0/blob/master/md/spark-data-source.md

Spark 2.4.0 源碼分析(建設中)

Spark2.4.0 Dataset head 源碼分析

• Spark2.4.0 Dataset head 源碼分析(文檔說明): https://github.com/opensourceteams/spark-scala-maven-2.4.0/blob/master/md/sql/Dataset-head.md

<iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=38193405&cid=68636905&page=6" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe> <iframe width="800" height="500" src="//player.bilibili.com/player.html?aid=38193405&cid=68636905&page=7" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> </iframe>