spark1.4 Windows local調試環境搭建總結

時間 2019-12-05

標籤 spark1.4 spark windows local 調試環境搭建總結欄目 Spark 简体版

原文原文鏈接

spark1.4 Windows local調試環境搭建總結ide

1.scala版本 scala-2.10.4 官方推薦 scala-2.11.7【不推薦，非sbt項目、須要後加載】oop

2.spark版本 spark-1.4.0-bin-hadoop2.6.tgzurl

3.hadoop 3.1版本 hadoop-2.6.0.tar.gz 3.2環境變量 HADOOP_HOME=E:/ysg.tools/spark/hadoop-2.6.0 或 System.setProperty("hadoop.home.dir", "E:\ysg.tools\spark\hadoop-2.6.0"); 3.3winutils.exeidea

winutils.exe拷貝至spark/hadoop-2.6.0/bin

文件下載地址 http://files.cnblogs.com/files/yjmyzz/hadoop2.6%28x64%29V0.2.zipspa

4.idea 新建 NO-SBT項目scala

libraties 增長 scala sdk 
spark-1.4.0-bin-hadoop2.6\lib\spark-assembly-1.4.0-hadoop2.6.0.jar

spark.test.iml 先加載 spark-assembly-1.4.0-hadoop2.6.0 再加載 scala-sdk-2.11.7調試

<?xml version="1.0" encoding="UTF-8"?>
<module type="JAVA_MODULE" version="4">
  <component name="NewModuleRootManager" inherit-compiler-output="true">
    <exclude-output />
    <content url="file://$MODULE_DIR$">
      <sourceFolder url="file://$MODULE_DIR$/src" isTestSource="false" />
    </content>
    <orderEntry type="inheritedJdk" />
    <orderEntry type="sourceFolder" forTests="false" />
    <orderEntry type="library" name="spark-assembly-1.4.0-hadoop2.6.0" level="project" />

    <orderEntry type="library" name="scala-sdk-2.11.7" level="project" />
  </component>
</module>