（二）win7下用Intelij IDEA 遠程調試spark standalone 集羣

時間 2019-11-08

標籤 win7 win 下用 intelij idea 遠程調試 spark standalone 集羣欄目 Windows 简体版

原文原文鏈接

關於這個spark的環境搭建了很久，踩了一堆坑，今天html

環境： WIN7筆記本java

　　　 spark 集羣（4個虛擬機搭建的）node

　　　 Intelij IDEA15web

　　　 scala-2.10.4apache

　　　 java-1.7.0windows

版本問題：api

我的選擇的是hadoop2.6.0 spark1.5.0 scala2.10.4 jdk1.7.0app

關於搭建集羣環境，見我的的上一篇博客：(一) Spark Standalone集羣環境搭建，接下來就是用Intelij IDEA來遠程鏈接spark集羣，這樣就能夠方便的在本機上進行調試。dom

首先須要注意windows能夠設置hosts，在 C:\Windows\System32\drivers\etc 有個hosts，把如下映射地址填進去，這樣能省去很多事tcp

172.21.75.102 spark1

172.21.75.194 spark2

172.21.75.122 spark3

172.21.75.95 spark4

1）首先在我的WIN7本上搭好java，scala環境，並配置好環境變量，安裝好Intelij IDEA，並安裝好scala插件。

2）新建Scala項目，選擇Scala：

3）分別引入 java 與 Scala SDK，並對項目命名，這裏一會咱們運行SparkPi的程序，名字能夠隨意

4）進入主界面，雙擊src，或者File->Project Structer,進入程序配置界面

5）點擊library裏「+」，點擊java，添加spark-1.5.0-hadoop-2.6.0的jar包

6）點擊library裏「+」，點擊Scala SDK 添加Scala SDK

7）以上步驟點擊OK退出，在src新建 SparkPi.scala 的scala object文件

8）寫代碼以前，先進行一個jar包設置

9) 這裏的路徑必定要設置好，爲jar包的輸出路徑，一會要寫到程序裏，使得spark集羣的查找

10）選中這裏的Build on make，程序就會編譯後自動打包

11）注意以上的路徑，這個路徑就是提交給spark的jar包

.setJars(List("F:\\jar_package\\job\\SparkPi.jar"))

12）複製以下代碼到SparkPi.scala

import scala.math.random
import org.apache.spark.{SparkConf, SparkContext}

/**
  * Created by Administrator on 2016/5/13.
  */
//alt+Enter自動引入缺失的包
object SparkPi {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Spark Pi").setMaster("spark://172.21.75.102:7077")
      .setJars(List("F:\\jar_package\\job\\SparkPi.jar"))
    val spark = new SparkContext(conf)
    val slices = if (args.length > 0) args(0).toInt else 2
    val n = 100000 * slices
    val count = spark.parallelize(1 to n, slices).map { i =>
      val x = random * 2 - 1
      val y = random * 2 - 1
      if (x * x + y * y < 1) 1 else 0
    }.reduce(_ + _)
    println("Pi is roughly " + 4.0 * count / n)
    spark.stop()
  }
}

View Code

13）如今大功告成，設置Run 的Edit Configuration，點擊+，Application，設置MainClass，點擊OK！

14）點擊Run便可運行程序了，程序會在剛纔的路徑生成對應的jar，而後會啓動spark集羣，去運行該jar文件，如下爲執行結果：

"C:\Program Files\Java\jdk1.7.0_09\bin\java" -Didea.launcher.port=7534 "-Didea.launcher.bin.path=D:\IntelliJ IDEA Community Edition 2016.1.2\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.7.0_09\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\jce.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\jfxrt.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\resources.jar;C:\Program Files\Java\jdk1.7.0_09\jre\lib\rt.jar;F:\IDEA\SparkPi\out\production\SparkPi;C:\Program Files (x86)\scala\lib\scala-actors-migration.jar;C:\Program Files (x86)\scala\lib\scala-actors.jar;C:\Program Files (x86)\scala\lib\scala-library.jar;C:\Program Files (x86)\scala\lib\scala-reflect.jar;C:\Program Files (x86)\scala\lib\scala-swing.jar;F:\jar_package\spark-assembly-1.5.0-hadoop2.6.0.jar;D:\IntelliJ IDEA Community Edition 2016.1.2\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain SparkPi
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/05/13 17:47:43 INFO SparkContext: Running Spark version 1.5.0
16/05/13 17:47:53 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/05/13 17:47:55 INFO SecurityManager: Changing view acls to: Administrator
16/05/13 17:47:55 INFO SecurityManager: Changing modify acls to: Administrator
16/05/13 17:47:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Administrator); users with modify permissions: Set(Administrator)
16/05/13 17:47:58 INFO Slf4jLogger: Slf4jLogger started
16/05/13 17:47:58 INFO Remoting: Starting remoting
16/05/13 17:48:00 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@172.21.75.63:62339]
16/05/13 17:48:00 INFO Utils: Successfully started service 'sparkDriver' on port 62339.
16/05/13 17:48:00 INFO SparkEnv: Registering MapOutputTracker
16/05/13 17:48:00 INFO SparkEnv: Registering BlockManagerMaster
16/05/13 17:48:00 INFO DiskBlockManager: Created local directory at C:\Users\Administrator\AppData\Local\Temp\blockmgr-0046600a-5752-4cd5-89d6-cde41f7011d1
16/05/13 17:48:01 INFO MemoryStore: MemoryStore started with capacity 484.8 MB
16/05/13 17:48:01 INFO HttpFileServer: HTTP File server directory is C:\Users\Administrator\AppData\Local\Temp\spark-4d4d665e-45ad-4ea9-b664-c95eeeb5f8b5\httpd-756f1b24-34a1-48a2-969c-6cc7a5d4cb57
16/05/13 17:48:01 INFO HttpServer: Starting HTTP Server
16/05/13 17:48:01 INFO Utils: Successfully started service 'HTTP file server' on port 62340.
16/05/13 17:48:01 INFO SparkEnv: Registering OutputCommitCoordinator
16/05/13 17:48:02 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/05/13 17:48:02 INFO SparkUI: Started SparkUI at http://172.21.75.63:4040
16/05/13 17:48:03 INFO SparkContext: Added JAR F:\jar_package\job\SparkPi.jar at http://172.21.75.63:62340/jars/SparkPi.jar with timestamp 1463132883308
16/05/13 17:48:04 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/05/13 17:48:04 INFO AppClient$ClientEndpoint: Connecting to master spark://172.21.75.102:7077...
16/05/13 17:48:06 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160513024433-0002
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor added: app-20160513024433-0002/0 on worker-20160513012923-172.21.75.102-44267 (172.21.75.102:44267) with 1 cores
16/05/13 17:48:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20160513024433-0002/0 on hostPort 172.21.75.102:44267 with 1 cores, 1024.0 MB RAM
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor added: app-20160513024433-0002/1 on worker-20160513012924-172.21.75.95-54009 (172.21.75.95:54009) with 1 cores
16/05/13 17:48:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20160513024433-0002/1 on hostPort 172.21.75.95:54009 with 1 cores, 1024.0 MB RAM
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor added: app-20160513024433-0002/2 on worker-20160513012924-172.21.75.194-35992 (172.21.75.194:35992) with 1 cores
16/05/13 17:48:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20160513024433-0002/2 on hostPort 172.21.75.194:35992 with 1 cores, 1024.0 MB RAM
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor added: app-20160513024433-0002/3 on worker-20160513012923-172.21.75.122-39901 (172.21.75.122:39901) with 1 cores
16/05/13 17:48:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20160513024433-0002/3 on hostPort 172.21.75.122:39901 with 1 cores, 1024.0 MB RAM
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor updated: app-20160513024433-0002/1 is now LOADING
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor updated: app-20160513024433-0002/0 is now LOADING
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor updated: app-20160513024433-0002/2 is now LOADING
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor updated: app-20160513024433-0002/3 is now LOADING
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor updated: app-20160513024433-0002/0 is now RUNNING
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor updated: app-20160513024433-0002/1 is now RUNNING
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor updated: app-20160513024433-0002/2 is now RUNNING
16/05/13 17:48:06 INFO AppClient$ClientEndpoint: Executor updated: app-20160513024433-0002/3 is now RUNNING
16/05/13 17:48:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 62360.
16/05/13 17:48:07 INFO NettyBlockTransferService: Server created on 62360
16/05/13 17:48:07 INFO BlockManagerMaster: Trying to register BlockManager
16/05/13 17:48:07 INFO BlockManagerMasterEndpoint: Registering block manager 172.21.75.63:62360 with 484.8 MB RAM, BlockManagerId(driver, 172.21.75.63, 62360)
16/05/13 17:48:07 INFO BlockManagerMaster: Registered BlockManager
16/05/13 17:48:08 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/05/13 17:48:09 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@172.21.75.194:57560/user/Executor#-786956451]) with ID 2
16/05/13 17:48:10 INFO BlockManagerMasterEndpoint: Registering block manager 172.21.75.194:48333 with 530.3 MB RAM, BlockManagerId(2, 172.21.75.194, 48333)
16/05/13 17:48:10 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@172.21.75.102:60131/user/Executor#1889839276]) with ID 0
16/05/13 17:48:10 INFO BlockManagerMasterEndpoint: Registering block manager 172.21.75.102:33896 with 530.3 MB RAM, BlockManagerId(0, 172.21.75.102, 33896)
16/05/13 17:48:10 INFO SparkContext: Starting job: reduce at SparkPi.scala:19
16/05/13 17:48:10 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:19) with 2 output partitions
16/05/13 17:48:10 INFO DAGScheduler: Final stage: ResultStage 0(reduce at SparkPi.scala:19)
16/05/13 17:48:10 INFO DAGScheduler: Parents of final stage: List()
16/05/13 17:48:10 INFO DAGScheduler: Missing parents: List()
16/05/13 17:48:11 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:15), which has no missing parents
16/05/13 17:48:11 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@172.21.75.95:42263/user/Executor#1076811589]) with ID 1
16/05/13 17:48:11 INFO BlockManagerMasterEndpoint: Registering block manager 172.21.75.95:50679 with 530.3 MB RAM, BlockManagerId(1, 172.21.75.95, 50679)
16/05/13 17:48:12 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@172.21.75.122:36331/user/Executor#-893021210]) with ID 3
16/05/13 17:48:12 INFO MemoryStore: ensureFreeSpace(1832) called with curMem=0, maxMem=508369305
16/05/13 17:48:12 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free 484.8 MB)
16/05/13 17:48:12 INFO MemoryStore: ensureFreeSpace(1189) called with curMem=1832, maxMem=508369305
16/05/13 17:48:12 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1189.0 B, free 484.8 MB)
16/05/13 17:48:12 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.21.75.63:62360 (size: 1189.0 B, free: 484.8 MB)
16/05/13 17:48:12 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:861
16/05/13 17:48:12 INFO BlockManagerMasterEndpoint: Registering block manager 172.21.75.122:59662 with 530.3 MB RAM, BlockManagerId(3, 172.21.75.122, 59662)
16/05/13 17:48:12 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:15)
16/05/13 17:48:12 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/05/13 17:48:13 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.21.75.194, PROCESS_LOCAL, 2137 bytes)
16/05/13 17:48:13 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 172.21.75.102, PROCESS_LOCAL, 2194 bytes)
16/05/13 17:49:21 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.21.75.102:33896 (size: 1189.0 B, free: 530.3 MB)
16/05/13 17:49:22 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 68937 ms on 172.21.75.102 (1/2)
16/05/13 17:49:42 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.21.75.194:48333 (size: 1189.0 B, free: 530.3 MB)
16/05/13 17:49:42 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 90038 ms on 172.21.75.194 (2/2)
16/05/13 17:49:42 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:19) finished in 90.071 s
16/05/13 17:49:42 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
16/05/13 17:49:42 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:19, took 92.205022 s
Pi is roughly 3.13816
16/05/13 17:49:42 INFO SparkUI: Stopped Spark web UI at http://172.21.75.63:4040
16/05/13 17:49:42 INFO DAGScheduler: Stopping DAGScheduler
16/05/13 17:49:42 INFO SparkDeploySchedulerBackend: Shutting down all executors
16/05/13 17:49:42 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
16/05/13 17:49:43 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/05/13 17:49:43 INFO MemoryStore: MemoryStore cleared
16/05/13 17:49:43 INFO BlockManager: BlockManager stopped
16/05/13 17:49:43 INFO BlockManagerMaster: BlockManagerMaster stopped
16/05/13 17:49:43 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/05/13 17:49:43 INFO SparkContext: Successfully stopped SparkContext
16/05/13 17:49:43 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/05/13 17:49:43 INFO ShutdownHookManager: Shutdown hook called
16/05/13 17:49:43 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/05/13 17:49:43 INFO ShutdownHookManager: Deleting directory C:\Users\Administrator\AppData\Local\Temp\spark-4d4d665e-45ad-4ea9-b664-c95eeeb5f8b5

Process finished with exit code 0

View Code

看着真是有點小激動！

15）去172.21.75.102:8080查看運行的痕跡

16）搭建調試環境過程當中的錯誤

null\bin\winutils.exe，這個錯誤很簡單，是由於本win7壓根就沒裝hadoop系統，解決辦法是從集羣上覆制一份過來,放到F盤，而且配置好環境變量

HADOOP_HOME=F:\hadoop-2.6.0

Path=%HADOOP_HOME%\bin

接下來下載對應的版本的winutils放到 F:\hadoop-2.6.0\bin 文件夾下，應該就解決了

SparkUncaughtExceptionHandler: Uncaught exception in thread Thread

這個錯誤好坑，查了很久的資料，才解決，原來是搭建集羣時候spark-env.sh設置的問題

將SPARK_MASTER_IP=spark1改爲

SPARK_MASTER_IP=172.21.75.102便可解決，改了以後再網頁裏也能查出來

Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException : spark1

以上是當須要操做HDFS時候，寫上HDFS地址 hdfs://spark1:9000,會出現，後來發現原來windows也能夠設置hosts，在 C:\Windows\System32\drivers\etc 有個hosts，把須要映射的地址填進去便可

172.21.75.102 spark1

FAILED: RuntimeException org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=dbs, access=WRITE, inode="/opt/hadoop-1.0.1":hadoop:supergroup:drwxr-xr-x

解決辦法：

在 hdfs-site.xml 總添加參數：

<property>
        <name>dfs.permissions</name>
        <value>false</value>
</property>
</configuration>

改完後記得重啓HDFS