StandaloneAppClient是什麼?

StandaloneAppClient是什麼?這個很容易搞混淆。其實StandaloneAppClient不是SparkApplication,它主要是用在ScheduleBackend中的。app

獨立集羣環境中,ScheduleBackend是用的StandaloneScheduleBackend,它繼承了CoarseGrainedSchedulerBackend類。ide

StandaloneScheduleBackend裏面用了一個叫StandaloneAppClient的類,這個StandaloneAppClient很具備迷惑性,其實它的主要功能是替換CoarseGrainedSchedulerBackend的資源申請的方法,改成向Master申請資源,咱們看看相關代碼片斷就好了。spa

先看他啓動的時候:code

private def tryRegisterAllMasters(): Array[JFuture[_]] = {
      for (masterAddress <- masterRpcAddresses) yield {
        registerMasterThreadPool.submit(new Runnable {
          override def run(): Unit = try {
            if (registered.get) {
              return
            }
            logInfo("Connecting to master " + masterAddress.toSparkURL + "...")
            val masterRef = rpcEnv.setupEndpointRef(masterAddress, Master.ENDPOINT_NAME)
            masterRef.send(RegisterApplication(appDescription, self))
          } catch {
          }
        })
      }
    }

向Master發送RegisterApplication消息,將本appDesc註冊給Master,這個和DriverDescription註冊到Master是有點區別的。繼承

再好比資源申請的代碼:ip

def requestTotalExecutors(requestedTotal: Int): Future[Boolean] = {
    if (endpoint.get != null && appId.get != null) {
      endpoint.get.ask[Boolean](RequestExecutors(appId.get, requestedTotal))
    } else {
      logWarning("Attempted to request executors before driver fully initialized.")
      Future.successful(false)
    }
  }

就是向Master發送RequestExecutor消息申請Executor資源。資源

這裏爲啥要註冊Application到Master呢?主要是當Master失效或者Master更改時,能通知到Application,這樣就能從新鏈接新的Master了,從新運行spark程序。不然就很脆弱,很容易崩潰,這是個人理解哦,不必定正確,^~^rpc

相關文章
相關標籤/搜索