Spark集羣奇怪的問題之Workers只顯示一個

時間 2019-11-06

標籤 spark 集羣奇怪問題 workers 顯示一個欄目 Spark 简体版

原文原文鏈接

首先問一個問題引出問題的根源所在：

hostname 僅僅是在 /etc/hosts文件這裏控制的嗎？ java

答案：不是web

那麼開始今天個人問題記錄吧...apache

環境前置說明：bootstrap

Ubuntu 16.04 LTS版 (三臺機器 ,hostname分別是: hadoop-master, hadoop-s1, hadoop-s2, 其中 master機器自己也作了一個slave)ubuntu

jdk 1.8bash

Hadoop-2.7.3app

spark-2.1.1-bin-hadoop2.7 - standalone集羣socket

Scala 2.12.xoop

我在三臺虛擬機上分別輸入 jps 查看進程，徹底沒有問題spa

都有work進程在裏面

可是我打開 hadoop-master:8080 web界面卻只能看到一個workers，如圖

只有一個，這是怎麼回事呢

咱們master機器執行 $SPARK_HOME/sbin/start-all.sh 的時候打開一下啓動的日誌看看，日誌位於

$SPARK_HOME/logs，看看slave1,slave2節點的啓動日誌

異常有點長，其實有用的就是這第一行！

19/09/13 21:49:35 WARN worker.Worker: Failed to connect to master ubuntu:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
   at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
   at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
   at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
   at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
   at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
   at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
   at org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deploy$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:218)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to connect to ubuntu:7077
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
   at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
   ... 4 more
Caused by: java.nio.channels.UnresolvedAddressException
   at sun.nio.ch.Net.checkAddress(Net.java:101)
   at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
   at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:242)
   at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
   at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:517)
   at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
   at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:215)
   at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
   at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)
   at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
   at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
   ... 1 more
19/09/13 21:49:43 INFO worker.Worker: Retrying connection to master (attempt # 1)
19/09/13 21:49:43 INFO worker.Worker: Connecting to master ubuntu:7077...
19/09/13 21:49:43 WARN worker.Worker: Failed to connect to master ubuntu:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
   at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
   at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
   at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
   at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
   at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
   at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
   at org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deploy$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:218)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to connect to ubuntu:7077
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
   at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
   ... 4 more
Caused by: java.nio.channels.UnresolvedAddressException
   at sun.nio.ch.Net.checkAddress(Net.java:101)
   at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
   at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:242)
   at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
   at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:517)
   at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
   at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:215)
   at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
   at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)
   at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
   at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
   ... 1 more
19/09/13 21:49:52 INFO worker.Worker: Retrying connection to master (attempt # 2)
19/09/13 21:49:52 INFO worker.Worker: Connecting to master ubuntu:7077...
19/09/13 21:49:52 WARN worker.Worker: Failed to connect to master ubuntu:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
   at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
   at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
   at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
   at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
   at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
   at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
   at org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deploy$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:218)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to connect to ubuntu:7077
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
   at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
   ... 4 more
Caused by: java.nio.channels.UnresolvedAddressException
   at sun.nio.ch.Net.checkAddress(Net.java:101)
   at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
   at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:242)
   at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
   at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:517)
   at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
   at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:215)
   at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
   at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)
   at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
   at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
   ... 1 more
19/09/13 21:50:01 INFO worker.Worker: Retrying connection to master (attempt # 3)
19/09/13 21:50:01 INFO worker.Worker: Connecting to master ubuntu:7077...
19/09/13 21:50:01 WARN worker.Worker: Failed to connect to master ubuntu:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
   at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
   at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
   at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
   at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
   at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
   at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
   at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
   at org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deploy$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:218)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to connect to ubuntu:7077
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
   at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
   ... 4 more
Caused by: java.nio.channels.UnresolvedAddressException
   at sun.nio.ch.Net.checkAddress(Net.java:101)
   at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
   at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:242)
   at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
   at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
   at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:550)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:535)
   at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:517)
   at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
   at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:215)
   at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
   at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)
   at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
   at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
   ... 1 more

重點在這行！：

19/09/13 21:49:35 WARN worker.Worker: Failed to connect to master ubuntu:7077

但是 !!!!!!!!!!!!!!!!!

我已經把master機器的hostname改成了 hadoop-master啊。這裏爲何還顯示主機名 ubuntu沒法解析呢

我再次檢查 /etc/hosts 文件，也徹底木有問題啊

個人hosts文件是這樣的

127.0.0.1	localhost

192.168.0.45    hadoop-master
192.168.0.46    hadoop-s1
192.168.0.42    hadoop-s2

我決定閱讀一下$SPARK_HOME/sbin 下面的這些啓動腳本文件，看看裏面有什麼樣的玄機

首先我打開start-all.sh

代碼以下

#!/usr/bin/env bash

#這裏有一大段註釋，我去掉了

if [ -z "${SPARK_HOME}" ]; then
  export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
fi

# Load the Spark configuration
. "${SPARK_HOME}/sbin/spark-config.sh"

# Start Master
"${SPARK_HOME}/sbin"/start-master.sh

# Start Workers
"${SPARK_HOME}/sbin"/start-slaves.sh

從上面代碼能夠看出： start-all.sh啓動的是 start-master.sh 和 start-slaves.sh (注意是 slaves 而不是 slave)

那麼start-master.sh 從名字能夠看出，只是啓動主節點的，和slaves半毛錢關係沒有，

那麼接下來咱們看看start-slaves.sh

代碼以下:

#!/usr/bin/env bash
# Starts a slave instance on each machine specified in the conf/slaves file.

if [ -z "${SPARK_HOME}" ]; then
  export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
fi

. "${SPARK_HOME}/sbin/spark-config.sh"
. "${SPARK_HOME}/bin/load-spark-env.sh"

# Find the port number for the master
if [ "$SPARK_MASTER_PORT" = "" ]; then
  SPARK_MASTER_PORT=7077
fi

if [ "$SPARK_MASTER_HOST" = "" ]; then
  case `uname` in
      (SunOS)
	  SPARK_MASTER_HOST="`/usr/sbin/check-hostname | awk '{print $NF}'`"
	  ;;
      (*)
	  SPARK_MASTER_HOST="`hostname -f`"
	  ;;
  esac
fi

# Launch the slaves
"${SPARK_HOME}/sbin/slaves.sh" cd "${SPARK_HOME}" \; "${SPARK_HOME}/sbin/start-slave.sh" "spark://$SPARK_MASTER_HOST:$SPARK_MASTER_PORT"

在終端執行一下 hostsname 發現竟然輸出的就是 "ubuntu" !!!

我擦，這下問題找到了！

那麼問題來了如何修改這個hostname呢，這個我搜索了一下，發現ubuntu和Centos系列的修改方法還不同

Ubuntu修改須要修改 /etc/hostname 文件

改完我重啓機器，問題解決