spark 集羣搭建

本案例是搭建spark Standalone Cluster，沒有用到yarn等其它第三方資源管理框架或者zk

spark版本號：2.1.1

一共是3節點集羣

主要是修改spark/conf下的

1. slaves

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# A Spark Worker will be started on each of the machines listed below.
dscn1
dscn2
dscn3

2. spark-env.sh

export SPARK_PID_DIR={$SPARK_HOME}
# 修改spark的UI端口號爲10010
export SPARK_MASTER_WEBUI_PORT=10010

3. spark-defaults.conf

# Example:
spark.master                     spark://dscn1:7077
# spark.eventLog.enabled           true
# spark.eventLog.dir               hdfs://namenode:8021/directory
spark.serializer                 org.apache.spark.serializer.KryoSerializer
spark.driver.memory              10g
spark.driver.extraJavaOptions -XX:PermSize=512M -XX:MaxPermSize=1024M
spark.dynamicAllocation.minExecutors=1
spark.dynamicAllocation.maxExecutors=100
spark.executor.memory=16g
spark.cores.max=24

在/etc/profile中添加：

export SPARK_HOME=/home/hadmin/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin:

記得source /etc/profile

在spark目錄下添加logs目錄

而後執行./spark/sbin/start-all.sh 就將集羣起來了

界面以下：