Spark集羣配置-Mesosphere方案

時間 2019-11-17

原文原文鏈接

#操做系統

CentOS 7

JDK 1.6以上，如：/usr/share/jdk1.7.0_45

#修改節點名字

本次搭建Spark集羣機器主從節點，修改host（/etc/hosts）以下：html

	Hostnamenode	Ip shell
Masterexpress	xd-uiapache	192.168.1.5centos
Slave 1瀏覽器	Xd-1服務器	192.168.1.6分佈式
Slave 2oop	Xd-2	192.168.1.7
Slave 3	Xd-3	192.168.1.8

#安裝mesosphere repo

sudo rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm

#下載apache mesos

wget http://downloads.mesosphere.io/master/centos/7/mesos-0.21.1-1.1.centos701406.x86_64.rpm

sudo rpm -Uvh mesos-0.21.1-1.1.centos701406.x86_64.rpm

#安裝marathon

sudo yum -y install marathon

#安裝chronos

sudo yum -y install chronos

ZooKeeper配置，此處略，詳細看zook配置文檔

On each node, replacing the IP addresses below with each master's IP address, set /etc/mesos/zk to:

zk://1.1.1.1:2181,2.2.2.2:2181,3.3.3.3:2181/mesos

#配置Mesosphere master節點

quorum

設置：/etc/mesos-master/quorum內容： 1

目前我的理解Quorum主節點個數，相似hadoop臨時主節點，你們有意見的能夠反饋我。

Hostname

設置/etc/mesos-master/hostname爲xd-ui

Work_dir

設置/etc/mesos-master/work_dir爲工做目前，默認爲：/var/lib/mesos，我這裏設置/alidata1/mesos

配置結構顯示以下

[dev@xd-ui ~]$ tree /etc/mesos-master/
/etc/mesos-master/
├── hostname     à xd-ui
├── quorum   à 1
└── work_dir   à /alidata1/mesos

重啓Mesos Master:

[dev@xd-ui ~]$sudo service mesos-master restart

#配置Mesosphere slave節點

Xd-1

配置各個節點的Slaves
[dev@spark-1 ~]$ tree /etc/mesos-slave/
/etc/mesos-slave/
├── hostname xd-1
└── work_dir /alidata1/mesos

Xd-2

配置各個節點的Slaves
[dev@spark-2 ~]$ tree /etc/mesos-slave/
/etc/mesos-slave/
├── hostname xd-2
└── work_dir /alidata1/mesos

Xd-3

配置各個節點的Slaves
[dev@spark-3 ~]$ tree /etc/mesos-slave/
/etc/mesos-slave/
├── hostname xd-3
└── work_dir /alidata1/mesos

在節點[xd-1，xd-2，xd-3]上重啓各個Mesos Slaves:

[dev@xd-1 ~]$sudo service mesos-slave restart

檢查各個節點是否起來

[dev@xd-ui ~]$sudo ps -ef | grep mesos

/usr/sbin/mesos-slave --master=zk://xd-1:2181,xd-2:2181,xd-3:2181/mesos --log_dir=/var/log/mesos --hostname=xd-3 --work_dir=/alidata1/mesos

出現相似內容則啓動正常

#訪問Mesosphere主頁,確認各個節點是否正常

瀏覽器輸入 http://xd-ui:5050/，點擊slaves標籤頁

#Spark配置

下載spark分佈式二進制套件

注意：按實際須要，選擇對應的spark和hadoop版本

解壓spark-1.2.1-bin-hadoop2.4.tgz

tar -czvf spark-1.2.1-bin-hadoop2.4.tgz spark-1.2.1-bin-hadoop2.4

修改${SPARK_HOME}/bin/spark-class或者conf/spark-env.sh，在首行添加：

export JAVA_HOME=/usr/share/jdk1.7.0_45

肯定全部的Mesos 節點的JDK，按照目錄是一致的.

再壓縮成spark-1.2.1-bin-hadoop2.4.tgz

tar -czvf spark-1.2.1-bin-haoop2.4.tgz

再發布壓縮tgz包到hdfs或者http。

能夠使用Nginx作一個簡單的下載服務器，好比個人下載地址是：

http://192.168.0.7/download/ spark-1.2.1-bin-haoop2.4.tgz

把解壓的spark目錄放到mesos-master上。

而且配置${SPARK_HOME}/conf/spark-env.sh，在文件末尾添加：

export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI=http://192.168.0.7/download/ spark-1.2.1-bin-haoop2.4.tgz
export MASTER=mesos://xd-ui:5050

#啓動spark shell

[dev@xd-ui spark-1.2.0-dist]$ ./bin/spark-shell

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

15/01/22 16:07:46 INFO SecurityManager: Changing view acls to: dev

15/01/22 16:07:46 INFO SecurityManager: Changing modify acls to: dev

15/01/22 16:07:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(dev); users with modify permissions: Set(dev)

15/01/22 16:07:46 INFO HttpServer: Starting HTTP Server

15/01/22 16:07:46 INFO Utils: Successfully started service 'HTTP class server' on port 50256.

Welcome to

____ __

/ __/__ ___ _____/ /__

_\ \/ _ \/ _ `/ __/ '_/

/___/ .__/\_,_/_/ /_/\_\ version 1.2.0

/_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_45)

Type in expressions to have them evaluated.

輸入測試代碼

scala> val data = 1 to 10000; val distData = sc.parallelize(data); distData.filter(_< 10).collect()

data: scala.collection.immutable.Range.Inclusive = Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170...