【原創】大數據基礎之Mesos+Marathon+Docker部署nginx

一 安裝

安裝docker：http://www.javashuo.com/article/p-yeqxuuta-hd.html
安裝mesos：http://www.javashuo.com/article/p-ylfysgvx-km.html
安裝marathon：http://www.javashuo.com/article/p-ykqdvphd-kk.html

二 部署nginx

使用docker官方nginx image，若是想用私有docker registry，能夠參考：http://www.javashuo.com/article/p-acnvfrvq-hn.html

打開marathon頁面，使用docker image建立app

 

1 開啓mesos對docker的支持

# echo 'docker,mesos' > /ec/mesos-slave/containerizers
# service mesos-slave restart

2 建立app以後nginx正常運行，可是增長端口映射以後（即--network bridge -p 80:80），app一直卡住沒法運行，查看mesos master日誌以下：

# tail -f /var/log/mesos/mesos-master.log.INFO
I0216 01:01:35.815131 25339 master.cpp:9533] Sending offers [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O171, e11dfbd1-316e-4635-9267-8d5089f5ecfa-O172, e11dfbd1-316e-4635-9267-8d5089f5ecfa-O173, e11dfbd1-316e-4635-9267-8d5089f5ecfa-O174 ] to framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.821087 25338 master.cpp:5886] Processing DECLINE call for offers: [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O171 ] for framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.821200 25338 master.cpp:11531] Removing offer e11dfbd1-316e-4635-9267-8d5089f5ecfa-O171
I0216 01:01:35.821494 25335 master.cpp:5886] Processing DECLINE call for offers: [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O173 ] for framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.821557 25335 master.cpp:11531] Removing offer e11dfbd1-316e-4635-9267-8d5089f5ecfa-O173
I0216 01:01:35.821578 25335 master.cpp:5886] Processing DECLINE call for offers: [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O174 ] for framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.821614 25335 master.cpp:11531] Removing offer e11dfbd1-316e-4635-9267-8d5089f5ecfa-O174
I0216 01:01:35.823659 25334 master.cpp:5886] Processing DECLINE call for offers: [ e11dfbd1-316e-4635-9267-8d5089f5ecfa-O172 ] for framework 46ef421b-55ad-4b9f-876c-5d454abd2e6b-0000 (marathon) at scheduler-2655dbf9-0e11-4d1a-934f-5012c504ab07@192.168.0.54:38562
I0216 01:01:35.823747 25334 master.cpp:11531] Removing offer e11dfbd1-316e-4635-9267-8d5089f5ecfa-O172

可見mesos-master一直向marathon提供resource offer，可是一直被拒絕，再仔細查看日誌發現

I0216 01:01:12.527853 25338 master.cpp:7503] Re-registered agent 0548c9c0-dd5f-4b3a-8aca-4668f8aacb69-S9 at slave(1)@192.168.0.1:5051 (server1.bj) with cpus:8; mem:14861; disk:35829; ports:[31000-32000]

發現mesos-slave的資源範圍有問題，ports:[31000-32000]，nginx是須要80端口的，因此一直被拒絕，關於resources官方文檔描述以下：

--resources=VALUE

Total consumable resources per agent. Can be provided in JSON format or as a semicolon-delimited list of key:value pairs, with the role optionally specified.
As a key:value list: name(role):value;name:value...
To use JSON, pass a JSON-formatted string or use --resources=filepath to specify the resources via a file containing a JSON-formatted string. 'filepath' can be of the form file:///path/to/file or /path/to/file.
Example JSON:
[
{
"name": "cpus",
"type": "SCALAR",
"scalar": {
"value": 24
}
},
{
"name": "mem",
"type": "SCALAR",
"scalar": {
"value": 24576
}
}
]

參考：http://mesos.apache.org/documentation/latest/configuration/agent/

 

修改端口範圍

# echo 'ports(*):[80-80000]' > /etc/mesos-slave/resources
# service mesos-slave restart

以前的錯誤沒了，可是又出現新的錯

Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: E0216 01:37:17.314812 21733 slave.cpp:7311] EXIT with status 1: Failed to perform recovery: Incompatible agent info detected.
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: resources {
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: name: "ports"
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: type: RANGES
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: ranges {
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: range {
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: begin: 31000
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: end: 32000
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: }
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: }
Feb 16 01:37:17 cdp-test-server-04.bj mesos-slave[21714]: }

這個錯是由於修改resources以後須要充值agent並重啓，詳見：http://www.javashuo.com/article/p-scgkqdpn-hk.html

# rm -f /var/lib/mesos/meta/slaves/latest
# service mesos-slave start

3 若是須要制定在哪臺機器上執行，能夠設置constraints

只容許在一臺機器上執行

"constraints": [
[
"hostname",
"CLUSTER",
"server2.bj"
]
],

容許在兩臺機器上執行，可是每臺機器只能啓動一個

"constraints": [
[
"hostname",
"UNIQUE"
],
[
"hostname",
"LIKE",
"server[1-2].bj"
]
],

參考：https://mesosphere.github.io/marathon/docs/constraints.html

 

部署完成後

 

還有一點，注意到上邊截圖中的status是waiting，由於我作了一個restart操做，可是我每一個app都指定只部署在一臺服務器上，這時其實會一直卡在waiting的，由於marathon在restart時並不會將app先stop再start，而是不斷等待mesos-master的resource offer，只有當有可用資源啓動新的app以後纔會把舊的app停掉，而我這裏的app會佔用80端口，因此是一個死循環，這時只能手工從docker上去stop container；

 

這樣就ok了