Docker1.12.1之swarm集羣搭建與使用

1：前言

在docker1.12版本以前，衆所周知dokcer自己只能單機上運行，而集羣則要依賴mesos、kubernetes、swarm等集羣管理方案。其中swarm是docker公司本身的容器集羣管理工具，在當時的熱度仍是低於前二者。docker1.12.0版本發佈中，Docker公司出於戰略眼光，將swarm集成到docker-engine中，使docker內置了集羣解決方案。因而swarm這個「親兒子」的江湖地位迅速提高，在docker集羣方案中與mesos，k8s造成三足鼎立之勢，在將來則大有趕超之勢。

2：規劃

   IP           主機名          角色   

10.10.32.245    swarm1      swarm manager

10.10.32.246    swarm2      worker node

10.10.32.247    swarm3      worker node

10.10.32.248    swarm4      worker node

3：集羣通訊原理

經過/usr/lib/system.d/system/docker.service配置開啓2375管理端口

ExecStart=/usr/bin/dockerd -s overlay --insecure-registry registry.cntv.net  -H tcp://0.0.0.0:2375 -H unix:///va

每一個node節點上，都開啓2375遠程管理端口，swarm集羣管理經過2375管理端口來執行

4：swarm調度原理

swarm中實現調度功能的主要是scheduler模塊，其中包括Filter和strategy。

Filter：根據下發的任務的資源需求，用來過濾節點，從集羣全部節點中找出知足條件的節點（好比資源足夠，節點正常等等）

Strategy：用來在過濾出的節點中根據策略選擇一個最優的節點（好比對找出的節點進行對比，找到資源最多的節點等等）

5：swarm集羣建立

（1）初始化manager節點

• 10.10.32.245 
  

$docker swarm init --advertise-addr 10.10.32.245
Swarm initialized: current node (b73yii7s7rn321ejz8n3ch7ay) is now a manager.
To add a worker to this swarm, run the following command:
        docker swarm join \
        --token SWMTKN-1-2z3obrzww2zcnwaiq5dzocpmvz9kty25usroy3gh3xux0l32uo-8dxurdbms8yli2crdxbmyk7y2 \
        10.10.32.245:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

（2）manage查看節點swarm狀態

• 10.10.32.245 

$docker info
  Swarm: active
  NodeID: ax13h89zlb5dqw980ddo1ox4f
  Is Manager: true
  ClusterID: d5nam0mink4wdk506o1ybbs9e
  Managers: 1
  Nodes: 1

（3）manage查看swarm 節點

• 10.10.32.245 

$ docker node ls
  ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS    
  ax13h89zlb5dqw980ddo1ox4f *  swarm1    Ready   Active        Leader

（4）node節點加入swarm集羣

• 10.10.32.246 10.10.32.247 10.10.32.248 

$ docker swarm join-token manager   #manage上查詢加入命令及token
To add a manager to this swarm, run the following command:
    docker swarm join \
    --token SWMTKN-1-2z3obrzww2zcnwaiq5dzocpmvz9kty25usroy3gh3xux0l32uo-8dxurdbms8yli2crdxbmyk7y2 \
    10.10.32.245:2377
$ docker swarm join \
    --token SWMTKN-1-2z3obrzww2zcnwaiq5dzocpmvz9kty25usroy3gh3xux0l32uo-8dxurdbms8yli2crdxbmyk7y2 \ 
     10.10.32.245:2377
This node joined a swarm as a worker

• （5）manage查看swarm節點

$ docker node ls
  ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
  03b5w49n7uz663xwjuof3dsk0    swarm4    Ready   Active       
  0g039jomhpad819a39yvkow6y    swarm2    Ready   Active       
  4kos85v5z6jh6lahwv8yjpku8    swarm3    Ready   Active       
  ax13h89zlb5dqw980ddo1ox4f *  swarm1    Ready   Active        Leader

• （6）使節點離開集羣

$ docker swarm leave
Node left the swarm.

6：在swarm集羣上建立服務

• （1）建立服務

$docker service create --network net3 --replicas 4 -p 8099:80 --name time-php registry.cntv.net/heqin/tvtime-php:v0.85dongsi
bdkwsgchcydz5ixatrbiyr1fx

• （2）查看服務

$ docker service ls        #查看服務
ID            NAME            REPLICAS  IMAGE                               COMMAND     
bdkwsgchcydz  time-php        4/4       registry.cntv.net/heqin/tvtime-php:v0.85dongsi

$docker service ps time-php    #查看服務的實例
ID                         NAME        IMAGE                                              NODE    DESIRED STATE  CURRENT STATE          ERROR
d15ambrdd7pyi22951vi497xa  time-php.1  registry.cntv.net/heqin/tvtime-php:v0.85dongsi  swarm4  Running        Running 5 minutes ago 
3wtmmggo65cmyutg7zwikb94s  time-php.2  registry.cntv.net/heqin/tvtime-php:v0.85dongsi  swarm3  Running        Running 5 minutes ago 
5ulbhndwgswpicx8g62af8k5z  time-php.3  registry.cntv.net/heqin/tvtime-php:v0.85dongsi  swarm2  Running        Running 5 minutes ago 
4b9rh8aiwfi88xrdgmyss9ili  time-php.4  registry.cntv.net/heqin/tvtime-php:v0.85dongsi  swarm2  Running        Running 5 minutes ago   
$docker service inspect time-php     #查看服務的詳細信息
[
    {
        "ID": "bdkwsgchcydz5ixatrbiyr1fx",
        "Version": {
            "Index": 3074
        },
        "CreatedAt": "2016-08-31T01:41:18.892483521Z",
        "UpdatedAt": "2016-08-31T01:41:18.909164422Z",
        "Spec": {
            "Name": "time-php",
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "registry.cntv.net/heqin/tvtime-php:v0.85dongsi"
                },
                "Resources": {
                    "Limits": {},
                    "Reservations": {}
                },
                "RestartPolicy": {
                    "Condition": "any",
                    "MaxAttempts": 0
                },
                "Placement": {}
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 4
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause"
            },
            "Networks": [
                {
                    "Target": "d83qc9rgkkj1ws8kvmod4x759"
                }
            ],
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 8099
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 8099
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 80,
                    "PublishedPort": 8099
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "biu7m9hi8fgbbihfzg48whxqi",
                    "Addr": "10.255.0.19/16"
                },
                {
                    "NetworkID": "d83qc9rgkkj1ws8kvmod4x759",
                    "Addr": "10.88.0.2/24"
                }
            ]
        },
        "UpdateStatus": {
            "StartedAt": "0001-01-01T00:00:00Z",
            "CompletedAt": "0001-01-01T00:00:00Z"
        }
    }
]

• （3）擴展服務實例數

$ docker service scale time-php=10
time-php scaled to 10
$ docker service ls
ID            NAME            REPLICAS  IMAGE                                                                                
bdkwsgchcydz  time-php        6/6       registry.cntv.net/heqin/tvtime-php:v0.85dongsi

• （4）更新服務

$ docker service update --p_w_picpath registry.cntv.net/heqin/tvtime-php:v0.84xidan --log-driver=syslog time-php
time-php
$ docker service ls
ID            NAME            REPLICAS  IMAGE                                                                               
bdkwsgchcydz  time-php        6/6       registry.cntv.net/heqin/tvtime-php:v0.84xidan

• （5）刪除服務

$docker service rm time-php
time-php

7：網絡

（1）查看容器網絡

$docker network ls
NETWORK ID        NAME          DRIVER        SCOPE
a17f16650bf9      bridge        bridge        local   #容器使用獨立網絡Namespace，鏈接docker0虛擬網卡（默認模式）
f1c102babcf8      host          host          local   #容器與主機共享網絡Namespace，擁有主機相同網卡 
b6a2efce65ef      none          null          local   #容器沒有任何網卡，適合不須要與外部通訊的容器
asa2hv41mtci      ingress       overlay       swarm   #swarm集羣的的overlay網絡，容器能夠跨主機網絡通訊

注：上面三個網絡是docker安裝後默認帶的三個網絡，第四個overlay網絡是啓動swarm以後默認建立的overlay網絡。

（2）查看overlay網絡信息

$docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "asa2hv41mtci2qzjkaonusnc8",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.255.0.0/16",
                    "Gateway": "10.255.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "258"
        },
        "Labels": null
    }
]

（3）建立自定義的overlay網絡

$docker network create --driver=overlay --subnet=10.88.0.0/24  --gateway=10.88.0.1 net3
c654pb76q9jnni5bpdvd34rg4
$docker network ls
NETWORK ID          NAME                DRIVER              SCOPE              
c654pb76q9jn        net3                overlay             swarm
$docker network inspect net3
[
    {
        "Name": "net3",
        "Id": "c654pb76q9jnni5bpdvd34rg4",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.88.0.0/24",
                    "Gateway": "10.88.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "258"
        },
        "Labels": null
    }
]

注：在manage上建立overlay網絡後，各node上並無同步建立該網絡，只有當使用該overlay網絡的容器在node節點上運行時，纔會在node節點上自動建立,容器刪除後，網絡也會在node節點上刪除。

（4）將容器綁定到自定義的網絡中

$docker service create --network net3 --replicas 4 --name time-php registry.cntv.net/heqin/tvtime-php:v0.85dongsi
5qiv4hhv7ra5p65fpq2v6ok0x
$docker service ls
5qiv4hhv7ra5  time-php        4/4       registry.cntv.net/heqin/tvtime-php:v0.85dongsi   
$docker service ps time-php
ID                         NAME        IMAGE                                              NODE    DESIRED STATE  CURRENT STATE           ERROR
529kuyazgxxl8l0eds4f740nu  time-php.1  registry.cntv.net/heqin/tvtime-php:v0.85dongsi  swarm3  Running        Running 32 seconds ago 
brdelpbuwm3p9cujy6k78cpzo  time-php.2  registry.cntv.net/heqin/tvtime-php:v0.85dongsi  swarm3  Running        Running 33 seconds ago 
3csp6fmrsg5cvski270732q98  time-php.3  registry.cntv.net/heqin/tvtime-php:v0.85dongsi  swarm2  Running        Running 34 seconds ago 
2ismj5zt9o3t2hfe1j1lx7dcw  time-php.4  registry.cntv.net/heqin/tvtime-php:v0.85dongsi  swarm4  Running        Running 33 seconds ago

進入swarm3 ，能夠看到net3網絡下有兩個容器。

$docker docker network inspect net3
[
    {
        "Name": "net3",
        "Id": "d83qc9rgkkj1ws8kvmod4x759",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.88.0.0/24",
                    "Gateway": "10.88.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "d465611683b69f20c28ec9eb276a9e4088867e35508ecf2bf28c07c5d3e1b33a": {
                "Name": "time-php.1.529kuyazgxxl8l0eds4f740nu",
                "EndpointID": "ca667c89550f88eabf54eeaaaa3c55c678ec8184f121fc845bbaab1cd4be20ff",
                "MacAddress": "02:42:0a:58:00:05",
                "IPv4Address": "10.88.0.5/24",
                "IPv6Address": ""
            },
            "d8fa64c4f8357418304ed1f19a049e4aac196c94ef50a27a0686ebe4ee734225": {
                "Name": "time-php.2.brdelpbuwm3p9cujy6k78cpzo",
                "EndpointID": "ab45bcf8e73d114e9bdf97cac128750a6c7a073149783746a9962c6cd4c599a8",
                "MacAddress": "02:42:0a:58:00:06",
                "IPv4Address": "10.88.0.6/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "257"
        },
        "Labels": {}
    }
]

用swarm3的容器去ping swarm2主機上的容器，是能夠ping通的

$docker exec -it d8fa64c4f835  ping 10.88.0.3
PING 10.88.0.3 (10.88.0.3) 56(84) bytes of data.
64 bytes from 10.88.0.3: icmp_seq=1 ttl=64 time=0.611 ms
64 bytes from 10.88.0.3: icmp_seq=2 ttl=64 time=5.45 ms
64 bytes from 10.88.0.3: icmp_seq=3 ttl=64 time=0.439 ms
64 bytes from 10.88.0.3: icmp_seq=4 ttl=64 time=0.586 ms
64 bytes from 10.88.0.3: icmp_seq=5 ttl=64 time=0.603 ms

8：swarm基本命令

• swarm集羣建立與管理

docker swarm Command

Commands:

  init        Initialize a swarm

  join        Join a swarm as a node and/or manager

  join-token  Manage join tokens

  update      Update the swarm

  leave       Leave a swarm

• swarm服務建立與管理

docker service Command

Commands:

  create      Create a new service

  inspect     Display detailed information on one or more services

  ps          List the tasks of a service

  ls          List services

  rm          Remove one or more services

  scale       Scale one or multiple services

  update      Update a service

• swarm服務建立選項

docker service create [OPTIONS] IMAGE 

Options:

      --constraint value               Placement constraints (default [])

      --container-label value          Container labels (default [])

      --endpoint-mode string           Endpoint mode (vip or dnsrr)

  -e, --env value                      Set environment variables (default [])

      --help                           Print usage

  -l, --label value                    Service labels (default [])

      --limit-cpu value                Limit CPUs (default 0.000)

      --limit-memory value             Limit Memory (default 0 B)

      --log-driver string              Logging driver for service

      --log-opt value                  Logging driver options (default [])

      --mode string                    Service mode (replicated or global) (default "replicated")

      --mount value                    Attach a mount to the service

      --name string                    Service name

      --network value                  Network p_w_uploads (default [])

  -p, --publish value                  Publish a port as a node port (default [])

      --replicas value                 Number of tasks (default none)

      --reserve-cpu value              Reserve CPUs (default 0.000)

      --reserve-memory value           Reserve Memory (default 0 B)

      --restart-condition string       Restart when condition is met (none, on-failure, or any)

      --restart-delay value            Delay between restart attempts (default none)

      --restart-max-attempts value     Maximum number of restarts before giving up (default none)

      --restart-window value           Window used to evaluate the restart policy (default none)

      --stop-grace-period value        Time to wait before force killing a container (default none)

      --update-delay duration          Delay between updates

      --update-failure-action string   Action on update failure (pause|continue) (default "pause")

      --update-parallelism uint        Maximum number of tasks updated simultaneously (0 to update all at once) (default 1)

  -u, --user string                    Username or UID

9：利用2375端口遠程管理

注：-H 指定遠程地址和端口，便可在任意一臺有docker客戶端的機器遠程操做swarm集羣

$docker  -H 10.00.32.245:2375  service ls
ID            NAME            REPLICAS  IMAGE                                                                                 COMMAND
3e0ihi2lnnaa  test_api        1/1       registry.cntv.net/heqin/api-web:1.1                             
5mdqw6r53tom  test_apimem     1/1       registry.cntv.net/heqin/memcache-1.4    
5ucxo9wqywo9  test_tomcatmem  1/1       registry.cntv.net/heqin/memcache-1.4 
cjd1z14ug28g  HQtest_tvtime   5/5       registry.cntv.net/heqin/tvtime-php:v0.85dongsi                                     
ex4mahzt5k8u  test_tomcat     1/1       registry.cntv.net/heqin/jdk7-tomcat7.0.52:201608011wq

10：swarm與mesos對比

（1）：部署配置簡單，集羣管理便利，可是沒有web管理界面

（2）：每一個節點無需手動安裝agent，集羣間只須要經過暴露tcp：//2375端口來保證節點間的通訊。

（3）：無需bamboo來作負載均衡，swarm集羣使用IPVS負載均衡

（4）：無需marathon、chrons等調度組件，swarm內置調度模塊。