vagrant+docker搭建consul集羣開發環境

HashiCorp 公司推出的Consul是一款分佈式高可用服務治理與服務配置的工具。關於其配置與使用能夠參考這篇文章 consul 簡介與配置說明。

通常，咱們會在多臺主機上安裝並啓動 consul，在開發時這可能會比較不方便，因此這裏介紹如何使用 vagrant 和 docker 來簡化開發環境的搭建。

利用 vagrant 建立虛擬機

---

Vagrant 是 HashiCorp 公司的產品， 用於建立和部署虛擬化開發環境，支持常見的操做系統。因爲其安裝比較簡單，參照官方文檔便可，此處再也不贅述。

安裝完成後，咱們建立文件夾consul，在consul文件夾下建立一個文件Vagrantfile，內如以下：

# -*- mode: ruby -*- # vi: set ft=ruby : $script = <<SCRIPT echo "Installing dependencies ..." # 使用阿里雲鏡像 sudo sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list sudo apt-get update sudo apt-get install -y unzip curl jq SCRIPT # Specify a custom Vagrant box for the demo DEMO_BOX_NAME = "ubuntu/xenial64" # Vagrantfile API/syntax version. # NB: Don't touch unless you know what you're doing! VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = DEMO_BOX_NAME config.vm.provision "shell", inline: $script config.vm.define "n1" do |n1| n1.vm.hostname = "n1" n1.vm.network "private_network", ip: "172.20.20.10" end config.vm.define "n2" do |n2| n2.vm.hostname = "n2" n2.vm.network "private_network", ip: "172.20.20.11" end config.vm.define "n3" do |n3| n3.vm.hostname = "n3" n3.vm.network "private_network", ip: "172.20.20.12" end end # -*- mode: ruby -*- # vi: set ft=ruby : $script = <<SCRIPT echo "Installing dependencies ..." # 使用阿里雲鏡像 sudo sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list sudo apt-get update sudo apt-get install -y unzip curl jq SCRIPT # Specify a custom Vagrant box for the demo DEMO_BOX_NAME = "ubuntu/xenial64" # Vagrantfile API/syntax version. # NB: Don't touch unless you know what you're doing! VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = DEMO_BOX_NAME config.vm.provision "shell", inline: $script config.vm.define "n1" do |n1| n1.vm.hostname = "n1" n1.vm.network "private_network", ip: "172.20.20.10" end config.vm.define "n2" do |n2| n2.vm.hostname = "n2" n2.vm.network "private_network", ip: "172.20.20.11" end config.vm.define "n3" do |n3| n3.vm.hostname = "n3" n3.vm.network "private_network", ip: "172.20.20.12" end end

這裏咱們建立了三個虛擬機，hostname爲 n1，n2，n3；ip分別爲 172.20.20.10，172.20.20.11，172.20.20.10，使用鏡像 ubuntu/xenial64，而且將鏡像更新爲了阿里雲，安裝了一些必須的軟件。

編輯好該文件，在命令行運行 vagrant up，出去喝杯咖啡，你的機器上就會啓動三臺對應的虛擬機了。若是你不想使用ubuntu 16.04，還能夠更換其餘的操做系統。

在 consul 文件夾下使用 vagrant ssh命令，就能夠登陸到對應的虛擬機了：

➜  consul vagrant ssh n2
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-66-generic x86_64)

0 packages can be updated.
0 updates are security updates.

Last login: Tue Mar 28 04:13:00 2017 from 10.0.2.2
ubuntu@n2:~$

配置運行consul的docker容器

---

docker的安裝請參照這裏docker安裝。須要注意，鑑於 dockerhub 在國內喪心病狂的訪問速度，咱們通常會選擇使用國內的鏡像，如阿里雲、網易蜂巢等。

安裝好了docker後，咱們以 n1 虛擬機爲例，說明如何在虛擬機上使用docker容器來運行consul。

首先，建立一個consul文件夾，而後編輯 Dockerfile：

FROM phusion/baseimage:latest MAINTAINER millions <chenzhesp@gmail.com> RUN DEBIAN_FRONTEND=noninteractive RUN locale-gen en_US.UTF-8 ENV LANGUAGE=en_US.UTF-8 ENV LC_ALL=en_US.UTF-8 ENV LC_CTYPE=UTF-8 ENV LANG=en_US.UTF-8 ENV TERM xterm RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list RUN apt-get update -y RUN apt-get upgrade -y RUN apt-get -qq install curl unzip ADD https://releases.hashicorp.com/consul/0.7.5/consul_0.7.5_linux_amd64.zip /tmp/consul.zip RUN cd /usr/sbin && unzip /tmp/consul.zip && chmod +x /usr/sbin/consul && rm /tmp/consul.zip ADD consul.json /config/ ADD https://releases.hashicorp.com/consul/0.7.5/consul_0.7.5_web_ui.zip /tmp/webui.zip RUN mkdir /webui && unzip /tmp/webui.zip -d /webui/ && rm /tmp/webui.zip EXPOSE 8300 8301 8301/udp 8302 8302/udp 8400 8500 53/udp VOLUME ["/data"] ENTRYPOINT [ "/usr/sbin/consul", "agent", "-config-dir=/config" ] CMD [] FROM phusion/baseimage:latest MAINTAINER millions <chenzhesp@gmail.com> RUN DEBIAN_FRONTEND=noninteractive RUN locale-gen en_US.UTF-8 ENV LANGUAGE=en_US.UTF-8 ENV LC_ALL=en_US.UTF-8 ENV LC_CTYPE=UTF-8 ENV LANG=en_US.UTF-8 ENV TERM xterm RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list RUN apt-get update -y RUN apt-get upgrade -y RUN apt-get -qq install curl unzip ADD https://releases.hashicorp.com/consul/0.7.5/consul_0.7.5_linux_amd64.zip /tmp/consul.zip RUN cd /usr/sbin && unzip /tmp/consul.zip && chmod +x /usr/sbin/consul && rm /tmp/consul.zip ADD consul.json /config/ ADD https://releases.hashicorp.com/consul/0.7.5/consul_0.7.5_web_ui.zip /tmp/webui.zip RUN mkdir /webui && unzip /tmp/webui.zip -d /webui/ && rm /tmp/webui.zip EXPOSE 8300 8301 8301/udp 8302 8302/udp 8400 8500 53/udp VOLUME ["/data"] ENTRYPOINT [ "/usr/sbin/consul", "agent", "-config-dir=/config" ] CMD []

大約解釋一下上面的Dockerfile。首先，咱們選擇phusion/baseimage爲基礎構建docker容器。而後設置使用阿里雲鏡像，安裝各類工具，下載 consul 並建立 consul 使用的 ui 文件夾。而後聲明暴露8個 consul 須要使用的 tcp 或 udp 端口，並將 /data 設置爲一個 VOLUME （經過--volumes-from能夠在容器之間共享數據）。 最後，注意 ENTRYPOINT 聲明瞭容器啓動時執行的命令，而在啓動容器時加上的命令會被添加在 ENTRYPOINT 指定的命令後，好比ENTRYPOINT爲 [ "/usr/sbin/consul", "agent", "-config-dir=/config" ]，啓動鏡像時的命令爲 sudo docker run <鏡像名稱> XXXX，則容器啓動時執行的命令爲 /usr/sbin/consul agent -config-dir=/config XXXX 。

另外，爲了使用 consul，咱們也須要編輯其配置文件 consul.json。這裏咱們使用一個比較簡單的配置，有特殊須要的請參閱 consul 官方文檔的配置說明。

{
  "data_dir": "/data",
  "ui_dir": "/webui",
  "client_addr": "0.0.0.0",
  "ports": {
    "dns": 53
  },
  "recursor": "8.8.8.8"
}

編輯好這2個文件後，在consul文件夾下執行命令 sudo docker build -t millions/consul . 便可建立鏡像 millions/consul，經過 docker images 命令能夠查看。

ubuntu@n1:~/consul$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE millions/consul latest 5fc1597b91cf 23 hours ago 372 MB phusion/baseimage latest b6b482650b50 7 days ago 247 MB ubuntu@n1:~/consul$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE millions/consul latest 5fc1597b91cf 23 hours ago 372 MB phusion/baseimage latest b6b482650b50 7 days ago 247 MB

然後在其餘虛擬機使用一樣方法建立鏡像。爲了方便啓動 docker 鏡像，咱們再在每臺虛擬機上建立如下環境變量：

# 在 Vagrantfile 中指定的ip，對於n1爲172.20.20.10
export PUBLIC_IP="$(ifconfig enp0s8 | awk -F ' *|:' '/inet addr/{print $4}')"

然後，因爲咱們打算用n1作 bootstrap 時的leader，因此在 n二、n3 上建立環境變量：

export JOIN_IP=172.20.20.10

接下來，咱們就能夠準備啓動 consul 實例們啦~

啓動 consul 實例

注意啓動時須要使用 -h $HOSTNAME 指定容器的hostname。首先咱們啓動 n1，因爲用其作 bootstrap， 因此咱們使用了 -bootstrap-expect 3。其餘相關啓動參數若是不熟悉的話能夠查看 consul 的文檔，這裏不作詳細解釋。比較重要的是須要額外設置dns，首先使用本地 docker0 來使用 consul 解析 DNS，而後使用 8.8.8.8解析其餘請求的dns，最後爲 consul 的查詢指定搜索域。

sudo docker run -d -h $HOSTNAME -p 8300:8300  \
 -p 8301:8301 -p 8301:8301/udp -p 8302:8302  \
 -p 8302:8302/udp -p 8400:8400 -p 8500:8500  \
 -p 53:53/udp --name n1 \
--dns 172.17.0.1 --dns 8.8.8.8 \
 --dns-search 'service.consul' millions/consul   \
-server -advertise $PUBLIC_IP -bootstrap-expect 3

這是查看 n1 的輸出發現其已經啓動，可是還沒法組成集羣：

ubuntu@n1:~/consul$ sudo docker logs n1 ==> WARNING: Expect Mode enabled, expecting 3 servers ==> Starting Consul agent... ==> Starting Consul agent RPC... ==> Consul agent running! Version: 'v0.7.5' Node ID: 'cedd93d8-4ff4-4bde-8bb4-391f5a129ed4' Node name: 'n1' Datacenter: 'dc1' Server: true (bootstrap: false) Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400) Cluster Addr: 172.20.20.10 (LAN: 8301, WAN: 8302) Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false Atlas: <disabled> ubuntu@n1:~/consul$ sudo docker logs n1 ==> WARNING: Expect Mode enabled, expecting 3 servers ==> Starting Consul agent... ==> Starting Consul agent RPC... ==> Consul agent running! Version: 'v0.7.5' Node ID: 'cedd93d8-4ff4-4bde-8bb4-391f5a129ed4' Node name: 'n1' Datacenter: 'dc1' Server: true (bootstrap: false) Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400) Cluster Addr: 172.20.20.10 (LAN: 8301, WAN: 8302) Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false Atlas: <disabled>

接下來，咱們來啓動 n2 還有 n3（注意修改 --name 參數）：

sudo docker run -d -h $HOSTNAME \
-p 8300:8300 -p 8301:8301 -p 8301:8301/udp \
-p 8302:8302 -p 8302:8302/udp -p 8400:8400 \
-p 8500:8500 -p 53:53/udp --name n2 \
-dns 172.17.0.1 --dns 8.8.8.8 \
 --dns-search 'service.consul' millions/consul   \
-server -advertise $PUBLIC_IP -join $JOIN_IP

這裏做爲後來加入者咱們使用 -join參數而不是 bootstrap-expect。都完成啓動後在物理機上訪問 http://172.20.20.10:8500 便可看到熟悉的consul web-ui 界面。至此，咱們的 consul 集羣開發環境就搭建完成了，咱們可使用其餘各類程序將本身做爲服務註冊到 consul 集羣中，而且訪問集羣獲取其餘服務地址。

測試

---

爲了測試集羣是否能夠正常工做，咱們再建立其餘2個docker 鏡像。

distributed_app

Dockerfile爲：

ROM phusion/baseimage:latest MAINTAINER millions <chenzhesp@gmail.com> RUN DEBIAN_FRONTEND=noninteractive RUN locale-gen en_US.UTF-8 ENV LANGUAGE=en_US.UTF-8 ENV LC_ALL=en_US.UTF-8 ENV LC_CTYPE=UTF-8 ENV LANG=en_US.UTF-8 ENV TERM xterm RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list RUN apt-get update -y RUN apt-get upgrade -y RUN apt-get -qq install ruby-dev git libcurl4-openssl-dev curl build-essential python RUN gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/ RUN gem install --no-ri --no-rdoc uwsgi sinatra RUN mkdir -p /opt/distributed_app WORKDIR /opt/distributed_app RUN uwsgi --build-plugin https://github.com/unbit/uwsgi-consul ADD uwsgi-consul.ini /opt/distributed_app/ ADD config.ru /opt/distributed_app/ ENTRYPOINT [ "uwsgi", "--ini", "uwsgi-consul.ini", "--ini", "uwsgi-consul.ini:server1", "--ini", "uwsgi-consul.ini:server2" ] CMD [] ROM phusion/baseimage:latest MAINTAINER millions <chenzhesp@gmail.com> RUN DEBIAN_FRONTEND=noninteractive RUN locale-gen en_US.UTF-8 ENV LANGUAGE=en_US.UTF-8 ENV LC_ALL=en_US.UTF-8 ENV LC_CTYPE=UTF-8 ENV LANG=en_US.UTF-8 ENV TERM xterm RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list RUN apt-get update -y RUN apt-get upgrade -y RUN apt-get -qq install ruby-dev git libcurl4-openssl-dev curl build-essential python RUN gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/ RUN gem install --no-ri --no-rdoc uwsgi sinatra RUN mkdir -p /opt/distributed_app WORKDIR /opt/distributed_app RUN uwsgi --build-plugin https://github.com/unbit/uwsgi-consul ADD uwsgi-consul.ini /opt/distributed_app/ ADD config.ru /opt/distributed_app/ ENTRYPOINT [ "uwsgi", "--ini", "uwsgi-consul.ini", "--ini", "uwsgi-consul.ini:server1", "--ini", "uwsgi-consul.ini:server2" ] CMD []

config.ru 的內容爲：

require 'rubygems' require 'sinatra' get '/' do "Hello World!" end run Sinatra::Application require 'rubygems' require 'sinatra' get '/' do "Hello World!" end run Sinatra::Application

uwsgi-consul.ini 的內容爲：

[uwsgi] plugins = consul socket = 127.0.0.1:9999 master = true enable-threads = true [server1] consul-register = url=http://%h.node.consul:8500,name=distributed_app,id=server1,port=2001 mule = config.ru [server2] consul-register = url=http://%h.node.consul:8500,name=distributed_app,id=server2,port=2002 mule = config.ru [uwsgi] plugins = consul socket = 127.0.0.1:9999 master = true enable-threads = true [server1] consul-register = url=http://%h.node.consul:8500,name=distributed_app,id=server1,port=2001 mule = config.ru [server2] consul-register = url=http://%h.node.consul:8500,name=distributed_app,id=server2,port=2002 mule = config.ru

然後建立鏡像：

sudo docker build -t millions/distributed_app .

咱們建立好鏡像後，在 n1 和 n2 上分別啓動該容器：

sudo docker run -h $HOSTNAME -d --name n1-distributed \
--dns 172.17.0.1 --dns 8.8.8.8 \
 --dns-search 'service.consul'  millions/distributed_app

完成以後能夠在 web-ui 上看到 n1 和 n2 上各有2個名爲distributed_app的服務。

web-ui

咱們使用一個腳原本進行測試，爲了再溫習一下 docker，咱們仍是將其放在一個 docker 容器中運行，Dockerfile 以下，注意咱們替換了使用了淘寶的 ruby source 來加速：

FROM phusion/baseimage:latest MAINTAINER millions <chenzhesp@gmail.com> RUN DEBIAN_FRONTEND=noninteractive RUN locale-gen en_US.UTF-8 ENV LANGUAGE=en_US.UTF-8 ENV LC_ALL=en_US.UTF-8 ENV LC_CTYPE=UTF-8 ENV LANG=en_US.UTF-8 ENV TERM xterm RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list RUN apt-get update -y RUN apt-get upgrade -y RUN apt-get -qq install ruby ruby-dev build-essential RUN gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/ RUN gem install --no-ri --no-rdoc json RUN mkdir -p /opt/distributed_client ADD client.rb /opt/distributed_client/ WORKDIR /opt/distributed_client ENTRYPOINT [ "ruby", "/opt/distributed_client/client.rb" ] CMD [] FROM phusion/baseimage:latest MAINTAINER millions <chenzhesp@gmail.com> RUN DEBIAN_FRONTEND=noninteractive RUN locale-gen en_US.UTF-8 ENV LANGUAGE=en_US.UTF-8 ENV LC_ALL=en_US.UTF-8 ENV LC_CTYPE=UTF-8 ENV LANG=en_US.UTF-8 ENV TERM xterm RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list RUN apt-get update -y RUN apt-get upgrade -y RUN apt-get -qq install ruby ruby-dev build-essential RUN gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/ RUN gem install --no-ri --no-rdoc json RUN mkdir -p /opt/distributed_client ADD client.rb /opt/distributed_client/ WORKDIR /opt/distributed_client ENTRYPOINT [ "ruby", "/opt/distributed_client/client.rb" ] CMD []

ruby 腳本以下，分別使用了http和dns兩種方式來獲取服務:

require "rubygems" require "json" require "net/http" require "uri" require "resolv" empty = "There are no distributed applications registered in Consul" uri = URI.parse("http://consul.service.consul:8500/v1/catalog/service/distributed_app") http = Net::HTTP.new(uri.host, uri.port) request = Net::HTTP::Get.new(uri.request_uri) while true response = http.request(request) if response.body == "{}" puts empty sleep(1) elsif result = JSON.parse(response.body) result.each do |service| puts "Application #{service['ServiceName']} with element #{service["ServiceID"]} on port #{service["ServicePort"]} found on node #{service["Node"]} (#{service["Address"]})." sleep(1) end end end require "rubygems" require "json" require "net/http" require "uri" require "resolv" empty = "There are no distributed applications registered in Consul" uri = URI.parse("http://consul.service.consul:8500/v1/catalog/service/distributed_app") http = Net::HTTP.new(uri.host, uri.port) request = Net::HTTP::Get.new(uri.request_uri) while true response = http.request(request) if response.body == "{}" puts empty sleep(1) elsif result = JSON.parse(response.body) result.each do |service| puts "Application #{service['ServiceName']} with element #{service["ServiceID"]} on port #{service["ServicePort"]} found on node #{service["Node"]} (#{service["Address"]})." sleep(1) end end end

建立好鏡像：

sudo docker build -t millions/distributed_client .

運行之：

sudo docker run -h $HOSTNAME -d --name n3-distributed \
 --dns 172.17.0.1 --dns 8.8.8.8  \
 --dns-search 'service.consul' millions/distributed_client

使用 sudo docker logs n3-distributed 命令，能夠查看到:

ubuntu@n3:~$ sudo docker logs n3-distributed Application distributed_app with element server1 on port 2001 found on node n1 (172.20.20.10). Application distributed_app with element server2 on port 2002 found on node n1 (172.20.20.10). Application distributed_app with element server1 on port 2001 found on node n2 (172.20.20.11). Application distributed_app with element server2 on port 2002 found on node n2 (172.20.20.11). ubuntu@n3:~$ sudo docker logs n3-distributed Application distributed_app with element server1 on port 2001 found on node n1 (172.20.20.10). Application distributed_app with element server2 on port 2002 found on node n1 (172.20.20.10). Application distributed_app with element server1 on port 2001 found on node n2 (172.20.20.11). Application distributed_app with element server2 on port 2002 found on node n2 (172.20.20.11).

或者可使用 dns 來向 consul 查詢 ip 以及 port，咱們利用 dig 命令查看 dns 的結果，能夠看到獲取了 port 以及 n1 和 n2 的 ip：

ubuntu@n3:~$ dig @172.17.0.1 distributed_app.service.consul SRV ; <<>> DiG 9.10.3-P4-Ubuntu <<>> @172.17.0.1 distributed_app.service.consul SRV ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30551 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 2 ;; QUESTION SECTION: ;distributed_app.service.consul. IN SRV ;; ANSWER SECTION: distributed_app.service.consul. 0 IN SRV 1 1 2001 n2.node.dc1.consul. distributed_app.service.consul. 0 IN SRV 1 1 2001 n1.node.dc1.consul. distributed_app.service.consul. 0 IN SRV 1 1 2002 n1.node.dc1.consul. ;; ADDITIONAL SECTION: n2.node.dc1.consul. 0 IN A 172.20.20.11 n1.node.dc1.consul. 0 IN A 172.20.20.10 ;; Query time: 0 msec ;; SERVER: 172.17.0.1#53(172.17.0.1) ;; WHEN: Tue Mar 28 14:41:34 UTC 2017 ;; MSG SIZE rcvd: 155 ubuntu@n3:~$ dig @172.17.0.1 distributed_app.service.consul SRV ; <<>> DiG 9.10.3-P4-Ubuntu <<>> @172.17.0.1 distributed_app.service.consul SRV ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30551 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 2 ;; QUESTION SECTION: ;distributed_app.service.consul. IN SRV ;; ANSWER SECTION: distributed_app.service.consul. 0 IN SRV 1 1 2001 n2.node.dc1.consul. distributed_app.service.consul. 0 IN SRV 1 1 2001 n1.node.dc1.consul. distributed_app.service.consul. 0 IN SRV 1 1 2002 n1.node.dc1.consul. ;; ADDITIONAL SECTION: n2.node.dc1.consul. 0 IN A 172.20.20.11 n1.node.dc1.consul. 0 IN A 172.20.20.10 ;; Query time: 0 msec ;; SERVER: 172.17.0.1#53(172.17.0.1) ;; WHEN: Tue Mar 28 14:41:34 UTC 2017 ;; MSG SIZE rcvd: 155

總結

---

至此，咱們的consul 集羣開發環境就搭建好了。關機前不要忘記用 vagrant halt 或者 vagrant suspend 關閉或暫停你的虛擬機哦~

做者：millions_chan 連接：https://www.jianshu.com/p/52f4ea31aa1b 來源：簡書 簡書著做權歸做者全部，任何形式的轉載都請聯繫做者得到受權並註明出處。