etcd和flannel實現docker跨物理機通訊

實驗目標

跨物理機的容器之間能直接訪問
docker經過Flannel能夠實現各容器間的相互通訊，即宿主機和容器，容器和容器之間都能相互通訊

實驗環境

192.168.3.50 //etcd、flannel、docker
192.168.3.51 //flannel、docker

00、系通通一設置

hosts\selinux\firewalld\iptalbes

systemctl stop firewalld.service

#放開iptables，讓flannel經過iptables實現跨網通訊

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -F

0一、etcd部署

https://github.com/etcd-io/etcd/releases
wget https://github.com/coreos/etcd/releases/download/v2.3.2/etcd-v2.3.2-linux-amd64.tar.gz
tar xvf etcd-v2.3.2-linux-amd64.tar.gz
cd etcd-v2.3.2-linux-amd64
mv etcd* /usr/local/bin/

加入systemd管理，而且爲了之後擴展，我還設置了集羣，你們能夠沒必要添加集羣設置
cat > /usr/lib/systemd/system/etcd.service <<-'EOF'
[Unit]
Description=etcd
[Service]
Environment=ETCD_NAME=kubernetes
Environment=ETCD_DATA_DIR=/var/lib/etcd
Environment=ETCD_LISTEN_CLIENT_URLS=http://192.168.3.50:4001,http://localhost:4001
Environment=ETCD_LISTEN_PEER_URLS=http://192.168.3.50:7001,http://localhost:7001
Environment=ETCD_INITIAL_ADVERTISE_PEER_URLS=http://192.168.3.50:7001,http://localhost:7001
Environment=ETCD_ADVERTISE_CLIENT_URLS=http://192.168.3.50:4001,http://localhost:4001
Environment=ETCD_INITIAL_CLUSTER_STATE=new
Environment=ETCD_INITIAL_CLUSTER_TOKEN=Kubernetes
Environment=ETCD_INITIAL_CLUSTER=kubernetes=http://192.168.3.50:7001,kubernetes=http://localhost:7001
ExecStart=/usr/local/bin/etcd
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl start etcd

etcd添加網段設置分配給docker網絡的網段
etcdctl mk /coreos.com/network/config '{"Network":"172.20.0.0/16", "SubnetMin": "172.20.1.0", "SubnetMax": "172.20.254.0"}'
etcdctl get /coreos.com/network/config

 0二、docker-ce部署

參考以下文章：

https://www.cnblogs.com/xiaochina/p/10469715.html

https://www.cnblogs.com/xiaochina/p/11518007.html

0三、flannel部署

注意：全部主機均須要安裝flannel
二進制安裝flannel
wget https://github.com/coreos/flannel/releases/download/v0.5.5/flannel-0.5.5-linux-amd64.tar.gz
[root@c_3 ~]# tar zxf flannel-0.5.5-linux-amd64.tar.gz
[root@c_3 ~]# mv flannel-0.5.5 /opt/flannel
[root@c_3 ~]#
[root@c_3 ~]# ll /opt/flannel/
total 16212
-rwxr-xr-x. 1 1000 1000 16581152 Nov 13 2015 flanneld
-rwxrwxr-x. 1 1000 1000 2008 Nov 13 2015 mk-docker-opts.sh
-rw-rw-r--. 1 1000 1000 11843 Nov 13 2015 README.md

添加一個flannel服務的System單元，簡單的就能夠
cat > /usr/lib/systemd/system/flanneld.service <<-'EOF'
[Unit]
Description=flannel
[Service]
ExecStart=/opt/flannel/flanneld \
　　-etcd-endpoints=http://192.168.3.50:4001
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl start flanneld && systemctl status flanneld

[root@c_3 ~]# ip a |grep flannel
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN group default qlen 500
inet 172.20.56.0/16 scope global flannel0

ot@c_3 opt]# etcdctl ls /coreos.com/network/subnets //etcd查看已經分配的網段
/coreos.com/network/subnets/172.20.6.0-24
/coreos.com/network/subnets/172.20.56.0-24

 0四、docker-ce修改配置

cd /opt/flannel

[root@c_3 flannel]# ./mk-docker-opts.sh --help
./mk-docker-opts.sh: illegal option -- -
./mk-docker-opts.sh [-f FLANNEL-ENV-FILE] [-d DOCKER-ENV-FILE] [-i] [-c] [-m] [-k COMBINED-KEY]

Generate Docker daemon options based on flannel env file
OPTIONS:
-f Path to flannel env file. Defaults to /run/flannel/subnet.env //flannel0網卡的配置信息
-d Path to Docker env file to write to. Defaults to /run/docker_opts.env
-i Output each Docker option as individual var. e.g. DOCKER_OPT_MTU=1500
-c Output combined Docker options into DOCKER_OPTS var //網絡信息寫入 /run/docker_opts.env
-k Set the combined options key to this value (default DOCKER_OPTS=)
-m Do not output --ip-masq (useful for older Docker version)

[root@c_3 flannel]# ./mk-docker-opts.sh -c    //建立並寫入 /run/docker_opts.env
[root@c_3 flannel]# cat /run/docker_opts.env //保存環境鍵值對
DOCKER_OPTS=" --bip=172.20.56.1/24 --ip-masq=true --mtu=1472 "

docker使用flannel的網絡傳遞數據修改docker啓動參數
[root@c_3 ~]# systemctl cat docker | head -1 //查看docker讀取的配置文件
# /usr/lib/systemd/system/docker.service

/usr/lib/systemd/system/docker.service
EnvironmentFile=-/run/docker_opts.env
ExecStart=/usr/bin/dockerd $DOCKER_OPTS

systemctl daemon-reload && systemctl restart docker

[root@c_3 flannel]# ip a|grep docker0 //docker0網絡信息已經變化
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
inet 172.20.56.1/24 brd 172.20.56.255 scope global docker0

[root@c_3 flannel]# docker network ls
NETWORK ID NAME DRIVER SCOPE
3c6b97123f83 bridge bridge local
4501ec9545f6 host host local
ccd8eef9f50b none null local
[root@c_3 flannel]# docker network inspect 3c6b97123f83 |grep -i subnet
"Subnet": "172.20.56.1/24",

0五、測試跨主機，容器通訊

docker run --rm -it busybox sh   //各個host上運行

參考：

　　https://www.cnblogs.com/linjiqin/p/11549329.html