Overlay Network:覆蓋網絡,在基礎網絡上疊加的一種虛擬網絡技術模式,該網絡中的主機經過虛擬鏈路鏈接起來。
VXLAN:將源數據包封裝到UDP中,並使用基礎網絡的IP/MAC做爲外層報文頭進行封裝,而後在以太網上傳輸,到達目的地後由隧道端點解封並將數據發送給目的地址。
Fannel:Overlay網絡的一種,也是將源數據包封裝在另外一種網絡包裏面進行路由轉發和通訊,目前已經支持UDP、VXLAN、AWS VPC和GCE路由等數據轉發方式。
多主機容器網絡通訊其餘主流方案:隧道方案(Weave、OpenvSwitch)、路由方案(Calico)等。
node
一、寫入分配的子網段到etcd,供flanneld使用linux
[root@master ~]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints="https://192.168.238.130:2379,https://192.168.238.129:2379,https://192.168.238.128:2379" set /coreos.com/network/config '{"Network":"172.17.0.0/16","Backend":{"Type":"vxlan"}}'
二、下載二進制包git
[root@master ~]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz [root@master ~]# ls flannel-v0.11.0-linux-amd64.tar.gz [root@master ~]# tar -zxf flannel-v0.11.0-linux-amd64.tar.gz [root@master ~]# ls mk-docker-opts.sh flanneld [root@master ~]# mv flanneld mk-docker-opts.sh /opt/kubernetes/bin [root@master ~]# ls /opt/kubernetes/bin/ etcd etcdctl flanneld mk-docker-opts.sh 在node01和node02重複上述操做。
三、配置flannelgithub
[root@node01 ~]# cat /opt/kubernetes/cfg/flanneld FIANNEL_OPTIONS="--etcd-endpoints=https://192.168.238.129:2380,https://192.168.238.128:2380,https://192.168.238.130:2380 -etcd-cafile=/opt/kubernetes/ssl/ca.pem -etcd-certfile=/opt/kubernetes/ssl/server.pem --etcd-keyfile=/opt/kubernetes/ssl/server-key.pem"
四、systemd管理flanneldocker
[root@node01 ~]# cat /usr/lib/systemd/system/flanneld.service [Unit] Description=Flanneld overlay address etcd agent After=network-online.target network.target Before=docker.service [Service] Type=notify EnvironmentFile=/opt/kubernetes/cfg/flanneld ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIOS ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env Restart=on-failure [Install] WantedBy=multi-user.target
五、配置docker啓動指定子網段
六、啓動vim
加載配置 [root@node01 ~]# systemctl daemon-reload [root@node01 ~]# systemctl start flanneld Job for flanneld.service failed because the control process exited with error code. See "systemctl status flanneld.service" and "journalctl -xe" for details. 查看系統日誌 [root@node01 ~]# tail -n 20 /var/log/messages Jul 4 20:15:24 localhost etcd: c858c42725f38881 received MsgVoteResp from c858c42725f38881 at term 16130 Jul 4 20:15:24 localhost etcd: c858c42725f38881 [logterm: 7765, index: 18] sent MsgVote request to a7e9807772a004c5 at term 16130 Jul 4 20:15:24 localhost etcd: c858c42725f38881 [logterm: 7765, index: 18] sent MsgVote request to 203750a5948d27da at term 16130 Jul 4 20:15:25 localhost etcd: c858c42725f38881 is starting a new election at term 16130 Jul 4 20:15:25 localhost etcd: c858c42725f38881 became candidate at term 16131 Jul 4 20:15:25 localhost etcd: c858c42725f38881 received MsgVoteResp from c858c42725f38881 at term 16131 Jul 4 20:15:25 localhost etcd: c858c42725f38881 [logterm: 7765, index: 18] sent MsgVote request to 203750a5948d27da at term 16131 Jul 4 20:15:25 localhost etcd: c858c42725f38881 [logterm: 7765, index: 18] sent MsgVote request to a7e9807772a004c5 at term 16131 Jul 4 20:15:27 localhost etcd: c858c42725f38881 is starting a new election at term 16131 Jul 4 20:15:27 localhost etcd: c858c42725f38881 became candidate at term 16132 Jul 4 20:15:27 localhost etcd: c858c42725f38881 received MsgVoteResp from c858c42725f38881 at term 16132 Jul 4 20:15:27 localhost etcd: c858c42725f38881 [logterm: 7765, index: 18] sent MsgVote request to 203750a5948d27da at term 16132 Jul 4 20:15:27 localhost etcd: c858c42725f38881 [logterm: 7765, index: 18] sent MsgVote request to a7e9807772a004c5 at term 16132 Jul 4 20:15:28 localhost etcd: c858c42725f38881 is starting a new election at term 16132 Jul 4 20:15:28 localhost etcd: c858c42725f38881 became candidate at term 16133 Jul 4 20:15:28 localhost etcd: c858c42725f38881 received MsgVoteResp from c858c42725f38881 at term 16133 Jul 4 20:15:28 localhost etcd: c858c42725f38881 [logterm: 7765, index: 18] sent MsgVote request to 203750a5948d27da at term 16133 Jul 4 20:15:28 localhost etcd: c858c42725f38881 [logterm: 7765, index: 18] sent MsgVote request to a7e9807772a004c 5 at term 16133 Jul 4 20:15:28 localhost etcd: health check for peer 203750a5948d27da could not connect: dial tcp 192.168.238.128:2380: getsockopt: no route to host Jul 4 20:15:28 localhost etcd: health check for peer a7e9807772a004c5 could not connect: dial tcp 192.168.238.130:2380: i/o timeout 初步斷定防火牆致使,關閉防火牆 [root@node01 ~]# systemctl stop firewalld.service [root@node01 ~]# systemctl start flanneld Job for flanneld.service failed because a timeout was exceeded. See "systemctl status flanneld.service" and "journalctl -xe" for details. 網絡故障緣由 [root@node01 ~]# tail -n 20 /var/log/messages Jul 6 08:49:15 localhost systemd: flanneld.service failed. Jul 6 08:49:15 localhost systemd: flanneld.service holdoff time over, scheduling restart. Jul 6 08:49:15 localhost systemd: Stopped Flanneld overlay address etcd agent. Jul 6 08:49:15 localhost systemd: Starting Flanneld overlay address etcd agent... Jul 6 08:49:15 localhost flanneld: I0706 08:49:15.831267 8741 main.go:514] Determining IP address of default interface Jul 6 08:49:15 localhost flanneld: I0706 08:49:15.831870 8741 main.go:527] Using interface with name eno16777736 and address 192.168.238.129 Jul 6 08:49:15 localhost flanneld: I0706 08:49:15.831905 8741 main.go:544] Defaulting external address to interface address (192.168.238.129) Jul 6 08:49:15 localhost flanneld: I0706 08:49:15.831987 8741 main.go:244] Created subnet manager: Etcd Local Manager with Previous Subnet: None Jul 6 08:49:15 localhost flanneld: I0706 08:49:15.831992 8741 main.go:247] Installing signal handlers Jul 6 08:49:15 localhost flanneld: E0706 08:49:15.834924 8741 main.go:382] Couldn't fetch network config: 100: Key not found (/coreos.com) [16] Jul 6 08:49:16 localhost flanneld: timed out Jul 6 08:49:16 localhost flanneld: E0706 08:49:16.837394 8741 main.go:382] Couldn't fetch network config: 100: Key not found (/coreos.com) [16] Jul 6 08:49:17 localhost flanneld: timed out Jul 6 08:49:17 localhost flanneld: E0706 08:49:17.840183 8741 main.go:382] Couldn't fetch network config: 100: Key not found (/coreos.com) [16] Jul 6 08:49:18 localhost flanneld: timed out Jul 6 08:49:18 localhost flanneld: E0706 08:49:18.842579 8741 main.go:382] Couldn't fetch network config: 100: Key not found (/coreos.com) [16] Jul 6 08:49:19 localhost flanneld: timed out Jul 6 08:49:19 localhost flanneld: E0706 08:49:19.845302 8741 main.go:382] Couldn't fetch network config: 100: Key not found (/coreos.com) [16] Jul 6 08:49:20 localhost flanneld: timed out Jul 6 08:49:20 localhost flanneld: E0706 08:49:20.848554 8741 main.go:382] Couldn't fetch network config: 100: Key not found (/coreos.com) [16] 測試網絡是否正常 [root@node01 ~]# telnet 192.168.238.130 2379 Trying 192.168.238.130... Connected to 192.168.238.130. Escape character is '^]'. quit Connection closed by foreign host. 檢查key是否存在 [root@master ~]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints="https://192.168.238.130:2379,https://192.168.238.129:2379,https://192.168.238.128:2379" get /coreos.com/network/config Error: 100: Key not found (/coreos.com) [16] 主節點從新添加網絡步驟一 [root@master ~]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints="https://192.168.238.130:2379,https://192.168.238.129:2379,https://192.168.238.128:2379" set /coreos.com/network/config '{"Network":"172.17.0.0/16","Backend":{"Type":"vxlan"}}' {"Network":"172.17.0.0/16","Backend":{"Type":"vxlan"}} 再次啓動 [root@node01 ~]# systemctl start flanneld [root@node01 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:29:11:0e brd ff:ff:ff:ff:ff:ff inet 192.168.238.129/24 brd 192.168.238.255 scope global dynamic eno16777736 valid_lft 1633sec preferred_lft 1633sec inet6 fe80::20c:29ff:fe29:110e/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:aa:0a:b1:a5 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN link/ether 16:22:a1:7a:3a:99 brd ff:ff:ff:ff:ff:ff inet 172.17.64.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::1422:a1ff:fe7a:3a99/64 scope link valid_lft forever preferred_lft forever 查看flannel分配的子網信息 [root@node01 ~]# cat /run/flannel/subnet.env DOCKER_OPT_BIP="--bip=172.17.64.1/24" DOCKER_OPT_IPMASQ="--ip-masq=false" DOCKER_OPT_MTU="--mtu=1450" DOCKER_NETWORK_OPTIONS=" --bip=172.17.64.1/24 --ip-masq=false --mtu=1450" 配置docker,註釋相同選項,新增以下內容 [root@node01 ~]# vi /usr/lib/systemd/system/docker.service EnvironmentFile=/run/flannel/subnet.env ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS 重啓docker [root@node01 ~]# systemctl daemon-reload [root@node01 ~]# systemctl restart docker [root@node01 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:29:11:0e brd ff:ff:ff:ff:ff:ff inet 192.168.238.129/24 brd 192.168.238.255 scope global dynamic eno16777736 valid_lft 1400sec preferred_lft 1400sec inet6 fe80::20c:29ff:fe29:110e/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:aa:0a:b1:a5 brd ff:ff:ff:ff:ff:ff inet 172.17.64.1/24 brd 172.17.64.255 scope global docker0 valid_lft forever preferred_lft forever 4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN link/ether 16:22:a1:7a:3a:99 brd ff:ff:ff:ff:ff:ff inet 172.17.64.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::1422:a1ff:fe7a:3a99/64 scope link valid_lft forever preferred_lft forever 此時docker0與flannel.1在同一網段內 節點2重複上述操做進行配置 [root@node02 ~]# systemctl start flanneld [root@node02 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:5a:c2:eb brd ff:ff:ff:ff:ff:ff inet 192.168.238.128/24 brd 192.168.238.255 scope global dynamic eno16777736 valid_lft 1496sec preferred_lft 1496sec inet6 fe80::20c:29ff:fe5a:c2eb/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:63:4f:0b:45 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN link/ether ea:b7:55:da:3b:a7 brd ff:ff:ff:ff:ff:ff inet 172.17.89.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::e8b7:55ff:feda:3ba7/64 scope link valid_lft forever preferred_lft forever 設置docker [root@node02 ~]# vim /usr/lib/systemd/system/docker.service EnvironmentFile=/run/flannel/subnet.env ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS [root@node02 ~]# systemctl daemon-reload [root@node02 ~]# systemctl restart docker [root@node02 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:5a:c2:eb brd ff:ff:ff:ff:ff:ff inet 192.168.238.128/24 brd 192.168.238.255 scope global dynamic eno16777736 valid_lft 1191sec preferred_lft 1191sec inet6 fe80::20c:29ff:fe5a:c2eb/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:63:4f:0b:45 brd ff:ff:ff:ff:ff:ff inet 172.17.89.1/24 brd 172.17.89.255 scope global docker0 valid_lft forever preferred_lft forever 4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN link/ether ea:b7:55:da:3b:a7 brd ff:ff:ff:ff:ff:ff inet 172.17.89.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::e8b7:55ff:feda:3ba7/64 scope link valid_lft forever preferred_lft forever 測試網絡是否正常 [root@node02 ~]# ping 172.17.64.1 PING 172.17.64.1 (172.17.64.1) 56(84) bytes of data. 64 bytes from 172.17.64.1: icmp_seq=1 ttl=64 time=0.508 ms 64 bytes from 172.17.64.1: icmp_seq=2 ttl=64 time=0.336 ms [root@node01 ~]# ping 172.17.64.1 PING 172.17.64.1 (172.17.64.1) 56(84) bytes of data. 64 bytes from 172.17.64.1: icmp_seq=1 ttl=64 time=0.032 ms 64 bytes from 172.17.64.1: icmp_seq=2 ttl=64 time=0.030 ms 啓用防火牆的狀況下須要配置防火牆策略 [root@master ~]# iptables -I INPUT -s 192.168.0.0/24 -j ACCEPT 列出存儲的信息 [root@master ~]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints="https://192.168.238.130:2379,https://192.168.238.129:2379,https://192.168.238.128:2379" ls /coreos.com/network/ /coreos.com/network/subnets /coreos.com/network/config 列出配置的網絡 [root@master ~]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints="https://192.168.238.130:2379,https://192.168.238.129:2379,https://192.168.238.128:2379" ls /coreos.com/network/subnets /coreos.com/network/subnets/172.17.64.0-24 /coreos.com/network/subnets/172.17.89.0-24 獲取key [root@master ~]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints="https://192.168.238.130:2379,https://192.168.238.129:2379,https://192.168.238.128:2379" get /coreos.com/network/subnets/172.17.64.0-24 {"PublicIP":"192.168.238.129","BackendType":"vxlan","BackendData":{"VtepMAC":"16:22:a1:7a:3a:99"}} 查看路由表信息 [root@node01 ~]# ip route show default via 192.168.238.2 dev eno16777736 proto static metric 100 172.17.64.0/24 dev docker0 proto kernel scope link src 172.17.64.1 172.17.89.0/24 via 172.17.89.0 dev flannel.1 onlink 192.168.238.0/24 dev eno16777736 proto kernel scope link src 192.168.238.129 metric 100