Kubernetes集羣搭建之CNI-Flanneld部署篇

本次系列使用的所需部署包版本都使用的目前最新的或最新穩定版，安裝包地址請到公衆號內回覆【K8s實戰】獲取

Flannel是CoreOS提供用於解決Dokcer集羣跨主機通信的覆蓋網絡工具。它的主要思路是：預先留出一個網段，每一個主機使用其中一部分，而後每一個容器被分配不一樣的ip；讓全部的容器認爲你們在同一個直連的網絡，底層經過UDP/VxLAN等進行報文的封裝和轉發。

架構介紹

---

Flannel默認使用8285端口做爲UDP封裝報文的端口，VxLan使用8472端口。

K8s的有不少CNI(網絡網絡接口)組件，好比Flannel、Calico，我司目前使用的是Flannel，穩定性還能夠。因此我這裏先只介紹Flannel，Calico後續有機會會分享。

etcd和docker部署在前兩篇文章已經結束，這裏就不過多展開了。

Flanneld部署

---

因爲flanneld須要依賴etcd來保證集羣IP分配不衝突的問題，因此首先要在etcd中設置 flannel節點所使用的IP段。

[root@master-01 ~]# etcdctl --ca-file=/etc/etcd/ssl/ca.pem --cert-file=/etc/etcd/ssl/server.pem --key-file=/etc/etcd/ssl/server-key.pem --endpoints="https://192.168.209.130:2379,https://192.168.209.130:2379,https://192.168.209.130:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}

注： flanneld默認的Backend 類型是udp   這裏改爲vxlan 性能要比udp好一些

1. 解壓flanneld

[root@master-01 ~]# tar xf flannel-v0.11.0-linux-amd64.tar.gz [root@master-01 ~]# mv flanneld mk-docker-opts.sh /usr/bin/

2. 配置flanneld

[root@master-01 ~]# cat /etc/kubernetes/flanneld FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.209.130:2379,https://192.168.209.130:2379,https://192.168.209.130:2379 -etcd-cafile=/etc/etcd/ssl/ca.pem -etcd-certfile=/etc/etcd/ssl/server.pem -etcd-keyfile=/etc/etcd/ssl/server-key.pem -etcd-prefix=/coreos.com/network"

3.  配置flanneld啓動文件

[root@master-01 ~]# cat /usr/lib/systemd/system/flanneld.service[Unit]Description=Flanneld overlay address etcd agentAfter=network-online.target network.targetBefore=docker.service[Service]Type=notifyEnvironmentFile=/etc/kubernetes/flanneldExecStart=/usr/bin/flanneld --ip-masq $FLANNEL_OPTIONSExecStartPost=/usr/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.envRestart=on-failure[Install]WantedBy=multi-user.target

注: mk-docker-opts.sh 腳本將分配給 flanneld 的 Pod 子網網段信息寫入 /run/flannel/docker 文件，後續 docker 啓動時 使用這個文件中的環境變量配置 docker0 網橋； flanneld 使用系統缺省路由所在的接口與其它節點通訊，對於有多個網絡接口（如內網和公網）的節點，能夠用 -iface 參數指定通訊接口; flanneld 運行時須要 root 權限；

4. 配置Docker啓動參數

在各個節點安裝好之後最後要更改Docker的啓動參數，使其可以使用flannel進行IP分配，以及網絡通信。

修改docker的啓動參數，並使其啓動後使用由flannel生成的配置參數，修改以下:

[root@master-01 ~]# cat /usr/lib/systemd/system/docker.service|grep -v "^#"[Unit]Description=Docker Application Container EngineDocumentation=https://docs.docker.comBindsTo=containerd.serviceAfter=network-online.target firewalld.service containerd.serviceWants=network-online.targetRequires=docker.socket[Service]Type=notifyEnvironmentFile=/run/flannel/subnet.envExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock $DOCKER_NETWORK_OPTIONSExecReload=/bin/kill -s HUP $MAINPIDTimeoutSec=0RestartSec=2Restart=alwaysStartLimitBurst=3StartLimitInterval=60sLimitNOFILE=infinityLimitNPROC=infinityLimitCORE=infinityTasksMax=infinityDelegate=yesKillMode=process[Install]WantedBy=multi-user.target

5. 同步配置和二進制

如今把配置和二進制文件都拷貝到另外幾個節點上

# 二進制 and 腳本[root@master-01 ~]# scp /usr/bin/flanneld /usr/bin/mk-docker-opts.sh 192.168.209.131:/usr/bin/[root@master-01 ~]# scp /usr/bin/flanneld /usr/bin/mk-docker-opts.sh 192.168.209.132:/usr/bin/[root@master-01 ~]# scp /usr/bin/flanneld /usr/bin/mk-docker-opts.sh 192.168.209.133:/usr/bin/# 配置文件[root@master-01 ~]# scp /etc/kubernetes/flanneld 192.168.209.131:/etc/kubernetes/[root@master-01 ~]# scp /etc/kubernetes/flanneld 192.168.209.132:/etc/kubernetes/[root@master-01 ~]# scp /etc/kubernetes/flanneld 192.168.209.133:/etc/kubernetes/# 啓動文件[root@master-01 ~]# scp /usr/lib/systemd/system/flanneld.service /usr/lib/systemd/system/docker.service 192.168.209.131:/usr/lib/systemd/system/[root@master-01 ~]# scp /usr/lib/systemd/system/flanneld.service /usr/lib/systemd/system/docker.service 192.168.209.132:/usr/lib/systemd/system/[root@master-01 ~]# scp /usr/lib/systemd/system/flanneld.service /usr/lib/systemd/system/docker.service 192.168.209.133:/usr/lib/systemd/system/

6. 啓動服務

相繼啓動各節點的服務 注意啓動flannel再重啓docker 纔會覆蓋docker0網橋

[root@master-01 ~]# systemctl daemon-reload[root@master-01 ~]# systemctl start flanneld[root@master-01 ~]# systemctl enable flanneld[root@master-01 ~]# systemctl restart dockerCreated symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.[root@master-01 ~]# systemctl restart docker

flannel服務啓動時主要作了如下幾步的工做： 從etcd中獲取network的配置信息 劃分subnet，並在etcd中進行註冊 將子網信息記錄到/run/flannel/subnet.env中，以保證各個節點的flanneld IP不會重複分配

7. 驗證服務

# master-01[root@master-01 ~]# ip addr2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 00:0c:29:72:13:72 brd ff:ff:ff:ff:ff:ffinet 192.168.209.130/24 brd 192.168.209.255 scope global noprefixroute ens33valid_lft forever preferred_lft foreverinet6 fe80::5bba:68b5:e255:a636/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::23a0:4782:75a7:43b/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::46a2:a117:34d9:db1a/64 scope link noprefixroute valid_lft forever preferred_lft forever3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:3e:8f:c7:39 brd ff:ff:ff:ff:ff:ffinet 172.17.58.1/24 brd 172.17.58.255 scope global docker0valid_lft forever preferred_lft forever4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 7e:2b:b9:3c:ce:36 brd ff:ff:ff:ff:ff:ffinet 172.17.58.0/32 scope global flannel.1valid_lft forever preferred_lft foreverinet6 fe80::7c2b:b9ff:fe3c:ce36/64 scope link valid_lft forever preferred_lft forever# master-02[root@master-02 ~]# ip addr2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 00:0c:29:ac:58:9d brd ff:ff:ff:ff:ff:ffinet 192.168.209.131/24 brd 192.168.209.255 scope global noprefixroute ens33valid_lft forever preferred_lft foreverinet6 fe80::5bba:68b5:e255:a636/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::23a0:4782:75a7:43b/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::46a2:a117:34d9:db1a/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:32:c7:25:ca brd ff:ff:ff:ff:ff:ffinet 172.17.44.1/24 brd 172.17.44.255 scope global docker0valid_lft forever preferred_lft forever4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 62:c3:35:ec:17:54 brd ff:ff:ff:ff:ff:ffinet 172.17.44.0/32 scope global flannel.1valid_lft forever preferred_lft foreverinet6 fe80::60c3:35ff:feec:1754/64 scope link valid_lft forever preferred_lft forever

須要確保docker0和flanneld在同一個網段

測試不一樣節點互通，在master-01上ping另外幾個節點的docker0 ip

[root@master-01 ~]# ping 172.17.44.1PING 172.17.44.1 (172.17.44.1) 56(84) bytes of data.64 bytes from 172.17.44.1: icmp_seq=1 ttl=64 time=2.03 ms^C--- 172.17.44.1 ping statistics ---2 packets transmitted, 2 received, 0% packet loss, time 1001msrtt min/avg/max/mdev = 0.441/1.236/2.032/0.796 ms[root@master-01 ~]# ping 172.17.79.1PING 172.17.79.1 (172.17.79.1) 56(84) bytes of data.64 bytes from 172.17.79.1: icmp_seq=1 ttl=64 time=0.919 ms^C--- 172.17.79.1 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.919/0.919/0.919/0.000 ms[root@master-01 ~]# ping 172.17.47.1PING 172.17.47.1 (172.17.47.1) 56(84) bytes of data.64 bytes from 172.17.47.1: icmp_seq=1 ttl=64 time=0.369 ms^C--- 172.17.47.1 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.369/0.369/0.369/0.000 ms

若是能通說明Flannel部署成功。若是不通檢查下日誌：journalctl -u flannel或 tailf /var/log/messages

最後咱們來看下etcd中保存的網段信息

[root@master-01 ~]# etcdctl --ca-file=/etc/etcd/ssl/ca.pem --cert-file=/etc/etcd/ssl/server.pem --key-file=/etc/etcd/ssl/server-key.pem --endpoints="https://192.168.209.130:2379,https://192.168.209.131:2379,https://192.168.209.132:2379" ls /coreos.com/network/subnets/coreos.com/network/subnets/172.17.47.0-24/coreos.com/network/subnets/172.17.58.0-24/coreos.com/network/subnets/172.17.44.0-24/coreos.com/network/subnets/172.17.79.0-24[root@master-01 ~]# etcdctl --ca-file=/etc/etcd/ssl/ca.pem --cert-file=/etc/etcd/ssl/server.pem --key-file=/etc/etcd/ssl/server-key.pem --endpoints="https://192.168.209.130:2379,https://192.168.209.131:2379,https://192.168.209.132:2379" get /coreos.com/network/subnets/172.17.47.0-24{"PublicIP":"192.168.209.133","BackendType":"vxlan","BackendData":{"VtepMAC":"8a:b1:a7:26:fd:33"}}[root@master-01 ~]# etcdctl --ca-file=/etc/etcd/ssl/ca.pem --cert-file=/etc/etcd/ssl/server.pem --key-file=/etc/etcd/ssl/server-key.pem --endpoints="https://192.168.209.130:2379,https://192.168.209.131:2379,https://192.168.209.132:2379" get /coreos.com/network/subnets/172.17.58.0-24{"PublicIP":"192.168.209.130","BackendType":"vxlan","BackendData":{"VtepMAC":"7e:2b:b9:3c:ce:36"}}[root@master-01 ~]# etcdctl --ca-file=/etc/etcd/ssl/ca.pem --cert-file=/etc/etcd/ssl/server.pem --key-file=/etc/etcd/ssl/server-key.pem --endpoints="https://192.168.209.130:2379,https://192.168.209.131:2379,https://192.168.209.132:2379" get /coreos.com/network/subnets/172.17.44.0-24{"PublicIP":"192.168.209.131","BackendType":"vxlan","BackendData":{"VtepMAC":"62:c3:35:ec:17:54"}}[root@master-01 ~]# etcdctl --ca-file=/etc/etcd/ssl/ca.pem --cert-file=/etc/etcd/ssl/server.pem --key-file=/etc/etcd/ssl/server-key.pem --endpoints="https://192.168.209.130:2379,https://192.168.209.131:2379,https://192.168.209.132:2379" get /coreos.com/network/subnets/172.17.79.0-24{"PublicIP":"192.168.209.132","BackendType":"vxlan","BackendData":{"VtepMAC":"02:a0:14:9f:ff:fe"}}

好了，到這一步咱們就完成了flanneld的部署，下一章介紹master集羣的部署，敬請期待，謝謝！

END

若是你以爲文章還不錯，請你們點**『好看』**分享下。你的確定是我最大的鼓勵和支持。

原文出處：https://www.cnblogs.com/guigujun/p/10504205.html