Kubernetes學習之路（五）之Flannel網絡二進制部署和測試

時間  2019-11-07
標籤 kubernetes 學習 之路 flannel 網絡 二進制 部署 測試 欄目 系統網絡 简体版
原文   原文鏈接

1、K8S的ip地址

Node IP：節點設備的IP，如物理機，虛擬機等容器宿主的實際IP。

Pod IP：Pod的IP地址，是根據docker0網絡IP段進行分配的。node

Cluster IP：Service的IP，是一個虛擬IP，僅做用於service對象，由K8S管理和分配，須要結合service port才能使用，單獨的IP沒有通訊功能，集羣外訪問須要一些修改。linux

在K8S集羣內部，node ip、pod ip、clustere ip的通訊機制是由k8s指定的路由規則，不是IP路由。git

[root@linux-node1 ~]# kubectl get service
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.1.0.1     <none>        443/TCP   3h

 2、Flannel網絡部署

（1）爲Flannel生成證書

[root@linux-node1 ssl]# vim flanneld-csr.json
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

（2）生成證書

[root@linux-node1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
>    -ca-key=/opt/kubernetes/ssl/ca-key.pem \
>    -config=/opt/kubernetes/ssl/ca-config.json \
>    -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
[root@linux-node1 ssl]# ll flannel*
-rw-r--r-- 1 root root  997 May 31 11:13 flanneld.csr
-rw-r--r-- 1 root root  221 May 31 11:13 flanneld-csr.json
-rw------- 1 root root 1675 May 31 11:13 flanneld-key.pem
-rw-r--r-- 1 root root 1391 May 31 11:13 flanneld.pem

（3）分發證書

[root@linux-node1 ssl]# cp flanneld*.pem /opt/kubernetes/ssl/
[root@linux-node1 ssl]# scp flanneld*.pem 192.168.56.120:/opt/kubernetes/ssl/
flanneld-key.pem                                           100% 1675   127.2KB/s   00:00    
flanneld.pem                                               100% 1391   308.3KB/s   00:00    
[root@linux-node1 ssl]# scp flanneld*.pem 192.168.56.130:/opt/kubernetes/ssl/
flanneld-key.pem                                           100% 1675   291.1KB/s   00:00    
flanneld.pem                                               100% 1391    90.4KB/s   00:00

（4）下載Flannel軟件包

[root@linux-node1 ~]# cd /usr/local/src
# wget
 https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
[root@linux-node1 src]# tar zxf flannel-v0.10.0-linux-amd64.tar.gz
[root@linux-node1 src]# cp flanneld mk-docker-opts.sh /opt/kubernetes/bin/
複製到linux-node2和linux-node3節點
[root@linux-node1 src]# scp flanneld mk-docker-opts.sh 192.168.56.120:/opt/kubernetes/bin/
[root@linux-node1 src]# scp flanneld mk-docker-opts.sh 192.168.56.130:/opt/kubernetes/bin/

複製對應腳本到/opt/kubernetes/bin目錄下。
[root@linux-node1 ~]# cd /usr/local/src/kubernetes/cluster/centos/node/bin/
[root@linux-node1 bin]# cp remove-docker0.sh /opt/kubernetes/bin/
[root@linux-node1 bin]# scp remove-docker0.sh 192.168.56.120:/opt/kubernetes/bin/
[root@linux-node1 bin]# scp remove-docker0.sh 192.168.56.130:/opt/kubernetes/bin/

（5）配置Flannel

[root@linux-node1 ~]# vim /opt/kubernetes/cfg/flannel
FLANNEL_ETCD="-etcd-endpoints=https://192.168.56.110:2379,https://192.168.56.120:2379,https://192.168.56.130:2379"
FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network"
FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem"
FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem"
FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem"
複製配置到其它節點上
[root@linux-node1 ~]# scp /opt/kubernetes/cfg/flannel 192.168.56.120:/opt/kubernetes/cfg/
[root@linux-node1 ~]# scp /opt/kubernetes/cfg/flannel 192.168.56.130:/opt/kubernetes/cfg/

（6）設置Flannel系統服務

[root@linux-node1 ~]# vim /usr/lib/systemd/system/flannel.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
Before=docker.service

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/flannel
ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh
ExecStart=/opt/kubernetes/bin/flanneld ${FLANNEL_ETCD} ${FLANNEL_ETCD_KEY} ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE}
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker

Type=notify

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
複製系統服務腳本到其它節點上
# scp /usr/lib/systemd/system/flannel.service 192.168.56.120:/usr/lib/systemd/system/
# scp /usr/lib/systemd/system/flannel.service 192.168.56.130:/usr/lib/systemd/system/

3、Flannel CNI集成

（1）下載CNI插件

https://github.com/containernetworking/plugins/releases
wget https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz
[root@linux-node1 ~]# mkdir /opt/kubernetes/bin/cni
[root@linux-node2 ~]# mkdir /opt/kubernetes/bin/cni
[root@linux-node3 ~]# mkdir /opt/kubernetes/bin/cni
[root@linux-node1 src]# tar zxf cni-plugins-amd64-v0.7.1.tgz -C /opt/kubernetes/bin/cni
[root@linux-node1 src]# scp -r /opt/kubernetes/bin/cni/* 192.168.56.120:/opt/kubernetes/bin/cni/
[root@linux-node1 src]# scp -r /opt/kubernetes/bin/cni/* 192.168.56.130:/opt/kubernetes/bin/cni/

（2）建立Etcd的key

此步的操做是爲了建立POD的網段，並在ETCD中存儲，然後FLANNEL從ETCD中取出並進行分配github

[root@linux-node1 src]# /opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file /opt/kubernetes/ssl/flanneld-key.pem \
      --no-sync -C https://192.168.56.110:2379,https://192.168.56.120:2379,https://192.168.56.130:2379 \
mk /kubernetes/network/config '{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}' >/dev/null 2>&1

（3）啓動flannel

[root@linux-node1 ~]# systemctl daemon-reload
[root@linux-node1 ~]# systemctl enable flannel
[root@linux-node1 ~]# chmod +x /opt/kubernetes/bin/*
[root@linux-node1 ~]# systemctl start flannel

[root@linux-node2 ~]# systemctl daemon-reload
[root@linux-node2 ~]# systemctl enable flannel
[root@linux-node2 ~]# chmod +x /opt/kubernetes/bin/*
[root@linux-node2 ~]# systemctl start flannel

[root@linux-node3 ~]# systemctl daemon-reload
[root@linux-node3 ~]# systemctl enable flannel
[root@linux-node3 ~]# chmod +x /opt/kubernetes/bin/*
[root@linux-node3 ~]# systemctl start flannel

能夠看到每一個節點上會多出一個flannel.1的網卡，不一樣的節點都在不一樣網段。docker

[root@linux-node1 ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.2.46.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::f4e6:1aff:fe7e:575b  prefixlen 64  scopeid 0x20<link>
        ether f6:e6:1a:7e:57:5b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

[root@linux-node2 ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.2.87.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::d4e5:72ff:fe3e:7309  prefixlen 64  scopeid 0x20<link>
        ether d6:e5:72:3e:73:09  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

[root@linux-node3 ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.2.33.0  netmask 255.255.255.255  broadcast 0.0.0.0
        ether be:cd:5a:4f:6b:d1  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0

（4）遇到的問題：Flannel沒法啓動

檢查/opt/kubernetes/cfg/etcd.conf配置文件中的ETCD_LISTEN_CLIENT_URLS是否配置監聽127.0.0.1:2379。依舊沒法啓動flannel，從新輸入了一遍，正常了，暫時沒發現其餘緣由，至於etcdctl沒法獲取key值，有待研究!!!json

4、配置Docker使用Flannel

[root@linux-node1 ~]# vim /usr/lib/systemd/system/docker.service
[Unit] #在Unit下面修改After和增長Requires
After=network-online.target firewalld.service flannel.service  #讓docker在flannel網絡後面啓動
Wants=network-online.target
Requires=flannel.service 

[Service] #增長EnvironmentFile=-/run/flannel/docker
Type=notify
EnvironmentFile=-/run/flannel/docker #加載環境文件，設置docker0的ip地址爲flannel分配的ip地址
ExecStart=/usr/bin/dockerd $DOCKER_OPTS
[root@linux-node1 ~]# systemctl daemon-reload
[root@linux-node1 ~]# systemctl restart docker
[root@linux-node1 ~]# ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 10.2.46.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 02:42:1f:ef:9f:b5  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@linux-node2 ~]# ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 10.2.87.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 02:42:8a:a5:42:d7  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@linux-node3 ~]# ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 10.2.33.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 02:42:57:90:05:47  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

總結

在kubectl get node時，會看到節點的狀態READY，若是狀態爲NotReady，能夠查看節點上的kubelet是否已經啓動，若是未啓動，進行啓動。kubelet沒法啓動，要進行查看systemctl status kubelet或journalctl -xe看看是什麼緣由致使沒法啓動。遇到的一種狀況是依賴docker，查看docker沒法啓動。再進一步排查docker沒法啓動的緣由。vim

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程