k8s集羣部署v1.15實踐5:部署flannel網絡:
參考文檔node
部署flannel網絡
1.下載和分發flannellinux
下載地址git
下載github
[root@k8s-node1 flannel]# wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz --2019-11-03 22:24:27-- https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz Resolving github.com (github.com)... 140.82.114.3 Connecting to github.com (github.com)|140.82.114.3|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/21704134/596e76e2-002c-11e8-9359-36689058e7af?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191104T032428Z&X-Amz-Expires=300&X-Amz-Signature=a12144bfeb51e306a89eda1205fa7f17935e12130a3f12456926eb96792422ef&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dflannel-v0.10.0-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following] --2019-11-03 22:24:28-- https://github-production-release-asset-2e65be.s3.amazonaws.com/21704134/596e76e2-002c-11e8-9359-36689058e7af?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191104T032428Z&X-Amz-Expires=300&X-Amz-Signature=a12144bfeb51e306a89eda1205fa7f17935e12130a3f12456926eb96792422ef&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dflannel-v0.10.0-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.114.171 Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.114.171|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 9706487 (9.3M) [application/octet-stream] Saving to: ‘flannel-v0.10.0-linux-amd64.tar.gz’ 100%[==============================================================================================>] 9,706,487 570KB/s in 15s 2019-11-03 22:24:44 (652 KB/s) - ‘flannel-v0.10.0-linux-amd64.tar.gz’ saved [9706487/9706487] [root@k8s-node1 flannel]# pwd /opt/k8s/k8s_software/flannel [root@k8s-node1 flannel]# ls flannel-v0.10.0-linux-amd64.tar.gz [root@k8s-node1 flannel]#
解壓縮web
[root@k8s-node1 flannel]# tar -zxvf flannel-v0.10.0-linux-amd64.tar.gz flanneld mk-docker-opts.sh README.md [root@k8s-node1 flannel]# ls flanneld flannel-v0.10.0-linux-amd64.tar.gz mk-docker-opts.sh README.md [root@k8s-node1 flannel]#
分發docker
[root@k8s-node1 flannel]# cp flanneld mk-docker-opts.sh /opt/k8s/bin [root@k8s-node1 flannel]# scp flanneld mk-docker-opts.sh root@k8s-node2:/opt/k8s/bin flanneld 100% 35MB 105.7MB/s 00:00 mk-docker-opts.sh 100% 2139 2.5MB/s 00:00 [root@k8s-node1 flannel]# scp flanneld mk-docker-opts.sh root@k8s-node3:/opt/k8s/bin flanneld 100% 35MB 82.7MB/s 00:00 mk-docker-opts.sh 100% 2139 2.9MB/s 00:00 [root@k8s-node1 flannel]#
2.修改文件屬主和權限json
[root@k8s-node1 flannel]# ll /opt/k8s/bin total 130744 -rwxr-xr-x 1 root root 10376657 Oct 29 23:59 cfssl -rwxr-xr-x 1 root root 6595195 Oct 29 23:59 cfssl-certinfo -rwxr-xr-x 1 root root 2277873 Oct 29 23:59 cfssljson -rwxr-xr-x. 1 k8s root 1749 Oct 29 23:06 environment.sh -rwxr-xr-x 1 root root 19266976 Nov 3 21:45 etcd -rwxr-xr-x 1 root root 16018720 Nov 3 21:45 etcdctl -rwxr-xr-x 1 root root 36327752 Nov 3 22:27 flanneld -rwxr-xr-x 1 root root 42993696 Oct 30 00:41 kubectl -rwxr-xr-x 1 root root 2139 Nov 3 22:27 mk-docker-opts.sh [root@k8s-node1 flannel]# chown -R k8s /opt/k8s/bin [root@k8s-node1 flannel]# ssh root@k8s-node2 "chown -R k8s /opt/k8s/bin" [root@k8s-node1 flannel]# ssh root@k8s-node3 "chown -R k8s /opt/k8s/bin" [root@k8s-node1 flannel]# ll /opt/k8s/bin total 130744 -rwxr-xr-x 1 k8s root 10376657 Oct 29 23:59 cfssl -rwxr-xr-x 1 k8s root 6595195 Oct 29 23:59 cfssl-certinfo -rwxr-xr-x 1 k8s root 2277873 Oct 29 23:59 cfssljson -rwxr-xr-x. 1 k8s root 1749 Oct 29 23:06 environment.sh -rwxr-xr-x 1 k8s root 19266976 Nov 3 21:45 etcd -rwxr-xr-x 1 k8s root 16018720 Nov 3 21:45 etcdctl -rwxr-xr-x 1 k8s root 36327752 Nov 3 22:27 flanneld -rwxr-xr-x 1 k8s root 42993696 Oct 30 00:41 kubectl -rwxr-xr-x 1 k8s root 2139 Nov 3 22:27 mk-docker-opts.sh [root@k8s-node1 flannel]#
3.建立flannel的證書和密鑰網絡
flannel 從 etcd 集羣存取網段分配信息,而 etcd 集羣啓用了雙向 x509 證書認證,因此須要爲 flanneld 生成證書和私鑰.app
建立證書籤名請求:ssh
[root@k8s-node1 flannel]# cat flanneld-csr.json
{
"CN": "flanneld",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "SZ",
"O": "k8s",
"OU": "4Paradigm"
}
]
}
[root@k8s-node1 flannel]#
該證書只會被 kubectl 當作 client 證書使用,因此 hosts 字段爲空.
建立證書
[root@k8s-node1 flannel]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
2019/11/04 01:20:45 [INFO] generate received request
2019/11/04 01:20:45 [INFO] received CSR
2019/11/04 01:20:45 [INFO] generating key: rsa-2048
2019/11/04 01:20:45 [INFO] encoded CSR
2019/11/04 01:20:45 [INFO] signed certificate with serial number 53627868829031143026717597514039929210087848401
2019/11/04 01:20:45 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s-node1 flannel]# ls
flanneld flanneld-csr.json flanneld.pem mk-docker-opts.sh
flanneld.csr flanneld-key.pem flannel-v0.10.0-linux-amd64.tar.gz README.md
[root@k8s-node1 flannel]#
4.分發證書
建立證書存放目錄並修改屬主
[root@k8s-node1 flannel]# mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld/cert [root@k8s-node1 flannel]# ssh root@k8s-node2 "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld/cert" [root@k8s-node1 flannel]# ssh root@k8s-node3 "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld/cert"
分發證書
[root@k8s-node1 flannel]# cp flanneld.pem flanneld-key.pem /etc/flanneld/cert/ [root@k8s-node1 flannel]# scp flanneld.pem flanneld-key.pem root@k8s-node2:/etc/flanneld/cert/ flanneld.pem 100% 1371 376.0KB/s 00:00 flanneld-key.pem 100% 1679 1.2MB/s 00:00 [root@k8s-node1 flannel]# scp flanneld.pem flanneld-key.pem root@k8s-node3:/etc/flanneld/cert/ flanneld.pem 100% 1371 1.4MB/s 00:00 flanneld-key.pem 100% 1679 1.4MB/s 00:00 [root@k8s-node1 flannel]#
修改文件屬主
[root@k8s-node1 flannel]# chown -R k8s /etc/flanneld/cert/ [root@k8s-node1 flannel]# ssh root@k8s-node2 "chown -R k8s /etc/flanneld/cert/" [root@k8s-node1 flannel]# ssh root@k8s-node3 "chown -R k8s /etc/flanneld/cert/"
5.把pod網絡信息寫入etcd集羣
先檢查下前面定義的集羣變量,source再檢索
注意:flanneld 當前版本 (v0.10.0) 不支持 etcd v3,故使用 etcd v2 API 寫入配置 key 和網段數據；
寫入的 Pod 網段 ${CLUSTER_CIDR} 必須是 /16 段地址,必須與 kube-controllermanager的 --cluster-cidr 參數值一致.
[root@k8s-node1 flannel]# source /opt/k8s/bin/environment.sh
[root@k8s-node1 flannel]# echo ${ETCD_ENDPOINTS}
https://192.168.174.128:2379,https://192.168.174.129:2379,https://192.168.174.130:2379
[root@k8s-node1 flannel]# echo ${FLANNEL_ETCD_PREFIX}
/kubernetes/network
[root@k8s-node1 flannel]# echo ${CLUSTER_CIDR}
172.30.0.0/16
[root@k8s-node1 flannel]# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'",
> "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
{"Network":"172.30.0.0/16",
"SubnetLen": 24, "Backend": {"Type": "vxlan"}}
[root@k8s-node1 flannel]#
6.建立flanneld的systemd unit文件
mk-docker-opts.sh 腳本將分配給 flanneld 的 Pod 子網網段信息寫入/run/flannel/docker 文件,後續 docker 啓動時使用這個文件中的環境變量配置 docker0 網橋.
flanneld 使用系統缺省路由所在的接口與其它節點通訊,對於有多個網絡接口(如內網和公網)的節點,能夠用 -iface 參數指定通訊接口,以下面的ens33接口;
flanneld 運行時須要 root 權限.
[root@k8s-node1 flannel]# cat flanneld.service [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify ExecStart=/opt/k8s/bin/flanneld \ -etcd-cafile=/etc/kubernetes/cert/ca.pem \ -etcd-certfile=/etc/flanneld/cert/flanneld.pem \ -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \ -etcd-endpoints=https://192.168.174.128:2379,https://192.168.174.129:2379,https://192.168.174.130:2379 \ -etcd-prefix=/kubernetes/network \ -iface=ens33 ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service [root@k8s-node1 flannel]#
分發文件到節點
root@k8s-node1 flannel]# cp flanneld.service /etc/systemd/system/ [root@k8s-node1 flannel]# scp flanneld.service root@k8s-node2:/etc/systemd/system/ flanneld.service 100% 702 588.5KB/s 00:00 [root@k8s-node1 flannel]# scp flanneld.service root@k8s-node3:/etc/systemd/system/ flanneld.service 100% 702 922.7KB/s 00:00 [root@k8s-node1 flannel]#
修改權限
[root@k8s-node1 flannel]# ll /etc/systemd/system total 12 drwxr-xr-x. 2 root root 57 Oct 28 05:47 basic.target.wants lrwxrwxrwx. 1 root root 57 Oct 28 05:47 dbus-org.freedesktop.nm-dispatcher.service -> /usr/lib/systemd/system/NetworkManager-dispatcher.service lrwxrwxrwx. 1 root root 37 Oct 28 05:51 default.target -> /lib/systemd/system/multi-user.target drwxr-xr-x. 2 root root 87 Oct 28 05:47 default.target.wants -rw-r--r-- 1 root root 1137 Nov 3 21:32 etcd.service -rw-r--r-- 1 root root 702 Nov 4 01:37 flanneld.service drwxr-xr-x. 2 root root 32 Oct 28 05:47 getty.target.wants drwxr-xr-x. 2 root root 35 Oct 28 05:47 local-fs.target.wants drwxr-xr-x. 2 root root 4096 Oct 30 04:18 multi-user.target.wants drwxr-xr-x. 2 root root 48 Oct 28 05:47 network-online.target.wants drwxr-xr-x. 2 root root 29 Oct 28 05:47 sockets.target.wants drwxr-xr-x. 2 root root 254 Oct 28 05:47 sysinit.target.wants drwxr-xr-x. 2 root root 44 Oct 28 05:47 system-update.target.wants drwxr-xr-x. 2 root root 29 Oct 28 05:47 vmtoolsd.service.requires
[root@k8s-node1 flannel]# chmod -R +x /etc/systemd/system [root@k8s-node1 flannel]# ssh root@k8s-node2 "chmod -R +x /etc/systemd/system" [root@k8s-node1 flannel]# ssh root@k8s-node3 "chmod -R +x /etc/systemd/system"
7.全部節點啓動flanneld服務
[root@k8s-node1 flannel]# systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /etc/systemd/system/flanneld.service. Created symlink from /etc/systemd/system/docker.service.requires/flanneld.service to /etc/systemd/system/flanneld.service.
[root@k8s-node1 flannel]# systemctl status flanneld
● flanneld.service - Flanneld overlay address etcd agent
Loaded: loaded (/etc/systemd/system/flanneld.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-11-04 01:40:41 EST; 56s ago
Process: 2151 ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
Main PID: 2139 (flanneld)
Tasks: 7
Memory: 8.6M
CGroup: /system.slice/flanneld.service
└─2139 /opt/k8s/bin/flanneld -etcd-cafile=/etc/kubernetes/cert/ca.pem -etcd-certfile=/etc/flanneld/cert/flanneld.pem -etcd...
Nov 04 01:40:41 k8s-node1 flanneld[2139]: I1104 01:40:41.135624 2139 main.go:300] Wrote subnet file to /run/flannel/subnet.env
Nov 04 01:40:41 k8s-node1 flanneld[2139]: I1104 01:40:41.135689 2139 main.go:304] Running backend.
Nov 04 01:40:41 k8s-node1 flanneld[2139]: I1104 01:40:41.143691 2139 iptables.go:115] Some iptables rules are missing; dele...g rules
Nov 04 01:40:41 k8s-node1 flanneld[2139]: I1104 01:40:41.143708 2139 iptables.go:137] Deleting iptables rule: -s 172.30.0.0... ACCEPT
Nov 04 01:40:41 k8s-node1 flanneld[2139]: I1104 01:40:41.145236 2139 iptables.go:137] Deleting iptables rule: -d 172.30.0.0... ACCEPT
Nov 04 01:40:41 k8s-node1 flanneld[2139]: I1104 01:40:41.149018 2139 iptables.go:125] Adding iptables rule: -s 172.30.0.0/1... ACCEPT
Nov 04 01:40:41 k8s-node1 flanneld[2139]: I1104 01:40:41.150804 2139 vxlan_network.go:60] watching for new subnet leases
Nov 04 01:40:41 k8s-node1 flanneld[2139]: I1104 01:40:41.153796 2139 main.go:396] Waiting for 22h59m59.979034011s to renew lease
Nov 04 01:40:41 k8s-node1 flanneld[2139]: I1104 01:40:41.153827 2139 iptables.go:125] Adding iptables rule: -d 172.30.0.0/1... ACCEPT
Nov 04 01:40:41 k8s-node1 systemd[1]: Started Flanneld overlay address etcd agent.
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-node1 flannel]#
檢查etcd裏分配給各個節點的pod網絡信息
[root@k8s-node1 flannel]# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem ls kubernetes/network
/kubernetes/network/config
/kubernetes/network/subnets
[root@k8s-node1 flannel]# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem ls kubernetes/network/subnets
/kubernetes/network/subnets/172.30.78.0-24
/kubernetes/network/subnets/172.30.22.0-24
/kubernetes/network/subnets/172.30.86.0-24
[root@k8s-node1 flannel]# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem get kubernetes/network/subnets/172.30.78.0-24
{"PublicIP":"192.168.174.128","BackendType":"vxlan","BackendData":{"VtepMAC":"02:c5:0a:aa:0b:fc"}}
flannel接口信息
[root@k8s-node1 flannel]# ip a | grep -A 10 flannel
6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 02:c5:0a:aa:0b:fc brd ff:ff:ff:ff:ff:ff
inet 172.30.78.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::c5:aff:feaa:bfc/64 scope link
valid_lft forever preferred_lft forever
[root@k8s-node1 flannel]#
[root@k8s-node2 ~]# ip a | grep -A 10 flannel
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 8e:d8:c2:66:df:a0 brd ff:ff:ff:ff:ff:ff
inet 172.30.22.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::8cd8:c2ff:fe66:dfa0/64 scope link
valid_lft forever preferred_lft forever
[root@k8s-node2 ~]#
[root@k8s-node3 ~]# ip a | grep -A 10 flannel
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether e2:ac:1e:ff:1d:ae brd ff:ff:ff:ff:ff:ff
inet 172.30.86.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::e0ac:1eff:feff:1dae/64 scope link
valid_lft forever preferred_lft forever
[root@k8s-node3 ~]#
互相ping測試下
[root@k8s-node1 flannel]# ping 172.30.22.0 PING 172.30.22.0 (172.30.22.0) 56(84) bytes of data. 64 bytes from 172.30.22.0: icmp_seq=1 ttl=64 time=0.894 ms 64 bytes from 172.30.22.0: icmp_seq=2 ttl=64 time=1.26 ms ^C --- 172.30.22.0 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.894/1.080/1.266/0.186 ms [root@k8s-node1 flannel]# ping 172.30.86.0 PING 172.30.86.0 (172.30.86.0) 56(84) bytes of data. 64 bytes from 172.30.86.0: icmp_seq=1 ttl=64 time=0.848 ms 64 bytes from 172.30.86.0: icmp_seq=2 ttl=64 time=0.489 ms 64 bytes from 172.30.86.0: icmp_seq=3 ttl=64 time=0.369 ms 64 bytes from 172.30.86.0: icmp_seq=4 ttl=64 time=1.53 ms 64 bytes from 172.30.86.0: icmp_seq=5 ttl=64 time=1.22 ms 64 bytes from 172.30.86.0: icmp_seq=6 ttl=64 time=2.68 ms ^C --- 172.30.86.0 ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 5005ms rtt min/avg/max/mdev = 0.369/1.191/2.683/0.777 ms [root@k8s-node1 flannel]#
8.部署完成
相關文章
- 1. k8s集羣部署v1.15實踐4:部署etcd集羣
- 2. k8s集羣部署v1.15實踐7:部署kube-apiserver組件
- 3. k8s集羣部署四（部署Flannel網絡）
- 4. k8s集羣部署v1.15實踐8:部署高可用 kube-controller-manager 集羣
- 5. k8s集羣部署v1.15實踐9:部署高可用 kube-scheduler 集羣
- 6. k8s部署flannel網絡
- 7. Kubernetes V1.15 二進制部署k8s集羣
- 8. k8s集羣部署v1.15實踐3: 部署kubectl命令工具行
- 9. k8s部署Flannel網絡篇（k8s篇二）
- 10. k8s實踐19:kubernetes二進制部署集羣v1.12升級v1.15
- 更多相關文章...
- • Maven 自動化部署 - Maven教程
- • ionic 頭部與底部 - ionic 教程
- • 使用阿里雲OSS+CDN部署前端頁面與加速靜態資源
- • TiDB 在摩拜單車在線數據業務的應用和實踐
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. k8s集羣部署v1.15實踐4:部署etcd集羣
- 2. k8s集羣部署v1.15實踐7:部署kube-apiserver組件
- 3. k8s集羣部署四（部署Flannel網絡）
- 4. k8s集羣部署v1.15實踐8:部署高可用 kube-controller-manager 集羣
- 5. k8s集羣部署v1.15實踐9:部署高可用 kube-scheduler 集羣
- 6. k8s部署flannel網絡
- 7. Kubernetes V1.15 二進制部署k8s集羣
- 8. k8s集羣部署v1.15實踐3: 部署kubectl命令工具行
- 9. k8s部署Flannel網絡篇（k8s篇二）
- 10. k8s實踐19:kubernetes二進制部署集羣v1.12升級v1.15