二進制部署K8s集羣第19節addons之flannel三種模型安裝部署詳解

時間 2020-10-07

標籤 linux nginx git github docker shell 服務器網絡架構 ide 欄目負載均衡简体版

原文原文鏈接

架構圖

kubernetes設計了網絡模型，但卻將它的實現交給了網絡插件，CNI網絡插件最主要的功能主是實現pod資源可以跨宿主機進行通訊linux

常見的CNI網絡插件：nginx

Flannel
Calico
Canal
Contiv
OpenContrail
NSX-T
Kube-router

Flannel三種模型：git

host-gw
VxLan
Directrouting

1.1 host-gw模型

host-gw模型即網關模式，在服務器直接添加一條靜態路由便可，效率高，各節點必須在同一網段github

10.4.7.21經過172.7.22.0這條靜態路由鏈接10.4.7.22這臺主機再鏈接172.7.21網段，反之亦是docker

/opt/etcd/etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}' 
/opt/etcd/etcdctl set /coreos.com/network/config # 查看
/opt/etcd/etcdctl rm /coreos.com/network/config # 刪除

1.2 VxLAN模型

在不一樣網段，能夠用VxLAN模式，主機A會生成一個flannel.1網卡，經過封裝頭從flannel.1網卡出去經過flanne隧道傳出，從flannel.1網卡傳入拆包，到達目標主機網卡到指向的靜態路由，效率低shell

/opt/etcd/etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "VxLAN"}}'

1.3 Directrouting模型

直接路由模式，結合了VxLAN和host-gw模型服務器
自動識別服務器，若是同網段，則使用host-gw模型，若是不一樣網段則使用VxLAN模型

/opt/etcd/etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "VxLAN","Directrouting": true}}'

1.4 Flannel部署

1.4.1 集羣規劃

主機名	IP	角色	節點
hdss21-host.com	10.4.7.21	Flannel	Node
hdss22-host.com	10.4.4.22	Flannel	Node

1.4.2 下載解壓

在hdss21-host.com操做，hdss22-host.com操做相似網絡

下載連接：https://github.com/coreos/flannel/releases架構

cd /opt/src
rz ==> flannel-v0.12.0-linux-amd64.tar.gz
mkdir /opt/flanneld-v0.12.0
tar xf flannel-v0.12.0-linux-amd64.tar.gz -C /opt/flanneld-v0.12.0
ln -s /opt/flanneld-v0.12.0 /opt/flannel

1.4.3 建立配置

cat > /opt/flannel/subnet.env <<'eof'
FLANNEL_NETWORK=172.7.0.0/16
FLANNEL_SUBNET=172.7.21.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false
eof

不一樣地方ide

172.4.7.21 配置 FLANNEL_SUBNET=172.7.21.1/24

172.4.7.22 配置 FLANNEL_SUBNET=172.7.22.1/24

1.4.4 拷貝證書

mkdir /opt/flannel/certs
scp hdss7-200:/opt/certs/ca.pem /opt/flannel/certs/
scp hdss7-200:/opt/certs/client.pem /opt/flannel/certs/
scp hdss7-200:/opt/certs/client-key.pem /opt/flannel/certs/

1.4.5 建立啓動腳本

cat > /opt/flannel/flanneld.sh <<'eof'
#!/bin/sh
./flanneld \
  --public-ip=10.4.7.21 \
  --etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
  --etcd-keyfile=./certs/client-key.pem \
  --etcd-certfile=./certs/client.pem \
  --etcd-cafile=./certs/ca.pem \
  --iface=ens33 \
  --subnet-file=./subnet.env \
  --healthz-port=2401
eof
chmod +x /opt/flannel/flanneld.sh 
mkdir -p /data/logs/flanneld

不一樣地方：

172.4.7.21 配置 -public-ip=10.4.7.21

172.4.7.22 配置 -public-ip=10.4.7.22

--iface=ens33 配置爲服務器網卡名

1.4.6 建立supervisor配置

cat > /etc/supervisord.d/flanneld.ini << 'eof'
[program:flanneld-7-21]
command=/opt/flannel/flanneld.sh                             ; the program (relative uses PATH, can take args)
numprocs=1                                                   ; number of processes copies to start (def 1)
directory=/opt/flannel                                       ; directory to cwd to before exec (def no cwd)
autostart=true                                               ; start at supervisord start (default: true)
autorestart=true                                             ; retstart at unexpected quit (default: true)
startsecs=22                                                 ; number of secs prog must stay running (def. 1)
startretries=3                                               ; max # of serial start failures (default 3)
exitcodes=0,2                                                ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                              ; signal used to kill process (default TERM)
stopwaitsecs=10                                              ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                    ; setuid to this UNIX account to run the program
redirect_stderr=false                                        ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/flanneld/flanneld.stdout.log       ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                 ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                     ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                  ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                  ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/flanneld/flanneld.stderr.log       ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                 ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                     ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                  ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false  
eof

不一樣地方：[program:flanneld-7-21]

1.5 使用host-gw模型

hdss21-host.com操做，任意一臺etcd機器操做均可以

/opt/etcd/etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
supervisorctl update
supervisorctl status
[root@hdss7-22 ~]# route -n # 在22主機上可看到添加了一條10.4.7.21的靜態路由
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.4.7.254      0.0.0.0         UG    100    0        0 ens33
10.4.7.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.7.21.0      10.4.7.21       255.255.255.0   UG    0      0        0 ens33
172.7.22.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
[root@hdss7-21 ~]# route -n # 在21主機上可看到添加了一條10.4.7.22的靜態路由
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.4.7.254      0.0.0.0         UG    100    0        0 ens33
10.4.7.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.7.21.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.7.22.0      10.4.7.22       255.255.255.0   UG    0      0        0 ens33
[root@hdss7-21 ~]# kubectl get pods -o wide
NAME             READY   STATUS    RESTARTS   AGE     IP           NODE                NOMINATED NODE   READINESS GATES
my-nginx-h88nr   1/1     Running   1          3h46m   172.7.22.2   hdss7-22.host.com   <none>           <none>
my-nginx-vrlsk   1/1     Running   0          3h46m   172.7.21.2   hdss7-21.host.com   <none>           <none>
[root@hdss7-21 ~]# ping 172.7.22.2
[root@hdss7-22 ~]# ping 172.7.21.2

因爲flanned網絡的gw模型是直接幫你添加一條靜態路由，若不用flanned插件，可自行添加路由打通網絡

10.4.7.21上操做

route add -net 172.7.22.0/24 gw 10.4.7.22

iptables -I FORWARD -d 172.7.21.0/24 -j ACCEPT

10.4.7.22上操做

route add -net 172.7.21.0/24 gw 10.4.7.21

iptables -I FORWARD -d 172.7.22.0/24 -j ACCEPT

1.6 使用VxLAN模型

關閉兩臺機flannel
刪除兩臺機先前添加的靜態路由
刪除etc先前的設置從新添加新模型保存
啓動flannel查看網卡

[root@hdss7-21 ~]# supervisorctl stop flanneld 
[root@hdss7-22 ~]# supervisorctl stop flanneld 
[root@hdss7-22 ~]# route del -net 172.7.21.0/24 gw 10.4.7.21
[root@hdss7-21 ~]# route del -net 172.7.22.0/24 gw 10.4.7.22
[root@hdss7-21 ~]# /opt/etcd/etcdctl rm /coreos.com/network/config
[root@hdss7-21 ~]# /opt/etcd/etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "VxLAN"}}'
[root@hdss7-21 ~]# /opt/etcd/etcdctl get /coreos.com/network/config
[root@hdss7-21 ~]# supervisorctl start flanneld
[root@hdss7-22 ~]# supervisorctl start flanneld
[root@hdss7-21 ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.7.21.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::803:ff:fe6d:1f33  prefixlen 64  scopeid 0x20<link>
        ether 0a:03:00:6d:1f:33  txqueuelen 0  (Ethernet)
        RX packets 10  bytes 2234 (2.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 892 (892.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
[root@hdss7-22 etcd]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.7.22.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::d0c2:4bff:fe29:df00  prefixlen 64  scopeid 0x20<link>
        ether d2:c2:4b:29:df:00  txqueuelen 0  (Ethernet)
        RX packets 14  bytes 892 (892.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10  bytes 2234 (2.1 KiB)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
[root@hdss7-22 etcd]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.4.7.254      0.0.0.0         UG    100    0        0 ens33
10.4.7.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.7.21.0      172.7.21.0      255.255.255.0   UG    0      0        0 flannel.1
172.7.22.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
[root@hdss7-21 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.4.7.254      0.0.0.0         UG    100    0        0 ens33
10.4.7.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.7.21.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.7.22.0      172.7.22.0      255.255.255.0   UG    0      0        0 flannel.1
[root@hdss7-21 ~]# ping 172.7.22.2
[root@hdss7-22 ~]# ping 172.7.21.2

1.7 使用Directrouting模型

hdss7-21主機操做爲例，7-22操做相似

supervisorctl stop flanneld
[root@hdss7-21 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.4.7.254      0.0.0.0         UG    100    0        0 ens33
10.4.7.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.7.21.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.7.22.0      172.7.22.0      255.255.255.0   UG    0      0        0 flannel.1
route del -net 172.7.22.0/24 gw 172.7.22.0 # 刪除舊flannel路由
/opt/etcd/etcdctl rm /coreos.com/network/config # 刪除舊數據
/opt/etcd/etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "VxLAN","Directrouting": true}}' #設置新數據
/opt/etcd/etcdctl get /coreos.com/network/config
supervisorctl start flanneld
[root@hdss7-21 ~]# route -n # 因爲服務器同網段直接路由模型使用host-gw模型
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.4.7.254      0.0.0.0         UG    100    0        0 ens33
10.4.7.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.7.21.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.7.22.0      10.4.7.22       255.255.255.0   UG    0      0        0 ens33
[root@hdss7-21 ~]# ping 172.7.22.2