kubernetes（四）二進制安裝-flannel安裝

時間 2020-03-30

標籤 kubernetes 二進制安裝 flannel 简体版

原文原文鏈接

部署 flannel 網絡(在master節點上執行）

kubernetes組件kubelet服務依賴docker服務，docker網絡須要用flannel來配置docker0網橋的ip地址，因此須要先安裝flannel網絡組建node

flannel 使用 vxlan 技術爲各節點建立一個能夠互通的 Pod 網絡，使用的端口爲 UDP 8472（須要開放該端口，如公有云 AWS 等）。linux

flanneld 第一次啓動時，從 etcd 獲取配置的 Pod 網段信息，爲本節點分配一個未使用的地址段，而後建立 flannedl.1 網絡接口（也多是其它名稱，如 flannel1 等）。git

flannel 將分配給本身的 Pod 網段信息寫入 /run/flannel/docker 文件，docker 後續使用這個文件中的環境變量設置 docker0 網橋，從而從這個地址段爲本節點的全部 Pod 容器分配 IPgithub

下載和安裝flanneld 二進制文件docker

cd /opt/k8s/work
mkdir flannel
wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel

cp flannel/{flanneld,mk-docker-opts.sh} /opt/k8s/bin/

export node_ip=192.168.0.114
scp flannel/{flanneld,mk-docker-opts.sh} root@${192.168.0.114}:/opt/k8s/bin/

建立 flanneld 證書和私鑰json

flanneld 從 etcd 集羣存取網段分配信息，而 etcd 集羣啓用了雙向 x509 證書認證，因此須要爲 flanneld 生成證書和私鑰。網絡

建立證書籤名請求ssh

cd /opt/k8s/work
cat > flanneld-csr.json <<EOF
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "NanJing",
      "L": "NanJing",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF

生成證書和私鑰oop

cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
ls flanneld*pem

將生成的證書和私鑰分發到全部節點ui

cd /opt/k8s/work
mkdir -p /etc/flanneld/cert
cp flanneld*.pem /etc/flanneld/cert

export node_ip=192.168.0.114
ssh root@${node_ip} "mkdir -p /etc/flanneld/cert"
scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert

向 etcd 寫入集羣 Pod 網段信息

cd /opt/k8s/work

export FLANNEL_ETCD_PREFIX="/kubernetes/network"
export ETCD_ENDPOINTS="https://192.168.0.107:2379"

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/opt/k8s/work/ca.pem \
  --cert-file=/opt/k8s/work/flanneld.pem \
  --key-file=/opt/k8s/work/flanneld-key.pem \
  mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'

寫入的 Pod 網段 Network 網絡段對應的數值（如 /16）必須小於 SubnetLen對應的值（如24）

建立 flanneld 服務的啓動文件
```
cd /opt/k8s/work
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
export ETCD_ENDPOINTS="https://192.168.0.107:2379"

cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
  -etcd-endpoints=${ETCD_ENDPOINTS} \\
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
  -ip-masq
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
```
- mk-docker-opts.sh 腳本將分配給 flanneld 的 Pod 子網段信息，經過-d參數寫入 /run/flannel/docker 文件，後續 docker 啓動時使用這個文件中的環境變量配置 docker0 網橋， -k 參數控制生成文件中變量的名稱，下面docker啓動時會用到這個變量；
- flanneld 使用系統缺省路由所在的接口與其它節點通訊，對於有多個網絡接口（如內網和公網）的節點，能夠用 -iface 參數指定通訊接口;
- -ip-masq: flanneld 爲訪問 Pod 網絡外的流量設置 SNAT 規則，同時將傳遞給 Docker 的變量 --ip-masq（/run/flannel/docker 文件中）設置爲 false，這樣 Docker 將再也不建立 SNAT 規則； Docker 的 --ip-masq 爲 true 時，建立的 SNAT 規則比較「暴力」：將全部本節點 Pod 發起的、訪問非 docker0 接口的請求作 SNAT，這樣訪問其餘節點 Pod 的請求來源 IP 會被設置爲 flannel.1 接口的 IP，致使目的 Pod 看不到真實的來源 Pod IP。 flanneld 建立的 SNAT 規則比較溫和，只對訪問非 Pod 網段的請求作 SNAT

分發flanneld服務

cd /opt/k8s/work

cp flanneld.service /etc/systemd/system/

export node_ip=192.168.0.114
scp flanneld.service root@${node_ip}:/etc/systemd/system/

啓動flanneld服務

systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld

ssh root@${node_ip) "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"

檢查啓動結果

systemctl status flanneld|grep Active

export node_ip=192.168.0.114
ssh root@${node_ip} "systemctl status flanneld|grep Active"

確保狀態爲 active (running)，不然查看日誌，確認緣由
若是出現異常，經過以下命令查看
```
journalctl -u flanneld
```

檢查分配給各 flanneld 的 Pod 網段信息

export FLANNEL_ETCD_PREFIX="/kubernetes/network"
export ETCD_ENDPOINTS="https://192.168.0.107:2379"


etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/config

輸出結果

{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}

查看已分配的 Pod 子網段列表

export FLANNEL_ETCD_PREFIX="/kubernetes/network"
export ETCD_ENDPOINTS="https://192.168.0.107:2379"

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  ls ${FLANNEL_ETCD_PREFIX}/subnets

輸出結果

/kubernetes/network/subnets/172.30.22.0-24
/kubernetes/network/subnets/172.30.78.0-24

檢查節點 flannel 網絡信息

root@master:/opt/k8s/work# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 04:92:26:13:92:2b brd ff:ff:ff:ff:ff:ff
3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d0:c5:d3:57:73:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.107/24 brd 192.168.0.255 scope global dynamic noprefixroute wlp3s0
       valid_lft 6385sec preferred_lft 6385sec
    inet6 fe80::1fda:e90a:207a:67e4/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 12:cb:66:43:de:36 brd ff:ff:ff:ff:ff:ff
    inet 172.30.22.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::10cb:66ff:fe43:de36/64 scope link
       valid_lft forever preferred_lft forever
  
root@master:/opt/k8s/work# ip route show |grep flannel.1
172.30.78.0/24 via 172.30.78.0 dev flannel.1 onlink

驗證各節點能經過 Pod 網段互通

root@master:/opt/k8s/work# ip addr show flannel.1 |grep -w inet
    inet 172.30.22.0/32 scope global flannel.1
root@master:/opt/k8s/work# ssh 192.168.0.114 "/sbin/ip addr show flannel.1|grep -w inet"
    inet 172.30.78.0/32 scope global flannel.1
root@master:/opt/k8s/work# ping -c 1 172.30.78.0
PING 172.30.78.0 (172.30.78.0) 56(84) bytes of data.
64 bytes from 172.30.78.0: icmp_seq=1 ttl=64 time=80.7 ms

--- 172.30.78.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 80.707/80.707/80.707/0.000 ms
root@master:/opt/k8s/work# ssh 192.168.0.114 "ping -c 1 172.30.22.0"
PING 172.30.22.0 (172.30.22.0) 56(84) bytes of data.
64 bytes from 172.30.22.0: icmp_seq=1 ttl=64 time=4.09 ms

--- 172.30.22.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.094/4.094/4.094/0.000 ms

生成文件內容，注意DOCKER_NETWORK_OPTIONS的值

root@master:/opt/k8s/work# cat /run/flannel/subnet.env
FLANNEL_NETWORK=172.30.0.0/16
FLANNEL_SUBNET=172.30.22.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
root@master:/opt/k8s/work# cat /run/flannel/docker
DOCKER_OPT_BIP="--bip=172.30.22.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.30.22.1/24 --ip-masq=false --mtu=1450"

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。