kubeadm部署kubernetes v1.14.1高可用集羣
第1章 高可用簡介
kubernetes高可用部署參考：
https://kubernetes.io/docs/setup/independent/high-availability/
https://github.com/kubernetes-sigs/kubespray
https://github.com/wise2c-devops/breeze
https://github.com/cookeem/kubeadm-hahtml
1.1 拓撲選擇
配置高可用（HA）Kubernetes集羣，有如下兩種可選的etcd拓撲：node
- 集羣master節點與etcd節點共存，etcd也運行在控制平面節點上
- 使用外部etcd節點，etcd節點與master在不一樣節點上運行
1.1.1 堆疊的etcd拓撲
堆疊HA集羣是這樣的拓撲，其中etcd提供的分佈式數據存儲集羣與由kubeamd管理的運行master組件的集羣節點堆疊部署。
每一個master節點運行kube-apiserver，kube-scheduler和kube-controller-manager的一個實例。kube-apiserver使用負載平衡器暴露給工做節點。
每一個master節點建立一個本地etcd成員，該etcd成員僅與本節點kube-apiserver通訊。這一樣適用於本地kube-controller-manager 和kube-scheduler實例。
該拓撲將master和etcd成員耦合在相同節點上。比設置具備外部etcd節點的集羣更簡單，而且更易於管理複製。
可是，堆疊集羣存在耦合失敗的風險。若是一個節點發生故障，則etcd成員和master實例都將丟失，而且冗餘會受到影響。您能夠經過添加更多master節點來下降此風險。
所以，您應該爲HA羣集運行至少三個堆疊的master節點。
這是kubeadm中的默認拓撲。使用kubeadm init和kubeadm join --experimental-control-plane命令時，在master節點上自動建立本地etcd成員。linux
1.1.2 外部etcd拓撲
具備外部etcd的HA集羣是這樣的拓撲，其中由etcd提供的分佈式數據存儲集羣部署在運行master組件的節點造成的集羣外部。
像堆疊ETCD拓撲結構，在外部ETCD拓撲中的每一個master節點運行一個kube-apiserver，kube-scheduler和kube-controller-manager實例。而且kube-apiserver使用負載平衡器暴露給工做節點。可是，etcd成員在不一樣的主機上運行，每一個etcd主機與kube-apiserver每一個master節點進行通訊。
此拓撲將master節點和etcd成員分離。所以，它提供了HA設置，其中丟失master實例或etcd成員具備較小的影響而且不像堆疊的HA拓撲那樣影響集羣冗餘。
可是，此拓撲須要兩倍於堆疊HA拓撲的主機數。具備此拓撲的HA羣集至少須要三個用於master節點的主機和三個用於etcd節點的主機。git
1.2 部署要求
使用kubeadm部署高可用性Kubernetes集羣的兩種不一樣方法：github
- 使用堆疊master節點。這種方法須要較少的基礎設施，etcd成員和master節點位於同一位置。
- 使用外部etcd集羣。這種方法須要更多的基礎設施， master節點和etcd成員是分開的。
在繼續以前，您應該仔細考慮哪一種方法最能知足您的應用程序和環境的需求。docker
部署要求shell
- 至少3個master節點
- 至少3個worker節點
- 全部節點網絡所有互通（公共或私有網絡）
- 全部機器都有sudo權限
- 從一個設備到系統中全部節點的SSH訪問
- 全部節點安裝kubeadm和kubelet，kubectl是可選的。
- 針對外部etcd集羣，你須要爲etcd成員額外提供3個節點
1.3 負載均衡
部署集羣前首選須要爲kube-apiserver建立負載均衡器。
注意：負載平衡器有許多中配置方式。能夠根據你的集羣要求選擇不一樣的配置方案。在雲環境中，您應將master節點做爲負載平衡器TCP轉發的後端。此負載平衡器將流量分配到其目標列表中的全部健康master節點。apiserver的運行情況檢查是對kube-apiserver偵聽的端口的TCP檢查（默認值:6443）。
負載均衡器必須可以與apiserver端口上的全部master節點通訊。它還必須容許其偵聽端口上的傳入流量。另外確保負載均衡器的地址始終與kubeadm的ControlPlaneEndpoint地址匹配。
haproxy/nignx+keepalived是其中可選的負載均衡方案，針對公有云環境能夠直接使用運營商提供的負載均衡產品。
部署時首先將第一個master節點添加到負載均衡器並使用如下命令測試鏈接：json
# nc -v LOAD_BALANCER_IP PORT
因爲apiserver還沒有運行，所以預計會出現鏈接拒絕錯誤。可是，超時意味着負載均衡器沒法與master節點通訊。若是發生超時，請從新配置負載平衡器以與master節點通訊。將剩餘的master節點添加到負載平衡器目標組。bootstrap
第2章 部署集羣
本次使用kubeadm部署kubernetes v1.14.1高可用集羣，包含3個master節點和1個node節點，部署步驟以官方文檔爲基礎，負載均衡部分採用haproxy+keepalived容器方式實現。全部組件版本以kubernetes v1.14.1爲準，其餘組件以當前最新版本爲準。vim
2.1 基本配置
節點信息：
| 主機名 | IP地址 | 角色 | OS | CPU/MEM | 磁盤 | 網卡 |
| k8s-master01 | 192.168.92.10 | master | CentOS7.6 | 2C2G | 60G | x1 |
| k8s-master02 | 192.168.92.11 | master | CentOS7.6 | 2C2G | 60G | x1 |
| k8s-master03 | 192.168.92.12 | master | CentOS7.6 | 2C2G | 60G | x1 |
| k8s-node01 | 192.168.92.13 | node | CentOS7.6 | 2C2G | 60G | x1 |
| K8S VIP | 192.168.92.30 | - | - | - | - | - |
如下操做在全部節點執行
#配置主機名 hostnamectl set-hostname k8s-master01 hostnamectl set-hostname k8s-master02 hostnamectl set-hostname k8s-master03 hostnamectl set-hostname k8s-node01 #修改/etc/hosts cat >> /etc/hosts << EOF 192.168.92.10 k8s-master01 192.168.92.11 k8s-master02 192.168.92.12 k8s-master03 192.168.92.13 k8s-node01 EOF # 開啓firewalld防火牆並容許全部流量 systemctl start firewalld && systemctl enable firewalld firewall-cmd --set-default-zone=trusted firewall-cmd --complete-reload # 關閉selinux sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0 #關閉swap swapoff -a yes | cp /etc/fstab /etc/fstab_bak cat /etc/fstab_bak | grep -v swap > /etc/fstab
配置時間同步
使用chrony同步時間，centos7默認已安裝，這裏修改時鐘源，全部節點與網絡時鐘源同步：
# 安裝chrony：
yum install -y chrony
cp /etc/chrony.conf{,.bak}
# 註釋默認ntp服務器
sed -i 's/^server/#&/' /etc/chrony.conf
# 指定上游公共 ntp 服務器
cat >> /etc/chrony.conf << EOF
server 0.asia.pool.ntp.org iburst
server 1.asia.pool.ntp.org iburst
server 2.asia.pool.ntp.org iburst
server 3.asia.pool.ntp.org iburst
EOF
# 設置時區
timedatectl set-timezone Asia/Shanghai
# 重啓chronyd服務並設爲開機啓動：
systemctl enable chronyd && systemctl restart chronyd
#驗證,查看當前時間以及存在帶*的行
timedatectl && chronyc sources
加載IPVS模塊
在全部的Kubernetes節點執行如下腳本（若內核大於4.19替換nf_conntrack_ipv4爲nf_conntrack）: cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF #執行腳本 chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4 #安裝相關管理工具 yum install ipset ipvsadm -y
配置內核參數
cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 vm.swappiness=0 EOF sysctl --system
2.2 安裝docker
CRI安裝參考：https://kubernetes.io/docs/setup/cri/
要在Pod中運行容器，Kubernetes使用容器運行時。如下是可選的容器運行時。
- Docker
- CRI-O
- Containerd
- Other CRI runtimes: frakti
2.2.1 Cgroup驅動程序簡介
當systemd被選爲Linux發行版的init系統時，init進程會生成並使用根控制組（cgroup）並充當cgroup管理器。Systemd與cgroup緊密集成，並將爲每一個進程分配cgroup。能夠配置容器運行時和要使用的kubelet cgroupfs。cgroupfs與systemd一塊兒使用意味着將有兩個不一樣的cgroup管理器。
Control groups用於約束分配給進程的資源。單個cgroup管理器將簡化正在分配的資源的視圖，而且默認狀況下將具備更可靠的可用和使用資源視圖。
當咱們有兩個managers時，咱們最終會獲得兩個這些資源的視圖。咱們已經看到了現場的狀況，其中配置cgroupfs用於kubelet和Docker systemd 的節點以及在節點上運行的其他進程在資源壓力下變得不穩定。
更改設置，使容器運行時和kubelet systemd用做cgroup驅動程序，從而使系統穩定。請注意native.cgroupdriver=systemd下面Docker設置中的選項。
2.2.2 安裝並配置docker
如下操做在全部節點執行。
# 安裝依賴軟件包
yum install -y yum-utils device-mapper-persistent-data lvm2
# 添加Docker repository，這裏改成國內阿里雲yum源
yum-config-manager \
--add-repo \
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安裝docker-ce
yum update -y && yum install -y docker-ce
## 建立 /etc/docker 目錄
mkdir /etc/docker
# 配置 daemon.
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"registry-mirrors": ["https://uyah70su.mirror.aliyuncs.com"]
}
EOF
#注意，因爲國內拉取鏡像較慢，配置文件最後追加了阿里雲鏡像加速配置。
mkdir -p /etc/systemd/system/docker.service.d
# 重啓docker服務
systemctl daemon-reload && systemctl restart docker && systemctl enable docker
2.3 安裝負載均衡
kubernetes master 節點運行以下組件：
- kube-apiserver
- kube-scheduler
- kube-controller-manager
kube-scheduler 和 kube-controller-manager 能夠以集羣模式運行，經過 leader 選舉產生一個工做進程，其它進程處於阻塞模式。
kube-apiserver能夠運行多個實例，但對其它組件須要提供統一的訪問地址，該地址須要高可用。本次部署使用 keepalived+haproxy 實現 kube-apiserver VIP 高可用和負載均衡。
haproxy+keepalived配置vip，實現了api惟一的訪問地址和負載均衡。keepalived 提供 kube-apiserver 對外服務的 VIP。haproxy 監聽 VIP，後端鏈接全部 kube-apiserver 實例，提供健康檢查和負載均衡功能。
運行 keepalived 和 haproxy 的節點稱爲 LB 節點。因爲 keepalived 是一主多備運行模式，故至少兩個 LB 節點。
本次部署複用 master 節點的三臺機器，在全部3個master節點部署haproxy和keepalived組件，以達到更高的可用性，haproxy 監聽的端口(6444) 須要與 kube-apiserver的端口 6443 不一樣，避免衝突。
keepalived 在運行過程當中週期檢查本機的 haproxy 進程狀態，若是檢測到 haproxy 進程異常，則觸發從新選主的過程，VIP 將飄移到新選出來的主節點，從而實現 VIP 的高可用。
全部組件（如 kubeclt、apiserver、controller-manager、scheduler 等）都經過 VIP +haproxy 監聽的6444端口訪問 kube-apiserver 服務。
負載均衡架構圖以下：
2.4 運行HA容器
使用的容器鏡像爲睿雲智合開源項目breeze相關鏡像，具體使用方法請訪問：
https://github.com/wise2c-devops
其餘選擇：haproxy鏡像也可使用dockerhub官方鏡像，但keepalived未提供官方鏡像，可自行構建或使用dockerhub他人已構建好的鏡像，本次部署所有使用breeze提供的鏡像。
在3個master節點以容器方式部署haproxy，容器暴露6444端口，負載均衡到後端3個apiserver的6443端口，3個節點haproxy配置文件相同。
如下操做在master01節點執行。
2.4.1 建立haproxy啓動腳本
編輯start-haproxy.sh文件，修改Kubernetes Master節點IP地址爲實際Kubernetes集羣所使用的值（Master Port默認爲6443不用修改）：
mkdir -p /data/lb cat > /data/lb/start-haproxy.sh << "EOF" #!/bin/bash MasterIP1=192.168.92.10 MasterIP2=192.168.92.11 MasterIP3=192.168.92.12 MasterPort=6443 docker run -d --restart=always --name HAProxy-K8S -p 6444:6444 \ -e MasterIP1=$MasterIP1 \ -e MasterIP2=$MasterIP2 \ -e MasterIP3=$MasterIP3 \ -e MasterPort=$MasterPort \ wise2c/haproxy-k8s EOF
2.4.2 建立keepalived啓動腳本
編輯start-keepalived.sh文件，修改虛擬IP地址VIRTUAL_IP、虛擬網卡設備名INTERFACE、虛擬網卡的子網掩碼NETMASK_BIT、路由標識符RID、虛擬路由標識符VRID的值爲實際Kubernetes集羣所使用的值。（CHECK_PORT的值6444通常不用修改，它是HAProxy的暴露端口，內部指向Kubernetes Master Server的6443端口）
cat > /data/lb/start-keepalived.sh << "EOF" #!/bin/bash VIRTUAL_IP=192.168.92.30 INTERFACE=ens33 NETMASK_BIT=24 CHECK_PORT=6444 RID=10 VRID=160 MCAST_GROUP=224.0.0.18 docker run -itd --restart=always --name=Keepalived-K8S \ --net=host --cap-add=NET_ADMIN \ -e VIRTUAL_IP=$VIRTUAL_IP \ -e INTERFACE=$INTERFACE \ -e CHECK_PORT=$CHECK_PORT \ -e RID=$RID \ -e VRID=$VRID \ -e NETMASK_BIT=$NETMASK_BIT \ -e MCAST_GROUP=$MCAST_GROUP \ wise2c/keepalived-k8s EOF
複製啓動腳本到其餘2個master節點
[root@k8s-master02 ~]# mkdir -p /data/lb [root@k8s-master03 ~]# mkdir -p /data/lb [root@k8s-master01 ~]# scp start-haproxy.sh start-keepalived.sh 192.168.92.11:/data/lb/ [root@k8s-master01 ~]# scp start-haproxy.sh start-keepalived.sh 192.168.92.12:/data/lb/
分別在3個master節點運行腳本啓動haproxy和keepalived容器：
sh /data/lb/start-haproxy.sh && sh /data/lb/start-keepalived.sh
2.4.3 驗證HA狀態
查看容器運行狀態
[root@k8s-master01 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c1d1901a7201 wise2c/haproxy-k8s "/docker-entrypoint.…" 5 days ago Up 3 hours 0.0.0.0:6444->6444/tcp HAProxy-K8S 2f02a9fde0be wise2c/keepalived-k8s "/usr/bin/keepalived…" 5 days ago Up 3 hours Keepalived-K8S
查看網卡綁定的vip 爲192.168.92.30
[root@k8s-master01 ~]# ip a | grep ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.92.10/24 brd 192.168.92.255 scope global noprefixroute ens33 inet 192.168.92.30/24 scope global secondary ens33
查看監聽端口爲6444
[root@k8s-master01 ~]# netstat -tnlp | grep 6444 tcp6 0 0 :::6444 :::* LISTEN 11695/docker-proxy
keepalived配置文件中配置了vrrp_script腳本，使用nc命令對haproxy監聽的6444端口進行檢測，若是檢測失敗即認定本機haproxy進程異常，將vip漂移到其餘節點。
因此不管本機keepalived容器異常或haproxy容器異常都會致使vip漂移到其餘節點，能夠停掉vip所在節點任意容器進行測試。
[root@k8s-master01 ~]# docker stop HAProxy-K8S HAProxy-K8S #能夠看到vip漂移到k8s-master02節點 [root@k8s-master02 ~]# ip a | grep ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.92.11/24 brd 192.168.92.255 scope global noprefixroute ens33 inet 192.168.92.30/24 scope global secondary ens33
也能夠在本地執行該nc命令查看結果
[root@k8s-master02 ~]# yum install -y nc [root@k8s-master02 ~]# nc -v -w 2 -z 127.0.0.1 6444 2>&1 | grep 'Connected to' | grep 6444 Ncat: Connected to 127.0.0.1:6444.
關於haproxy和keepalived配置文件能夠在github源文件中參考Dockerfile，或使用docker exec -it xxx sh命令進入容器查看，容器中的具體路徑：
- /etc/keepalived/keepalived.conf
- /usr/local/etc/haproxy/haproxy.cfg
負載均衡部分配置完成後便可開始部署kubernetes集羣。
2.5 安裝kubeadm
如下操做在全部節點執行。
#因爲官方源國內沒法訪問，這裏使用阿里雲yum源進行替換：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF #安裝kubeadm、kubelet、kubectl,注意這裏默認安裝當前最新版本v1.14.1: yum install -y kubeadm kubelet kubectl systemctl enable kubelet && systemctl start kubelet
2.6 初始化master節點
初始化參考：
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1
建立初始化配置文件
可使用以下命令生成初始化配置文件
kubeadm config print init-defaults > kubeadm-config.yaml
根據實際部署環境修改信息：
[root@k8s-master01 kubernetes]# vim kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta1 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.92.10 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: k8s-master01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta1 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: "192.168.92.30:6444" controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: v1.14.1 networking: dnsDomain: cluster.local podSubnet: "10.244.0.0/16" serviceSubnet: 10.96.0.0/12 scheduler: {} --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration featureGates: SupportIPVSProxyMode: true mode: ipvs
配置說明：
- controlPlaneEndpoint：爲vip地址和haproxy監聽端口6444
- imageRepository:因爲國內沒法訪問google鏡像倉庫k8s.gcr.io，這裏指定爲阿里雲鏡像倉庫registry.aliyuncs.com/google_containers
- podSubnet:指定的IP地址段與後續部署的網絡插件相匹配，這裏須要部署flannel插件，因此配置爲10.244.0.0/16
- mode: ipvs:最後追加的配置爲開啓ipvs模式。
在集羣搭建完成後可使用以下命令查看生效的配置文件：
kubectl -n kube-system get cm kubeadm-config -oyaml
2.7 初始化Master01節點
這裏追加tee命令將初始化日誌輸出到kubeadm-init.log中以備用（可選）。
kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log
該命令指定了初始化時須要使用的配置文件，其中添加–experimental-upload-certs參數能夠在後續執行加入節點時自動分發證書文件。
初始化示例
[root@k8s-master01 ~]# kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log [init] Using Kubernetes version: v1.14.1 [preflight] Running pre-flight checks [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.92.10 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.92.10 127.0.0.1 ::1] [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "ca" certificate and key [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.92.10 192.168.92.30] [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "kubelet.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 19.020444 seconds [upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in ConfigMap "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: 11def25d624a2150b57715e21b0c393695bc6a70d932e472f75d24f747eb657e [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: abcdef.0123456789abcdef [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:7b232b343577bd5fac312996b9fffb3c88f8f8bb39f46bf865ac9f9f52982b82 \ --experimental-control-plane --certificate-key 11def25d624a2150b57715e21b0c393695bc6a70d932e472f75d24f747eb657e Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --experimental-upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:7b232b343577bd5fac312996b9fffb3c88f8f8bb39f46bf865ac9f9f52982b82
kubeadm init主要執行了如下操做：
- [init]：指定版本進行初始化操做
- [preflight] ：初始化前的檢查和下載所須要的Docker鏡像文件
- [kubelet-start]：生成kubelet的配置文件」/var/lib/kubelet/config.yaml」，沒有這個文件kubelet沒法啓動，因此初始化以前的kubelet實際上啓動失敗。
- [certificates]：生成Kubernetes使用的證書，存放在/etc/kubernetes/pki目錄中。
- [kubeconfig] ：生成 KubeConfig 文件，存放在/etc/kubernetes目錄中，組件之間通訊須要使用對應文件。
- [control-plane]：使用/etc/kubernetes/manifest目錄下的YAML文件，安裝 Master 組件。
- [etcd]：使用/etc/kubernetes/manifest/etcd.yaml安裝Etcd服務。
- [wait-control-plane]：等待control-plan部署的Master組件啓動。
- [apiclient]：檢查Master組件服務狀態。
- [uploadconfig]：更新配置
- [kubelet]：使用configMap配置kubelet。
- [patchnode]：更新CNI信息到Node上，經過註釋的方式記錄。
- [mark-control-plane]：爲當前節點打標籤，打了角色Master，和不可調度標籤，這樣默認就不會使用Master節點來運行Pod。
- [bootstrap-token]：生成token記錄下來，後邊使用kubeadm join往集羣中添加節點時會用到
- [addons]：安裝附加組件CoreDNS和kube-proxy
說明：不管是初始化失敗或者集羣已經徹底搭建成功，你均可以直接執行kubeadm reset命令清理集羣或節點，而後從新執行kubeadm init或kubeadm join相關操做便可。
2.8 配置kubectl命令
不管在master節點或node節點，要可以執行kubectl命令必須進行如下配置：
root用戶執行如下命令
cat << EOF >> ~/.bashrc export KUBECONFIG=/etc/kubernetes/admin.conf EOF source ~/.bashrc
普通用戶執行如下命令（參考init時的輸出結果）
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
等集羣配置完成後，能夠在全部master節點和node節點進行以上配置，以支持kubectl命令。針對node節點複製任意master節點/etc/kubernetes/admin.conf到本地。
查看當前狀態
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady master 81s v1.14.1
[root@k8s-master01 ~]# kubectl -n kube-system get pod
NAME READY STATUS RESTARTS AGE
coredns-8686dcc4fd-cbrc5 0/1 Pending 0 64s
coredns-8686dcc4fd-wqpwr 0/1 Pending 0 64s
etcd-k8s-master01 1/1 Running 0 16s
kube-apiserver-k8s-master01 1/1 Running 0 13s
kube-controller-manager-k8s-master01 1/1 Running 0 25s
kube-proxy-4vwbb 1/1 Running 0 65s
kube-scheduler-k8s-master01 1/1 Running 0 4s
[root@k8s-master01 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
因爲未安裝網絡插件，coredns處於pending狀態，node處於notready狀態。
2.9 安裝網絡插件
kubernetes支持多種網絡方案，這裏簡單介紹經常使用的flannel和calico安裝方法，選擇其中一種方案進行部署便可。
如下操做在master01節點執行便可。
2.9.1 安裝flannel網絡插件：
因爲kube-flannel.yml文件指定的鏡像從coreos鏡像倉庫拉取，可能拉取失敗，能夠從dockerhub搜索相關鏡像進行替換，另外能夠看到yml文件中定義的網段地址段爲10.244.0.0/16。
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml cat kube-flannel.yml | grep image cat kube-flannel.yml | grep 10.244 sed -i 's#quay.io/coreos/flannel:v0.11.0-amd64#willdockerhub/flannel:v0.11.0-amd64#g' kube-flannel.yml kubectl apply -f kube-flannel.yml
再次查看node和 Pod狀態，所有爲Running
[root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready master 9m8s v1.14.1 [root@k8s-master01 ~]# kubectl -n kube-system get pod NAME READY STATUS RESTARTS AGE coredns-8686dcc4fd-cbrc5 1/1 Running 0 8m53s coredns-8686dcc4fd-wqpwr 1/1 Running 0 8m53s etcd-k8s-master01 1/1 Running 0 8m5s kube-apiserver-k8s-master01 1/1 Running 0 8m2s kube-controller-manager-k8s-master01 1/1 Running 0 8m14s kube-flannel-ds-amd64-vtppf 1/1 Running 0 115s kube-proxy-4vwbb 1/1 Running 0 8m54s kube-scheduler-k8s-master01 1/1 Running 0 7m53s
2.9.2 安裝calico網絡插件（可選）：
安裝參考：https://docs.projectcalico.org/v3.6/getting-started/kubernetes/
kubectl apply -f \ https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
注意該yaml文件中默認CIDR爲192.168.0.0/16，須要與初始化時kube-config.yaml中的配置一致，若是不一樣請下載該yaml修改後運行。
2.10 加入master節點
從初始化輸出或kubeadm-init.log中獲取命令
kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:c0a1021e5d63f509a0153724270985cdc22e46dc76e8e7b84d1fbb5e83566ea8 \ --experimental-control-plane --certificate-key 52f64a834454c3043fe7a0940f928611b6970205459fa19cb1193b33a288e7cc
依次將k8s-master02和k8s-master03加入到集羣中，示例
[root@k8s-master02 ~]# kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef \ > --discovery-token-ca-cert-hash sha256:7b232b343577bd5fac312996b9fffb3c88f8f8bb39f46bf865ac9f9f52982b82 \ > --experimental-control-plane --certificate-key 11def25d624a2150b57715e21b0c393695bc6a70d932e472f75d24f747eb657e [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [preflight] Running pre-flight checks before initializing the new control plane instance [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master02 localhost] and IPs [192.168.92.11 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master02 localhost] and IPs [192.168.92.11 127.0.0.1 ::1] [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master02 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.92.11 192.168.92.30] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Valid certificates and keys now exist in "/etc/kubernetes/pki" [certs] Using the existing "sa" key [kubeconfig] Generating kubeconfig files [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [check-etcd] Checking that the etcd cluster is healthy [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... [etcd] Announced new etcd member joining to the existing etcd cluster [etcd] Wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml" [etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s [upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [mark-control-plane] Marking the node k8s-master02 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node k8s-master02 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] This node has joined the cluster and a new control plane instance was created: * Certificate signing request was sent to apiserver and approval was received. * The Kubelet was informed of the new secure connection details. * Control plane (master) label and taint were applied to the new node. * The Kubernetes control plane instances scaled up. * A new etcd member was added to the local/stacked etcd cluster. To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Run 'kubectl get nodes' to see this node join the cluster.
2.11 加入node節點
從kubeadm-init.log中獲取命令
kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:c0a1021e5d63f509a0153724270985cdc22e46dc76e8e7b84d1fbb5e83566ea8
示例
[root@k8s-node01 ~]# kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef \ > --discovery-token-ca-cert-hash sha256:7b232b343577bd5fac312996b9fffb3c88f8f8bb39f46bf865ac9f9f52982b82 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
2.12 驗證集羣狀態
查看nodes運行狀況
[root@k8s-master01 ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-master01 Ready master 10h v1.14.1 192.168.92.10 <none> CentOS Linux 7 (Core) 3.10.0-957.10.1.el7.x86_64 docker://18.9.5 k8s-master02 Ready master 10h v1.14.1 192.168.92.11 <none> CentOS Linux 7 (Core) 3.10.0-957.10.1.el7.x86_64 docker://18.9.5 k8s-master03 Ready master 10h v1.14.1 192.168.92.12 <none> CentOS Linux 7 (Core) 3.10.0-957.10.1.el7.x86_64 docker://18.9.5 k8s-node01 Ready <none> 10h v1.14.1 192.168.92.13 <none> CentOS Linux 7 (Core) 3.10.0-957.10.1.el7.x86_64 docker://18.9.5
查看pod運行狀況
[root@k8s-master03 ~]# kubectl -n kube-system get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-8686dcc4fd-6ttgv 1/1 Running 1 22m 10.244.2.3 k8s-master03 <none> <none> coredns-8686dcc4fd-dzvsx 1/1 Running 0 22m 10.244.3.3 k8s-node01 <none> <none> etcd-k8s-master01 1/1 Running 1 6m23s 192.168.92.10 k8s-master01 <none> <none> etcd-k8s-master02 1/1 Running 0 37m 192.168.92.11 k8s-master02 <none> <none> etcd-k8s-master03 1/1 Running 1 36m 192.168.92.12 k8s-master03 <none> <none> kube-apiserver-k8s-master01 1/1 Running 1 48m 192.168.92.10 k8s-master01 <none> <none> kube-apiserver-k8s-master02 1/1 Running 0 37m 192.168.92.11 k8s-master02 <none> <none> kube-apiserver-k8s-master03 1/1 Running 2 36m 192.168.92.12 k8s-master03 <none> <none> kube-controller-manager-k8s-master01 1/1 Running 2 48m 192.168.92.10 k8s-master01 <none> <none> kube-controller-manager-k8s-master02 1/1 Running 1 37m 192.168.92.11 k8s-master02 <none> <none> kube-controller-manager-k8s-master03 1/1 Running 1 35m 192.168.92.12 k8s-master03 <none> <none> kube-flannel-ds-amd64-d86ct 1/1 Running 0 37m 192.168.92.11 k8s-master02 <none> <none> kube-flannel-ds-amd64-l8clz 1/1 Running 0 36m 192.168.92.13 k8s-node01 <none> <none> kube-flannel-ds-amd64-vtppf 1/1 Running 1 42m 192.168.92.10 k8s-master01 <none> <none> kube-flannel-ds-amd64-zg4z5 1/1 Running 1 37m 192.168.92.12 k8s-master03 <none> <none> kube-proxy-4vwbb 1/1 Running 1 49m 192.168.92.10 k8s-master01 <none> <none> kube-proxy-gnk2v 1/1 Running 0 37m 192.168.92.11 k8s-master02 <none> <none> kube-proxy-kqm87 1/1 Running 0 36m 192.168.92.13 k8s-node01 <none> <none> kube-proxy-n5mdh 1/1 Running 2 37m 192.168.92.12 k8s-master03 <none> <none> kube-scheduler-k8s-master01 1/1 Running 2 48m 192.168.92.10 k8s-master01 <none> <none> kube-scheduler-k8s-master02 1/1 Running 1 37m 192.168.92.11 k8s-master02 <none> <none> kube-scheduler-k8s-master03 1/1 Running 2 36m 192.168.92.12 k8s-master03 <none> <none>
查看service
[root@k8s-master03 ~]# kubectl -n kube-system get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 51m
2.13 驗證IPVS
查看kube-proxy日誌，第一行輸出Using ipvs Proxier.
[root@k8s-master01 ~]# kubectl -n kube-system logs -f kube-proxy-4vwbb I0426 16:05:03.156092 1 server_others.go:177] Using ipvs Proxier. W0426 16:05:03.156501 1 proxier.go:381] IPVS scheduler not specified, use rr by default I0426 16:05:03.156788 1 server.go:555] Version: v1.14.1 I0426 16:05:03.166269 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0426 16:05:03.169022 1 config.go:202] Starting service config controller I0426 16:05:03.169103 1 controller_utils.go:1027] Waiting for caches to sync for service config controller I0426 16:05:03.169182 1 config.go:102] Starting endpoints config controller I0426 16:05:03.169200 1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller I0426 16:05:03.269760 1 controller_utils.go:1034] Caches are synced for endpoints config controller I0426 16:05:03.270123 1 controller_utils.go:1034] Caches are synced for service config controller I0426 16:05:03.352400 1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/192.168.92.11:6443 I0426 16:05:03.352478 1 graceful_termination.go:174] Deleting rs: 10.96.0.1:443/TCP/192.168.92.11:6443 ......
查看代理規則
[root@k8s-master01 ~]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.96.0.1:443 rr -> 192.168.92.10:6443 Masq 1 3 0 -> 192.168.92.11:6443 Masq 1 0 0 -> 192.168.92.12:6443 Masq 1 0 0 TCP 10.96.0.10:53 rr -> 10.244.0.5:53 Masq 1 0 0 -> 10.244.0.6:53 Masq 1 0 0 TCP 10.96.0.10:9153 rr -> 10.244.0.5:9153 Masq 1 0 0 -> 10.244.0.6:9153 Masq 1 0 0 UDP 10.96.0.10:53 rr -> 10.244.0.5:53 Masq 1 0 0 -> 10.244.0.6:53 Masq 1 0 0
2.14 etcd集羣
執行如下命令查看etcd集羣狀態
kubectl -n kube-system exec etcd-k8s-master01 -- etcdctl \ --endpoints=https://192.168.92.10:2379 \ --ca-file=/etc/kubernetes/pki/etcd/ca.crt \ --cert-file=/etc/kubernetes/pki/etcd/server.crt \ --key-file=/etc/kubernetes/pki/etcd/server.key cluster-health
示例
[root@k8s-master01 ~]# kubectl -n kube-system exec etcd-k8s-master01 -- etcdctl \ > --endpoints=https://192.168.92.10:2379 \ > --ca-file=/etc/kubernetes/pki/etcd/ca.crt \ > --cert-file=/etc/kubernetes/pki/etcd/server.crt \ > --key-file=/etc/kubernetes/pki/etcd/server.key cluster-health member a94c223ced298a9 is healthy: got healthy result from https://192.168.92.12:2379 member 1db71d0384327b96 is healthy: got healthy result from https://192.168.92.11:2379 member e86955402ac20700 is healthy: got healthy result from https://192.168.92.10:2379 cluster is healthy
2.15 驗證HA
在master01上執行關機操做，建議提早在其餘節點配置kubectl命令支持。
[root@k8s-master01 ~]# shutdown -h now
在任意運行節點驗證集羣狀態，master01節點NotReady，集羣可正常訪問：
[root@k8s-master02 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 NotReady master 19m v1.14.1 k8s-master02 Ready master 11m v1.14.1 k8s-master03 Ready master 10m v1.14.1 k8s-node01 Ready <none> 9m21s v1.14.1
查看網卡，vip自動漂移到master03節點
[root@k8s-master03 ~]# ip a |grep ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.92.12/24 brd 192.168.92.255 scope global noprefixroute ens33 inet 192.168.92.30/24 scope global secondary ens33
————————————————
版權聲明：本文爲CSDN博主「willblog」的原創文章，遵循 CC 4.0 BY-SA 版權協議，轉載請附上原文出處連接及本聲明。
原文連接：
http://www.javashuo.com/article/p-txxxfqhc-v.html
https://blog.csdn.net/networken/article/details/89599004