k8s實踐(十四)：Pod驅逐遷移和Node節點維護

時間 2021-01-19

標籤 node nginx git github docker api bash app 運維 ide 欄目 Nginx 简体版

原文原文鏈接

環境說明：node

主機名	操做系統版本	ip	docker version	kubelet version	配置	備註
master	Centos 7.6.1810	172.27.9.131	Docker 18.09.6	V1.14.2	2C2G	master主機
node01	Centos 7.6.1810	172.27.9.135	Docker 18.09.6	V1.14.2	2C2G	node節點
node02	Centos 7.6.1810	172.27.9.136	Docker 18.09.6	V1.14.2	2C2G	node節點

k8s集羣部署詳見：Centos7.6部署k8s(v1.14.2)集羣 nginx

k8s學習資料詳見：基本概念、kubectl命令和資料分享 git

emptyDir詳見：存儲卷和數據持久化(Volumes and Persistent Storage) github

k8s高可用集羣部署詳見：Centos7.6部署k8s v1.16.4高可用集羣(主備模式) docker

k8s集羣高可用部署詳見：lvs+keepalived部署k8s v1.16.4高可用集羣 api

1、背景

當node節點進行如打補丁、操做系統升級等操做時，需停機維護，這就涉及pod驅逐遷移，本文將詳細介紹node節點維護的整個過程。bash

2、pdb簡介

pdb爲poddisruptionbudgets縮寫，意爲主動驅逐保護；

沒有pdb。當進行節點維護時，若是某個服務的多個pod在該節點上，則節點的停機可能會形成服務中斷或者服務降級。舉個例子，某服務有5個pod，最低3個pod能保證服務質量，不然會形成響應慢等影響，此時該服務的4個pod在node01上，若是對node01進行停機維護，此時只有1個pod能正常對外服務，在node01的4個pod遷移過程當中，就會影響該服務正常響應；

pdb能保證應用在節點維護時不低於必定數量的pod運行，從而保持服務質量；

3、準備工做

1.新建pod

[root@master ~]# more nginx-master.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-master
spec:
  replicas: 10 
  template:
    metadata:
      labels:
        app: nginx
    spec:
      restartPolicy: Always
      containers:
      - name: nginx
        image: nginx:latest
[root@master ~]# kubectl apply -f nginx-master.yml 
deployment.extensions/nginx-master created
[root@master ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-47vfj   1/1     Running   0          28s   10.244.0.129   master   <none>           <none>
nginx-master-9d4cf4f77-69jn6   1/1     Running   0          28s   10.244.2.206   node02   <none>           <none>
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          28s   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          28s   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-fxsjd   1/1     Running   0          28s   10.244.2.204   node02   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          28s   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          28s   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pcznk   1/1     Running   0          28s   10.244.2.203   node02   <none>           <none>
nginx-master-9d4cf4f77-px98b   1/1     Running   0          28s   10.244.2.205   node02   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          28s   10.244.1.220   node01   <none>           <none>

新建pod，鏡像爲最新版的nginx，deployment爲nginx-master，數量爲10。能夠看到10個pod分佈在node0一、node02和master 3臺不一樣主機上。app

2.新建pdb

[root@master ~]# more pdb-nginx.yaml 
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: pdb-nginx
spec:
  minAvailable: 9
  selector:
    matchLabels:
      app: nginx
[root@master ~]# kubectl apply -f pdb-nginx.yaml 
poddisruptionbudget.policy/pdb-nginx created
[root@master ~]# kubectl get pdb
NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
pdb-nginx   9               N/A               1                     8s

新建pdb pdb-nginx，Label Selector和deployment同樣都爲app: nginx，minAvailable: 9意爲存活的nginx pod至少爲9個。運維

4、節點維護

本文以節點node02維護爲例介紹。ide

1.設置節點不可調度

[root@master ~]# kubectl cordon node02
node/node02 cordoned
[root@master ~]# kubectl get node
NAME     STATUS                     ROLES    AGE    VERSION
master   Ready                      master   184d   v1.14.2
node01   Ready                      <none>   183d   v1.14.2
node02   Ready,SchedulingDisabled   <none>   182d   v1.14.2
[root@master ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-47vfj   1/1     Running   0          30m   10.244.0.129   master   <none>           <none>
nginx-master-9d4cf4f77-69jn6   1/1     Running   0          30m   10.244.2.206   node02   <none>           <none>
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          30m   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          30m   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-fxsjd   1/1     Running   0          30m   10.244.2.204   node02   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          30m   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          30m   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pcznk   1/1     Running   0          30m   10.244.2.203   node02   <none>           <none>
nginx-master-9d4cf4f77-px98b   1/1     Running   0          30m   10.244.2.205   node02   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          30m   10.244.1.220   node01   <none>           <none>

設置node02不可調度，查看各節點狀態，發現node02爲SchedulingDisabled，此時master不會將新的pod調度到該節點上，可是node02上pod仍是正常運行。

2.驅逐節點上的pod

[root@master ~]# kubectl drain node02 --delete-local-data --ignore-daemonsets --force 
node/node02 already cordoned

參數說明：

--delete-local-data 即便pod使用了emptyDir也刪除

--ignore-daemonsets 忽略deamonset控制器的pod，若是不忽略，deamonset控制器控制的pod被刪除後可能立刻又在此節點上啓動起來,會成爲死循環；

--force 不加force參數只會刪除該NODE上由ReplicationController, ReplicaSet, DaemonSet,StatefulSet or Job建立的Pod，加了後還會刪除'裸奔的pod'(沒有綁定到任何replication controller)

能夠看到同一時刻只有一個pod進行遷移，對外提供服務的pod始終有9個。

遷移pod nginx-master-9d4cf4f77-pcznk到node01

遷移pod nginx-master-9d4cf4f77-px98b到master，此時前一個pod nginx-master-9d4cf4f77-pcznk已經遷移完成。

遷移pod nginx-master-9d4cf4f77-69jn6到master

遷移pod nginx-master-9d4cf4f77-fxsjd到master

這個也再次驗證了同一時刻只有一個pod遷移，nginx服務始終有9個pod對外提供服務。

3.維護結束

[root@master ~]# kubectl uncordon node02
node/node02 uncordoned
[root@master ~]# kubectl get nodes      
NAME     STATUS   ROLES    AGE    VERSION
master   Ready    master   184d   v1.14.2
node01   Ready    <none>   183d   v1.14.2
node02   Ready    <none>   183d   v1.14.2

維護結束，從新將node02節點置爲可調度狀態。

5、pod回遷

pod回遷貌似還沒什麼好的辦法，這裏採用delete而後重建的方式回遷。

[root@master ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-2vnvk   1/1     Running   0          33m   10.244.1.222   node01   <none>           <none>
nginx-master-9d4cf4f77-47vfj   1/1     Running   0          73m   10.244.0.129   master   <none>           <none>
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          73m   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-7n7pt   1/1     Running   0          32m   10.244.0.131   master   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          73m   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          73m   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          73m   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pdkst   1/1     Running   0          32m   10.244.0.130   master   <none>           <none>
nginx-master-9d4cf4f77-pskmp   1/1     Running   0          32m   10.244.0.132   master   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          73m   10.244.1.220   node01   <none>           <none>
[root@master ~]# kubectl delete po nginx-master-9d4cf4f77-47vfj
pod "nginx-master-9d4cf4f77-47vfj" deleted
[root@master ~]# kubectl delete po nginx-master-9d4cf4f77-2vnvk
pod "nginx-master-9d4cf4f77-2vnvk" deleted
[root@master ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          76m   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-7n7pt   1/1     Running   0          35m   10.244.0.131   master   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          76m   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-f92hp   1/1     Running   0          44s   10.244.2.207   node02   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          76m   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          76m   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pdkst   1/1     Running   0          35m   10.244.0.130   master   <none>           <none>
nginx-master-9d4cf4f77-pskmp   1/1     Running   0          35m   10.244.0.132   master   <none>           <none>
nginx-master-9d4cf4f77-tdghn   1/1     Running   0          15s   10.244.2.208   node02   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          76m   10.244.1.220   node01   <none>           <none>

在業務低峯delete pod nginx-master-9d4cf4f77-47vfj和nginx-master-9d4cf4f77-2vnvk，因爲node02上的pod以前都被驅逐，此時資源使用率最低，因此pod重建時會調度值該節點，完成pod回遷。

6、節點刪除

1.刪除節點

實際運維過程當中可能會刪除某個node節點，本文仍是以node02爲例，介紹若是刪除節點。

[root@master ~]# kubectl cordon node02
[root@master ~]# kubectl drain node02 --delete-local-data --ignore-daemonsets --force 
[root@master ~]# kubectl delete node node02

[root@node02 ~]# kubeadm reset

2.節點從新加入

master節點上運行

[root@master ~]# kubeadm token create --print-join-command
kubeadm join 172.27.9.131:6443 --token kpz40z.tuxb4t4m1q37vwl1     --discovery-token-ca-cert-hash sha256:5f656ae26b5e7d4641a979cbfdffeb7845cc5962bbfcd1d5435f00a25c02ea50

node02從新加入集羣

[root@node02 ~]# kubeadm join 172.27.9.131:6443 --token svrip0.lajrfl4jgal0ul6i     --discovery-token-ca-cert-hash sha256:5f656ae26b5e7d4641a979cbfdffeb7845cc5962bbfcd1d5435f00a25c02ea50

查看node

本文全部腳本和配置文件已上傳：Pode Eviction and Node Manage