Kubernetes集羣實踐（10）滾動更新和回滾

Kubernetes集羣的證書有效期是1年，若是超過有效期，kublet服務會無效，此時繼續使用kubelet命令，將會獲得相似這樣的提示「Unable to connect to the server: x509: certificate has expired or is not yet valid.」所以，咱們須要對其進行維護。

1. 登錄Master節點查看證書有效期

kubeadm alpha certs check-expiration

2. 備份現有證書

mkdir -p $HOME/k8s-old-certs/pki
cp -p /etc/kubernetes/pki/*.* $HOME/k8s-old-certs/pki
ls -l $HOME/k8s-old-certs/pki/

結果：

total 56
-rw-r--r-- 1 root root 1261 Sep  4  2019 apiserver.crt
-rw-r--r-- 1 root root 1090 Sep  4  2019 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Sep  4  2019 apiserver-etcd-client.key
-rw------- 1 root root 1679 Sep  4  2019 apiserver.key
-rw-r--r-- 1 root root 1099 Sep  4  2019 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Sep  4  2019 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1025 Sep  4  2019 ca.crt
-rw------- 1 root root 1675 Sep  4  2019 ca.key
-rw-r--r-- 1 root root 1038 Sep  4  2019 front-proxy-ca.crt
-rw------- 1 root root 1675 Sep  4  2019 front-proxy-ca.key
-rw-r--r-- 1 root root 1058 Sep  4  2019 front-proxy-client.crt
-rw------- 1 root root 1679 Sep  4  2019 front-proxy-client.key
-rw------- 1 root root 1675 Sep  4  2019 sa.key
-rw------- 1 root root  451 Sep  4  2019 sa.pub

3. 備份配置文件

cp -p /etc/kubernetes/*.conf $HOME/k8s-old-certs
ls -ltr $HOME/k8s-old-certs

結果：

total 36
-rw------- 1 root root 5451 Sep  4  2019 admin.conf
-rw------- 1 root root 5595 Sep  4  2019 kubelet.conf
-rw------- 1 root root 5483 Sep  4  2019 controller-manager.conf
-rw------- 1 root root 5435 Sep  4  2019 scheduler.conf
drwxr-xr-x 2 root root 4096 Dec 19 21:21 pki

4. 備份Home目錄的配置文件

mkdir -p $HOME/k8s-old-certs/.kube
cp -p ~/.kube/config $HOME/k8s-old-certs/.kube/.
ls -l $HOME/k8s-old-certs/.kube/.

結果：

-rw------- 1 root root 5451 Sep  4  2019 config

5. 更新Kubernetes證書

kubeadm alpha certs renew all

結果

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healtcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

6. 在次檢查證書有效期是不是364天后無效

kubeadm alpha certs check-expiration

7. 確保kubelet服務器正常，同時work和master間通訊正常。
8. 等待幾分鐘，使用以下命令確保work可用

kubectl get nodes

若是輸出如下信息

The connection to the server 9.37.21.119:6443 was refused - did you specify the right host or port?

則須要執行下面步驟繼續進行修復

9. 檢查比/etc/kubernetes/kubelet.conf文件

diff $HOME/k8s-old-certs/kubelet.conf /etc/kubernetes/kubelet.conf

若是沒有任何輸出，則更新證書操做沒有影響此文件，繼續下面的步驟手動修復。

10. 更新/etc/kubernetes/kubelet.conf文件

cd /etc/kubernetes
sudo kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf
diff $HOME/k8s-old-certs/kubelet.conf /etc/kubernetes/kubelet.conf

更新後能夠查看與備份文件的區別.

11. 檢查對比~/.kube/config文件

diff ~/.kube/config $HOME/fcik8s-old-certs/.kube/config

若是沒有任何輸出，則該文件包含過時的key和證書，繼續下面的步驟手動修復。

12. 使用當前更新後的/etc/kubernetes/kubelet.conf中的'client-certificate-data '和'client-key-data'的值更新~/.kube/config文件中對應的值。
13. 重啓kubelet服務

systemctl daemon-reload&&systemctl restart kubelet

14. 再次查看節點和Pod是否正常

kubectl get nodes
...
kubectl get pods

參考資料：https://www.ibm.com/docs/en/fci/1.1.0?topic=kubernetes-renewing-cluster-certificates