k8s集羣部署v1.15實踐8:部署高可用 kube-controller-manager 集羣

參考文檔

部署高可用 kube-controller-manager 集羣

備註:

本文檔介紹部署高可用 kube-controller-manager 集羣的步驟.

集羣包含 3 個節點,啓動後將經過競爭選舉機制產生一個 leader 節點,其它節點爲阻塞狀態.當 leader 節點不可用後,剩餘節點將再次進行選舉產生新的 leader 節點,從而保證服務的可用性.

二進制文件,在前面部署api時,已經下載好且已經配置到指定位置了.

[root@k8s-node1 ~]# cd /opt/k8s/bin
[root@k8s-node1 bin]# ls
cfssl           cfssljson       etcd     flanneld        kube-controller-manager  kube-scheduler
cfssl-certinfo  environment.sh  etcdctl  kube-apiserver  kubectl                  mk-docker-opts.sh
[root@k8s-node1 bin]#

*1.建立 kube-controller-manager 證書和私鑰

建立證書籤名請求:

hosts 列表包含全部 kube-controller-manager 節點 IP.

CN 爲 system:kube-controller-manager.

O 爲 system:kube-controller-manager,kubernetes 內置的 ClusterRoleBindings system:kube-controller-manager 賦予kube-controller-manager 工做所需的權限.

[root@k8s-node1 kube-controller-manager]# pwd
/opt/k8s/k8s_software/server/kube-controller-manager

[root@k8s-node1 kube-controller-manager]# cat kube-controller-manager-csr.json 
{
"CN": "system:kube-controller-manager",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"127.0.0.1",
"192.168.174.128",
"192.168.174.129",
"192.168.174.130"
],
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "SZ",
"O": "system:kube-controller-manager",
"OU": "4Paradigm"
}
]
}
[root@k8s-node1 kube-controller-manager]#

生成證書和密鑰

[root@k8s-node1 kube-controller-manager]#  cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
2019/11/04 21:47:27 [INFO] generate received request
2019/11/04 21:47:27 [INFO] received CSR
2019/11/04 21:47:27 [INFO] generating key: rsa-2048
2019/11/04 21:47:27 [INFO] encoded CSR
2019/11/04 21:47:27 [INFO] signed certificate with serial number 87619246134042005891041298787368847333016054812
2019/11/04 21:47:27 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@k8s-node1 kube-controller-manager]# ls
kube-controller-manager.csr  kube-controller-manager-csr.json  kube-controller-manager-key.pem  kube-controller-manager.pem
[root@k8s-node1 kube-controller-manager]#

把證書添加到全部節點

kube-controller-manager.csr  kube-controller-manager-csr.json  kube-controller-manager-key.pem  kube-controller-manager.pem
[root@k8s-node1 kube-controller-manager]# cp kube-controller-manager-key.pem kube-controller-manager.pem /etc/kubernetes/cert/
[root@k8s-node1 kube-controller-manager]# scp kube-controller-manager-key.pem kube-controller-manager.pem root@k8s-node2:/etc/kubernetes/cert/
kube-controller-manager-key.pem                                                                       100% 1675     1.6MB/s   00:00    
kube-controller-manager.pem                                                                           100% 1489     1.2MB/s   00:00    
[root@k8s-node1 kube-controller-manager]# scp kube-controller-manager-key.pem kube-controller-manager.pem root@k8s-node3:/etc/kubernetes/cert/
kube-controller-manager-key.pem                                                                       100% 1675     1.6MB/s   00:00    
kube-controller-manager.pem                                                                           100% 1489     1.2MB/s   00:00    
[root@k8s-node1 kube-controller-manager]#

修改文件屬主和加x權限

[root@k8s-node1 kube-controller-manager]# chown -R k8s /etc/kubernetes/cert/ && chmod -R +x /etc/kubernetes/cert/
[root@k8s-node1 kube-controller-manager]# ssh root@k8s-node2 "chown -R k8s /etc/kubernetes/cert/ && chmod -R +x /etc/kubernetes/cert/"
[root@k8s-node1 kube-controller-manager]# ssh root@k8s-node3 "chown -R k8s /etc/kubernetes/cert/ && chmod -R +x /etc/kubernetes/cert/"

2.建立和分發kube-controller-manager使用的kubeconfig文件

kubeconfig 文件包含訪問 apiserver 的全部信息,如 apiserver 地址,CA 證書和自身使用的證書,和訪問api的用戶名和rbac權限等等.

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.174.127:8443 --kubeconfig=kube-controller-manager.kubeconfig
Cluster "kubernetes" set.

這條是寫入集羣信息,集羣名字爲kubernetes,認證的證書爲ca.pem,集羣的地址爲vip地址192.168.174.147.把這些信息寫入kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig
User "system:kube-controller-manager" set.

這條是寫入用戶信息,用戶爲system:kube-controller-manager,用戶的證書爲kube-controller-manager.pem,私鑰爲kube-controller-manager-key.pem,把這些信息寫入kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
Context "system:kube-controller-manager" created.

這條是配置上下文參數,上下文名字爲system:kube-controller-manager,使用的集羣爲kubernetes,用戶爲system:kube-controller-manager,把這些信息寫入kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
Switched to context "system:kube-controller-manager".

使用上面配置的上下文.

分發kubeconfig 到全部節點

[root@k8s-node1 kube-controller-manager]# cp kube-controller-manager.kubeconfig /etc/kubernetes/
[root@k8s-node1 kube-controller-manager]# scp kube-controller-manager.kubeconfig root@k8s-node2:/etc/kubernetes/
kube-controller-manager.kubeconfig                                                                    100% 6445     5.3MB/s   00:00    
[root@k8s-node1 kube-controller-manager]# scp kube-controller-manager.kubeconfig root@k8s-node3:/etc/kubernetes/
kube-controller-manager.kubeconfig

修改下權限

[root@k8s-node1 kube-controller-manager]# chown -R k8s /etc/kubernetes/ && chmod -R +x /etc/kubernetes/
[root@k8s-node1 kube-controller-manager]# ssh root@k8s-node2 "chown -R k8s /etc/kubernetes/ && chmod -R +x /etc/kubernetes/"
[root@k8s-node1 kube-controller-manager]# ssh root@k8s-node3 "chown -R k8s /etc/kubernetes/ && chmod -R +x /etc/kubernetes/"

3.建立和分發kube-controller-manager的system unit文件

參數說明

--port=0 ：關閉監聽 http /metrics 的請求.同時 --address 參數無效,--bind-address 參數有效.這項必須取消禁止,不然報錯.
解釋:

--secure-port=10252 、 --bind-address=0.0.0.0:在全部網絡接口監聽10252 端口的 https /metrics 請求,這兩項也要禁止.

--kubeconfig:指定 kubeconfig 文件路徑,kube-controller-manager 使用它鏈接和驗證 kube-apiserver；

--cluster-signing--file:簽名 TLS Bootstrap 建立的證書.

--experimental-cluster-signing-duration:指定 TLS Bootstrap 證書的有效期.

--root-ca-file ：放置到容器 ServiceAccount 中的 CA 證書,用來對 kubeapiserver的證書進行校驗.

--service-account-private-key-file:簽名 ServiceAccount 中 Token 的私鑰文件,必須和 kube-apiserver 的 --service-account-key-file 指定的公鑰文件配對使用.

--service-cluster-ip-range:指定 Service Cluster IP 網段,必須和 kubeapiserver中的同名參數一致.

--leader-elect=true:集羣運行模式,啓用選舉功能.被選爲 leader 的節點負責處理工做,其它節點爲阻塞狀態.

--feature-gates=RotateKubeletServerCertificate=true:開啓 kublet server 證書的自動更新特性.

--controllers=,bootstrapsigner,tokencleaner:啓用的控制器列表,tokencleaner 用於自動清理過時的 Bootstrap token.

--horizontal-pod-autoscaler-*:custom metrics 相關參數,支持autoscaling/v2alpha1.

--tls-cert-file 、 --tls-private-key-file:使用 https 輸出 metrics 時使用的 Server 證書和祕鑰；

--use-service-account-credentials=true:ClusteRole:system:kube-controller-manager 的權限很小,只能建立 secret,serviceaccount 等資源對象.各 controller 的權限分散到 ClusterRolesystem:controller:XXX 中.須要在 kube-controller-manager 的啓動參數中添加 --use-service-accountcredentials=true 參數,這樣 main controller 會爲各 controller 建立對應的ServiceAccount XXX-controller.內置的 ClusterRoleBinding system:controller:XXX 將賦予各 XXX-controllerServiceAccount 對應的 ClusterRole system:controller:XXX 權限.

User=k8s:使用 k8s 帳戶運行.

kube-controller-manager 不對請求 https metrics 的 Client 證書進行校驗,故不須要指定--tls-ca-file 參數,並且該參數已被淘汰.

[root@k8s-node1 kube-controller-manager]# source /opt/k8s/bin/environment.sh

[root@k8s-node1 kube-controller-manager]# cat kube-controller-manager.service 
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/opt/k8s/bin/kube-controller-manager \
#--port=0 \
#--secure-port=10252 \
#--bind-address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
--service-cluster-ip-range=10.254.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \
--experimental-cluster-signing-duration=8760h \
--root-ca-file=/etc/kubernetes/cert/ca.pem \
--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \
--leader-elect=true \
--feature-gates=RotateKubeletServerCertificate=true \
--controllers=*,bootstrapsigner,tokencleaner \
--horizontal-pod-autoscaler-use-rest-clients=true \
--horizontal-pod-autoscaler-sync-period=10s \
--tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \
--tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \
--use-service-account-credentials=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2
Restart=on
Restart=on-failure
RestartSec=5
User=k8s
[Install]
WantedBy=multi-user.target
[root@k8s-node1 kube-controller-manager]#

分發文件都全部節點

[root@k8s-node1 kube-controller-manager]# cp kube-controller-manager.service /etc/systemd/system
[root@k8s-node1 kube-controller-manager]# scp kube-controller-manager.service root@k8s-node2:/etc/systemd/system
kube-controller-manager.service                                                                       100% 1231     1.4MB/s   00:00    
[root@k8s-node1 kube-controller-manager]# scp kube-controller-manager.service root@k8s-node3:/etc/systemd/system
kube-controller-manager.service                                                                       100% 1231     1.5MB/s   00:00    
[root@k8s-node1 kube-controller-manager]#

修改權限

[root@k8s-node1 kube-controller-manager]# chmod +x /etc/systemd/system/kube-controller-manager.service
[root@k8s-node1 kube-controller-manager]# ssh root@k8s-node2 "chmod +x /etc/systemd/system/kube-controller-manager.service"
[root@k8s-node1 kube-controller-manager]# ssh root@k8s-node3 "chmod +x /etc/systemd/system/kube-controller-manager.service"

4.啓動服務

systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager

[root@k8s-node1 kube-controller-manager]# systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/etc/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-11-04 22:36:49 EST; 2min 38s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 17475 (kube-controller)
    Tasks: 6
   Memory: 15.0M
   CGroup: /system.slice/kube-controller-manager.service
           └─17475 /opt/k8s/bin/kube-controller-manager #--port=0 #--secure-port=10252 #--bind-address=127.0.0.1 --kubeconfig=/etc/ku...

Nov 04 22:36:49 k8s-node1 kube-controller-manager[17475]: I1104 22:36:49.847823   17475 flags.go:33] FLAG: --v="2"
Nov 04 22:36:49 k8s-node1 kube-controller-manager[17475]: I1104 22:36:49.847826   17475 flags.go:33] FLAG: --version="false"
Nov 04 22:36:49 k8s-node1 kube-controller-manager[17475]: I1104 22:36:49.847831   17475 flags.go:33] FLAG: --vmodule=""
Nov 04 22:36:50 k8s-node1 kube-controller-manager[17475]: W1104 22:36:50.112321   17475 authentication.go:249] No authentication...work.
Nov 04 22:36:50 k8s-node1 kube-controller-manager[17475]: W1104 22:36:50.112555   17475 authentication.go:252] No authentication...work.
Nov 04 22:36:50 k8s-node1 kube-controller-manager[17475]: W1104 22:36:50.112575   17475 authorization.go:146] No authorization-k...work.
Nov 04 22:36:50 k8s-node1 kube-controller-manager[17475]: I1104 22:36:50.112597   17475 controllermanager.go:164] Version: v1.15.5
Nov 04 22:36:50 k8s-node1 kube-controller-manager[17475]: I1104 22:36:50.113013   17475 secure_serving.go:116] Serving securely ...10257
Nov 04 22:36:50 k8s-node1 kube-controller-manager[17475]: I1104 22:36:50.115743   17475 deprecated_insecure_serving.go:53] Servi...10252
Nov 04 22:36:50 k8s-node1 kube-controller-manager[17475]: I1104 22:36:50.115793   17475 leaderelection.go:235] attempting to acq...er...
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-node1 kube-controller-manager]#

執行命令測試下

[root@k8s-node1 kube-controller-manager]# kubectl get cs
NAME                 STATUS      MESSAGE                                                                                     ERROR
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused   
controller-manager   Healthy     ok                                                                                          
etcd-2               Healthy     {"health":"true"}                                                                           
etcd-1               Healthy     {"health":"true"}                                                                           
etcd-0               Healthy     {"health":"true"}                                                                           
[root@k8s-node1 kube-controller-manager]#

檢索那個是Leader

[root@k8s-node1 kube-controller-manager]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-node2_c38fd29c-84e3-47c2-b2c0-6c19952c3e86","leaseDurationSeconds":15,"acquireTime":"2019-11-05T03:37:03Z","renewTime":"2019-11-05T03:42:42Z","leaderTransitions":1}'
creationTimestamp: "2019-11-05T03:25:45Z"
name: kube-controller-manager
namespace: kube-system
resourceVersion: "2991"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
uid: 0bf961ae-1219-41f4-98f8-bebd83875190
[root@k8s-node1 kube-controller-manager]#

中止k8s-node2的kube-controller-manager服務,看看Leader會切換嗎？

[root@k8s-node2 ~]# systemctl stop kube-controller-manager
[root@k8s-node2 ~]# systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/etc/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Mon 2019-11-04 22:43:26 EST; 1min 28s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 17267 ExecStart=/opt/k8s/bin/kube-controller-manager #--port=0 #--secure-port=10252 #--bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem --experimental-cluster-signing-duration=8760h --root-ca-file=/etc/kubernetes/cert/ca.pem --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --controllers=*,bootstrapsigner,tokencleaner --horizontal-pod-autoscaler-use-rest-clients=true --horizontal-pod-autoscaler-sync-period=10s --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem --use-service-account-credentials=true --alsologtostderr=true --logtostderr=false --log-dir=/var/log/kubernetes --v=2 (code=killed, signal=TERM)
 Main PID: 17267 (code=killed, signal=TERM)
Nov 04 22:37:07 k8s-node2 kube-controller-manager[17267]: I1104 22:37:07.941851   17267 controller_utils.go:1036] Caches are syn...oller
Nov 04 22:37:07 k8s-node2 kube-controller-manager[17267]: I1104 22:37:07.941931   17267 garbagecollector.go:240] synced garbage ...ector
Nov 04 22:37:07 k8s-node2 kube-controller-manager[17267]: I1104 22:37:07.965357   17267 controller_utils.go:1036] Caches are syn...oller
Nov 04 22:37:07 k8s-node2 kube-controller-manager[17267]: I1104 22:37:07.966845   17267 controller_utils.go:1036] Caches are syn...oller
Nov 04 22:37:07 k8s-node2 kube-controller-manager[17267]: I1104 22:37:07.966911   17267 controller_utils.go:1036] Caches are syn...oller
Nov 04 22:37:07 k8s-node2 kube-controller-manager[17267]: I1104 22:37:07.975179   17267 controller_utils.go:1036] Caches are syn...oller
Nov 04 22:37:07 k8s-node2 kube-controller-manager[17267]: I1104 22:37:07.975275   17267 garbagecollector.go:137] Garbage collect...rbage
Nov 04 22:37:08 k8s-node2 kube-controller-manager[17267]: I1104 22:37:08.014970   17267 controller_utils.go:1036] Caches are syn...oller
Nov 04 22:43:26 k8s-node2 systemd[1]: Stopping Kubernetes Controller Manager...
Nov 04 22:43:26 k8s-node2 systemd[1]: Stopped Kubernetes Controller Manager.
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-node2 ~]#

切換到k8s-node1了

[root@k8s-node2 ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-node1_8f0109d6-8ec3-4e60-8061-6b36e0fb9f79","leaseDurationSeconds":15,"acquireTime":"2019-11-05T03:43:43Z","renewTime":"2019-11-05T03:45:16Z","leaderTransitions":2}'
creationTimestamp: "2019-11-05T03:25:45Z"
name: kube-controller-manager
namespace: kube-system
resourceVersion: "3108"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
uid: 0bf961ae-1219-41f4-98f8-bebd83875190
[root@k8s-node2 ~]#