本文檔介紹部署高可用 kube-controller-manager 集羣的步驟。node
該集羣包含 3 個節點,啓動後將經過競爭選舉機制產生一個 leader 節點,其它節點爲阻塞狀態。當 leader 節點不可用後,剩餘節點將再次進行選舉產生新的 leader 節點,從而保證服務的可用性。git
爲保證通訊安全,本文檔先生成 x509 證書和私鑰,kube-controller-manager 在以下兩種狀況下使用該證書:github
注意:若是沒有特殊指明,本文檔的全部操做均在 k8s-master1 節點上執行,而後遠程分發文件和執行命令。json
建立證書籤名請求:bootstrap
cd /opt/k8s/work cat > kube-controller-manager-csr.json <<EOF { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.161.150", "192.168.161.151", "192.168.161.152" ], "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-controller-manager", "OU": "4Paradigm" } ] } EOF
生成證書和私鑰:api
cd /opt/k8s/work cfssl gencert -ca=/opt/k8s/work/ca.pem \ -ca-key=/opt/k8s/work/ca-key.pem \ -config=/opt/k8s/work/ca-config.json \ -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager ls kube-controller-manager*pem
將生成的證書和私鑰分發到全部 master 節點:安全
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" scp kube-controller-manager*.pem root@${node_ip}:/etc/kubernetes/cert/ done
kubeconfig 文件包含訪問 apiserver 的全部信息,如 apiserver 地址、CA 證書和自身使用的證書;網絡
cd /opt/k8s/work source /opt/k8s/bin/environment.sh kubectl config set-cluster kubernetes \ --certificate-authority=/opt/k8s/work/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-credentials system:kube-controller-manager \ --client-certificate=kube-controller-manager.pem \ --client-key=kube-controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-context system:kube-controller-manager \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
分發 kubeconfig 到全部 master 節點:ssh
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" scp kube-controller-manager.kubeconfig root@${node_ip}:/etc/kubernetes/ done
cd /opt/k8s/work source /opt/k8s/bin/environment.sh cat > kube-controller-manager.service <<EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] WorkingDirectory=${K8S_DIR}/kube-controller-manager ExecStart=/opt/k8s/bin/kube-controller-manager \\ --port=0 \\ --secure-port=10252 \\ --bind-address=127.0.0.1 \\ --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\ --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\ --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\ --service-cluster-ip-range=${SERVICE_CIDR} \\ --cluster-name=kubernetes \\ --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\ --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\ --experimental-cluster-signing-duration=8760h \\ --root-ca-file=/etc/kubernetes/cert/ca.pem \\ --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\ --leader-elect=true \\ --controllers=*,bootstrapsigner,tokencleaner \\ --horizontal-pod-autoscaler-use-rest-clients=true \\ --horizontal-pod-autoscaler-sync-period=10s \\ --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\ --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\ --use-service-account-credentials=true \\ --kube-api-qps=1000 \\ --kube-api-burst=2000 \\ --logtostderr=true \\ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
--port=0
:關閉監聽 http /metrics 的請求,同時 --address
參數無效,--bind-address
參數有效;--secure-port=10252
、--bind-address=0.0.0.0
: 在全部網絡接口監聽 10252 端口的 https /metrics 請求;--kubeconfig
:指定 kubeconfig 文件路徑,kube-controller-manager 使用它鏈接和驗證 kube-apiserver;--authentication-kubeconfig
和 --authorization-kubeconfig
:kube-controller-manager 使用它鏈接 apiserver,對 client 的請求進行認證和受權。kube-controller-manager
再也不使用 --tls-ca-file
對請求 https metrics 的 Client 證書進行校驗。若是沒有配置這兩個 kubeconfig 參數,則 client 鏈接 kube-controller-manager https 端口的請求會被拒絕(提示權限不足)。--cluster-signing-*-file
:簽名 TLS Bootstrap 建立的證書;--experimental-cluster-signing-duration
:指定 TLS Bootstrap 證書的有效期;--root-ca-file
:放置到容器 ServiceAccount 中的 CA 證書,用來對 kube-apiserver 的證書進行校驗;--service-account-private-key-file
:簽名 ServiceAccount 中 Token 的私鑰文件,必須和 kube-apiserver 的 --service-account-key-file
指定的公鑰文件配對使用;--service-cluster-ip-range
:指定 Service Cluster IP 網段,必須和 kube-apiserver 中的同名參數一致;--leader-elect=true
:集羣運行模式,啓用選舉功能;被選爲 leader 的節點負責處理工做,其它節點爲阻塞狀態;--controllers=*,bootstrapsigner,tokencleaner
:啓用的控制器列表,tokencleaner 用於自動清理過時的 Bootstrap token;--horizontal-pod-autoscaler-*
:custom metrics 相關參數,支持 autoscaling/v2alpha1;--tls-cert-file
、--tls-private-key-file
:使用 https 輸出 metrics 時使用的 Server 證書和祕鑰;--use-service-account-credentials=true
: kube-controller-manager 中各 controller 使用 serviceaccount 訪問 kube-apiserver;完整 unit 見 kube-controller-manager.servicetcp
分發 systemd unit 文件到全部 master 節點:
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" scp kube-controller-manager.service root@${node_ip}:/etc/systemd/system/ done
ClusteRole: system:kube-controller-manager 的權限很小,只能建立 secret、serviceaccount 等資源對象,各 controller 的權限分散到 ClusterRole system:controller:XXX 中。
須要在 kube-controller-manager 的啓動參數中添加 --use-service-account-credentials=true
參數,這樣 main controller 會爲各 controller 建立對應的 ServiceAccount XXX-controller。
內置的 ClusterRoleBinding system:controller:XXX 將賦予各 XXX-controller ServiceAccount 對應的 ClusterRole system:controller:XXX 權限。
source /opt/k8s/bin/environment.sh
for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager" ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager" done
source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl status kube-controller-manager|grep Active" done
確保狀態爲 active (running)
,不然查看日誌,確認緣由:
$ journalctl -u kube-controller-manager
注意:如下命令在 kube-controller-manager 節點上執行。
kube-controller-manager 監聽 10252 端口,接收 https 請求:
[root@k8s-master1 work]# sudo netstat -lnpt|grep kube-controll tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 19176/kube-controll
停掉一個或兩個節點的 kube-controller-manager 服務,觀察其它節點的日誌,看是否獲取了 leader 權限。
kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
[root@k8s-master1 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master3_60147cc2-758e-11e9-8fcb-000c2956b313","leaseDurationSeconds":15,"acquireTime":"2019-05-13T14:49:56Z","renewTime":"2019-05-13T14:53:10Z","leaderTransitions":0}' creationTimestamp: 2019-05-13T14:49:56Z name: kube-controller-manager namespace: kube-system resourceVersion: "5106" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: 6015837d-758e-11e9-89c6-000c2956b313 [root@k8s-master1 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master1_60ab86f0-758e-11e9-931c-000c29aa0a68","leaseDurationSeconds":15,"acquireTime":"2019-05-13T14:53:37Z","renewTime":"2019-05-13T14:53:39Z","leaderTransitions":1}' creationTimestamp: 2019-05-13T14:49:56Z name: kube-controller-manager namespace: kube-system resourceVersion: "5116" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: 6015837d-758e-11e9-89c6-000c2956b313
可見,當前的 leader 從master3切到了master1 節點。