繼續上一章部署。html
9、controller-manager部署git
9.1 下載二進制文件,參考 第三章github
9.2 建立 kube-controller-manager 證書和私鑰json
cat > kube-controller-manager-csr.json <<EOF { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.56.20", "192.168.56.21" ], "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-controller-manager", "OU": "4Paradigm" } ] } EOF
生成證書和私鑰:bootstrap
cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \ -ca-key=/etc/kubernetes/cert/ca-key.pem \ -config=/etc/kubernetes/cert/ca-config.json \ -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager ls kube-controller-manager*.pem
分發證書到其餘master節點:api
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" scp kube-controller-manager*.pem k8s@${master_ip}:/etc/kubernetes/cert/ done
9.3 建立和分發 kubeconfig 文件網絡
source /opt/k8s/bin/environment.sh kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/cert/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-credentials system:kube-controller-manager \ --client-certificate=kube-controller-manager.pem \ --client-key=kube-controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-context system:kube-controller-manager \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
分發 kubeconfig 文件到全部 master 節點:ssh
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" scp kube-controller-manager.kubeconfig k8s@${master_ip}:/etc/kubernetes/ done
9.4 建立和分發 kube-controller-manager systemd unit 文件curl
source /opt/k8s/bin/environment.sh cat > kube-controller-manager.service <<EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/opt/k8s/bin/kube-controller-manager \\ --port=0 \\ --secure-port=10252 \\ --bind-address=127.0.0.1 \\ --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\ --service-cluster-ip-range=${SERVICE_CIDR} \\ --cluster-name=kubernetes \\ --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\ --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\ --experimental-cluster-signing-duration=8760h \\ --root-ca-file=/etc/kubernetes/cert/ca.pem \\ --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\ --leader-elect=true \\ --feature-gates=RotateKubeletServerCertificate=true \\ --controllers=*,bootstrapsigner,tokencleaner \\ --horizontal-pod-autoscaler-use-rest-clients=true \\ --horizontal-pod-autoscaler-sync-period=10s \\ --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\ --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\ --use-service-account-credentials=true \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/var/log/kubernetes \\ --v=2 Restart=on Restart=on-failure RestartSec=5 User=k8s [Install] WantedBy=multi-user.target EOF
kube-controller-manager 不對請求 https metrics 的 Client 證書進行校驗,故不須要指定 --tls-ca-file 參數,並且該參數已被淘汰tcp
分發 systemd unit 文件到全部master節點:
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" scp kube-controller-manager.service root@${master_ip}:/etc/systemd/system/ done
kube-controller-manager 的權限
ClusteRole: system:kube-controller-manager 的權限很小,只能建立 secret、serviceaccount 等資源對象,各 controller 的權限分散到 ClusterRole system:controller:XXX 中。
須要在 kube-controller-manager 的啓動參數中添加 --use-service-account-credentials=true 參數,這樣 main controller 會爲各 controller 建立對應的 ServiceAccount XXX-controller。
內置的 ClusterRoleBinding system:controller:XXX 將賦予各 XXX-controller ServiceAccount 對應的 ClusterRole system:controller:XXX 權限。
9.5 啓動 kube-controller-manager 服務
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" ssh root@${master_ip} "mkdir -p /var/log/kubernetes && chown -R k8s /var/log/kubernetes" ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager" done
檢查 kube-controller-manager 服務狀態
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" ssh k8s@${master_ip} "systemctl status kube-controller-manager|grep Active" done
確保都是 Active: active (running) 狀態
查看日誌:journalctl -u kube-controller-manager
9.6 查看輸出的 Metric
注:下面的命令在 kube-controller-manager 上執行
kube-controller-manager 監聽10252端口,接收 https 請求
[root@k8s-m1 template]# sudo netstat -lnpt|grep kube-controll tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 23543/kube-controll [root@k8s-m1 template]# curl -s --cacert /etc/kubernetes/cert/ca.pem https://127.0.0.1:10252/metrics |head # HELP ClusterRoleAggregator_adds Total number of adds handled by workqueue: ClusterRoleAggregator # TYPE ClusterRoleAggregator_adds counter ClusterRoleAggregator_adds 7 # HELP ClusterRoleAggregator_depth Current depth of workqueue: ClusterRoleAggregator # TYPE ClusterRoleAggregator_depth gauge ClusterRoleAggregator_depth 0 # HELP ClusterRoleAggregator_queue_latency How long an item stays in workqueueClusterRoleAggregator before being requested. # TYPE ClusterRoleAggregator_queue_latency summary ClusterRoleAggregator_queue_latency{quantile="0.5"} NaN ClusterRoleAggregator_queue_latency{quantile="0.9"} NaN
9.7 測試 kube-controller-manager 高可用
停掉一個節點的 kube-controller-manager 服務,觀察其餘節點的日誌,看是否獲取了 leader 權限
查看當前的 leader:
[root@k8s-m1 template]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-m1_1a33e4f7-d83a-11e8-92ec-080027675bc1","leaseDurationSeconds":15,"acquireTime":"2018-10-25T09:41:09Z","renewTime":"2018-10-25T10:09:32Z","leaderTransitions":0}' creationTimestamp: 2018-10-25T09:41:09Z name: kube-controller-manager namespace: kube-system resourceVersion: "4022" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: 1a397e61-d83a-11e8-a1d2-080027675bc1
當前 leader 是 k8s-m1
參考: