k8s控制平面kube-controller-manager部署

說明:本部署文章參照了 https://github.com/opsnull/follow-me-install-kubernetes-cluster ,歡迎給做者star

本文檔介紹部署高可用 kube-controller-manager 集羣的步驟。node

該集羣包含 3 個節點,啓動後將經過競爭選舉機制產生一個 leader 節點,其它節點爲阻塞狀態。當 leader 節點不可用後,剩餘節點將再次進行選舉產生新的 leader 節點,從而保證服務的可用性。git

爲保證通訊安全,本文檔先生成 x509 證書和私鑰,kube-controller-manager 在以下兩種狀況下使用該證書:github

  1. 與 kube-apiserver 的安全端口通訊時;
  2. 在安全端口(https,10252) 輸出 prometheus 格式的 metrics;

注意:若是沒有特殊指明,本文檔的全部操做均在 k8s-master1 節點上執行,而後遠程分發文件和執行命令。json

 

建立 kube-controller-manager 證書和私鑰

建立證書籤名請求:bootstrap

cd /opt/k8s/work
cat > kube-controller-manager-csr.json <<EOF
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.161.150",
      "192.168.161.151",
      "192.168.161.152"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "system:kube-controller-manager",
        "OU": "4Paradigm"
      }
    ]
}
EOF
  • hosts 列表包含全部 kube-controller-manager 節點 IP;
  • CN 爲 system:kube-controller-manager、O 爲 system:kube-controller-manager,kubernetes 內置的 ClusterRoleBindings system:kube-controller-manager 賦予 kube-controller-manager 工做所需的權限。

生成證書和私鑰:api

cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
ls kube-controller-manager*pem

將生成的證書和私鑰分發到全部 master 節點:安全

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152
  do
    echo ">>> ${node_ip}"
    scp kube-controller-manager*.pem root@${node_ip}:/etc/kubernetes/cert/
  done

 

 

建立和分發 kubeconfig 文件

kubeconfig 文件包含訪問 apiserver 的全部信息,如 apiserver 地址、CA 證書和自身使用的證書;網絡

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/k8s/work/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=kube-controller-manager.pem \
  --client-key=kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

分發 kubeconfig 到全部 master 節點:ssh

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152
  do
    echo ">>> ${node_ip}"
    scp kube-controller-manager.kubeconfig root@${node_ip}:/etc/kubernetes/
  done

建立和分發 kube-controller-manager systemd unit 文件

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
WorkingDirectory=${K8S_DIR}/kube-controller-manager
ExecStart=/opt/k8s/bin/kube-controller-manager \\
  --port=0 \\
  --secure-port=10252 \\
  --bind-address=127.0.0.1 \\
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
  --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
  --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\
  --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\
  --experimental-cluster-signing-duration=8760h \\
  --root-ca-file=/etc/kubernetes/cert/ca.pem \\
  --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\
  --leader-elect=true \\
  --controllers=*,bootstrapsigner,tokencleaner \\
  --horizontal-pod-autoscaler-use-rest-clients=true \\
  --horizontal-pod-autoscaler-sync-period=10s \\
  --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\
  --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\
  --use-service-account-credentials=true \\
  --kube-api-qps=1000 \\
  --kube-api-burst=2000 \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
  • --port=0:關閉監聽 http /metrics 的請求,同時 --address 參數無效,--bind-address 參數有效;
  • --secure-port=10252--bind-address=0.0.0.0: 在全部網絡接口監聽 10252 端口的 https /metrics 請求;
  • --kubeconfig:指定 kubeconfig 文件路徑,kube-controller-manager 使用它鏈接和驗證 kube-apiserver;
  • --authentication-kubeconfig 和 --authorization-kubeconfig:kube-controller-manager 使用它鏈接 apiserver,對 client 的請求進行認證和受權。kube-controller-manager 再也不使用 --tls-ca-file 對請求 https metrics 的 Client 證書進行校驗。若是沒有配置這兩個 kubeconfig 參數,則 client 鏈接 kube-controller-manager https 端口的請求會被拒絕(提示權限不足)。
  • --cluster-signing-*-file:簽名 TLS Bootstrap 建立的證書;
  • --experimental-cluster-signing-duration:指定 TLS Bootstrap 證書的有效期;
  • --root-ca-file:放置到容器 ServiceAccount 中的 CA 證書,用來對 kube-apiserver 的證書進行校驗;
  • --service-account-private-key-file:簽名 ServiceAccount 中 Token 的私鑰文件,必須和 kube-apiserver 的 --service-account-key-file 指定的公鑰文件配對使用;
  • --service-cluster-ip-range :指定 Service Cluster IP 網段,必須和 kube-apiserver 中的同名參數一致;
  • --leader-elect=true:集羣運行模式,啓用選舉功能;被選爲 leader 的節點負責處理工做,其它節點爲阻塞狀態;
  • --controllers=*,bootstrapsigner,tokencleaner:啓用的控制器列表,tokencleaner 用於自動清理過時的 Bootstrap token;
  • --horizontal-pod-autoscaler-*:custom metrics 相關參數,支持 autoscaling/v2alpha1;
  • --tls-cert-file--tls-private-key-file:使用 https 輸出 metrics 時使用的 Server 證書和祕鑰;
  • --use-service-account-credentials=true: kube-controller-manager 中各 controller 使用 serviceaccount 訪問 kube-apiserver;

完整 unit 見 kube-controller-manager.servicetcp

分發 systemd unit 文件到全部 master 節點:

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152
  do
    echo ">>> ${node_ip}"
    scp kube-controller-manager.service root@${node_ip}:/etc/systemd/system/
  done

kube-controller-manager 的權限

ClusteRole: system:kube-controller-manager 的權限很小,只能建立 secret、serviceaccount 等資源對象,各 controller 的權限分散到 ClusterRole system:controller:XXX 中。

須要在 kube-controller-manager 的啓動參數中添加 --use-service-account-credentials=true 參數,這樣 main controller 會爲各 controller 建立對應的 ServiceAccount XXX-controller。

內置的 ClusterRoleBinding system:controller:XXX 將賦予各 XXX-controller ServiceAccount 對應的 ClusterRole system:controller:XXX 權限。

啓動 kube-controller-manager 服務

source /opt/k8s/bin/environment.sh
for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152 do echo ">>> ${node_ip}" ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager" ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager" done
  • 必須先建立日誌目錄;

檢查服務運行狀態

source /opt/k8s/bin/environment.sh
for node_ip in 192.168.161.150 192.168.161.151 192.168.161.152
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-controller-manager|grep Active"
  done

確保狀態爲 active (running),不然查看日誌,確認緣由:

$ journalctl -u kube-controller-manager

 

 

查看輸出的 metric

注意:如下命令在 kube-controller-manager 節點上執行。

kube-controller-manager 監聽 10252 端口,接收 https 請求:

[root@k8s-master1 work]# sudo netstat -lnpt|grep kube-controll
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      19176/kube-controll 

 

 

測試 kube-controller-manager 集羣的高可用

停掉一個或兩個節點的 kube-controller-manager 服務,觀察其它節點的日誌,看是否獲取了 leader 權限。

 

查看當前的 leader

kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml


[root@k8s-master1 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master3_60147cc2-758e-11e9-8fcb-000c2956b313","leaseDurationSeconds":15,"acquireTime":"2019-05-13T14:49:56Z","renewTime":"2019-05-13T14:53:10Z","leaderTransitions":0}'
  creationTimestamp: 2019-05-13T14:49:56Z
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "5106"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
  uid: 6015837d-758e-11e9-89c6-000c2956b313
[root@k8s-master1 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master1_60ab86f0-758e-11e9-931c-000c29aa0a68","leaseDurationSeconds":15,"acquireTime":"2019-05-13T14:53:37Z","renewTime":"2019-05-13T14:53:39Z","leaderTransitions":1}'
  creationTimestamp: 2019-05-13T14:49:56Z
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "5116"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
  uid: 6015837d-758e-11e9-89c6-000c2956b313

 

可見,當前的 leader 從master3切到了master1 節點。

相關文章
相關標籤/搜索