K8S集羣tls證書管理

在前文介紹的k8s master高可用實踐方案中，須要對kube-apiserver的證書進行更新，加入VIP和從節點的IP，而後從新下發證書。回顧K8S集羣整個搭建過程當中，最容易讓人懵圈的也就是配置證書環節，所以本文對K8S集羣所用到的證書進行梳理一下。

1、根證書

ca.pem 根證書公鑰文件
ca-key.pem 根證書私鑰文件
ca.csr 證書籤名請求，用於交叉簽名或從新簽名
ca-config.json 使用cfssl工具生成其餘類型證書須要引用的配置文件
ca.pem用於簽發後續其餘的證書文件，所以ca.pem文件須要分發到集羣中的每臺服務器上去。

證書生成命令，默認生成的證書有效期5年

# echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -
# echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json

2、flannel證書

證書生成命令，默認生成的證書有效期5年

# cfssl gencert -ca=/etc/ssl/etcd/ca.pem  \
 -ca-key=/etc/ssl/etcd/ca-key.pem   \
-config=/etc/ssl/etcd/ca-config.json  \
 -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

這裏生成證書須要flanneld-csr.json文件

# cat flanneld-csr.json 
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "FuZhou",
      "L": "FuZhou",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

flannel啓動文件配置

# cat /usr/lib/systemd/system/flanneld.service 
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/usr/local/bin/flanneld \
  -etcd-cafile=/etc/ssl/etcd/ca.pem \
  -etcd-certfile=/etc/ssl/flanneld/flanneld.pem \
  -etcd-keyfile=/etc/ssl/flanneld/flanneld-key.pem \
  -etcd-endpoints=https://192.168.115.5,https://192.168.115.6:2379,https://192.168.115.7:2379 \
  -etcd-prefix=/kubernetes/network
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

3、etcd證書

一、服務端證書

server.pem etcd服務端證書公鑰文件
server-key.pem etcd服務端證書私鑰文件
server.csr 證書籤名請求

證書生成命令，默認生成的證書有效期5年

# export ADDRESS=192.168.115.5,192.168.115.6,192.168.115.7,vm1,vm2,vm3
# export NAME=server
# echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |  \
cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem \
-hostname="$ADDRESS" - | cfssljson -bare $NAME

server.pem、server-key.pem文件用來etcd集羣間通訊加解密，所以全部的etcd服務器都須要有這兩個文件

# tail -15 /usr/lib/systemd/system/etcd.service   
--initial-cluster-token=etcd-cluster-token \
--initial-cluster-state=new \
--cert-file=/etc/ssl/etcd/server.pem \
--key-file=/etc/ssl/etcd/server-key.pem \
--peer-cert-file=/etc/ssl/etcd/server.pem \
--peer-key-file=/etc/ssl/etcd/server-key.pem \
--trusted-ca-file=/etc/ssl/etcd/ca.pem \
--peer-trusted-ca-file=/etc/ssl/etcd/ca.pem \
--peer-client-cert-auth=true \
--client-cert-auth=true"
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

二、客戶端證書

client.pem etcd客戶端證書公鑰文件
client-key.pem etcd客戶端證書私鑰文件
client.csr 證書籤名請求

證書生成命令，默認生成的證書有效期5年

# export ADDRESS=
# export NAME=client
# echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | \
 cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem \
-hostname="$ADDRESS" - | cfssljson -bare $NAME

client.pem、client-key.pem文件給etcdctl客戶端用來和etcd服務器進行通訊，可根據實際須要進行配置。

# export ETCDCTL_API=3
# etcdctl --write-out=table  \
--cert=/etc/ssl/etcd/client.pem \
--key=/etc/ssl/etcd/client-key.pem \
--cacert=/etc/ssl/etcd/ca.pem \
--endpoints=https://192.168.115.5:2379,https://192.168.115.6:2379,https://192.168.115.7:2379 \
member list

4、Master節點證書

Kube-apiserver證書
Kubernetes.pem kube-apiserver證書公鑰文件
Kubernetes-key.pem kube-apiserver證書私鑰文件
kuberentes.csr kube-apiserver證書籤名請求

證書生成命令，默認生成的證書有效期5年

# cfssl gencert -ca=/etc/ssl/etcd/ca.pem \
  -ca-key=/etc/ssl/etcd/ca-key.pem \
  -config=/etc/ssl/etcd/ca-config.json \
  -profile=kubernetes k8s-csr.json | cfssljson -bare kubernetes

這裏生成證書須要k8s-csr.json文件，其中定義了master節點的IP列表等信息

# cat k8s-csr.json 
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.115.4",
    "192.168.115.5",
    "192.168.115.6",
    "192.168.115.7",
    "10.254.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "FuZhou",
      "L": "FuZhou",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

5、Node節點證書

一、kube-proxy證書

Kube-proxy.pem kube-proxy證書公鑰文件
Kube-proxy-key.pem kube-proxy證書私鑰文件
Kube-proxy.csr kube-proxy證書籤名請求

證書生成命令，默認生成的證書有效期5年

# cfssl gencert -ca=/etc/ssl/etcd/ca.pem \
  -ca-key=/etc/ssl/etcd/ca-key.pem \
  -config=/etc/ssl/etcd/ca-config.json \
  -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy

這裏生成證書須要kube-proxy-csr.json文件

# cat kube-proxy-csr.json 
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "FuZhou",
      "L": "FuZhou",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

二、Kubelet證書

Kubelet-client.crt: kubectl客戶端證書公鑰文件
Kubelet-client.key: kubectl客戶端私鑰文件
Kubelet.crt：kubelet服務端證書公鑰文件
Kubelet.key：kubelet服務端證書私鑰文件

kubelet-client.crt 文件在 kubelet 完成 TLS bootstrapping 後生成，有效期爲 1 年。此證書是由 controller manager 簽署的，此後 kubelet 將會加載該證書，用於與 apiserver 創建 TLS 通信，同時使用該證書的 CN 字段做爲用戶名，O 字段做爲用戶組向 apiserver 發起其餘請求。
kubelet.crt 文件在 kubelet 完成 TLS bootstrapping 後且沒有配置 --feature-gates=RotateKubeletServerCertificate=true 時纔會生成；該文件爲一個獨立於 apiserver CA 的自籤 CA 證書，有效期爲 1 年；被用做 kubelet 10250 api 端口

關於kubelet首次啓動 TLS bootstrapping的介紹（先有雞仍是先有蛋問題的解決方案）可參考文檔，https://mritd.me/2018/01/07/kubernetes-tls-bootstrapping-note/

6、配置證書自動續期

默認簽署kubectl客戶端和kubelet服務端證書只有 1 年有效期，若是想要調整證書有效期能夠經過設置 kube-controller-manager 的 --experimental-cluster-signing-duration 參數實現，該參數默認值爲 8760h0m0s。下面咱們來介紹一下如何實現證書到期的自動續簽。這個問題若是處理不當，證書過時以後會出現全部的node節點鏈接不上的狀況。

一、kcm服務，這裏爲了方便測試，過時時間修改成30分鐘

# egrep 'feature|experimental' /usr/lib/systemd/system/kube-controller-manager.service 
  --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \
  --experimental-cluster-signing-duration=30m0s \
# systemctl  daemon-reload
# systemctl restart kube-controller-manager

二、kubelet服務
配置完成刪掉Kubelet-client.crt、Kubelet-client.key、Kubelet.crt、Kubelet.key四個文件後重啓kubelet服務。

# egrep 'feature|rotate' /usr/lib/systemd/system/kubelet.service 
  --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \
  --rotate-certificates=true \
# systemctl  daemon-reload
# systemctl restart kubelet

三、手工簽發證書

# kubectl create clusterrolebinding kubelet-clinet \
--clusterrole=system:node  \
--user=system:anonymous

若是缺乏對system:anonymous用戶的受權，kubelet啓動的時候會報錯以下：
error: failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:anonymous" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope

# kubectl get csr
# kubectl certificate approve csr-fn946
# kubectl certificate approve csr-kwvg9

node節點將會從新生成kubectl客戶端和kubelet服務端證書

四、配置自動簽發證書，在大規模集羣下，這個配置是必須的

# cat rbac.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/selfnodeserver
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-autoapprove-certificate-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

# kubectl create -f  rbac.yaml

# kubectl create clusterrolebinding node-client-auto-approve-csr  \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient  \
--group=system:bootstrappers
# kubectl create clusterrolebinding node-client-auto-renew-crt \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient \
--group=system:nodes
# kubectl create clusterrolebinding node-server-auto-renew-crt  \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver  \
--group=system:nodes

這裏須要注意的是刪除掉Kubelet-client.crt、Kubelet-client.key兩個文件以後，啓動kubelet服務，首先會生成一個Kubelet-client.key文件，咱們須要對這個證書的crs請求進行approve，不然node節點沒法正常啓動。其次，若是kubelet.kubeconfig文件中配置的client-certificate、client-key目錄位置和kubelet的啓動參數--cert-dir不一致，則kubelet.kubeconfig文件中的配置文件會被自動更新。