K8S集羣證書已過時且etcd和apiserver已不能正常使用下的恢復方案

在這種比較極端的狀況下，要當心翼翼的規劃和操做，纔不會讓集羣完全死翹翹。首先，幾個ca根證書是10年期，應該尚未過時。咱們能夠基於這幾個根證書，來從新生成一套可用的各組件認證證書。

前期，先制定如下方案步驟，可否實現，待驗證。

一，製做證書的基本文件。

Ca-csr.json(由於根證書是OK的，因此這個文件，但是列在這裏，不會用上)

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "ca": {
    "expiry": "438000h"
  }
}

Ca-config.json(它用來從自簽名根ca.crt和ca.key生成新的證書，能夠共用)

{
  "signing": {
    "default": {
      "expiry": "43800h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "43800h"
      }
    }
  }
}

二，從新生成etcd系列證書((注意，這是依據/etc/kubernetes/pki/etcd/目錄下的ca證書)

Etcd-server.json

{
    "CN": "etcdServer",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "O": "etcd",
            "OU": "etcd Security",
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai"
        }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本機ip,小寫主機名 \
  -profile=kubernetes \
  etcd-server.json|cfssljson -bare server

etcd-peer.json

{
    "CN": "etcdPeer",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
        "O": "etcd",
        "OU": "etcd Security",
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai"
        }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本機ip,小寫主機名 \
  -profile=kubernetes \
  etcd-peer.json|cfssljson -bare peer

etcd-client.json

{
    "CN": "etcdClient",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
        "O": "etcd",
        "OU": "etcd Security",
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai"
        }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -profile=kubernetes \
  etcd-client.json |cfssljson -bare client

三，從新制做apiserver證書(注意，這是依據/etc/kubernetes/pki目錄下的ca證書)

Apiserver.json

{
    "CN": "kube-apiserver",
    "key": {
        "algo": "rsa",
        "size": 2048
    }
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1, kubernetes , kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local,本機ip,小寫主機名 \
  -profile=kubernetes \
  apiserver.json |cfssljson -bare apiserver

apiserver-kubelet-client.json

{
    "CN": "kube-apiserver-kubelet-client",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
        "O": "system:masters"
        }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -profile=kubernetes \
  apiserver-kubelet-client.json |cfssljson -bare apiserver-kubelet-client

三，從新制做front-proxy證書(注意，這是依據/etc/kubernetes/pki目錄下的front-proxy-ca證書，它必須和apiserver的ca不同，牽扯到apiserver的認證順序，切記)

Front-proxy-client.json

{
    "CN": "front-proxy-client",
    "key": {
        "algo": "rsa",
        "size": 2048
    }
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -profile=kubernetes \
  front-proxy-client.json |cfssljson -bare front-proxy-client

四，製做scheduler，controller-manager,admin,kubelet,bootstrap證書，此證書只存在於主節點。此證書主要用來生成controller-manager.conf, scheduler.conf, admin.conf, kubelet.conf bootstrap-kubelet.conf。

若是/etc/kubernetes/pki目錄下的sa.key,sa.pub存在，則無須更新，由於它沒有過時概念。

kube-scheduler-csr.json

{
    "CN": "system:kube-scheduler",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "O": "system:kube-scheduler",
      }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本機ip,小寫主機名 \
  -profile=kubernetes \
  kube-scheduler-csr.json|cfssljson -bare kube-scheduler

kube-controller-manager-csr.json

{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "O": "system:kube-controller-manager",
      }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本機ip,小寫主機名 \
  -profile=kubernetes \
  kube-controller-manager-csr.json |cfssljson -bare kube-controller-manager

admin-csr.json

{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "system:masters",
    }
  ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json |cfssljson -bare kube- admin

kubelet-csr.json(這個方法，只適合master上的kubelet運行，不用bootstrap的狀況)

{
  "CN": "system:node: 小寫主機名",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "system:nodes",
    }
  ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本機ip,小寫主機名 \
  -profile=kubernetes \
  kubelet-csr.json |cfssljson -bare kubelet

若是還須要bootstrap,能夠參考下面的url:

https://k2r2bai.com/2018/07/17/kubernetes/deploy/manual-install/

https://www.jianshu.com/p/6650954fa973?tdsourcetag=s_pctim_aiomsg

五，以上文件做好以後，須要根據如今的k8s命令規則更名，還要根據不一樣的文件，存放於不一樣的目錄。

六，這時，k8s master應該能夠啓動了。接下來，製做kubeconfig文件，參考url

http://www.javashuo.com/article/p-yqpeysxl-kq.html(配置bootstrap及kubelet認證)

http://www.javashuo.com/article/p-ylrsbnsn-kq.html（配置.kube/config文件）

# 設置集羣參數

kubectl config set-cluster

# 設置客戶端認證參數

kubectl config set-credentials

# 設置上下文參數

kubectl config set-context

# 設置默認上下文

kubectl config use-context

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/cert/ca.pem \
  --embed-certs=true \
  --server=https://ip:port \
  --kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=kube-controller-manager.pem \
  --client-key=kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context system:kube-controller-manager \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/cert/ca.pem \
  --embed-certs=true \
  --server=https://ip:port \
  --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \
  --client-certificate=kube-scheduler.pem \
  --client-key=kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context system:kube-scheduler \
  --cluster=kubernetes \
  --user=system:kube-scheduler \
  --kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-cluster kubernetes \
    --certificate-authority=${PKI_DIR}/ca.pem \
    --embed-certs=true \
    --server=https://ip:port \
    --kubeconfig=${K8S_DIR}/admin.conf

kubectl config set-credentials kubernetes-admin \
    --client-certificate=${PKI_DIR}/admin.pem \
    --client-key=${PKI_DIR}/admin-key.pem \
    --embed-certs=true \
    --kubeconfig=${K8S_DIR}/admin.conf

kubectl config set-context kubernetes-admin@kubernetes \
    --cluster=kubernetes \
    --user=kubernetes-admin \
    --kubeconfig=${K8S_DIR}/admin.conf

kubectl config use-context kubernetes-admin@kubernetes \
    --kubeconfig=${K8S_DIR}/admin.conf

kubectl config set-cluster kubernetes \
  --certificate-authority=${PKI_DIR}/ca.pem \
  --embed-certs=true \
  --server=https://ip:port \
  --kubeconfig=${K8S_DIR}/kubelet.conf && \
kubectl config set-credentials system:node:小寫主機名 \
  --client-certificate=${PKI_DIR}/kubelet.pem \
  --client-key=${PKI_DIR}/kubelet-key.pem \
  --embed-certs=true \
  --kubeconfig=${K8S_DIR}/kubelet.conf && \
kubectl config set-context system:node:小寫主機名@kubernetes \
  --cluster=kubernetes \
  --user=system:node:小寫主機名 \
  --kubeconfig=${K8S_DIR}/kubelet.conf && \
kubectl config use-context system:node:小寫主機名@kubernetes \
  --kubeconfig=${K8S_DIR}/kubelet.conf

七，當製做好這些文件以後，按k8s安裝的位置，分發文件，重啓kubelet，應該就能夠從新啓動好集羣了。