K8S從入門到放棄系列-(3)部署etcd集羣

摘要：etcd 是k8s集羣最重要的組件，用來存儲k8s的全部服務信息， etcd 掛了，集羣就掛了，咱們這裏把etcd部署在master三臺節點上作高可用，etcd集羣採用raft算法選舉Leader， 因爲Raft算法在作決策時須要多數節點的投票，因此etcd通常部署集羣推薦奇數個節點，推薦的數量爲三、5或者7個節點構成一個集羣。

官方地址 https://github.com/coreos/etcd/releases

1）下載etcd二進制文件

etcd命令爲下載的二進制文件，解壓後複製到指定目錄便可

[root@k8s-master01 ~]# cd k8s/
[root@k8s-master01 k8s]# wget https://github.com/etcd-io/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz
[root@k8s-master01 k8s]# tar -xf etcd-v3.3.12-linux-amd64.tar.gz  
[root@k8s-master01 k8s]# cd etcd-v3.3.12-linux-amd64  ##有2個文件，etcdctl是操做etcd的命令
##把etcd二進制文件傳輸到三個master節點
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/root/k8s/etcd-v3.3.12-linux-amd64/etcd dest=/usr/local/bin/ mode=0755'
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/root/k8s/etcd-v3.3.12-linux-amd64/etcdctl dest=/usr/local/bin/ mode=0755'
說明：如果不用ansible，能夠直接用scp把兩個文件傳輸到三個master節點的/usr/local/bin/目錄下

2）建立etcd證書請求模板文件

[root@k8s-master01 ~]# vim /opt/k8s/certs/etcd-csr.json  ##證書請求文件
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.10.0.18",
    "10.10.0.19",
    "10.10.0.20"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "ShangHai",
      "L": "ShangHai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
說明：hosts中的IP爲各etcd節點IP及本地127地址，在生產環境中hosts列表最好多預留幾個IP，這樣後續擴展節點或者因故障須要遷移時不須要再從新生成證書。(我生產環境使用阿里雲VPC網絡，因此會預留指定段的IP)

3）生成證書及私鑰

注意命令中使用的證書的具體位置

[root@k8s-master01 ~]# cd /opt/k8s/certs/
[root@k8s-master01 certs]# cfssl gencert -ca=/opt/k8s/certs/ca.pem \
     -ca-key=/opt/k8s/certs/ca-key.pem \
     -config=/opt/k8s/certs/ca-config.json \
     -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
2019/04/22 17:17:51 [INFO] generate received request
2019/04/22 17:17:51 [INFO] received CSR
2019/04/22 17:17:51 [INFO] generating key: rsa-2048
2019/04/22 17:17:51 [INFO] encoded CSR
2019/04/22 17:17:51 [INFO] signed certificate with serial number 335217685822754469090490767964903486042452749906
2019/04/22 17:17:51 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

4）查看證書

etcd.csr是簽署時用到的中間文件，若是你不打算本身簽署證書，而是讓第三方的CA機構簽署，只須要把etcd.csr文件提交給CA機構。

[root@k8s-master01 certs]# ll etcd*
-rw-r--r--. 1 root root 1066 Apr 22 17:17 etcd.csr
-rw-r--r--. 1 root root  293 Apr 22 17:10 etcd-csr.json
-rw-------. 1 root root 1679 Apr 22 17:17 etcd-key.pem
-rw-r--r--. 1 root root 1444 Apr 22 17:17 etcd.pem

5）證書分發

把生成的etcd證書複製到建立的證書目錄並放至另2臺etcd節點

正常狀況下只須要copy這三個文件便可，ca.pem(已經存在)、etcd-key.pem、etcd.pem

[root@k8s-master01 certs]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/etcd.pem dest=/etc/kubernetes/ssl/'
[root@k8s-master01 certs]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/etcd-key.pem dest=/etc/kubernetes/ssl/'

6）修改etcd配置參數

爲了安全性起我這裏使用單獨的用戶啓動 Etcd

##建立etcd用戶和組
[root@k8s-master01 ~]# ansible k8s-master -m group -a 'name=etcd'
[root@k8s-master01 ~]# ansible k8s-master -m user -a 'name=etcd group=etcd comment="etcd user" shell=/sbin/nologin home=/var/lib/etcd createhome=no'
##建立etcd數據存放目錄並受權
[root@k8s-master01 ~]# ansible k8s-master -m file -a 'path=/var/lib/etcd state=directory owner=etcd group=etcd'

說明：

以上步驟如果感受比較麻煩，能夠直接在對應三臺master主機執行如下命令便可

mkdir /etc/kubernetes/config

groupadd -r etcd

useradd -r -g etcd -d /var/lib/etcd -s /sbin/nologin -c "etcd user" etcd

mkdir /var/lib/etcd/

chown -R etcd:etcd /var/lib/etcd/

7）配置etcd配置文件

etcd.conf配置文件信息，配置文件中涉及證書，etcd用戶須要對其有可讀權限，不然會提示沒法獲取證書，644權限便可。

[root@k8s-master01 ~]# vim /opt/k8s/cfg/etcd.conf
#[member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://10.10.0.18:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.10.0.18:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
ETCD_AUTO_COMPACTION_RETENTION="1"
ETCD_QUOTA_BACKEND_BYTES="8589934592"
ETCD_MAX_REQUEST_BYTES="5242880"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.0.18:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd01=https://10.10.0.18:2380,etcd02=https://10.10.0.19:2380,etcd03=https://10.10.0.20:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://10.10.0.18:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/etc/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/etc/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/etc/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/etc/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/kubernetes/ssl/etcd-key.pem"

參數解釋：

1. ETCD_NAME：etcd節點成員名稱，在一個etcd集羣中必須惟一性，可以使用Hostname或者machine-id
2. ETCD_LISTEN_PEER_URLS：和其它成員節點間通訊地址，每一個節點不一樣，必須使用IP，使用域名無效
3. ETCD_LISTEN_CLIENT_URLS：對外提供服務的地址，一般爲本機節點。使用域名無效
4. ETCD_INITIAL_ADVERTISE_PEER_URLS：節點監聽地址，並會通告集羣其它節點
5. ETCD_INITIAL_CLUSTER：集羣中全部節點信息，格式爲：節點名稱+監聽的本地端口，及：ETCD_NAME：https://ETCD_INITIAL_ADVERTISE_PEER_URLS
6. ETCD_ADVERTISE_CLIENT_URLS：節點成員客戶端url列表，對外公告此節點客戶端監聽地址，可使用域名
7. ETCD_AUTO_COMPACTION_RETENTION:  在一個小時內爲mvcc鍵值存儲的自動壓實保留。0表示禁用自動壓縮
8. ETCD_QUOTA_BACKEND_BYTES: 事務中容許的最大操做數,默認1.5M,官方推薦10M，我這裏設置5M，你們根據本身實際業務設置
9. ETCD_MAX_REQUEST_BYTES: ETCDdb存儲數據大小，默認2G，推薦8G

       因爲咱們是三個節點etcd集羣，因此須要把etcd.conf配置文件複製到另外2個節點，並把上面參數解釋中紅色參數修改成對應主機IP。

 分發etcd.conf配置文件，固然你不用ansible，能夠直接用scp命令把配置文件傳輸到三臺機器對應位置，而後三臺機器分別修改IP、ETCD_NAME等參數。 

[root@k8s-master01 config]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/etcd.conf dest=/etc/kubernetes/config/etcd.conf'
##登錄對應主機修改配置文件，把對應IP修改成本地IP

編輯etcd.service 啓動文件 

[root@k8s-master01 ~]# vim /opt/k8s/unit/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/kubernetes/config/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/unit/etcd.service dest=/usr/lib/systemd/system/etcd.service'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl daemon-reload'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl enable etcd'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl start etcd'

注：

這裏須要三臺etcd服務同時啓動，在三臺機器上同時執行啓動命令，啓動其中一臺後，服務會卡在那裏，直到集羣中全部etcd節點都已啓動。我這裏由於是ansible遠程執行，因此沒有出現這個問題。

 8）驗證集羣

etcd3版本，查看集羣狀態時，須要指定對應的證書位置  

[root@k8s-master01 ~]# etcdctl --endpoints=https://10.10.0.18:2379,https://10.10.0.19:2379,https://10.10.0.20:2379 \
           --cert-file=/etc/kubernetes/ssl/etcd.pem \
           --ca-file=/etc/kubernetes/ssl/ca.pem \
           --key-file=/etc/kubernetes/ssl/etcd-key.pem \
           cluster-health
member 804ed05b4beec304 is healthy: got healthy result from https://10.10.0.20:2379
member 8a5b84381bee52dd is healthy: got healthy result from https://10.10.0.19:2379
member caba783185460428 is healthy: got healthy result from https://10.10.0.18:2379
cluster is healthy
[root@k8s-master01 ~]# etcdctl --endpoints=https://10.10.0.18:2379,https://10.10.0.19:2379,https://10.10.0.20:2379 \
           --cert-file=/etc/kubernetes/ssl/etcd.pem \
           --ca-file=/etc/kubernetes/ssl/ca.pem \
           --key-file=/etc/kubernetes/ssl/etcd-key.pem \
           member list
804ed05b4beec304: name=etcd03 peerURLs=https://10.10.0.20:2380 clientURLs=https://10.10.0.20:2379 isLeader=false
8a5b84381bee52dd: name=etcd02 peerURLs=https://10.10.0.19:2380 clientURLs=https://10.10.0.19:2379 isLeader=true
caba783185460428: name=etcd01 peerURLs=https://10.10.0.18:2380 clientURLs=https://10.10.0.18:2379 isLeader=false
## 能夠看到集羣顯示健康，並能夠看到isLeader=true 所在節點