etcd集羣搭建

時間  2019-11-12
標籤 etcd 集羣 搭建 欄目 負載均衡 简体版
原文   原文鏈接

主機

kubermaster1 192.168.4.11
kubermaster2 192.168.4.12
kubermaster3 192.163.4.13

系統

[root@kubermaster1 etcd-v3.2.11-linux-amd64]# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core)

TLS密鑰和證書

這裏部署的etcd集羣使用TLS證書對證書通訊進行加密，並開啓基於CA根證書籤名的雙向數字證書認證。node

安裝go語言組件

登陸 https://golang.org/dl/ 找到最新版的go並下載

cd /usr/local/src
wget http://redirector.gvt1.com/edgedl/go/go1.9.2.linux-amd64.tar.gz
tar -xvf go1.9.2.linux-amd64.tar.gz -C /usr/local

配置go環境

cat >> /etc/profile << EOF
#go的安裝路徑
export GOROOT=/usr/local/go
#go安裝的工具路徑
export GOPATH=/apps/local/go
export PATH=$GOROOT/bin:$PATH
EOF
source /etc/profile

GOPATH和GOROOT不能相同linux

配置生效git

[root@kubermaster2 bin]# go version
go version go1.9.2 linux/amd64

安裝cfssl

將會用使用cfssl生成所須要的私鑰和證書github

go get -u github.com/cloudflare/cfssl/cmd/...

會在$GOPATH/bin下安裝cfssl, cfssjosn, mkbundle等工具。golang

建立CA證書和私鑰，準備爲etcd和其它組件辦法證書和簽名

建立ca-config.json

{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "aspire": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

ca-config.json中能夠定義多個profile，分別設置不一樣的expiry和usages等參數。如上面的ca-config.json中定義了名稱爲aspire的profile，這個profile的expiry 87600h爲10年，useages中：shell

  • signing表示此CA證書能夠用於簽名其餘證書，ca.pem中的CA=TRUE
  • server auth表示TLS Server Authentication
  • client auth表示TLS Client Authentication

建立CA證書籤名請求配置ca-csr.json

{
  "CN": "aspire",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "aspire",
      "OU": "cloudnative"
    }
  ]
}

生成CA證書和私鑰json

cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

生成etcd證書和私鑰

建立etcd證書籤名請求配置etcd-csr.jsonapi

{
    "CN": "aspire.etcd",
    "hosts": [
      "127.0.0.1",
      "192.168.4.11",
      "192.168.4.12",
      "192.168.4.13",
      "kubermaster1",
      "kubermaster2",
      "kubermaster3"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "aspire.etcd",
            "OU": "Operation and maintenance center"
        }
    ]
}

該"hosts"是能夠使用該證書域名列表。‘CN’，kube-apiserver從證書中提取該字段做爲請求的用戶名 (User Name)；瀏覽器使用該字段驗證網站是否合法；瀏覽器

該"names"值其實是名稱對象的列表。每一個名稱對象應至少包含一個「C」，「L」，「O」，「OU」或「ST」值（或這些的任意組合）。這些值是：app

  • 「C」：國家
  • 「L」：地區或城市（如城市或城鎮名稱）
  • 「O」：組織 Organization，kube-apiserver從證書中提取該字段做爲請求用戶所屬的組 (Group)；
  • 「OU」：組織單位，如負責擁有密鑰的部門; 它也能夠用於「作生意」（DBS）的名稱
  • 「ST」：州或省

下面生成etcd的證書和私鑰：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=aspire etcd-csr.json | cfssljson -bare etcd

對生成的證書能夠使用cfssl或openssl查看

$ cfssl-certinfo -cert etcd.pem
{
  "subject": {
   ...
      "cloudnative",
      "aspire"
    ]
  },
  "serial_number": "555738010691550377350124675225187029254417657480",
  "sans": [
    "kubermaster1",
    "kubermaster2",
    "kubermaster3",
    "127.0.0.1",
    "192.168.4.11",
    "192.168.4.12",
    "192.168.4.13"
  ],
  "not_before": "2017-12-18T06:57:00Z",
  "not_after": "2027-12-16T06:57:00Z",
  "sigalg": "SHA256WithRSA",
  "authority_key_id": "DB:5D:58:25:31:D5:2A:D8:DB:C1:EF:C4:68:B4:B0:13:FA:6B:42:C3",
  "subject_key_id": "6D:9B:6E:6A:F8:40:4D:4C:03:A4:0F:05:58:E1:9A:72:2E:8E:AB:58",
  "pem": "-----BEGIN CERTIFICATE-----\nMIIETjCCAzagAwIBAgIUYVgnfkNJEfm75Tye3fynwTrvrogwDQYJKoZIhvcNAQEL\nBQAwaTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDA...
  "

}

將生成的CA證書ca.pem, etcd祕鑰etcd-key.pem, etcd證書etcd.pem拷貝到各節點的/etc/etcd/ssl目錄中

安裝etcd

下載安裝包

訪問github https://github.com/coreos/etcd/releases 找到最新安裝包並下載

cd /usr/local/src
wget https://github.com/coreos/etcd/releases/download/v3.2.11/etcd-v3.2.11-linux-amd64.tar.gz

解壓縮etcd-v3.2.11-linux-amd64.tar.gz，將其中的etcd和etcdctl兩個可執行文件複製到各節點的/usr/bin目錄。

在各節點建立etcd的數據目錄：

mkdir -p /var/lib/etcd

在每一個節點上建立etcd的systemd unit文件/usr/lib/systemd/system/etcd.service，注意替換ETCD_NAME和INTERNAL_IP變量的值：

export ETCD_NAME=kubermaster3
export INTERNAL_IP=192.168.4.13
cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
  --name ${ETCD_NAME} \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://${INTERNAL_IP}:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster node1=https://192.168.4.11:2380,node2=https://192.168.4.12:2380,node3=https://192.168.4.13:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

上面在啓動參數中指定了etcd的工做目錄和數據目錄是/var/lib/etcd

  • --cert-file和--key-file分別指定etcd的公鑰證書和私鑰
  • --peer-cert-file和--peer-key-file分別指定了etcd的Peers通訊的公鑰證書和私鑰。
  • --trusted-ca-file指定了客戶端的CA證書
  • --peer-trusted-ca-file指定了Peers的CA證書
  • --initial-cluster-state new表示這是新初始化集羣，--name指定的參數值必須在--initial-cluster中

啓動etcd

在各節點啓動etcd

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

檢查集羣是否健康

etcdctl \
  --ca-file=/etc/etcd/ssl/ca.pem \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --endpoints=https://node1:2379,https://node2:2379,https://node3:2379 \
  cluster-health
  
2017-04-24 19:53:40.545148 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-04-24 19:53:40.546127 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 4f2f99d70000fc19 is healthy: got healthy result from https://192.168.61.12:2379
member 99a756f799eb4163 is healthy: got healthy result from https://192.168.61.11:2379
member a9aff19397de2e4e is healthy: got healthy result from https://192.168.61.13:2379
cluster is healthy
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程