分佈式kv存儲系統之Etcd集羣
etcd是什麼？html
etcd是一個高可用的分佈式鍵值數據庫，可用於服務發現，etcd採用 raft 一致性算法，基於 Go 語言實現。其特色有簡單易用，所謂簡單易用是指安裝配置簡單，提供http/https接口；安全，安全是指etcd支持ssl證書認證，支持集羣各節點間使用對等證書認證；客戶端和服務端的雙向證書認證；可靠，可靠是指etcd使用raft協議實現分佈式系統數據的可用性和一致性；etcd主要有兩個版本v2和v3；v2和v3的api是互不兼容的，因此咱們在同一服務器上安裝多個版本的etcd時，咱們須要用ETCDCTL_API這個環境變量指定；node
etcd集羣部署linux
環境準備git
| 主機名稱 | ip地址 |
| master01.k8s.org | 192.168.0.41 |
| master02.k8s.org | 192.168.0.42 |
| master03.k8s.org | 192.168.0.43 |
各主機hosts文件解析github
[root@master01 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.0.99 time.test.org time-node 192.168.0.41 master01 master01.k8s.org etcd01 etcd01.k8s.org 192.168.0.42 master02 master02.k8s.org etcd02 etcd02.k8s.org 192.168.0.43 master03 master03.k8s.org etcd03 etcd03.k8s.org 192.168.0.44 node01 node01.k8s.org 192.168.0.45 node02 node02.k8s.org 192.168.0.46 node03 node03.k8s.org [root@master01 ~]#
關閉各主機的firewalld服務算法
[root@master01 ~]# systemctl stop firewalld
[root@master01 ~]# systemctl disable firewalld
[root@master01 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@master01 ~]#
各主機間時間同步數據庫
[root@master01 ~]# grep server /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
server time.test.org iburst
# Serve time even if not synchronized to any NTP server.
[root@master01 ~]# systemctl restart chronyd.service
[root@master01 ~]# systemctl status chronyd.service
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2021-01-30 15:41:25 CST; 11s ago
Docs: man:chronyd(8)
man:chrony.conf(5)
Process: 1411 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 1407 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 1409 (chronyd)
CGroup: /system.slice/chronyd.service
└─1409 /usr/sbin/chronyd
Jan 30 15:41:25 master01.k8s.org systemd[1]: Stopped NTP client/server.
Jan 30 15:41:25 master01.k8s.org systemd[1]: Starting NTP client/server...
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SI...+DEBUG)
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: commandkey directive is no longer supported
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: generatecommandkey directive is no longer supported
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: Frequency -25.600 +/- 2.450 ppm read from /var/lib/chrony/drift
Jan 30 15:41:25 master01.k8s.org systemd[1]: Started NTP client/server.
Jan 30 15:41:29 master01.k8s.org chronyd[1409]: Selected source 192.168.0.99
Hint: Some lines were ellipsized, use -l to show in full.
[root@master01 ~]#
提示：集羣內部可使用本身搭建的時間服務，把chrony.conf中的server 指向對應時間服務器，而後重啓chronyd便可；固然也可使用互聯網上公有的時間服務器；總之一個服務以集羣方式工做，其時間同步是很是重要；vim
各主機間ssh 互信centos
[root@master01 ~]# ssh master02 Last login: Sat Jan 30 15:34:33 2021 from master01 [root@master02 ~]# exit logout Connection to master02 closed. [root@master01 ~]# ssh master03 Last login: Sat Jan 30 15:34:37 2021 from master01 [root@master03 ~]# exit logout Connection to master03 closed. [root@master01 ~]#
提示：有關ssh互信的配置請參考本人博客：http://www.javashuo.com/article/p-rjibahxc-bk.html；各主機間實現ssh互信，其主要目的是方便各組件同步文件；作好以上準備之後，咱們就能夠下載etcd二進制包進行etcd集羣部署；這裏須要說明一下，在centos7上的extras倉庫中有etcd的rpm包，咱們可使用yum來安裝；可是extras倉庫中的版本不是最新的，要想使用最新的就須要到官方github倉庫中下載最新版本的etcd二進制包進行部署；兩種部署方式沒有什麼特別的不一樣；若是對版本要求不是特別新的環境中，建議使用yum安裝；api
下載etcd二進制包
[root@master01 ~]#wget https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz --2021-01-30 15:46:18-- https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz Resolving github.com (github.com)... 52.192.72.89 Connecting to github.com (github.com)|52.192.72.89|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-releases.githubusercontent.com/11225014/ad6a1d80-2f1a-11eb-8cb8-2f1ae35d5487?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210130T074619Z&X-Amz-Expires=300&X-Amz-Signature=47569782ddb8a1f70fbd28350433d3a045d22f040dd95b7de1055c96e7b4c359&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=11225014&response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.14-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following] --2021-01-30 15:46:19-- https://github-releases.githubusercontent.com/11225014/ad6a1d80-2f1a-11eb-8cb8-2f1ae35d5487?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210130T074619Z&X-Amz-Expires=300&X-Amz-Signature=47569782ddb8a1f70fbd28350433d3a045d22f040dd95b7de1055c96e7b4c359&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=11225014&response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.14-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.111.154, 185.199.109.154, 185.199.108.154, ... Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.111.154|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 17373058 (17M) [application/octet-stream] Saving to: ‘etcd-v3.4.14-linux-amd64.tar.gz’ 100%[=================================================================================================>] 17,373,058 24.9MB/s in 0.7s 2021-01-30 15:46:20 (24.9 MB/s) - ‘etcd-v3.4.14-linux-amd64.tar.gz’ saved [17373058/17373058] [root@master01 ~]#
解壓etcd二進制包
[root@master01 ~]# ls etcd-v3.4.14-linux-amd64.tar.gz [root@master01 ~]# tar xf etcd-v3.4.14-linux-amd64.tar.gz -C /usr/local/src/ [root@master01 ~]# cd /usr/local/src/ [root@master01 src]# ls etcd-v3.4.14-linux-amd64 [root@master01 src]# cd etcd-v3.4.14-linux-amd64/ [root@master01 etcd-v3.4.14-linux-amd64]# ls Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md [root@master01 etcd-v3.4.14-linux-amd64]#
把etcd和etcdctl軟鏈接至path環境變量下
[root@master01 etcd-v3.4.14-linux-amd64]# ls Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md [root@master01 etcd-v3.4.14-linux-amd64]# ln -s /usr/local/src/etcd-v3.4.14-linux-amd64/etcd /usr/bin/ [root@master01 etcd-v3.4.14-linux-amd64]# ln -s /usr/local/src/etcd-v3.4.14-linux-amd64/etcdctl /usr/bin/ [root@master01 etcd-v3.4.14-linux-amd64]# ll /usr/bin/etcd lrwxrwxrwx 1 root root 44 Jan 30 15:59 /usr/bin/etcd -> /usr/local/src/etcd-v3.4.14-linux-amd64/etcd [root@master01 etcd-v3.4.14-linux-amd64]# ll /usr/bin/etcdctl lrwxrwxrwx 1 root root 47 Jan 30 15:59 /usr/bin/etcdctl -> /usr/local/src/etcd-v3.4.14-linux-amd64/etcdctl [root@master01 etcd-v3.4.14-linux-amd64]#
編寫etcd.service unit文件
[root@master01 ~]# cat /usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target [Service] Type=simple WorkingDirectory=/var/lib/etcd EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd" Type=notify [Install] WantedBy=multi-user.target [root@master01 ~]#
提供etcd環境變量加載文件/etc/etcd/etcd.conf文件
[root@master01 ~]# mkdir /etc/etcd/ [root@master01 ~]# cd /etc/etcd/ [root@master01 etcd]# vim etcd.conf #[Member] #ETCD_CORS="" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_WAL_DIR="" #ETCD_LISTEN_PEER_URLS="http://localhost:2380" ETCD_LISTEN_CLIENT_URLS="http://localhost:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" ETCD_NAME="default" #ETCD_SNAPSHOT_COUNT="100000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD_QUOTA_BACKEND_BYTES="0" #ETCD_MAX_REQUEST_BYTES="1572864" #ETCD_GRPC_KEEPALIVE_MIN_TIME="5s" #ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s" #ETCD_GRPC_KEEPALIVE_TIMEOUT="20s" # #[Clustering] #ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380" ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379" #ETCD_DISCOVERY="" #ETCD_DISCOVERY_FALLBACK="proxy" #ETCD_DISCOVERY_PROXY="" #ETCD_DISCOVERY_SRV="" #ETCD_INITIAL_CLUSTER="default=http://localhost:2380" #ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #ETCD_INITIAL_CLUSTER_STATE="new" #ETCD_STRICT_RECONFIG_CHECK="true" #ETCD_ENABLE_V2="true" # #[Proxy] #ETCD_PROXY="off" #ETCD_PROXY_FAILURE_WAIT="5000" #ETCD_PROXY_REFRESH_INTERVAL="30000" #ETCD_PROXY_DIAL_TIMEOUT="1000" #ETCD_PROXY_WRITE_TIMEOUT="5000" #ETCD_PROXY_READ_TIMEOUT="0" # #[Security] #ETCD_CERT_FILE="" #ETCD_KEY_FILE="" #ETCD_CLIENT_CERT_AUTH="false" #ETCD_TRUSTED_CA_FILE="" #ETCD_AUTO_TLS="false" #ETCD_PEER_CERT_FILE="" #ETCD_PEER_KEY_FILE="" #ETCD_PEER_CLIENT_CERT_AUTH="false" #ETCD_PEER_TRUSTED_CA_FILE="" #ETCD_PEER_AUTO_TLS="false" # #[Logging] #ETCD_DEBUG="false" #ETCD_LOG_PACKAGE_LEVELS="" #ETCD_LOG_OUTPUT="default" # #[Unsafe] #ETCD_FORCE_NEW_CLUSTER="false" # #[Version] #ETCD_VERSION="false" #ETCD_AUTO_COMPACTION_RETENTION="0" # #[Profiling] #ETCD_ENABLE_PPROF="false" #ETCD_METRICS="basic" # #[Auth] #ETCD_AUTH_TOKEN="simple" "etcd.conf" [New] 69L, 1686C written [root@master01 etcd]#
更改etcd.conf文件
提示：ETCD_DATA_DIR用於指定etcd數據目錄；ETCD_LISTEN_PEER_URLS用於指定集羣節點通訊監聽的url地址；ETCD_LISTEN_CLIENT_URLS用戶指定客戶端鏈接使用時的url地址；ETCD_NAME用戶指定當前節點etcd實例的名稱；ETCD_INITIAL_ADVERTISE_PEER_URLS用於指定集羣事務通告的url地址；ETCD_ADVERTISE_CLIENT_URLS用戶指定客戶端事務通告url地址；ETCD_INITIAL_CLUSTER用戶指定集羣成員，一個成員由成員名稱=對應集羣間通訊的url地址，多個成員用逗號隔開；
建立/var/lib/etcd目錄
[root@master01 etcd]# mkdir /var/lib/etcd/ [root@master01 etcd]# ll -d /var/lib/etcd/ drwxr-xr-x 2 root root 6 Jan 30 16:20 /var/lib/etcd/ [root@master01 etcd]#
複製master01上的/usr/bin/etcd和etcdctl二進制文件到master02和master03的/usr/bin/目錄下
[root@master01 etcd]# scp /usr/bin/etcd /usr/bin/etcdctl master02:/usr/bin/ etcd 100% 23MB 43.6MB/s 00:00 etcdctl 100% 17MB 49.1MB/s 00:00 [root@master01 etcd]# scp /usr/bin/etcd /usr/bin/etcdctl master03:/usr/bin/ etcd 100% 23MB 42.2MB/s 00:00 etcdctl 100% 17MB 56.8MB/s 00:00 [root@master01 etcd]#
複製master01上的etcd.service到master02和master03的/usr/lib/systemd/system目錄下
[root@master01 etcd]# scp /usr/lib/systemd/system/etcd.service master02:/usr/lib/systemd/system/ etcd.service 100% 417 165.1KB/s 00:00 [root@master01 etcd]# scp /usr/lib/systemd/system/etcd.service master03:/usr/lib/systemd/system/ etcd.service 100% 417 175.7KB/s 00:00 [root@master01 etcd]#
在master02和master03上建立/etc/etcd/目錄和/var/lib/etcd/目錄
[root@master01 etcd]# ssh master02 'mkdir /etc/etcd/ && mkdir /var/lib/etcd' [root@master01 etcd]# ssh master03 'mkdir /etc/etcd/ && mkdir /var/lib/etcd' [root@master01 etcd]#
複製master01上的etcd.conf文件到master02和master03的/etc/etcd/目錄下
[root@master01 etcd]# scp /etc/etcd/etcd.conf master02:/etc/etcd/ etcd.conf 100% 1749 743.3KB/s 00:00 [root@master01 etcd]# scp /etc/etcd/etcd.conf master03:/etc/etcd/ etcd.conf 100% 1749 824.2KB/s 00:00 [root@master01 etcd]#
修改master02上的/etc/etcd/etcd.conf文件
修改master03上的/etc/etcd/etcd.conf文件
到此三個節點的配置文件和相關用戶以及目錄都準備就緒，接下來咱們要重載systemd的配置文件，加載etcd.service文件
[root@master01 ~]# systemctl daemon-reload [root@master01 ~]# ssh master02 'systemctl daemon-reload' [root@master01 ~]# ssh master03 'systemctl daemon-reload' [root@master01 ~]#
啓動etcd
[root@master01 ~]# systemctl start etcd [root@master01 ~]#
提示：在每一個節點上運行上述命令啓動etcd；第一個啓動的節點將阻塞，緣由是etcd以集羣方式工做，它必需要有足夠的得票才能正常工做，若是集羣節點有3個，那麼至少有兩個節點正常啓動etcd才能正常工做；
驗證：查看各節點的2379和2380端口是否都處於監聽？
[root@master01 ~]# ss -tnl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 192.168.0.41:2379 *:* LISTEN 0 128 192.168.0.41:2380 *:* LISTEN 0 128 *:22 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::* [root@master01 ~]# ssh master02 'ss -tnl' State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 192.168.0.42:2379 *:* LISTEN 0 128 192.168.0.42:2380 *:* LISTEN 0 128 *:22 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::* [root@master01 ~]# ssh master03 'ss -tnl' State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 192.168.0.43:2379 *:* LISTEN 0 128 192.168.0.43:2380 *:* LISTEN 0 128 *:22 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::* [root@master01 ~]#
驗證：使用etcdctl查看集羣狀態
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 endpoint status 192.168.0.41:2379, b8b747c74aaea686, 3.4.14, 4.5 MB, true, false, 13, 2163, 2163, 192.168.0.42:2379, b3504381e8ba3cb, 3.4.14, 4.5 MB, false, false, 13, 2163, 2163, 192.168.0.43:2379, f572fdfc5cb68406, 3.4.14, 4.5 MB, false, false, 13, 2163, 2163, [root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 member list b3504381e8ba3cb, started, etcd02, http://etcd02:2380, http://etcd02:2379, false b8b747c74aaea686, started, etcd01, http://etcd01:2380, http://etcd01:2379, false f572fdfc5cb68406, started, etcd03, http://etcd03:2380, http://etcd03:2379, false [root@master01 ~]#
提示：可以列出集羣成員和查看集羣成員狀態，就表示etcd集羣工做已經正常；
驗證：向etcd任意節點寫數據，看看是否可以正常寫？
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 put name "test" OK [root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 get name name test [root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 del name 1 [root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 get name [root@master01 ~]#
提示：使用etcdctl工具能夠正常向etcd集羣寫入數據；
爲etcd集羣生成證書
在某一節點上安裝git工具
[root@master01 ~]# yum install git -y
克隆生成證書的腳本工具
[root@master01 ~]# git clone https://github.com/iKubernetes/k8s-certs-generator.git Cloning into 'k8s-certs-generator'... remote: Enumerating objects: 58, done. remote: Total 58 (delta 0), reused 0 (delta 0), pack-reused 58 Unpacking objects: 100% (58/58), done. [root@master01 ~]# ls etcd-v3.4.14-linux-amd64.tar.gz k8s-certs-generator [root@master01 ~]# cd k8s-certs-generator/ [root@master01 k8s-certs-generator]# ls etcd-certs-gen.sh gencerts.sh k8s-certs-gen.sh openssl.conf README.md [root@master01 k8s-certs-generator]#
使用gencerts.sh腳本生成etcd所需證書
[root@master01 k8s-certs-generator]# sh gencerts.sh -h
Usage: ./gencerts.sh etcd|k8s
[root@master01 k8s-certs-generator]# sh gencerts.sh etcd
Enter Domain Name [ilinux.io]: k8s.org
Generating RSA private key, 4096 bit long modulus
.......++
.................................................................................................................................................................................................................................................................++
e is 65537 (0x10001)
Generating RSA private key, 2048 bit long modulus
.................................................+++
.........................+++
e is 65537 (0x10001)
Generating etcd/pki/peer.csr
Generating RSA private key, 2048 bit long modulus
...........................................................................................................................................+++
...............+++
e is 65537 (0x10001)
Generating etcd/pki/server.csr
Generating RSA private key, 2048 bit long modulus
..............................................................+++
............................+++
e is 65537 (0x10001)
Generating etcd/pki/apiserver-etcd-client.csr
Generating RSA private key, 2048 bit long modulus
............+++
.................................+++
e is 65537 (0x10001)
Generating etcd/pki/client.csr
Generating etcd/pki/peer.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4096 (0x1000)
Validity
Not Before: Jan 30 10:46:52 2021 GMT
Not After : Jan 28 10:46:52 2031 GMT
Subject:
commonName = etcd
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Server
Netscape Comment:
OpenSSL Generated Server Certificate
X509v3 Subject Key Identifier:
FC:BA:D7:73:4E:C7:1D:9D:73:12:E3:60:96:5B:69:58:CE:4F:14:FD
X509v3 Authority Key Identifier:
keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
DirName:/CN=etcd-ca
serial:BE:88:C0:B5:81:5D:6D:D6
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name:
DNS:*.k8s.org
Certificate is to be certified until Jan 28 10:46:52 2031 GMT (3650 days)
Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/server.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4097 (0x1001)
Validity
Not Before: Jan 30 10:46:53 2021 GMT
Not After : Jan 28 10:46:53 2031 GMT
Subject:
commonName = etcd
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Server
Netscape Comment:
OpenSSL Generated Server Certificate
X509v3 Subject Key Identifier:
1C:BE:22:C0:B7:5F:03:39:5C:E0:FC:47:88:8D:3A:FC:27:FA:0E:BC
X509v3 Authority Key Identifier:
keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
DirName:/CN=etcd-ca
serial:BE:88:C0:B5:81:5D:6D:D6
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:*.k8s.org
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days)
Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/apiserver-etcd-client.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4098 (0x1002)
Validity
Not Before: Jan 30 10:46:53 2021 GMT
Not After : Jan 28 10:46:53 2031 GMT
Subject:
commonName = etcd
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Client
Netscape Comment:
OpenSSL Generated Client Certificate
X509v3 Subject Key Identifier:
FD:52:EA:9F:84:72:35:46:9A:33:71:DE:D0:41:E6:8D:89:C0:62:AE
X509v3 Authority Key Identifier:
keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days)
Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/client.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4099 (0x1003)
Validity
Not Before: Jan 30 10:46:53 2021 GMT
Not After : Jan 28 10:46:53 2031 GMT
Subject:
commonName = etcd
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Client
Netscape Comment:
OpenSSL Generated Client Certificate
X509v3 Subject Key Identifier:
6B:31:50:84:00:9E:0F:6E:B8:56:7A:C1:57:82:F4:BB:12:57:52:B2
X509v3 Authority Key Identifier:
keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days)
Write out database with 1 new entries
Data Base Updated
[root@master01 k8s-certs-generator]# ls
etcd etcd-certs-gen.sh gencerts.sh k8s-certs-gen.sh openssl.conf README.md
[root@master01 k8s-certs-generator]# ls etcd
patches pki
[root@master01 k8s-certs-generator]# ls etcd/pki/
apiserver-etcd-client.crt apiserver-etcd-client.key ca.crt ca.key client.crt client.key peer.crt peer.key server.crt server.key
[root@master01 k8s-certs-generator]#
提示：server.crt和server.key用於etcd服務端的證書和密鑰；peer.crt和peer.key用於集羣內部各節間認證所需證書和密鑰；client.crt和client.key用於客戶端鏈接服務端所需的證書和密鑰；ca.crt和ca.key是用於集羣內部作認證和客戶端鏈接服務端所信任的ca的證書和密鑰；
複製證書文件到其餘節點
[root@master01 k8s-certs-generator]# cp -a etcd/pki/ /etc/etcd/ [root@master01 k8s-certs-generator]# cd /etc/etcd/ [root@master01 etcd]# ls etcd.conf pki [root@master01 etcd]# scp -r pki/ master02:/etc/etcd/ ca.key 100% 3247 1.8MB/s 00:00 ca.crt 100% 1814 1.2MB/s 00:00 peer.key 100% 1679 1.1MB/s 00:00 server.key 100% 1679 1.2MB/s 00:00 apiserver-etcd-client.key 100% 1675 1.3MB/s 00:00 client.key 100% 1675 1.1MB/s 00:00 peer.crt 100% 1659 75.0KB/s 00:00 server.crt 100% 1647 917.8KB/s 00:00 apiserver-etcd-client.crt 100% 1570 1.2MB/s 00:00 client.crt 100% 1570 902.2KB/s 00:00 [root@master01 etcd]# scp -r pki/ master03:/etc/etcd/ ca.key 100% 3247 1.1MB/s 00:00 ca.crt 100% 1814 695.0KB/s 00:00 peer.key 100% 1679 621.6KB/s 00:00 server.key 100% 1679 657.1KB/s 00:00 apiserver-etcd-client.key 100% 1675 950.4KB/s 00:00 client.key 100% 1675 1.0MB/s 00:00 peer.crt 100% 1659 916.3KB/s 00:00 server.crt 100% 1647 1.0MB/s 00:00 apiserver-etcd-client.crt 100% 1570 850.8KB/s 00:00 client.crt 100% 1570 872.7KB/s 00:00 [root@master01 etcd]#
配置etcd基於https協議提供服務
配置master01上的etcd啓用證書認證
[root@master01 etcd]# cat etcd.conf #[Member] #ETCD_CORS="" ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd" #ETCD_WAL_DIR="" ETCD_LISTEN_PEER_URLS="http://192.168.0.41:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.0.41:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" ETCD_NAME="etcd01" #ETCD_SNAPSHOT_COUNT="100000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD_QUOTA_BACKEND_BYTES="0" #ETCD_MAX_REQUEST_BYTES="1572864" #ETCD_GRPC_KEEPALIVE_MIN_TIME="5s" #ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s" #ETCD_GRPC_KEEPALIVE_TIMEOUT="20s" # #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd01:2380" ETCD_ADVERTISE_CLIENT_URLS="http://etcd01:2379" #ETCD_DISCOVERY="" #ETCD_DISCOVERY_FALLBACK="proxy" #ETCD_DISCOVERY_PROXY="" #ETCD_DISCOVERY_SRV="" ETCD_INITIAL_CLUSTER="etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" #ETCD_STRICT_RECONFIG_CHECK="true" #ETCD_ENABLE_V2="true" # #[Proxy] #ETCD_PROXY="off" #ETCD_PROXY_FAILURE_WAIT="5000" #ETCD_PROXY_REFRESH_INTERVAL="30000" #ETCD_PROXY_DIAL_TIMEOUT="1000" #ETCD_PROXY_WRITE_TIMEOUT="5000" #ETCD_PROXY_READ_TIMEOUT="0" # #[Security] ETCD_CERT_FILE="/etc/etcd/pki/server.crt" ETCD_KEY_FILE="/etc/etcd/pki/server.key" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/ca.crt" ETCD_AUTO_TLS="false" ETCD_PEER_CERT_FILE="/etc/etcd/pki/peer.crt" ETCD_PEER_KEY_FILE="/etc/etcd/pki/peer.key" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/ca.crt" ETCD_PEER_AUTO_TLS="false" # #[Logging] #ETCD_DEBUG="false" #ETCD_LOG_PACKAGE_LEVELS="" #ETCD_LOG_OUTPUT="default" # #[Unsafe] #ETCD_FORCE_NEW_CLUSTER="false" # #[Version] #ETCD_VERSION="false" #ETCD_AUTO_COMPACTION_RETENTION="0" # #[Profiling] #ETCD_ENABLE_PPROF="false" #ETCD_METRICS="basic" # #[Auth] #ETCD_AUTH_TOKEN="simple" [root@master01 etcd]#
提示：ETCD_CERT_FILE用於指定etcd服務端證書文件路徑；ETCD_KEY_FILE用戶指定服務端證書文件所對應的密鑰文件路徑；ETCD_CLIENT_CERT_AUTH用戶指定是否啓用客戶端證書認證；ETCD_TRUSTED_CA_FILE用戶指定客戶端認證信任的ca證書文件；
ETCD_AUTO_TLS用於指定是否自動生成證書文件；ETCD_PEER_CERT_FILE用於指定集羣間對等證書文件路徑；ETCD_PEER_KEY_FILE用於指定集羣間對等證書對應的密鑰文件；ETCD_PEER_CLIENT_CERT_AUTH用於指定是否啓用對等證書認證；ETCD_PEER_TRUSTED_CA_FILE用於指定對等證書認證所信賴的ca證書；ETCD_PEER_AUTO_TLS用於指定是否自動生成對等證書；
修改/etc/etcd/etcd.conf文件，將etcd01修改爲etcd01.k8s.org，將etcd02修改爲etcd02.k8s.org，將etcd03修改爲etcd03.k8s.org，將http修改爲https
中止etcd服務，刪除/var/lib/etcd/目錄下的全部文件
[root@master01 etcd]# systemctl stop etcd [root@master01 etcd]# rm -rf /var/lib/etcd/* [root@master01 etcd]# ll /var/lib/etcd/ total 0 [root@master01 etcd]#
配置master02啓用證書認證，並將對應http修改爲很https，把對應短格式名稱修改成相似etcd01.k8s.org名稱
中止etcd服務，刪除/var/lib/etcd/目錄下的全部文件
[root@master02 ~]# systemctl stop etcd [root@master02 ~]# rm -rf /var/lib/etcd/* [root@master02 ~]# ll /var/lib/etcd/ total 0 [root@master02 ~]#
配置master03啓用證書認證，並將對應http修改爲很https，把對應短格式名稱修改成長格式名稱
中止etcd服務，刪除/var/lib/etcd下的全部文件
[root@master03 ~]# systemctl stop etcd [root@master03 ~]# rm -rf /var/lib/etcd/* [root@master03 ~]# ll /var/lib/etcd/ total 0 [root@master03 ~]#
啓動各節點上的etcd
[root@master01 etcd]# systemctl start etcd [root@master01 etcd]#
提示：若是三個節點上的etcd都能正常啓動，說明咱們配置文件沒有問題；
驗證：查看全部節點的etcd服務是否都正常啓動，並監聽對應的的端口？
[root@master01 etcd]# ss -tnl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 192.168.0.41:2379 *:* LISTEN 0 128 192.168.0.41:2380 *:* LISTEN 0 128 *:22 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::* [root@master01 etcd]# ssh master02 'ss -tnl' State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 192.168.0.42:2379 *:* LISTEN 0 128 192.168.0.42:2380 *:* LISTEN 0 128 *:22 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::* [root@master01 etcd]# ssh master03 'ss -tnl' State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 192.168.0.43:2379 *:* LISTEN 0 128 192.168.0.43:2380 *:* LISTEN 0 128 *:22 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::* [root@master01 etcd]#
驗證：查看集羣成員
[root@master01 etcd]# etcdctl --endpoints="https://etcd01.k8s.org:2379,https://etcd02.k8s.org:2379,https://etcd03.k8s.org:2379" --cacert="/etc/etcd/pki/ca.crt" --cert="/etc/etcd/pki/client.crt" --key="/etc/etcd/pki/client.key" endpoint status https://etcd01.k8s.org:2379, 61d91b7ed8f88f32, 3.4.14, 20 kB, true, false, 6, 9, 9, https://etcd02.k8s.org:2379, ef13441fdfe8af38, 3.4.14, 20 kB, false, false, 6, 9, 9, https://etcd03.k8s.org:2379, f11ed09b6567910f, 3.4.14, 20 kB, false, false, 6, 9, 9, [root@master01 etcd]# etcdctl --endpoints="https://etcd01.k8s.org:2379" --cacert="/etc/etcd/pki/ca.crt" --cert="/etc/etcd/pki/client.crt" --key="/etc/etcd/pki/client.key" member list 61d91b7ed8f88f32, started, etcd01.k8s.org, https://etcd01.k8s.org:2380, https://etcd01.k8s.org:2379, false ef13441fdfe8af38, started, etcd02.k8s.org, https://etcd02.k8s.org:2380, https://etcd02.k8s.org:2379, false f11ed09b6567910f, started, etcd03.k8s.org, https://etcd03.k8s.org:2380, https://etcd03.k8s.org:2379, false [root@master01 etcd]#
提示：如今etcd啓用了ssl認證功能，客戶端訪問必須攜帶對應的客戶端證書和私鑰文件以及對應認證所信任的ca證書，才能夠正常訪問到etcd集羣；這裏須要注意指定的endpoints須要使用域名格式給出，給定ip地址是沒法正常經過認證的；
相關文章
- 1. 小米開源分佈式KV存儲系統Pegasus
- 2. 從 leveldb 到分佈式 kv 存儲系統
- 3. 分佈式存儲系統
- 4. Kubernetes 集羣基於 Rook 搭建 Ceph 分佈式存儲系統
- 5. 分佈式鍵值存儲系統ETCD調研
- 6. etcd 一個分佈式一致性鍵值存儲系統
- 7. 分佈式存儲ceph集羣部署
- 8. 集羣 分佈式存儲----Ceph
- 9. ETCD分佈式存儲部署
- 10. 分佈式健值存儲etcd 3.1.7
- 更多相關文章...
- • Swarm 集羣管理 - Docker教程
- • 操作系統（OS）平臺 統計 - 瀏覽器信息
- • 再有人問你分佈式事務，把這篇扔給他
- • 常用的分佈式事務解決方案
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
