Kubernetes羣集部署之ETCD

時間 2021-01-17

標籤 node linux git github web 算法 docker 數據庫 json vim 欄目 Linux 简体版

原文原文鏈接

一、Master組件node

●kube-apiserver
Kubernetes API,集羣的統一入口，各組件協調者，以RESTful API提供接口服務，全部對象資源的增刪改查和監聽操做都交給APIServer處理後再提交給Etcd存儲。linux

●kube-controller-manager
處理集羣中常規後臺任務，一個資源對應一個控制器，而ControllerManager就是負責管理這些控制器的。git

●kube-scheduler
根據調度算法爲新建立的Pod選擇一個Node節點，能夠任意部署，能夠部署在同一個節點上，也能夠部署在不一樣的節點上。github

●etcd
分佈式鍵值存儲系統。用於保存集羣狀態數據，好比Pod、Service 等對象信息。web

二、Node組件算法

●kubelet
kubelet是Master在Node節點上的Agent，管理本機運行容器的生命週期，好比建立容器、Pod掛載數據卷、下載secret、獲取容器和節點狀態等工做。kubelet將每一個Pod轉換成一組容器。docker

●kube-proxy
在Node節點上實現Pod網絡代理，維護網絡規則和四層負載均衡工做。數據庫

●docker或rocket
容器引擎，運行容器。json

工做原理：vim

一、準備包含應用程序的Deployment的yml文件，而後經過kubectl客戶端工具發送給ApiServer。

二、ApiServer接收到客戶端的請求並將資源內容存儲到數據庫(etcd)中。

三、Controller組件(包括scheduler、replication、endpoint)監控資源變化並做出反應。

四、ReplicaSet檢查數據庫變化，建立指望數量的pod實例。

五、Scheduler再次檢查數據庫變化，發現還沒有被分配到具體執行節點(node)的Pod，而後根據一組相關規則將pod分配到能夠運行它們的節點上，並更新數據庫，記錄pod分配狀況。

六、Kubelete監控數據庫變化，管理後續pod的生命週期，發現被分配到它所在的節點上運行的那些pod。若是找到新pod，則會在該節點上運行這個新pod。

附：kuberproxy運行在集羣各個主機上，管理網絡通訊，如服務發現、負載均衡。當有數據發送到主機時，將其路由到正確的pod或容器。對於從主機上發出的數據，它能夠基於請求地址發現遠程服務器，並將數據正確路由，在某些狀況下會使用輪循調度算法(Round-robin)將請求發送到集羣中的多個實例。

Kubernetes核心概念

一、Pod

●最小部署單元

●一組容器的集合

●一個Pod中的容器共享網絡命名空間

●Pod是短暫的

二、Controllers

●ReplicaSet :確保預期的Pod副本數量

●Deployment:無狀態應用部署

●StatefulSet :有狀態應用部署

●DaemonSet:確保全部Node運行同一個Pod

●Job:一次性任務

●Cronjob:定時任務

更高級層次對象，部署和管理Pod

三、Service

●防止Pod失聯

●定義一組Pod的訪問策略

●Label: 標籤，附加到某個資源上，用於關聯對象、查詢和篩選

●Namespaces :命名空間，將對象邏輯上隔離

●Annotations :註釋

Kubernetes集羣部署

1.官方提供的三種部署方式

●minikube

Minikube是一個工具，能夠在本地快速運行一個單點的Kubernetes，僅用於嘗試Kubernetes或平常開發的用戶使用。
部署地址: https://kubernetes.io/docs/setup/minikube/

●kubeadm

Kubeadm也是一個工具，提供kubeadm init和kubeadm join,用於快速部署Kubernetes集羣。

部署地址:https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

●二進制包

推薦，從官方下載發行版的二進制包，手動部署每一個組件，組成Kubernetes集羣。

下載地址:https://github.com/kubernetes/kubernetes/releases

Kubernetes 平臺環境規劃

3.自籤SSL證書

Etcd數據庫集羣部署

Kubernetes二進制部署
k8s軟件包：

連接：https://pan.baidu.com/s/1oN2wkGZ_7parS8sMaaogGw
提取碼：lbjx

k8s部署規劃：
master：192.168.35.128 kube-apiserver kube-controller-manager kube-scheduler etcd
node1：192.168.35.195 kubelet kube-proxy docker flannel etcd
node2：192.168.35.138 kubelet kube-proxy docker flannel etcd

master操做：
[root@localhost ~]# mkdir k8s
[root@localhost ~]# cd k8s/
[root@localhost k8s]# ls //從宿主機拖進來
etcd-cert.sh etcd.sh
[root@localhost k8s]# mkdir etcd-cert
[root@localhost k8s]# mv etcd-cert.sh etcd-cert
上面 etcd-cert.sh etcd.sh腳本

vim etcd.sh

#!/bin/bash
#example: ./etcd.sh etcd01 192.168.1.10 etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380

ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd

cat <<EOF >$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

vim etcd-cert.sh

cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF

cat > ca-csr.json <<EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------

cat > server-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"10.206.240.188",
"10.206.240.189",
"10.206.240.111"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

下載官方包

[root@localhost k8s]# vim cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

[root@localhost k8s]# bash cfssl.sh //下載cfssl官方包
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9.8M 100 9.8M 0 0 77052 0 0:02:14 0:02:14 --:--:-- 94447
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2224k 100 2224k 0 0 66701 0 0:00:34 0:00:34 --:--:-- 71949
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6440k 100 6440k 0 0 74368 0 0:01:28 0:01:28 --:--:-- 93942

[root@localhost k8s]# ls /usr/local/bin/
cfssl cfssl-certinfo cfssljson
//cfssl 生成證書工具、cfssljson經過傳入json文件生成證書、cfssl-certinfo查看證書信息
定義證書

[root@localhost k8s]# cd etcd-cert/

//定義ca證書
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF

實現證書籤名

//實現證書籤名
cat > ca-csr.json <<EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
生產證書，生成ca-key.pem ca.pem

//生產證書，生成ca-key.pem ca.pem
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

2020/01/15 18:15:15 [INFO] generating a new CA key and certificate from CSR
2020/01/15 18:15:15 [INFO] generate received request
2020/01/15 18:15:15 [INFO] received CSR
2020/01/15 18:15:15 [INFO] generating key: rsa-2048
2020/01/15 18:15:15 [INFO] encoded CSR
2020/01/15 18:15:15 [INFO] signed certificate with serial number 661808851940283859099066838380794010566731982441
指定etcd三個節點之間的通訊驗證

//指定etcd三個節點之間的通訊驗證
cat > server-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"192.168.35.128",
"192.168.35.195",
"192.168.35.138"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF

生成ETCD證書 server-key.pem server.pem

//生成ETCD證書 server-key.pem server.pem
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

www server-csr.json | cfssljson -bare server
2020/01/15 18:24:09 [INFO] generate received request
2020/01/15 18:24:09 [INFO] received CSR
2020/01/15 18:24:09 [INFO] generating key: rsa-2048
2020/01/15 18:24:09 [INFO] encoded CSR
2020/01/15 18:24:09 [INFO] signed certificate with serial number 613252568370198035643630635602034323043189506463
2020/01/15 18:24:09 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
複製軟件包到centos7中

[root@localhost etcd-cert]# cd /root/k8s/
[root@localhost k8s]# ls //直接拉取到目錄下
cfssl.sh etcd.sh flannel-v0.10.0-linux-amd64.tar.gz
etcd-cert etcd-v3.3.10-linux-amd64.tar.gz kubernetes-server-linux-amd64.tar.gz

解壓

[root@localhost k8s]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz

[root@localhost k8s]# ls etcd-v3.3.10-linux-amd64
Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md

[root@localhost k8s]# mkdir /opt/etcd/{cfg,bin,ssl} -p //配置文件，命令文件，證書

[root@localhost k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/
證書拷貝

//證書拷貝
[root@localhost k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/

//進入卡住狀態等待其餘節點加入
[root@localhost k8s]# bash etcd.sh etcd01 192.168.35.128 etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380

使用另一個會話打開，會發現etcd進程已經開啓

[root@localhost ~]# ps aux | grep etcd
root 4653 0.3 0.6 10523616 12140 ? Ssl 19:49 0:00 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://192.168.35.128:2380 --listen-client-urls=https://192.168.35.128:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.35.128:2379 --initial-advertise-peer-urls=https://192.168.35.128:2380 --initial-cluster=etcd01=https://192.168.35.128:2380,etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
root 4719 0.0 0.0 112676 984 pts/2 R+ 19:50 0:00 grep --color=auto etcd
關閉防火牆

systemctl stop firewalld.service
setenforce 0

拷貝證書去其餘節點

[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.35.195:/opt/
[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.35.138:/opt
啓動腳本拷貝其餘節點

[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.35.195:/usr/lib/systemd/system/
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.35.138:/usr/lib/systemd/system/
在node01節點修改
[root@localhost ~]# vim /opt/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.35.195:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.35.195:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.35.195:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.35.195:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.35.128:2380,etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

在node02節點修改
[root@localhost ~]# vim /opt/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.35.138:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.35.138:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.35.138:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.35.138:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.35.128:2380,etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

先在master上開啓服務
[root@localhost system]# cd /root/k8s/
[root@localhost k8s]# bash etcd.sh etcd01 192.168.35.128 etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380
在node1和node2上開啓服務
[root@localhost ~]# systemctl start etcd
[root@localhost ~]# systemctl status etcd
再去master查看，會發現同步成功
[root@localhost k8s]# bash etcd.sh etcd01 192.168.35.128 etcd02=https://192.168.35.195:2380,etcd03=https://192.168.35.138:2380

檢查羣集狀態
[root@localhost k8s]# cd etcd-cert/
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.35.128:2379,https://192.168.35.195:2379,https://192.168.35.138:2379" cluster-health
member 12a96220ac829a49 is healthy: got healthy result from https://192.168.35.195:2379
member 76797989afd0ecba is healthy: got healthy result from https://192.168.35.128:2379
member ff469df2baaba1da is healthy: got healthy result from https://192.168.35.138:2379cluster is healthy