kubernetes 1.18.3 版本二進制部署

時間 2020-06-15

標籤 kubernetes 1.18.3 版本二進制部署简体版

原文原文鏈接

kubernetes 1.18.3 版本二進制部署

標籤（空格分隔）： kubernetes系列node

一：系統環境介紹
二：部署Etcd集羣
三：安裝docker
四：部署k8s Master Node
五：部署k8s Worker Node
六：部署Dashboard和CoreDNS

一：系統環境介紹

1.1 環境準備

在開始以前，部署Kubernetes集羣機器須要知足如下幾個條件：

操做系統: CentOS7.8-86_x64
硬件配置：2GB或更多RAM，2個CPU或更多CPU，硬盤30GB或更多集羣中全部機器之間網絡互通

能夠訪問外網，須要拉取鏡像，若是服務器不能上網，須要提早下載鏡像並導入節點

禁止swap分區

1.2 軟件環境：

操做系統:   CentOS7.8_x64 （mini）

Docker: 19-ce

Kubernetes: 1.18.3

1.3 環境規劃

服務器總體規劃：

1.4 單Master架構圖：

1.5 單Master服務器規劃：

1.6 操做系統初始化配置

# 關閉防火牆
systemctl stop firewalld
systemctl disable firewalld

# 關閉selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config  # 永久
setenforce 0  # 臨時

# 關閉swap
swapoff -a  # 臨時
sed -ri 's/.*swap.*/#&/' /etc/fstab    # 永久

# 根據規劃設置主機名
hostnamectl set-hostname <hostname>

# 在master添加hosts
cat >> /etc/hosts << EOF
192.168.100.11 node01.flyfish
192.168.100.12 node02.flyfish
192.168.100.13 node03.flyfish
EOF

# 將橋接的IPv4流量傳遞到iptables的鏈
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system  # 生效

# 時間同步
yum install chronyd
server ntp1.aliyun.com

2、部署Etcd集羣

2.1 ETCD 集羣的概念

Etcd 是一個分佈式鍵值存儲系統，Kubernetes使用Etcd進行數據存儲，因此先準備一個Etcd數據庫，爲解決Etcd單點故障，應採用集羣方式部署，這裏使用3臺組建集羣，可容忍1臺機器故障，固然，你也可使用5臺組建集羣，可容忍2臺機器故障

注：爲了節省機器，這裏與K8s節點機器複用。也能夠獨立於k8s集羣以外部署，只要apiserver能鏈接到就行。

2.2準備cfssl證書生成工具

cfssl是一個開源的證書管理工具，使用json文件生成證書，相比openssl更方便使用。
找任意一臺服務器操做，這裏用Master節點。
---
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
---

2.3 生成Etcd證書

1. 自簽證書頒發機構（CA）

建立工做目錄：

mkdir -p ~/TLS/{etcd,k8s}

cd TLS/etcd

自籤CA：

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生成證書

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
ls *pem
ca-key.pem  ca.pem

2. 使用自籤CA簽發Etcd HTTPS證書

建立證書申請文件：

cat > server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.100.11",
    "192.168.100.12",
    "192.168.100.13",
    "192.168.100.14",
    "192.168.100.15",
    "192.168.100.16",
    "192.168.100.17",
    "192.168.100.100"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

生成證書：
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

ls server*pem
server-key.pem  server.pem

2.4 從Github下載二進制文件

下載地址：https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

如下在節點1上操做，爲簡化操做，待會將節點1生成的全部文件拷貝到節點2和節點3.

1. 建立工做目錄並解壓二進制包

mkdir /opt/etcd/{bin,cfg,ssl} -p
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

2.5 建立etcd配置文件

cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.11:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.11:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.100.11:2380,etcd-2=https://192.168.100.12:2380,etcd-3=https://192.168.100.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

---
ETCD_NAME：節點名稱，集羣中惟一
ETCD_DATA_DIR：數據目錄
ETCD_LISTEN_PEER_URLS：集羣通訊監聽地址
ETCD_LISTEN_CLIENT_URLS：客戶端訪問監聽地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集羣通告地址
ETCD_ADVERTISE_CLIENT_URLS：客戶端通告地址
ETCD_INITIAL_CLUSTER：集羣節點地址
ETCD_INITIAL_CLUSTER_TOKEN：集羣Token
ETCD_INITIAL_CLUSTER_STATE：加入集羣的當前狀態，new是新集羣，existing表示加入已有集羣

2.6. systemd管理etcd

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

4. 拷貝剛纔生成的證書

把剛纔生成的證書拷貝到配置文件中的路徑：
cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/

5. 啓動並設置開機啓動
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

scp -r /opt/etcd/ root@192.168.100.12:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.100.12:/usr/lib/systemd/system/
scp -r /opt/etcd/ root@192.168.100.13:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.100.13:/usr/lib/systemd/system/

而後在節點2和節點3分別修改etcd.conf配置文件中的節點名稱和當前服務器IP：
vi /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1"   # 修改此處，節點2改成etcd-2，節點3改成etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.11:2380"   # 修改此處爲當前服務器IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.11:2379" # 修改此處爲當前服務器IP

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.11:2380" # 修改此處爲當前服務器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.11:2379" # 修改此處爲當前服務器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.100.11:2380,etcd-2=https://192.168.100.12:2380,etcd-3=https://192.168.100.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

最後啓動etcd並設置開機啓動，同上。

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.100.11:2379,https://192.168.100.12:2379,https://192.168.100.13:2379" endpoint health

若是輸出上面信息，就說明集羣部署成功。若是有問題第一步先看日誌：/var/log/message 或 journalctl -u etcd

3、安裝Docker

下載地址：https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz

如下在全部節點操做。這裏採用二進制安裝，用yum安裝也同樣。
在 node01.flyfish,node02.flyfish 與 node03.flyfish 節點上面安裝

3.1 解壓二進制包

tar zxvf docker-19.03.9.tgz
mv docker/* /usr/bin

3.2 systemd管理docker

cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF

3.3 建立配置文件

mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

registry-mirrors 阿里雲鏡像加速器

3.4 啓動並設置開機啓動

systemctl daemon-reload
systemctl start docker
systemctl enable docker

4、部署Master Node

4.1 生成kube-apiserver證書

1. 自簽證書頒發機構（CA）

cd /root/TLS/k8s/

---
cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

生成證書：

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

ls *pem
ca-key.pem  ca.pem

---

2. 使用自籤CA簽發kube-apiserver HTTPS證書
建立證書申請文件：

cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.100.11",
      "192.168.100.12",
      "192.168.100.13",
      "192.168.100.14",
      "192.168.100.15",
      "192.168.100.16",
      "192.168.100.17",
      "192.168.100.100",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

注：上述文件hosts字段中IP爲全部Master/LB/VIP IP，一個都不能少！爲了方便後期擴容能夠多寫幾個預留的IP。

生成證書：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

ls server*pem
server-key.pem  server.pem

4.2 從Github下載二進制文件

下載地址： https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183

注：打開連接你會發現裏面有不少包，下載一個server包就夠了，包含了Master和Worker Node二進制文件。

4.3 解壓二進制包

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/

4.4 部署kube-apiserver

1. 建立配置文件

cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://192.168.100.11:2379,https://192.168.100.12:2379,https://192.168.100.13:2379 \\
--bind-address=192.168.100.11 \\
--secure-port=6443 \\
--advertise-address=192.168.100.11 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

---
注：上面兩個\ \ 第一個是轉義符，第二個是換行符，使用轉義符是爲了使用EOF保留換行符。
–logtostderr：啓用日誌
—v：日誌等級
–log-dir：日誌目錄
–etcd-servers：etcd集羣地址
–bind-address：監聽地址
–secure-port：https安全端口
–advertise-address：集羣通告地址
–allow-privileged：啓用受權
–service-cluster-ip-range：Service虛擬IP地址段
–enable-admission-plugins：准入控制模塊
–authorization-mode：認證受權，啓用RBAC受權和節點自管理
–enable-bootstrap-token-auth：啓用TLS bootstrap機制
–token-auth-file：bootstrap token文件
–service-node-port-range：Service nodeport類型默認分配端口範圍
–kubelet-client-xxx：apiserver訪問kubelet客戶端證書
–tls-xxx-file：apiserver https證書
–etcd-xxxfile：鏈接Etcd集羣證書
–audit-log-xxx：審計日誌
---

2. 拷貝剛纔生成的證書

把剛纔生成的證書拷貝到配置文件中的路徑：
cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

3. 啓用 TLS Bootstrapping 機制

TLS Bootstraping：Master apiserver啓用TLS認證後，Node節點kubelet和kube-proxy要與kube-apiserver進行通訊，必須使用CA簽發的有效證書才能夠，當Node節點不少時，這種客戶端證書頒發須要大量工做，一樣也會增長集羣擴展複雜度。爲了簡化流程，Kubernetes引入了TLS bootstraping機制來自動頒發客戶端證書，kubelet會以一個低權限用戶自動向apiserver申請證書，kubelet的證書由apiserver動態簽署。因此強烈建議在Node上使用這種方式，目前主要用於kubelet，kube-proxy仍是由咱們統一頒發一個證書。

TLS bootstraping 工做流程：

建立上述配置文件中token文件：

cat > /opt/kubernetes/cfg/token.csv << EOF
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

格式：token，用戶名，UID，用戶組

token也可自行生成替換：

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

4. systemd管理apiserver

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

5. 啓動並設置開機啓動

systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver

6. 受權kubelet-bootstrap用戶容許請求證書

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

4.5 部署kube-controller-manager

1. 建立配置文件

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--master=127.0.0.1:8080 \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"
EOF

–master：經過本地非安全本地端口8080鏈接apiserver。
–leader-elect：當該組件啓動多個時，自動選舉（HA）
–cluster-signing-cert-file/–cluster-signing-key-file：自動爲kubelet頒發證書的CA，與apiserver保持一致

2. systemd管理controller-manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

3. 啓動並設置開機啓動

systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager

4.6 部署kube-scheduler
1. 建立配置文件

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1"
EOF

–master：經過本地非安全本地端口8080鏈接apiserver。
–leader-elect：當該組件啓動多個時，自動選舉（HA）

2. systemd管理scheduler

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

3. 啓動並設置開機啓動

systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler

4. 查看集羣狀態

全部組件都已經啓動成功，經過kubectl工具查看當前集羣組件狀態：

kubectl get cs

如上輸出說明Master節點組件運行正常。

5、部署Worker Node

下面仍是在Master Node上操做，即同時做爲Worker Node

5.1 建立工做目錄並拷貝二進制文件

在全部worker node建立工做目錄：

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

從master節點拷貝

cd kubernetes/server/bin
cp kubelet kube-proxy /opt/kubernetes/bin   # 本地拷貝

在master 節點上面執行

5.2 部署kubelet
1. 建立配置文件

cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=node01.flyfish \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
EOF

-----
–hostname-override：顯示名稱，集羣中惟一
–network-plugin：啓用CNI
–kubeconfig：空路徑，會自動生成，後面用於鏈接apiserver
–bootstrap-kubeconfig：首次啓動向apiserver申請證書
–config：配置參數文件
–cert-dir：kubelet證書生成目錄
–pod-infra-container-image：管理Pod網絡容器的鏡像

---

2. 配置參數文件
cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

在 server節點上面執行

3. 生成bootstrap.kubeconfig文件
寫一個boot.sh 腳本 把下面的內容放進去
---

KUBE_APISERVER="https://192.168.100.11:6443" # apiserver IP:PORT
TOKEN="c47ffb939f5ca36231d9e3121a252940" # 與token.csv裏保持一致

# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
---

. ./boot.sh

拷貝到配置文件路徑：

cp bootstrap.kubeconfig /opt/kubernetes/cfg

4. systemd管理kubelet

cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

5. 啓動並設置開機啓動
systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet

5.3 批准kubelet證書申請並加入集羣
# 查看kubelet證書請求
kubectl get csr

# 批准申請

kubectl certificate approve node-csr--vTFwyeAv5dSatbGrgpJptwQ5Fc_WvLpmYgdQN4bvaI

注：因爲網絡插件尚未部署，節點會沒有準備就緒 NotReady

5.4 部署kube-proxy
1. 建立配置文件

cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF

2. 配置參數文件

cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: node01.flyfish
clusterCIDR: 10.0.0.0/24
EOF

# 切換工做目錄
cd TLS/k8s

# 建立證書請求文件
cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF
---
# 生成證書
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

ls kube-proxy*pem
kube-proxy-key.pem  kube-proxy.pem

cp -p kube-proxy-key.pem  kube-proxy.pem /opt/kubernetes/ssl/

生成kubeconfig文件：
cd /opt/kubernetes/ssl/
vim kubeconfig.sh 
---
KUBE_APISERVER="https://192.168.100.11:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
---

. ./kubeconfig.sh

cp -p kube-proxy.kubeconfig /opt/kubernetes/cfg/

4. systemd管理kube-proxy

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

5. 啓動並設置開機啓動

systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy

5.5 部署CNI網絡

先準備好CNI二進制文件：

下載地址：https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz

解壓二進制包並移動到默認工做目錄：
mkdir /opt/cni/bin -p
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin

部署CNI網絡：

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml

kubectl apply -f kube-flannel.yml

kubectl get pods -n kube-system

部署好網絡插件，Node準備就緒。

kubectl get node

5.6 受權apiserver訪問kubelet

cat > apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF
---

kubectl apply -f apiserver-to-kubelet-rbac.yaml

5.7 新增長Worker Node

1. 拷貝已部署好的Node相關文件到新節點

在master節點將Worker Node涉及文件拷貝到新節點192.168.100.13 上面

scp -r /opt/kubernetes root@192.168.100.12:/opt/

scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.100.12:/usr/lib/systemd/system

scp -r /opt/cni/ root@192.168.100.12:/opt/

scp /opt/kubernetes/ssl/ca.pem root@192.168.100.12:/opt/kubernetes/ssl

2. 刪除kubelet證書和kubeconfig文件
rm -rf /opt/kubernetes/cfg/kubelet.kubeconfig 
rm -rf /opt/kubernetes/ssl/kubelet*

3. 修改主機名

vim /opt/kubernetes/cfg/kubelet.conf
--hostname-override=node02.flyfish

vim /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: node02.flyfish

4. 啓動並設置開機啓動

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy

5. 在Master上批准新Node kubelet證書申請

kubectl get csr

kubectl certificate approve node-csr-qFbDvbTwo9SP2ZEDyiKfXCBGxO4n4Qe7FCehyPKiXNc

增長一個 work節點：

增長 一個 work節點：

   scp -r /opt/kubernetes root@192.168.100.13:/opt/

scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.100.13:/usr/lib/systemd/system

scp -r /opt/cni/ root@192.168.100.13:/opt/

scp /opt/kubernetes/ssl/ca.pem root@192.168.100.13:/opt/kubernetes/ssl

2. 刪除kubelet證書和kubeconfig文件
rm -rf /opt/kubernetes/cfg/kubelet.kubeconfig 
rm -rf /opt/kubernetes/ssl/kubelet*

3. 修改主機名

vim /opt/kubernetes/cfg/kubelet.conf
--hostname-override=node03.flyfish

vim /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: node03.flyfish

4. 啓動並設置開機啓動

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy

5. 在Master上批准新Node kubelet證書申請

kubectl get csr

kubectl certificate approve node-csr-5ZsKjw2Udxrc97q4MtShig83PUJww7E3y_2mpvkMZr0

kubectl get node

6、部署Dashboard和CoreDNS

6.1 部署Dashboard

$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

默認Dashboard只能集羣內部訪問，修改Service爲NodePort類型，暴露到外部：

vim recommended.yaml
----
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard
----

kubectl get pods,svc -n kubernetes-dashboard

訪問地址：https://NodeIP:30001

建立service account並綁定默認cluster-admin管理員集羣角色：

kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

6.2 部署CoreDNS

CoreDNS用於集羣內部Service名稱解析。
kubectl apply -f coredns.yaml

DNS解析測試：

kubectl run -it --rm dns-test --image=busybox:1.28.4 sh

nslookup kubernetes

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。

kubernetes 1.18.3 版本 二進制部署

kubernetes 1.18.3 版本 二進制部署

一： 系統環境介紹二： 部署Etcd集羣三： 安裝docker四： 部署k8s Master Node五： 部署k8s Worker Node六： 部署Dashboard和CoreDNS

一： 系統環境介紹

1.1 環境準備

1.2 軟件環境：

1.3 環境規劃

1.4 單Master架構圖：

1.5 單Master服務器規劃：

1.6 操做系統初始化配置

2、部署Etcd集羣

2.1 ETCD 集羣的概念

2.2準備cfssl證書生成工具

2.3 生成Etcd證書

2.4 從Github下載二進制文件

2.5 建立etcd配置文件

2.6. systemd管理etcd

3、安裝Docker

4、部署Master Node

5、部署Worker Node

增長 一個 work節點：

6、部署Dashboard和CoreDNS

6.2 部署CoreDNS

kubernetes 1.18.3 版本二進制部署

kubernetes 1.18.3 版本二進制部署

一：系統環境介紹
二：部署Etcd集羣
三：安裝docker
四：部署k8s Master Node
五：部署k8s Worker Node
六：部署Dashboard和CoreDNS

一：系統環境介紹

增長一個 work節點：