K8S集羣Master高可用實踐

時間  2021-01-04
標籤 node git github redis json bootstrap centos api bash 服務器 欄目 負載均衡 简体版
原文   原文鏈接

本文將在前文基礎上介紹k8s集羣的高可用實踐，通常來說，k8s集羣高可用主要包含如下幾個內容：
一、etcd集羣高可用
二、集羣dns服務高可用
三、kube-apiserver、kube-controller-manager、kube-scheduler等master組件的高可用node

其中etcd實現的辦法較爲容易，具體實現辦法可參考前文：
https://blog.51cto.com/ylw6006/2095871git

集羣dns服務高可用，能夠經過配置dns的pod副本數爲2，經過配置label實現2個副本運行在在不一樣的節點上實現高可用。github

kube-apiserver服務的高可用，可行的方案較多，具體介紹可參考文檔：
https://jishu.io/kubernetes/kubernetes-master-ha/redis

kube-controller-manager、kube-scheduler等master組件的高可用相對容易實現，運行多份實例便可。json

1、環境介紹

master節點1： 192.168.115.5/24 主機名：vm1
master節點2： 192.168.115.6/24 主機名：vm2
VIP地址： 192.168.115.4/24 (使用keepalived實現)
Node節點1: 192.168.115.6/24 主機名：vm2
Node節點2: 192.168.115.7/24 主機名：vm3bootstrap

操做系統版本：centos 7.2 64bit
K8s版本：1.9.6 二進制部署centos

本文演示環境是在前文的基礎上，已有k8s集羣（1個master節點、2個node節點上），實現k8s集羣master組件的高可用，關於k8s環境的部署請參考前文連接!
一、配置Etcd集羣和TLS認證 ——> https://blog.51cto.com/ylw6006/2095871
二、Flannel網絡組件部署 ——> http://www.javashuo.com/article/p-ypcitoss-gy.html
三、升級Docker服務 ——> https://blog.51cto.com/ylw6006/2103064
四、K8S二進制部署Master節點 ——> https://blog.51cto.com/ylw6006/2104031
五、K8S二進制部署Node節點 ——> https://blog.51cto.com/ylw6006/2104692api

2、證書更新

在vm1節點上完成證書的更新，重點是要把master相關ip所有所有加入到列表裏面bash

# mkdir api-ha && cd api-ha
# cat k8s-csr.json    
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.115.4",
    "192.168.115.5",
    "192.168.115.6",
    "10.254.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "FuZhou",
      "L": "FuZhou",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

# cfssl gencert -ca=/etc/ssl/etcd/ca.pem \
  -ca-key=/etc/ssl/etcd/ca-key.pem \
  -config=/etc/ssl/etcd/ca-config.json \
  -profile=kubernetes k8s-csr.json | cfssljson -bare kubernetes

# mv *.pem /etc/kubernetes/ssl/

3、配置master組件

一、複製vm1的kube-apiserver、kube-controller-manager、kube-scheduler文件到vm2節點上服務器

# cd /usr/local/sbin
# scp -rp  kube-apiserver  kube-controller-manager  kube-scheduler  vm2:/usr/local/sbin/

二、複製vm1的證書文件到vm2節點上

# cd /etc/kubernetes/ssl
# scp -rp ./* vm2:/etc/kubernetes/ssl

三、配置服務並啓動服務

# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/sbin/kube-apiserver \
  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --advertise-address=0.0.0.0 \
  --bind-address=0.0.0.0 \
  --insecure-bind-address=127.0.0.1 \
  --authorization-mode=RBAC \
  --runtime-config=rbac.authorization.k8s.io/v1alpha1 \
  --kubelet-https=true \
  --enable-bootstrap-token-auth=true \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-cluster-ip-range=10.254.0.0/16 \
  --service-node-port-range=1024-65535 \
  --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --client-ca-file=/etc/ssl/etcd/ca.pem \
  --service-account-key-file=/etc/ssl/etcd/ca-key.pem \
  --etcd-cafile=/etc/ssl/etcd/ca.pem \
  --etcd-certfile=/etc/ssl/etcd/server.pem \
  --etcd-keyfile=/etc/ssl/etcd/server-key.pem \
  --etcd-servers=https://192.168.115.5:2379,https://192.168.115.6:2379,https://192.168.115.7:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/lib/audit.log \
  --event-ttl=1h \
  --v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
# cat /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/sbin/kube-scheduler \
  --address=127.0.0.1 \
  --master=http://127.0.0.1:8080 \
  --leader-elect=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
# cat /usr/lib/systemd/system/kube-controller-manager.service 
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/sbin/kube-controller-manager \
  --address=127.0.0.1 \
  --master=http://127.0.0.1:8080 \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=10.254.0.0/16 \
  --cluster-cidr=172.30.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/ssl/etcd/ca.pem \
  --cluster-signing-key-file=/etc/ssl/etcd/ca-key.pem \
  --service-account-private-key-file=/etc/ssl/etcd/ca-key.pem \
  --root-ca-file=/etc/ssl/etcd/ca.pem \
  --leader-elect=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
# systemctl enable kube-apiserver
# systemctl enable kube-controller-manager
# systemctl enable kube-scheduler
# systemctl start kube-apiserver
# systemctl start kube-controller-manager
# systemctl start kube-scheduler

注意：

vm1上的api-server配置文件須要將--advertise-address、--bind-address兩個參數修改成全網監聽

4、安裝和配置keepalived

# yum -y install keepalived
# cat /etc/keepalived/keepalived.conf   
! Configuration File for keepalived  
global_defs {  
    notification_email {   
        ylw@fjhb.cn
    }   
    notification_email_from admin@fjhb.cn
    smtp_server 127.0.0.1  
    smtp_connect_timeout 30  
    router_id LVS_MASTER  
}  

vrrp_script check_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 3
}  

vrrp_instance VI_1 {  
    state MASTER
    interface ens33
    virtual_router_id 60  
    priority 100  
    advert_int 1  
    authentication {  
        auth_type PASS  
        auth_pass k8s.59iedu.com
    }  
    virtual_ipaddress {  
        192.168.115.4/24
    }

    track_script {   
        check_apiserver
    }
}
# cat /usr/lib/systemd/system/keepalived.service  
[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target kube-apiserver.service
Require=kube-apiserver.service

[Service]
Type=forking
PIDFile=/var/run/keepalived.pid
KillMode=process
EnvironmentFile=-/etc/sysconfig/keepalived
ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

注意:

vm2節點上須要修改state爲BACKUP, priority爲99 （priority值必須小於master節點配置值）

# cat /etc/keepalived/check_apiserver.sh
#!/bin/bash
flag=$(systemctl status kube-apiserver &> /dev/null;echo $?)
if [[ $flag != 0 ]];then
        echo "kube-apiserver is down,close the keepalived"
        systemctl stop keepalived
fi
# chmod +x /etc/keepalived/check_apiserver.sh 
# systemctl daemon-reload
# systemctl enable keepalived
# systemctl start keepalived

5、修改客戶端配置

一、Kubelet.kubeconfig 、bootstrap.kubeconfig、kube-proxy.kubeconfig 配置

# grep 'server' /etc/kubernetes/kubelet.kubeconfig 
server: https://192.168.115.4:6443
# grep 'server' /etc/kubernetes/bootstrap.kubeconfig 
server: https://192.168.115.4:6443
# grep 'server' /etc/kubernetes/kube-proxy.kubeconfig 
    server: https://192.168.115.4:6443

二、confing配置

# grep 'server' /root/.kube/config 
server: https://192.168.115.4:6443

三、重啓客戶端服務

# systemctl restart kubelet 
# systemctl restart kube-proxy

6、測試

一、關閉服務前的集羣狀態，VIP在vm1節點上
K8S集羣Master高可用實踐

K8S集羣Master高可用實踐

二、在vm1上將kube-apiserver服務中止，能夠看到VIP消息，但任何能夠鏈接master獲取pod信息
K8S集羣Master高可用實踐
日誌顯示vip被自動移除
K8S集羣Master高可用實踐
三、在vm2上能看到自動註冊上了VIP，且kubectl客戶端鏈接正常
K8S集羣Master高可用實踐
K8S集羣Master高可用實踐
四、在vm1上將kube-apiserver、keepalived服務啓動，因爲配置的是主從模式，因此會搶佔VIP
K8S集羣Master高可用實踐
五、在vm2上能夠看到VIP的釋放，keepalived從新進入backup狀態
K8S集羣Master高可用實踐
六、在整個過程當中能夠用其餘的客戶端來鏈接master VIP來測試服務器的連續性
K8S集羣Master高可用實踐

7、使用haproxy改進

只用keepalived實現master ha，當api-server的訪問量大的時候，會有性能瓶頸問題，經過配置haproxy，能夠同時實現master的ha和流量的負載均衡。
一、安裝和配置haproxy，兩臺master作一樣的配置

# yum -y install haproxy
# cat /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    mode                    tcp
    log                     global
    option                  tcplog
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout check           10s
    maxconn                 3000

listen stats
    mode   http
    bind :10086
    stats   enable
    stats   uri     /admin?stats
    stats   auth    admin:admin
    stats   admin   if TRUE

frontend  k8s_https *:8443
    mode      tcp
    maxconn      2000
    default_backend     https_sri

backend https_sri
    balance      roundrobin
    server s1 192.168.115.5:6443  check inter 10000 fall 2 rise 2 weight 1
    server s2 192.168.115.6:6443  check inter 10000 fall 2 rise 2 weight 1

二、修改kube-apiserver配置,ip地址根據實際狀況修改

# grep 'address' /usr/lib/systemd/system/kube-apiserver.service     
  --advertise-address=192.168.115.5 \
  --bind-address=192.168.115.5 \
  --insecure-bind-address=127.0.0.1 \

三、修改keepalived啓動腳本和配置文件,vrrp腳本的ip地址根據實際狀況修改

# cat /usr/lib/systemd/system/keepalived.service            
[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target 
Require=haproxy.service
########如下輸出省略#########
# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived  
global_defs {  
    notification_email {   
        ylw@fjhb.cn
    }   
    notification_email_from admin@fjhb.cn  
    smtp_server 127.0.0.1  
    smtp_connect_timeout 30  
    router_id LVS_MASTER  
}  

vrrp_script check_apiserver {
    script "curl -o /dev/null -s -w %{http_code} -k  https://192.168.115.5:6443"
    interval 3
    timeout 3
    fall 2
    rise 2
}  
########如下輸出省略#########

四、修改kubelet和kubectl客戶端配置文件，指向haproxy的端口8443

# grep '192' /etc/kubernetes/bootstrap.kubeconfig 
    server: https://192.168.115.4:8443
# grep '192' /etc/kubernetes/kubelet.kubeconfig 
    server: https://192.168.115.4:8443
# grep '192' /etc/kubernetes/kube-proxy.kubeconfig 
server: https://192.168.115.4:8443
# grep '192' /root/.kube/config 
server: https://192.168.115.4:8443

五、重啓服務驗證
master

# systemctl daemon-reload
# systemctl enable haproxy 
# systemctl start haproxy 
# systemctl restart keepalived 
# systemctl restart kube-apiserver

kubelet

# systemctl restart kubelet
# systemctl restart kube-proxy

K8S集羣Master高可用實踐
K8S集羣Master高可用實踐

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程