K8S多master部署二：部署LoadBlance

前情提要

如下全部操做均在單master羣集已完成部署的狀況下進行。

全部服務器均保證防火牆常閉，核心功能selinux關閉。

服務器角色分配

角色	地址	安裝組件
master	192.168.142.220	kube-apiserver kube-controller-manager kube-scheduler etcd
master02	192.168.142.120	kube-apiserver kube-controller-manager kube-scheduler
node1	192.168.142.136	kubelet kube-proxy docker flannel etcd
node2	192.168.142.132	kubelet kube-proxy docker flannel etcd
nginx1	192.168.142.130	nginx keepalived
nginx2	192.168.142.140	nginx keepalived
VIP	192.168.142.20	虛擬地址

1、nginx端部署

創建nginx的YUM源

[root@lb-ma ~]# cat > /etc/yum.repos.d/nginx.repo <<EOF
[nginx]
name=nginx-repo
baseurl=http://nginx.org/packages/centos/7/\$basearch/
gpgcheck=0

EOF

安裝nginx並進行配置

[root@lb-ma ~]# yum install nginx -y

#添加stream模塊實現四層轉發
[root@lb-ma ~]# vim /etc/nginx/nginx.conf
###添加
stream {

   log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
        server 192.168.142.220:6443;
        server 192.168.142.120:6443;
    }
    server {
                listen 6443;
                proxy_pass k8s-apiserver;
    }
    }

#開啓服務
[root@lb-ma ~]# systemctl start nginx
[root@lb-ma ~]# systemctl enable nginx

安裝Keppalived服務

[root@lb-ma ~]# yum -y install keepalived

##修改keepalived配置文件
[root@lb-ma ~]# vim /etc/keepalived/keepalived.conf
###本來的所有刪除，按下面新建
! Configuration File for keepalived

global_defs {
  # 接收郵件地址
     notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   # 郵件發送地址
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
    script "/usr/local/nginx/sbin/check_nginx.sh"
}

vrrp_instance VI_1 {
    state MASTER   #備服務器改成BACKUP
    interface ens33   #監控ens33網卡
    virtual_router_id 51 # VRRP 路由 ID實例，每一個實例是惟一的
    priority 100    # 優先級，備服務器設置 90
    advert_int 1    # 指定VRRP 心跳包通告間隔時間，默認1秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.142.20/24   #VIP地址
    }
    track_script {
        check_nginx
    }
}

創建監控腳本

一旦nginx處於down狀態，將會自動關閉keeplived

[root@lb-ma ~]# mkdir -p /usr/local/nginx/sbin/
[root@lb-ma ~]# vim /usr/local/nginx/sbin/check_nginx.sh
##手動進行編寫
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi
[root@lb-ma ~]# chmod +x /usr/local/nginx/sbin/check_nginx.sh

#開啓keeplived
[root@lb-ma ~]# systemctl start keepalived
[root@lb-ma ~]# systemctl enable keepalived

---

此時，前面的負載均衡已經配置完畢，可是並不能起到實際的做用，緣由就是後方node節點中負責進行身份識別的kubeconfig文件中的地址沒有改變，沒法識別。

2、node端修改

更改kubeconfig中的地址

[root@node1 ~]# vim /opt/kubernetes/cfg/bootstrap.kubeconfig
[root@node1 ~]# vim /opt/kubernetes/cfg/kube-proxy.kubeconfig
[root@node1 ~]# vim /opt/kubernetes/cfg/kubelet.kubeconfig
##三個文件所有改成
    server: https://192.168.142.20:6443    #指向VIP地址

重啓kubelet & kube-proxy服務

[root@node1 ~]# systemctl restart kubelet
[root@node1 ~]# systemctl restart kube-proxy

以上，就是nginx作負載均衡，keppalived作雙機熱備的所有部署過程

---

DEMO：創建POD進行檢測

master端創建測試pod

[root@master ~]# kubectl run nginx --image=nginx
##創建pod
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx created

#查看創建打的pod
[root@master ~]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-7tdvp   1/1     Running   0          21s

此時，pod只能進行簡單的查看，一旦查看日誌會報錯。爲了解決這個問題，可採用下面的辦法解決。

#注意日誌問題
[root@master ~]# kubectl logs nginx-dbddb74b8-7tdvp
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-7tdvp)

###解決辦法：
[root@master ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created