008.Kubernetes二進制部署Nginx實現高可用

一 Nginx代理實現kube-apiserver高可用

1.1 Nginx實現高可用

基於 nginx 代理的 kube-apiserver 高可用方案。

控制節點的 kube-controller-manager、kube-scheduler 是多實例部署，因此只要有一個實例正常，就能夠保證高可用；

集羣內的 Pod 使用 K8S 服務域名 kubernetes 訪問 kube-apiserver， kube-dns 會自動解析出多個 kube-apiserver 節點的 IP，因此也是高可用的；

在每一個節點起一個 nginx 進程，後端對接多個 apiserver 實例，nginx 對它們作健康檢查和負載均衡；

kubelet、kube-proxy、controller-manager、scheduler 經過本地的 nginx（監聽 127.0.0.1）訪問 kube-apiserver，從而實現 kube-apiserver 的高可用；

從而基於 nginx 4 層透明代理功能實現 K8S 節點( master 節點和 worker 節點)高可用訪問 kube-apiserver 。

1.2 下載編譯Nginx

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# wget http://nginx.org/download/nginx-1.15.3.tar.gz
  3 [root@k8smaster01 work]# tar -xzvf nginx-1.15.3.tar.gz
  4 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/
  5 [root@k8smaster01 nginx-1.15.3]# mkdir nginx-prefix
  6 [root@k8smaster01 nginx-1.15.3]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
  7 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/
  8 [root@k8smaster01 nginx-1.15.3]# make && make install

解釋：

• --with-stream：開啓 4 層透明轉發(TCP Proxy)功能；
• --without-xxx：關閉全部其餘功能，這樣生成的動態連接二進制程序依賴最小。
• [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/
• [root@k8smaster01 nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v

1.3 驗證編譯後的Nginx

  1 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3
  2 [root@k8smaster01 nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v
  3 nginx version: nginx/1.15.3
  4 [root@k8smaster01 nginx-1.15.3]# ldd ./nginx-prefix/sbin/nginx	#查看 nginx 動態連接的庫
  5         linux-vdso.so.1 =>  (0x00007ffdda980000)
  6         libdl.so.2 => /lib64/libdl.so.2 (0x00007feb37300000)
  7         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007feb370e4000)
  8         libc.so.6 => /lib64/libc.so.6 (0x00007feb36d17000)
  9         /lib64/ld-linux-x86-64.so.2 (0x00007feb37504000)

提示：因爲只開啓了 4 層透明轉發功能，因此除了依賴 libc 等操做系統核心 lib 庫外，沒有對其它 lib 的依賴(如 libz、libssl 等)，以便達到精簡編譯的目的。

1.4 安裝和部署Nginx

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}
  7   done						#建立Nginx目錄
  8 [root@k8smaster01 ~]# cd /opt/k8s/work
  9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
 10 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
 11   do
 12     echo ">>> ${master_ip}"
 13     scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${master_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
 14     ssh root@${master_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
 15     ssh root@${master_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
 16   done						#分發Nginx二進制

1.5 配置Nginx 四層透明轉發

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# cat > kube-nginx.conf <<EOF
  3 worker_processes 1;
  4 
  5 events {
  6     worker_connections  1024;
  7 }
  8 
  9 stream {
 10     upstream backend {
 11         hash $remote_addr consistent;
 12         server 172.24.8.71:6443        max_fails=3 fail_timeout=30s;
 13         server 172.24.8.72:6443        max_fails=3 fail_timeout=30s;
 14         server 172.24.8.73:6443        max_fails=3 fail_timeout=30s;
 15     }
 16 
 17     server {
 18         listen 127.0.0.1:8443;
 19         proxy_connect_timeout 1s;
 20         proxy_pass backend;
 21     }
 22 }
 23 EOF
 24 [root@k8smaster01 ~]# cd /opt/k8s/work
 25 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
 26 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
 27   do
 28     echo ">>> ${master_ip}"
 29     scp kube-nginx.conf  root@${master_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
 30   done						#分發Nginx四層透明代理配置文件

1.6 配置Nginx system

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# cat > kube-nginx.service <<EOF
  3 [Unit]
  4 Description=kube-apiserver nginx proxy
  5 After=network.target
  6 After=network-online.target
  7 Wants=network-online.target
  8 
  9 [Service]
 10 Type=forking
 11 ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
 12 ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
 13 ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
 14 PrivateTmp=true
 15 Restart=always
 16 RestartSec=5
 17 StartLimitInterval=0
 18 LimitNOFILE=65536
 19 
 20 [Install]
 21 WantedBy=multi-user.target
 22 EOF

1.7 分發Nginx systemd

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp kube-nginx.service  root@${master_ip}:/etc/systemd/system/
  7   done

二 啓動並驗證

2.1 啓動Nginx

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
  7   done

2.2 檢查Nginx服務

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "systemctl status kube-nginx |grep 'Active:'"
  7   done