基於K3S構建高可用Rancher集羣

若是你是條船，漂泊就是你的命運，可別靠岸

北島

K3S簡述：
html

K3s (輕量級 Kubernetes)：和 RKE 相似，也是通過認證的 Kubernetes 發行版。它比 RKE 更新，更易用且更輕量化，所有組件都在一個小於 100 MB 的二進制文件中。從 Rancher v2.4 開始，Rancher 能夠安裝在 K3s 集羣上。

詳情見：https://rancher2.docs.rancher.cn/docs/installation/_indexnode

Rancher簡述：

Rancher 是爲使用容器的公司打造的容器管理平臺。Rancher 簡化了使用 Kubernetes 的流程，開發者能夠隨處運行 Kubernetes（Run Kubernetes Everywhere），知足 IT 需求規範，賦能 DevOps 團隊。

詳情見：https://rancher2.docs.rancher.cn/docs/overview/_indexmysql

使用環境：

操做系統	主機名	IP地址	節點	配置
CentOS 7 1810	nginx-master	192.168.111.21	Nginx主服務器	2C4G
CentOS 7 1810	nginx-backup	192.168.111.22	Nginx備服務器	2C4G
ubuntu-18.04.3-live-server	k3s-node1	192.168.111.50	k3s節點1	4C8G
ubuntu-18.04.3-live-server	k3s-node2	192.168.111.51	k3s節點2	4C8G
CentOS 7 1810	k3s-mysql	192.168.111.52	mysql	4C8G

部署前系統環境準備：

關閉防火牆和SeLinux

爲防止因端口問題形成集羣組建失敗，咱們在這裏提早關閉防火牆以及selinuxlinux

centos ：nginx

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

Ubuntu：c++
```
sudo ufw disable
```

節點及Docker功能調優git

https://rancher2.docs.rancher.cn/docs/best-practices/optimize/os/_index

配置host文件：

192.168.111.21 nginx-master
192.168.111.22 nginx-backup
192.168.111.50 k3s-node1
192.168.111.51 k3s-node2
192.168.111.52 k3s-mysql

配置host文件，並確保每臺機器上均可以經過主機名互通sql

須要用到的工具：

此安裝須要如下 CLI 工具。請確保這些工具已經安裝並在$PATH中可用docker

CLI工具的安裝在k3s節點上進行數據庫

kubectl - Kubernetes 命令行工具.
helm - Kubernetes 的軟件包管理工具。
請參閱Helm 版本要求選擇 Helm 的版原本安裝 Rancher。

開始部署：

安裝 Kubectl：

安裝參考K8S官網，因爲某些特殊緣由，此處咱們使用snap

sudo apt-get install snapd
sudo snap install kubectl --classic # 此處安裝較慢，請耐心等待
# 驗證安裝
kubectl help

安裝 Helm：

安裝參考Helm官網，Helm是Kubernetes的包管理器，Helm的版本須要高於v3

# 下載安裝包
wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
# 解壓
tar zxvf helm-v3.2.1-linux-amd64.tar.gz
# 將二進制文件移動至/usr/local/bin/
sudo mv linux-amd64/helm /usr/local/bin/helm
# 驗證安裝
helm help

建立 Nginx+Keepalived 集羣：

此處在CentOS節點上進行

安裝 Nginx

# 下載Nginx安裝包
wget http://nginx.org/download/nginx-1.17.10.tar.gz
# 解壓安裝包
tar zxvf nginx-1.17.10.tar.gz
# 安裝編譯時必備的軟件包
yum install -y gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel libnl3-devel
# 進入nginx目錄，此處咱們須要使用https，全部在編譯時選擇 --with-http_ssl_module 模塊
cd nginx-1.17.10
mkdir -p /usr/local/nginx
./configure --prefix=/usr/local/nginx --with-stream
# 安裝nginx
make && make install
# 建立nginx命令軟鏈接
ln -s /usr/local/nginx/sbin/nginx /usr/local/bin/nginx
# 驗證安裝
nginx -V
# 啓動nginx
nginx

安裝 Keepalived

# 下載安裝包
wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz
# 解壓安裝包
tar zxvf keepalived-2.0.20.tar.gz
# 編譯安裝keepalived
cd keepalived-2.0.20
mkdir /usr/local/keepalived
./configure --prefix=/usr/local/keepalived/
make && make install
# 配置 keepalived 爲系統服務
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
touch /etc/init.d/keepalived
chmod +x /etc/init.d/keepalived # keepalived 中的內容見下文
vim /etc/init.d/keepalived
# 配置 keepalived
mkdir /etc/keepalived/
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
vim /etc/keepalived/keepalived.conf #keepalived.conf 中的內容見下文
# 啓動keepalived
systemctl start keepalived
systemctl enable keepalived
# 驗證
systemctl status keepalived
# 此時keepalived應該是運行，一個爲master，一個爲backup， master上執行 ip addr 命令時，應該存在一個虛擬ip地址，backup上不該該有
# 訪問 https://192.168.111.20 驗證配置

# /etc/init.d/keepalived文件內容
#!/bin/sh
#
# Startup script for the Keepalived daemon
#
# processname: keepalived
# pidfile: /var/run/keepalived.pid
# config: /etc/keepalived/keepalived.conf
# chkconfig: - 21 79
# description: Start and stop Keepalived

# Source function library
. /etc/rc.d/init.d/functions

# Source configuration file (we set KEEPALIVED_OPTIONS there)
. /etc/sysconfig/keepalived

RETVAL=0

prog="keepalived"

start() {
    echo -n $"Starting $prog: "
    daemon keepalived ${KEEPALIVED_OPTIONS}
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
}

stop() {
    echo -n $"Stopping $prog: "
    killproc keepalived
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog
}

reload() {
    echo -n $"Reloading $prog: "
    killproc keepalived -1
    RETVAL=$?
    echo
}

# See how we were called.
case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    reload)
        reload
        ;;
    restart)
        stop
        start
        ;;
    condrestart)
        if [ -f /var/lock/subsys/$prog ]; then
            stop
            start
        fi
        ;;
    status)
        status keepalived
        RETVAL=$?
        ;;
    *)
        echo "Usage: $0 {start|stop|reload|restart|condrestart|status}"
        RETVAL=1
esac

exit $RETVAL

# /etc/keepalived/keepalived.conf 中的內容
! Configuration File for keepalived

global_defs {
   router_id 192.168.111.21 # 此id在網絡中有且只有一個，不該有重複的id
}

vrrp_script chk_nginx {     #由於要檢測nginx服務狀態，因此建立一個檢查腳本
    script "/usr/local/keepalived/check_ng.sh"
    interval 3
}

vrrp_instance VI_1 {
    state MASTER    # 配置此節點爲master，備機上設置爲BACKUP
    interface ens33    # 設置綁定的網卡
    virtual_router_id 51    # vrrp 組， 主備的vrrp組應該同樣
    priority 120    # 優先級，優先級大的爲主
    advert_int 1    # 檢查間隔
    authentication { # 認證
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress { # 虛擬IP
        192.168.111.20
    }
    track_script {    # 執行腳本
        chk_nginx
    }
}

# /usr/local/keepalived/check_ng.sh 中的內容
#!/bin/bash
d=`date --date today +%Y%m%d_%H:%M:%S`
n=`ps -C nginx --no-heading|wc -l`
if [ $n -eq "0" ]; then
        nginx
        n2=`ps -C nginx --no-heading|wc -l`
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                systemctl stop keepalived
        fi
fi

安裝 docker-ce ：

此處在RKE節點上進行

# 移除舊版本Docker
sudo apt-get remove docker docker-engine docker.io containerd runc
# 安裝工具包
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
# 添加 Docker官方 GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# 添加 stable apt 源
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
# 安裝 Docker-ce
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
# 驗證安裝
docker info
# 將當前用戶加入"docker"用戶組，加入到該用戶組的帳號在隨後安裝過程會用到。用於節點訪問的SSH用戶必須是節點上docker組的成員
sudo usermod -aG docker $USER

配置四層負載均衡

此處在Nginx集羣操做

# 更新nginx配置文件
# vim /usr/local/nginx/conf/nginx.conf

#user  nobody;
worker_processes  4;
worker_rlimit_nofile 40000;

events {
    worker_connections  8192;
}

stream {
    upstream rancher_servers_http {
        least_conn;
        server 192.168.111.50:80 max_fails=3 fail_timeout=5s;
        server 192.168.111.51:80 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     80;
        proxy_pass rancher_servers_http;
    }

    upstream rancher_servers_https {
        least_conn;
        server 192.168.111.50:443 max_fails=3 fail_timeout=5s;
        server 192.168.111.51:443 max_fails=3 fail_timeout=5s;
    }

    server {
        listen     443;
        proxy_pass rancher_servers_https;
    }
}

部署 MySQL 5.7

# 下載地址：https://dev.mysql.com/downloads/mysql/5.7.html#downloads
# 建立運行MySQL數據庫的用戶和用戶組
groupadd -r mysql
useradd -r -g mysql mysql
# 解壓安裝包，更改目錄權限，建立數據庫目錄
tar zxvf mysql-5.7.30-linux-glibc2.12-x86_64.tar.gz
mkdir -p /app/mysql/data
mv mysql-5.7.30-linux-glibc2.12-x86_64/* /app/mysql/
chown -R mysql:mysql /app/mysql
# 初始化數據庫
cd /app/mysql
./bin/mysqld --initialize \
--user=mysql --basedir=/app/mysql/ \
--datadir=/app/mysql/data/
# !!注意最後一行的初始密碼
7Jlhi:gg?rE0
# 建立RSA private key
./bin/mysql_ssl_rsa_setup --datadir=/app/mysql/data/
# 添加 MySQL 到開機啓動，修改/etc/init.d/mysqld中的basedir和datadir
cp support-files/mysql.server /etc/init.d/mysqld

basedir=/app/mysql
datadir=/app/mysql/data

chkconfig mysqld on
# 修改環境變量
vim /etc/profile
# 添加
export PATH=/app/mysql/bin:$PATH
# 使環境變量生效
source /etc/profile

# 備份系統自帶的/etc/my.cnf，在/app/mysql/目錄新建my.cnf,而且將文件屬性調整爲mysql:mysql
mv /etc/my.cnf /etc/my.cnf.bak
touch /app/mysql/my.cnf    # 具體內容見下文

# 啓動mysql
/etc/init.d/mysqld start
# 建立mysql.sock軟連接
ln -s /app/mysql/mysql.sock /tmp/mysql.sock
# 使用初始密碼登錄
mysql -uroot -p

# 登錄成功後修改密碼
alter user 'root'@'localhost' identified by "12345678";
flush privileges;

# 配置數據庫遠程登陸
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '12345678' WITH GRANT OPTION;
flush privileges;
# 驗證 略

# my.cnf
[mysqld]
character-set-server=utf8
datadir=/app/mysql/data
socket=/app/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[client]
character-set-server=utf8
socket=/app/mysql/mysql.sock

[mysql]
character-set-server=utf8
socket=/app/mysql/mysql.sock

部署k3s：

# 啓動 k3s Server
# ！注意，全部k3s節點上都要運行此命令
curl -sfL https://docs.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - server \
--datastore-endpoint="mysql://root:12345678@tcp(192.168.111.52:3306)/k3s"
# 驗證
sudo k3s kubectl get nodes

# 在每一個 Rancher Server 節點上安裝 K3s 時，會在節點上/etc/rancher/k3s/k3s.yaml位置建立一個kubeconfig文件。該文件包含用於徹底訪問集羣的憑據。# 複製 k3s.yaml 到 ~/.kube/config
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
# 驗證 kubectl
sudo kubectl get pods --all-namespaces

kube-system   coredns-8655855d6-c26h8                  1/1     Running     0          11m
kube-system   metrics-server-7566d596c8-v65fd          1/1     Running     0          11m
kube-system   helm-install-traefik-ttrfg               0/1     Completed   0          11m
kube-system   svclb-traefik-hxmzw                      2/2     Running     0          8m16s
kube-system   svclb-traefik-zxmg2                      2/2     Running     0          8m16s
kube-system   traefik-758cd5fc85-xsxbm                 1/1     Running     0          8m16s
kube-system   local-path-provisioner-6d59f47c7-497rl   1/1     Running     0          11m

部署 Rancher：

添加 Helm Chart 倉庫

helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

爲 Rancher 建立 Namespace

sudo kubectl create namespace cattle-system

生成證書

mkdir certs
cd certs
touch ~/.rnd
cp /usr/lib/ssl/openssl.cnf ./ # openssl.cnf內容有改動，詳情見下文
vim openssl.cnf

openssl genrsa -out cakey.pem 2048

openssl req -x509 -new -nodes -key cakey.pem \
-days 36500 \
-out cacerts.pem \
-extensions v3_ca \
-subj "/CN=rancher.local.com" \
-config ./openssl.cnf

openssl genrsa -out server.key 2048

openssl req -new -key server.key \
-out server.csr \
-subj "/CN=rancher.local.com" \
-config ./openssl.cnf

openssl x509 -req -in server.csr \
-CA cacerts.pem \
-CAkey cakey.pem \
-CAcreateserial -out server.crt \
-days  36500 -extensions v3_req \
-extfile ./openssl.cnf

openssl x509 -noout -in server.crt -text | grep DNS

cp server.crt tls.crt
cp server.key tls.key

openssl修改部分

[req]
distinguished_name      = req_distinguished_name
req_extetions           = v3_req
x509_extensions         = v3_ca

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = rancher.local.com

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer:always
basicConstraints = critical,CA:true
subjectAltName = @alt_names

ca證書密文

sudo kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=./tls.crt --key=./tls.key
sudo kubectl -n cattle-system create secret generic tls-ca \
--from-file=cacerts.pem

部署 Rancher 集羣

sudo helm install rancher rancher-stable/rancher \
 --namespace cattle-system \
 --set hostname=rancher.local.com \
 --set ingress.tls.source=secret \
 --set privateCA=true

等待 Rancher 集羣運行

sudo kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
deployment "rancher" successfully rolled out

若是看到如下錯誤：error: deployment "rancher" exceeded its progress deadline, 能夠經過運行如下命令來檢查 deployment 的狀態
```
sudo kubectl -n cattle-system get deploy rancher
```
搭建完成，在你的hosts文件中，把域名解析到負載均衡器，訪問 https://rancher.local.com

本文分享自微信公衆號 - 運維開發故事（mygsdcsf）。
若有侵權，請聯繫 support@oschina.cn 刪除。
本文參與「OSC源創計劃」，歡迎正在閱讀的你也加入，一塊兒分享。