Kubernetes集羣生產環境搭建全過程

本文詳細講解如何搭建高可用的Kubernetes集羣，如下簡稱k8s

由三臺服務器(CentOS 7.0)組成master集羣,命名爲m1，m2，m3，ip用m1 m2 m3來代替

etcd集羣搭建

首先搭建etcd集羣,etcd爲k8s集羣的核心組成部分，負責全部集羣配置信息和服務信息的存儲，因此必需要保證高可用，此處採用etcd的靜態服務發現，即在etcd啓動的時候，肯定etcd node的ip。

yum安裝etcd yum install -y etcd

分別在三臺機器啓動etcd進程(實際操做中須要將m1 m2 m3替換成實際的ip地址)
m1:

etcd -name infra1 -initial-advertise-peer-urls http://m1:2380 -listen-peer-urls http://m1:2380 -listen-client-urls http://m1:2379,http://127.0.0.1:2379 -advertise-client-urls http://m1:2379 -initial-cluster-token etcd-cluster-1 -initial-cluster infra1=http://m1:2380,infra2=http://m2:2380,infra3=http://m3:2380 -initial-cluster-state new

m2:

etcd -name infra2 -initial-advertise-peer-urls http://m2:2380 -listen-peer-urls http://m2:2380 -listen-client-urls http://m2:2379,http://127.0.0.1:2379 -advertise-client-urls http://m2:2379 -initial-cluster-token etcd-cluster-1 -initial-cluster infra1=http://m1:2380,infra2=http://m2:2380,infra3=http://m3:2380 -initial-cluster-state new

m3:

etcd -name infra3 -initial-advertise-peer-urls http://m3:2380 -listen-peer-urls http://m3:2380 -listen-client-urls http://m3:2379,http://127.0.0.1:2379 -advertise-client-urls http://m3:2379 -initial-cluster-token etcd-cluster-1 -initial-cluster infra1=http://m1:2380,infra2=http://m2:2380,infra3=http://m3:2380 -initial-cluster-state new

集羣啓動後，選取一臺機器執行 etcdctl cluster-health

若是出現3個相似 member cbfa6350b369c3a is healthy 的字樣，說明etcd集羣部署成功。

實戰中採起了systemd來進行管理
新建文件 /usr/lib/systemd/system/etcd.service
內容以下：

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/root
ExecStart=etcd -name infra1 -initial-advertise-peer-urls http://m1:2380 -listen-peer-urls http://m1:2380 -listen-client-urls http://m1:2379,http://127.0.0.1:2379 -advertise-client-urls http://m1:2379 -initial-cluster-token etcd-cluster-1 -initial-cluster infra1=http://m1:2380,infra2=http://m2:2380,infra3=http://m3:2380 -initial-cluster-state new
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

這樣能夠使用 systemctl start etcd來啓動服務

Kubernete master搭建

master搭建是經過自動化搭建腳本實現的 腳本內容以下

#!/bin/bash
echo '################ Prerequisites...'
systemctl stop firewalld
systemctl disable firewalld
yum -y install ntp
systemctl start ntpd
systemctl enable ntpd

echo '################ Installing flannel...'
#安裝flannel
yum install flannel -y

echo '################ Add subnets for flannel...'
A_SUBNET=172.17.0.0/16
B_SUBNET=192.168.0.0/16
C_SUBNET=10.254.0.0/16

FLANNEL_SUBNET=$A_SUBNET
SERVICE_SUBNET=$B_SUBNET
OCCUPIED_IPs=(`ifconfig -a | grep 'inet ' | cut -d ':' -f 2 |cut -d ' ' -f 1 | grep -v '^127'`)
for ip in ${OCCUPIED_IPs[@]};do
    if [ $(ipcalc -n $ip/${A_SUBNET#*/}) == $(ipcalc -n ${A_SUBNET}) ];then
        FLANNEL_SUBNET=$C_SUBNET
        SERVICE_SUBNET=$B_SUBNET
        break
    fi
    if [ $(ipcalc -n $ip/${B_SUBNET#*/}) == $(ipcalc -n ${B_SUBNET}) ];then
        FLANNEL_SUBNET=$A_SUBNET
        SERVICE_SUBNET=$C_SUBNET
        break
    fi
    if [ $(ipcalc -n $ip/${C_SUBNET#*/}) == $(ipcalc -n ${C_SUBNET}) ];then
        FLANNEL_SUBNET=$A_SUBNET
        SERVICE_SUBNET=$B_SUBNET
        break
    fi
done

while ((1));do
    sleep 2
    etcdctl cluster-health
    flag=$?
    if [ $flag == 0 ];then
etcdctl mk /coreos.com/network/config '{"Network":"'${FLANNEL_SUBNET}'"}'
        break
    fi
done

echo '################ Starting flannel...'
#此處將m1 m2 m3 換成實際的ip
echo -e "FLANNEL_ETCD=\"http://m1:2379,http://m2:2379,http://m3:2379\"
FLANNEL_ETCD_KEY=\"/coreos.com/network\"" > /etc/sysconfig/flanneld
systemctl enable flanneld
systemctl start flanneld

echo '################ Installing K8S...'
yum -y install kubernetes
echo 'KUBE_API_ADDRESS="--address=0.0.0.0"
KUBE_API_PORT="--port=8080"
KUBELET_PORT="--kubelet_port=10250"
KUBE_ETCD_SERVERS="--etcd_servers=http://m1:2379,http://m2:2379,http://m3:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range='${SERVICE_SUBNET}'"
KUBE_ADMISSION_CONTROL="--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
KUBE_API_ARGS=""' > /etc/kubernetes/apiserver

echo '################ Start K8S components...'
for SERVICES in kube-apiserver kube-controller-manager kube-scheduler; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES
done

Kubernete node搭建

node的搭建也是經過自動化部署腳本實現的，腳本內容以下：

#!/bin/bash

echo '################ Prerequisites...'
#關閉firewall 開啓ntp時間同步
systemctl stop firewalld
systemctl disable firewalld
yum -y install ntp
systemctl start ntpd
systemctl enable ntpd

#安裝kubernetes所須要的幾個軟件
yum -y install kubernetes docker flannel bridge-utils


#此處使用了一個vip 命名爲vip 實際部署時須要替換爲你的集羣的vip 使用此ip的服務有 kube-master（8080） registry（5000） skydns（53）

echo '################ Configuring nodes...'
echo '################ Configuring nodes > Find Kube master...'
KUBE_REGISTRY_IP="vip"
KUBE_MASTER_IP="vip"
echo '################ Configuring nodes > Configuring Minion...'
echo -e "KUBE_LOGTOSTDERR=\"--logtostderr=true\"
KUBE_LOG_LEVEL=\"--v=0\"
KUBE_ALLOW_PRIV=\"--allow_privileged=false\"
KUBE_MASTER=\"--master=http://${KUBE_MASTER_IP}:8080\"" > /etc/kubernetes/config
echo '################ Configuring nodes > Configuring kubelet...'
#取每一個node機器的eth0的ip做爲標識
KUBE_NODE_IP=`ifconfig eth0 | grep "inet " | awk '{print $2}'`

#api_servers 使用master1 master2 master3的ip數組形式
echo -e "KUBELET_ADDRESS=\"--address=0.0.0.0\"
KUBELET_PORT=\"--port=10250\"
KUBELET_HOSTNAME=\"--hostname_override=${KUBE_NODE_IP}\"
KUBELET_API_SERVER=\"--api_servers=http://m1:8080,http://m2:8080,http://m3:8080\"
KUBELET_ARGS=\"--cluster-dns=vip --cluster-domain=k8s --pod-infra-container-image=${KUBE_REGISTRY_IP}:5000/pause:latest\"" > /etc/kubernetes/kubelet


#flannel讀取etcd配置信息 爲本機的docker0分配ip 保證node集羣子網互通
echo '################ Configuring flannel...'
echo -e "FLANNEL_ETCD=\"http://m1:2379,http://m2:2379,http://m3:2379\"
FLANNEL_ETCD_KEY=\"/coreos.com/network\"" > /etc/sysconfig/flanneld


echo '################ Accept private registry...'
echo "OPTIONS='--selinux-enabled --insecure-registry ${KUBE_REGISTRY_IP}:5000'
DOCKER_CERT_PATH=/etc/docker" > /etc/sysconfig/docker

echo '################ Start K8S Components...'
systemctl daemon-reload
for SERVICES in kube-proxy flanneld; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES
done

echo '################ Resolve interface conflicts...'
systemctl stop docker
ifconfig docker0 down
brctl delbr docker0

echo '################ Accept private registry...'
echo -e "OPTIONS='--selinux-enabled --insecure-registry ${KUBE_REGISTRY_IP}:5000'
DOCKER_CERT_PATH=/etc/docker" > /etc/sysconfig/docker

for SERVICES in docker kubelet; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES
done

至此，Kubernetes master和node的搭建就結束了。