本文檔使用haproxy+keepalived的方式部署爲集羣高可用模式，lvs+keepalived也能夠實現，理論上來講能夠照搬用於生產。本文檔最初是基於kubenetes1.6版本編寫的，對於kuberentes1.8及以上版本一樣適用，只是個別位置有稍許變更，變更的地方將特別註明版本要求。css
本系列文檔介紹使用二進制部署 kubernetes 集羣的全部步驟，而不是使用 kubeadm 等自動化方式來部署集羣，同時開啓了集羣的TLS安全認證，安裝時使用vmvare建立的虛擬機，理論上適用於全部bare metal環境、on-premise環境和公有云環境。html
在部署的過程當中，將詳細列出各組件的啓動參數，給出配置文件，詳解它們的含義和可能遇到的問題。前端
部署完成後，你將理解系統各組件的交互原理，進而能快速解決實際問題。node
因此本文檔主要適合於那些有必定 kubernetes 基礎，想經過一步步部署的方式來學習和了解系統配置、運行原理的人。mysql
在下面的步驟中，將在8臺CentOS系統的虛擬機上部署高可用集羣。linux
角色分配以下：
keepalived1+haproxy1+etcd1： 192.168.223.201
keepalived2+haproxy2+etcd2： 192.168.223.202
keepalived3+haproxy3+etcd3： 192.168.223.203
Master1： 192.168.223.204
Master2： 192.168.223.205
Node1： 192.168.223.206
Node2： 192.168.223.207
docker+hub： 192.168.223.208nginx
vip: 192.168.223.200
集羣訪問kube-apiserver使用此地址git
注意： etcd和keepalived+haproxy複用3臺主機，實際生產最好2臺單獨部署keepalived+haproxy，3臺單獨部署etcdgithub
修改 /etc/selinux/config 文件中設置 SELINUX=disabled setenforce 0
systemctl disable firewalld; systemctl stop firewalld;
192.168.223.208 上安裝harbor私有鏡像倉庫參考教程：https://github.com/vmware/harbor 須要使用到的全部docker images：https://pan.baidu.com/s/1YH6OCpmz8EiO1OlmmxLtfg 密碼：k2mrweb
如下正式開始部署
這一步是在安裝配置kubernetes的全部步驟中最容易出錯也最難於排查問題的一步，而這卻恰好是第一步，萬事開頭難，不要由於這點困難就望而卻步。
kubernetes 系統的各組件須要使用 TLS 證書對通訊進行加密，本文檔使用 CloudFlare 的 PKI 工具集 cfssl 來生成 Certificate Authority (CA) 和其它證書；
生成的 CA 證書和祕鑰文件以下：
使用證書的組件以下：
注意： 如下操做都在 192.168.223.201 主機上執行，而後分發到集羣全部主機，證書只須要建立一次便可，之後在向集羣中添加新節點時只要將 /etc/kubernetes/ 目錄下的證書拷貝到新節點上便可。
直接使用二進制源碼包安裝
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 chmod +x cfssl_linux-amd64 mv cfssl_linux-amd64 /usr/bin/cfssl wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 chmod +x cfssljson_linux-amd64 mv cfssljson_linux-amd64 /usr/bin/cfssljson wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x cfssl-certinfo_linux-amd64 mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
mkdir /root/ssl
cd /root/ssl
cat > ca-config.json << EOF
{
"signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF
建立 ca-csr.json 文件，內容以下：
cat > ca-csr.json << EOF
{
"CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca ls ca* ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
建立 kubernetes 證書籤名請求文件 kubernetes-csr.json：
cat > kubernetes-csr.json << EOF
{
"CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.223.200", "192.168.223.201", "192.168.223.202", "192.168.223.203", "192.168.223.204", "192.168.223.205", "192.168.223.206", "192.168.223.207", "192.168.223.208", "10.254.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes ls kubernetes* kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem
建立 admin 證書籤名請求文件 admin-csr.json：
cat > admin-csr.json << EOF
{
"CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF
注意： 這個admin 證書，是未來生成管理員用的kube config 配置文件用的，如今咱們通常建議使用RBAC 來對kubernetes 進行角色權限控制， kubernetes 將證書中的CN 字段 做爲User， O 字段做爲 Group。
在搭建完 kubernetes 集羣后，咱們能夠經過命令: kubectl get clusterrolebinding cluster-admin -o yaml ,查看到 clusterrolebinding cluster-admin 的 subjects 的 kind 是 Group，name 是 system:masters。 roleRef 對象是 ClusterRole cluster-admin。 意思是凡是 system:masters Group 的 user 或者 serviceAccount 都擁有 cluster-admin 的角色。 所以咱們在使用 kubectl 命令時候，才擁有整個集羣的管理權限。可使用 kubectl get clusterrolebinding cluster-admin -o yaml 來查看。
kubectl get clusterrolebinding cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" creationTimestamp: 2017-04-11T11:20:42Z labels: kubernetes.io/bootstrapping: rbac-defaults name: cluster-admin resourceVersion: "52" selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin uid: e61b97b2-1ea8-11e7-8cd7-f4e9d49f8ed0 roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:masters
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json|cfssljson -bare admin ls admin* admin.csr admin-csr.json admin-key.pem admin.pem
建立 kube-proxy 證書籤名請求文件 kube-proxy-csr.json：
cat > kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy ls kube-proxy* kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem
opsnssl 命令openssl x509 -noout -text -in kubernetes.pem
... Signature Algorithm: sha256WithRSAEncryption Issuer: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=Kubernetes Validity Not Before: Apr 5 05:36:00 2017 GMT Not After : Apr 5 05:36:00 2018 GMT Subject: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=kubernetes ... X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: DD:52:04:43:10:13:A9:29:24:17:3A:0E:D7:14:DB:36:F8:6C:E0:E0 X509v3 Authority Key Identifier: keyid:44:04:3B:60:BD:69:78:14:68:AF:A0:41:13:F6:17:07:13:63:58:CD X509v3 Subject Alternative Name: DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:192.168.223.200, IP Address:192.168.223.201, IP Address:192.168.223.202, IP Address:192.168.223.203, IP Address:192.168.223.204, IP Address:192.168.223.205, IP Address:192.168.223.206, IP Address:192.168.223.207, IP Address:192.168.223.208, IP Address:10.254.0.1 ...
cfssl-certinfo 命令cfssl-certinfo -cert kubernetes.pem
...
{
"subject": { "common_name": "kubernetes", "country": "CN", "organization": "k8s", "organizational_unit": "System", "locality": "BeiJing", "province": "BeiJing", "names": [ "CN", "BeiJing", "BeiJing", "k8s", "System", "kubernetes" ] }, "issuer": { "common_name": "Kubernetes", "country": "CN", "organization": "k8s", "organizational_unit": "System", "locality": "BeiJing", "province": "BeiJing", "names": [ "CN", "BeiJing", "BeiJing", "k8s", "System", "Kubernetes" ] }, "serial_number": "174360492872423263473151971632292895707129022309", "sans": [ "127.0.0.1", "192.168.223.200", "192.168.223.201", "192.168.223.202", "192.168.223.203", "192.168.223.204", "192.168.223.205", "192.168.223.206", "192.168.223.207", "192.168.223.208", "10.254.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "not_before": "2017-04-05T05:36:00Z", "not_after": "2018-04-05T05:36:00Z", "sigalg": "SHA256WithRSA", ...
將生成的證書和祕鑰文件（後綴名爲.pem）拷貝到全部機器的 /etc/kubernetes/ssl 目錄下備用；
mkdir -p /etc/kubernetes/ssl
cp *.pem /etc/kubernetes/ssl
ssh 192.168.223.202 "mkdir -p /etc/kubernetes/ssl" ssh 192.168.223.203 "mkdir -p /etc/kubernetes/ssl" ssh 192.168.223.204 "mkdir -p /etc/kubernetes/ssl" ssh 192.168.223.205 "mkdir -p /etc/kubernetes/ssl" ssh 192.168.223.206 "mkdir -p /etc/kubernetes/ssl" ssh 192.168.223.207 "mkdir -p /etc/kubernetes/ssl" scp *.pem 192.168.223.202:/etc/kubernetes/ssl scp *.pem 192.168.223.203:/etc/kubernetes/ssl scp *.pem 192.168.223.204:/etc/kubernetes/ssl scp *.pem 192.168.223.205:/etc/kubernetes/ssl scp *.pem 192.168.223.206:/etc/kubernetes/ssl scp *.pem 192.168.223.207:/etc/kubernetes/ssl
通常只需在兩臺master主機安裝便可
注意請下載對應的Kubernetes版本的安裝包。
wget https://dl.k8s.io/v1.6.0/kubernetes-client-linux-amd64.tar.gz tar -xzvf kubernetes-client-linux-amd64.tar.gz cp kubernetes/client/bin/kube* /usr/bin/ chmod a+x /usr/bin/kube*
export KUBE_APISERVER="https://192.168.223.200:6443" # 設置集羣參數 kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} # 設置客戶端認證參數 kubectl config set-credentials admin \ --client-certificate=/etc/kubernetes/ssl/admin.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/ssl/admin-key.pem # 設置上下文參數 kubectl config set-context kubernetes \ --cluster=kubernetes \ --user=admin # 設置默認上下文 kubectl config use-context kubernetes
注意： ~/.kube/config文件擁有對該集羣的最高權限，請妥善保管。若是node節點上須要使用kubelet工具，只需將此文件拷貝過去。
kubelet、kube-proxy 等 Node 機器上的進程與 Master 機器的 kube-apiserver 進程通訊時須要認證和受權；
kuberetes 1.4 開始支持由 kube-apiserver 爲客戶端生成 TLS 證書的 TLS Bootstrapping 功能，這樣就不須要爲每一個客戶端生成證書了；該功能當前僅支持爲 kubelet 生成證書；
如下操做只須要在 master1： 192.168.223.204 節點上執行，生成的 *.kubeconfig 文件能夠直接拷貝到其餘節點的 /etc/kubernetes 目錄下。
Token能夠是任意的包含128 bit的字符串，可使用安全的隨機數發生器生成。
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ') cat > token.csv <<EOF ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF
注意： 請檢查 token.csv 文件，確認其中的 ${BOOTSTRAP_TOKEN} 環境變量已經被真實的值替換。 **BOOTSTRAP_TOKEN ** 將被寫入到 kube-apiserver 使用的 token.csv 文件和 kubelet 使用的 bootstrap.kubeconfig 文件，若是後續從新生成了 BOOTSTRAP_TOKEN，則須要：
更新 token.csv 文件，分發到全部機器 (master 和 node）的 /etc/kubernetes/ 目錄下，分發到node節點上非必需； 從新生成 bootstrap.kubeconfig 文件，分發到全部 node 機器的 /etc/kubernetes/ 目錄下； 重啓 kube-apiserver 和 kubelet 進程； 從新 approve kubelet 的 csr 請求；
cp token.csv /etc/kubernetes/
scp token.csv 192.168.223.205:/etc/kubernetes/ scp token.csv 192.168.223.206:/etc/kubernetes/ scp token.csv 192.168.223.207:/etc/kubernetes/
cd /etc/kubernetes
export KUBE_APISERVER="https://192.168.223.200:6443" # 設置集羣參數 kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=bootstrap.kubeconfig # 設置客戶端認證參數 kubectl config set-credentials kubelet-bootstrap \ --token=${BOOTSTRAP_TOKEN} \ --kubeconfig=bootstrap.kubeconfig # 設置上下文參數 kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig # 設置默認上下文 kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
export KUBE_APISERVER="https://192.168.223.200:6443" # 設置集羣參數 kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-proxy.kubeconfig # 設置客戶端認證參數 kubectl config set-credentials kube-proxy \ --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \ --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig # 設置上下文參數 kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig # 設置默認上下文 kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
將兩個 kubeconfig 文件分發到全部 節點機器的 /etc/kubernetes/ 目錄
scp bootstrap.kubeconfig kube-proxy.kubeconfig 192.168.223.205:/etc/kubernetes/ scp bootstrap.kubeconfig kube-proxy.kubeconfig 192.168.223.206:/etc/kubernetes/ scp bootstrap.kubeconfig kube-proxy.kubeconfig 192.168.223.207:/etc/kubernetes/
kuberntes 系統使用 etcd 存儲全部數據，本次部署一個三節點高可用 etcd 集羣的步驟，分別爲：192.168.223.20一、192.168.223.20二、192.168.223.203。
須要爲 etcd 集羣建立加密通訊的 TLS 證書，這裏複用之前建立的 kubernetes 證書
ls /etc/kubernetes/ssl/*.pem
admin-key.pem admin.pem ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem kubernetes-key.pem kubernetes.pem
到 https://github.com/coreos/etcd/releases 頁面下載最新版本的二進制文件
wget https://github.com/coreos/etcd/releases/download/v3.1.5/etcd-v3.1.5-linux-amd64.tar.gz tar -xvf etcd-v3.1.5-linux-amd64.tar.gz mv etcd-v3.1.5-linux-amd64/etcd* /usr/sbin
或者直接使用yum命令安裝：
yum install etcd -y
vi /usr/lib/systemd/system/etcd.service，內容以下。注意替換IP地址爲你本身的etcd集羣的主機IP。
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/sbin/etcd \
--name ${ETCD_NAME} \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \ --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \ --initial-cluster infra1=https://192.168.223.201:2380,infra2=https://192.168.223.202:2380,infra3=https://192.168.223.203:2380 \ --initial-cluster-state new \ --data-dir=${ETCD_DATA_DIR} Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target
mkdir -p /var/lib/etcd，不然啓動服務的時候會報錯「Failed at step CHDIR spawning /usr/bin/etcd: No such file or directory」；環境變量配置文件 vi /etc/etcd/etcd.conf
# [member] ETCD_NAME=infra1 ETCD_DATA_DIR="/var/lib/etcd" ETCD_LISTEN_PEER_URLS="https://192.168.223.201:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.223.201:2379" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.223.201:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.223.201:2379"
這是192.168.223.201節點的配置，其餘兩個etcd節點只要將上面的IP地址改爲相應節點的IP地址便可。ETCD_NAME換成對應節點的infra1/2/3。
systemctl daemon-reload systemctl enable etcd systemctl start etcd systemctl status etcd
在全部的 kubernetes master 節點重複上面的步驟，直到全部機器的 etcd 服務都已啓動。
etcdctl \
--ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ cluster-health member 9a2ec640d25672e5 is healthy: got healthy result from https://192.168.223.201:2379 member bc6f27ae3be34308 is healthy: got healthy result from https://192.168.223.202:2379 member e5c92ea26c4edba0 is healthy: got healthy result from https://192.168.223.203:2379 cluster is healthy
結果最後一行爲 cluster is healthy 時表示集羣服務正常。
本次部署一個三節點高可用 haproxy+keepalived 集羣，分別爲：192.168.223.20一、192.168.223.20二、192.168.223.203。VIP 地址 192.168.223.200
yum install -y haproxy keepalived
注： 3臺 haproxy+keepalived 節點都需安裝
節點1 192.168.223.201 配置文件 vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { notification_email { test@sina.com } notification_email_from admin@test.com smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_MASTER } vrrp_script check_haproxy { script "/etc/keepalived/check_haproxy.sh" interval 3 } vrrp_instance VI_1 { state MASTER # 若是配置主從，從服務器改成BACKUP便可 interface ens33 virtual_router_id 60 priority 100 # 從服務器設置小於100的數便可 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.223.200/24 } track_script { check_haproxy } }
節點2 192.168.223.202 配置文件 vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { notification_email { test@sina.com } notification_email_from admin@test.com smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_MASTER } vrrp_script check_haproxy { script "/etc/keepalived/check_haproxy.sh" interval 3 } vrrp_instance VI_1 { state BACKUP # 若是配置主從，從服務器改成BACKUP便可 interface ens33 virtual_router_id 60 priority 90 # 從服務器設置小於100的數便可 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.223.200/24 } track_script { check_haproxy } }
節點3 192.168.223.203 配置文件 vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { notification_email { test@sina.com } notification_email_from admin@test.com smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_MASTER } vrrp_script check_haproxy { script "/etc/keepalived/check_haproxy.sh" interval 3 } vrrp_instance VI_1 { state BACKUP # 若是配置主從，從服務器改成BACKUP便可 interface ens33 virtual_router_id 60 priority 80 # 從服務器設置小於100的數便可 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.223.200/24 } track_script { check_haproxy } }
檢測腳本 vi /etc/keepalived/check_haproxy.sh
#!/bin/bash flag=$(systemctl status haproxy &> /dev/null;echo $?) if [[ $flag != 0 ]];then echo "haproxy is down,close the keepalived" systemctl stop keepalived fi
修改keepalived啓動文件 vi /usr/lib/systemd/system/keepalived.service 如下部分:
[Unit]
Description=LVS and VRRP High Availability Monitor After=syslog.target network-online.target haproxy.service Requires=haproxy.service
3臺節點配置如出一轍 配置文件 vim /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode tcp log global option tcplog option dontlognull option redispatch retries 3 timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout check 10s maxconn 3000 listen stats mode http bind :10086 stats enable stats uri /admin?stats stats auth admin:admin stats admin if TRUE frontend k8s_http *:8080 mode tcp maxconn 2000 default_backend http_sri backend http_sri balance roundrobin server s1 192.168.223.204:8080 check inter 10000 fall 2 rise 2 weight 1 server s2 192.168.223.205:8080 check inter 10000 fall 2 rise 2 weight 1 frontend k8s_https *:6443 mode tcp maxconn 2000 default_backend https_sri backend https_sri balance roundrobin server s1 192.168.223.204:6443 check inter 10000 fall 2 rise 2 weight 1 server s2 192.168.223.205:6443 check inter 10000 fall 2 rise 2 weight 1
3個節點都啓動
systemctl daemon-reload
systemctl enable haproxy systemctl enable keepalived systemctl start haproxy systemctl start keepalived
若是沒有什麼報錯，那應該就能夠在主節點 192.168.223.201 上面看到ens33網卡已綁定VIP: 192.168.223.200
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:2b:74:46 brd ff:ff:ff:ff:ff:ff inet 192.168.223.201/24 brd 192.168.223.255 scope global ens33 valid_lft forever preferred_lft forever inet 192.168.223.200/24 scope global secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::435e:5e98:6d14:6c40/64 scope link valid_lft forever preferred_lft forever
全部的 node 節點都須要安裝網絡插件才能讓全部的Pod加入到同一個局域網中，若是想要在master節點上也能訪問 pods的ip，master 節點也安裝。
建議直接使用 yum 安裝 flanneld ，除非對版本有特殊需求，默認安裝的是0.7.1版本的 flannel 。
yum install -y flannel
service配置文件 vi /usr/lib/systemd/system/flanneld.service。
[Unit]
Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify EnvironmentFile=/etc/sysconfig/flanneld EnvironmentFile=-/etc/sysconfig/docker-network ExecStart=/usr/bin/flanneld-start \ -etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS} \ -etcd-prefix=${FLANNEL_ETCD_PREFIX} \ $FLANNEL_OPTIONS ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service
vi /etc/sysconfig/flanneld 配置文件：
# Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD_ENDPOINTS="https://192.168.223.201:2379,https://192.168.223.202:2379,https://192.168.223.203:2379" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_PREFIX="/kube-centos/network" # Any additional options that you want to pass FLANNEL_OPTIONS="-etcd-cafile=/etc/kubernetes/ssl/ca.pem -etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem -etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem"
注： 若是是主機是多網卡，則須要在FLANNEL_OPTIONS中增長指定的外網出口的網卡，例如-iface=eth2
在etcd中建立網絡配置
執行下面的命令爲docker分配IP地址段。
etcdctl --endpoints=https://192.168.223.201:2379,https://192.168.223.202:2379,https://192.168.223.203:2379 \
--ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ mkdir /kube-centos/network etcdctl --endpoints=https://192.168.223.201:2379,https://192.168.223.202:2379,https://192.168.223.203:2379 \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ mk /kube-centos/network/config '{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}'
若是你要使用host-gw模式，能夠直接將vxlan改爲host-gw便可。根據原做者測試，使用host-gw模式時網絡性能好一些。
systemctl daemon-reload systemctl enable flanneld systemctl start flanneld systemctl status flanneld
注： 啓動flannel前，請先中止docker，flannel啓動好後，再啓動docker。
如今查詢etcd中的內容能夠看到：
etcdctl --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ ls /kube-centos/network/subnets /kube-centos/network/subnets/172.30.14.0-24 /kube-centos/network/subnets/172.30.38.0-24 /kube-centos/network/subnets/172.30.46.0-24 /kube-centos/network/subnets/172.30.91.0-24 etcdctl --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ get /kube-centos/network/config { "Network": "172.30.0.0/16", "SubnetLen": 24, "Backend": { "Type": "vxlan" } } etcdctl --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ get /kube-centos/network/subnets/172.30.14.0-24 {"PublicIP":"192.168.223.204","BackendType":"vxlan","BackendData":{"VtepMAC":"56:27:7d:1c:08:22"}} etcdctl --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ get /kube-centos/network/subnets/172.30.38.0-24 {"PublicIP":"192.168.223.205","BackendType":"vxlan","BackendData":{"VtepMAC":"12:82:83:59:cf:b8"}} etcdctl --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ get /kube-centos/network/subnets/172.30.46.0-24 {"PublicIP":"192.168.223.206","BackendType":"vxlan","BackendData":{"VtepMAC":"e6:b2:fd:f6:66:96"}} etcdctl --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ get /kube-centos/network/subnets/172.30.91.0-24 {"PublicIP":"192.168.223.207","BackendType":"vxlan","BackendData":{"VtepMAC":"e3:b1:43:f6:34:67"}}
若是能夠查看到以上內容證實flannel已經安裝完成，而且已經正常分配kubernetes網段
kubernetes master 節點包含的組件：
如下pem證書文件咱們在建立TLS證書和祕鑰這一步中已經建立過了，token.csv文件在建立kubeconfig文件的時候建立。咱們再檢查一下。
cd /etc/kubernetes/ssl
ls
admin-key.pem admin.pem ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem kubernetes-key.pem kubernetes.pem
有兩種下載方式，請注意下載對應的Kubernetes版本。
方式一
從 github release 頁面 下載發佈版 tarball，解壓後再執行下載腳本
wget https://github.com/kubernetes/kubernetes/releases/download/v1.6.0/kubernetes.tar.gz tar -xzvf kubernetes.tar.gz cd kubernetes ./cluster/get-kube-binaries.sh
方式二
從 CHANGELOG頁面 下載 client 或 server tarball 文件 server 的 tarball kubernetes-server-linux-amd64.tar.gz 已經包含了 client(kubectl) 二進制文件，因此不用單獨下載kubernetes-client-linux-amd64.tar.gz文件；
# wget https://dl.k8s.io/v1.6.0/kubernetes-client-linux-amd64.tar.gz wget https://dl.k8s.io/v1.6.0/kubernetes-server-linux-amd64.tar.gz tar -xzvf kubernetes-server-linux-amd64.tar.gz cp -r kubernetes/server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/bin/ chmod +x /usr/bin/kube*
建立 kube-apiserver的service配置文件
service配置文件 vi /usr/lib/systemd/system/kube-apiserver.service內容：
[Unit]
Description=Kubernetes API Service Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target After=etcd.service [Service] EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/apiserver ExecStart=/usr/bin/kube-apiserver \ $KUBE_LOGTOSTDERR \ $KUBE_LOG_LEVEL \ $KUBE_ETCD_SERVERS \ $KUBE_API_ADDRESS \ $KUBE_API_PORT \ $KUBELET_PORT \ $KUBE_ALLOW_PRIV \ $KUBE_SERVICE_ADDRESSES \ $KUBE_ADMISSION_CONTROL \ $KUBE_API_ARGS Restart=on-failure Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target
vi /etc/kubernetes/config 文件的內容爲：
### # kubernetes system config # # The following values are used to configure various aspects of all # kubernetes services, including # # kube-apiserver.service # kube-controller-manager.service # kube-scheduler.service # kubelet.service # kube-proxy.service # logging to stderr means we get it in the systemd journal KUBE_LOGTOSTDERR="--logtostderr=true" # journal message level, 0 is debug KUBE_LOG_LEVEL="--v=0" # Should this cluster be allowed to run privileged docker containers KUBE_ALLOW_PRIV="--allow-privileged=true" # How the controller-manager, scheduler, and proxy find the apiserver #KUBE_MASTER="--master=http://sz-pg-oam-docker-test-001.tendcloud.com:8080" KUBE_MASTER="--master=http://192.168.223.200:8080"
注： 該配置文件同時被kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy使用。KUBE_MASTER 填寫 VIP 地址
apiserver配置文件 vi /etc/kubernetes/apiserver 內容爲：
### ## kubernetes system config ## ## The following values are used to configure the kube-apiserver ## # ## The address on the local server to listen to. #KUBE_API_ADDRESS="--insecure-bind-address=sz-pg-oam-docker-test-001.tendcloud.com" KUBE_API_ADDRESS="--advertise-address=0.0.0.0 --bind-address=0.0.0.0 --insecure-bind-address=0.0.0.0" # ## The port on the local server to listen on. #KUBE_API_PORT="--port=8080" # ## Port minions listen on #KUBELET_PORT="--kubelet-port=10250" # ## Comma separated list of nodes in the etcd cluster KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.223.201:2379,https://192.168.223.202:2379,https://192.168.223.203:2379" # ## Address range to use for services KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" # ## default admission control policies KUBE_ADMISSION_CONTROL="--admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota" # ## Add your own! KUBE_API_ARGS="--authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h"
Kubernetes 1.9不一樣點
啓動kube-apiserver
systemctl daemon-reload systemctl enable kube-apiserver systemctl start kube-apiserver systemctl status kube-apiserver
建立 kube-controller-manager的serivce配置文件
文件路徑 vi /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/controller-manager ExecStart=/usr/bin/kube-controller-manager \ $KUBE_LOGTOSTDERR \ $KUBE_LOG_LEVEL \ $KUBE_MASTER \ $KUBE_CONTROLLER_MANAGER_ARGS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target
配置文件 vi /etc/kubernetes/controller-manager。
### # The following values are used to configure the kubernetes controller-manager # defaults from config and apiserver should be adequate # Add your own! KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem --leader-elect=true"
啓動 kube-controller-manager
systemctl daemon-reload systemctl enable kube-controller-manager systemctl start kube-controller-manager systemctl status kube-controller-manager
咱們啓動每一個組件後能夠經過執行命令 kubectl get componentstatuses，來查看各個組件的狀態;
kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused
controller-manager Healthy ok
etcd-2 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"}
注： 目前scheduler未啓動，報錯是正常的
建立 kube-scheduler的serivce配置文件
文件路徑 vi /usr/lib/systemd/system/kube-scheduler.service。
[Unit]
Description=Kubernetes Scheduler Plugin Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/scheduler ExecStart=/usr/bin/kube-scheduler \ $KUBE_LOGTOSTDERR \ $KUBE_LOG_LEVEL \ $KUBE_MASTER \ $KUBE_SCHEDULER_ARGS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target
配置文件 vi /etc/kubernetes/scheduler。
### # kubernetes scheduler config # default config should be adequate # Add your own! KUBE_SCHEDULER_ARGS="--leader-elect=true --address=127.0.0.1"
啓動 kube-scheduler
systemctl daemon-reload
systemctl enable kube-scheduler systemctl start kube-scheduler systemctl status kube-scheduler
驗證 master 節點功能
kubectl get componentstatuses NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"}
注： 兩個master節點安裝方式與配置同樣
Kubernetes node節點包含以下組件：
注意： 每臺 node 上都須要安裝 flannel，master 節點上選裝。
步驟簡介
目錄和文件
咱們再檢查一下三個節點上，通過前幾步操做咱們已經建立了以下的證書和配置文件。
cd /etc/kubernetes/ssl
ls
admin-key.pem admin.pem ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem kubernetes-key.pem kubernetes.pem
ls /etc/kubernetes/
apiserver bootstrap.kubeconfig config controller-manager kubelet kube-proxy.kubeconfig proxy scheduler ssl token.csv
若是您使用yum的方式安裝的flannel則不須要執行mk-docker-opts.sh文件這一步，參考Flannel官方文檔中的Docker Integration。
若是你不是使用yum安裝的flannel，那麼須要下載flannel github release中的tar包，解壓後會得到一個 mk-docker-opts.sh 文件，到flannel release頁面下載對應版本的安裝包，該腳本見mk-docker-opts.sh，由於咱們使用yum安裝因此不須要執行這一步。這個文件是用來 Generate Docker daemon options based on flannel env file。 使用systemctl命令啓動flanneld後，會自動執行./mk-docker-opts.sh -i生成以下兩個文件環境變量文件：
FLANNEL_NETWORK=172.30.0.0/16 FLANNEL_SUBNET=172.30.46.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=false
DOCKER_OPT_BIP="--bip=172.30.46.1/24" DOCKER_OPT_IPMASQ="--ip-masq=true" DOCKER_OPT_MTU="--mtu=1450"
Docker將會讀取這兩個環境變量文件做爲容器啓動參數。
注意： 安裝docker-ce-17.12.1.ce版本的rpm包時，需給docker.service額外添加$DOCKER_NETWORK_OPTIONS --exec-opt native.cgroupdriver=systemd
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS --exec-opt native.cgroupdriver=systemd
注意： 不論您用什麼方式安裝的flannel，下面這一步是必不可少的。
yum方式安裝的flannel
修改docker的配置文件 vi /usr/lib/systemd/system/docker.service，增長一條環境變量配置
EnvironmentFile=-/run/flannel/docker
/run/flannel/docker文件是flannel啓動後自動生成的，其中包含了docker啓動時須要的參數。
二進制方式安裝的flannel
修改docker的配置文件 vi /usr/lib/systemd/system/docker.service，增長以下幾條環境變量配置：
EnvironmentFile=-/run/docker_opts.env EnvironmentFile=-/run/flannel/subnet.env
這兩個文件是mk-docker-opts.sh腳本生成環境變量文件默認的保存位置，docker啓動的時候須要加載這幾個配置文件才能夠加入到flannel建立的虛擬網絡裏。
因此不論您使用何種方式安裝的flannel，將如下配置加入到docker.service中可確保萬無一失。
EnvironmentFile=-/run/flannel/docker EnvironmentFile=-/run/docker_opts.env EnvironmentFile=-/run/flannel/subnet.env EnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/etc/sysconfig/docker-storage EnvironmentFile=-/etc/sysconfig/docker-network EnvironmentFile=-/run/docker_opts.env
docker安裝方式也分yum和rpm包安裝
方式一： yum 安裝
版本1.13.1-53
yum install docker -y
而後修改配置 vi /etc/sysconfig/docker 中OPTIONS參數以下：
OPTIONS='--log-driver=json-file --signature-verification=false --insecure-registry 192.168.223.208:80' # 附：192.168.223.208:80 爲harbor私有鏡像倉庫
修改 vi /etc/sysconfig/docker-storage 以下：
DOCKER_STORAGE_OPTIONS="--storage-driver overlay "
修改docker pull源 vi /etc/docker/daemon.json
{
"registry-mirrors":["https://registry.docker-cn.com"] }
修改 vi /usr/lib/systemd/system/docker.service:
[Unit]
Description=Docker Application Container Engine Documentation=http://docs.docker.com After=network.target rhel-push-plugin.socket registries.service Wants=docker-storage-setup.service Requires=docker-cleanup.timer [Service] Type=notify NotifyAccess=all EnvironmentFile=-/run/containers/registries.conf EnvironmentFile=-/run/docker_opts.env EnvironmentFile=-/etc/sysconfig/docker-network EnvironmentFile=-/etc/sysconfig/docker-storage EnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/run/flannel/subnet.env EnvironmentFile=-/run/docker_opts.env EnvironmentFile=-/run/flannel/docker Environment=GOTRACEBACK=crash Environment=DOCKER_HTTP_HOST_COMPAT=1 Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin ExecStart=/usr/bin/dockerd-current \ --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \ --default-runtime=docker-runc \ --exec-opt native.cgroupdriver=systemd \ --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \ --seccomp-profile=/etc/docker/seccomp.json \ $OPTIONS \ $DOCKER_STORAGE_OPTIONS \ $DOCKER_NETWORK_OPTIONS \ $ADD_REGISTRY \ $BLOCK_REGISTRY \ $INSECURE_REGISTRY \ $REGISTRIES ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=1048576 LimitNPROC=1048576 LimitCORE=infinity TimeoutStartSec=0 Restart=on-abnormal MountFlags=slave KillMode=process [Install] WantedBy=multi-user.target
方式二： rpm安裝
版本：ce-17.12.1
rpm -ivh docker-ce-17.12.1.ce-1.el7.centos.x86_64.rpm
而後修改配置 vi /etc/sysconfig/docker 中OPTIONS參數以下：
# /etc/sysconfig/docker # Modify these options if you want to change the way the docker daemon runs OPTIONS='--log-driver=json-file --insecure-registry 192.168.223.208:80' # 附：192.168.223.208:80 爲harbor私有鏡像倉庫，-signature-verification=false選項在此版本已不存在 if [ -z "${DOCKER_CERT_PATH}" ]; then DOCKER_CERT_PATH=/etc/docker fi # Do not add registries in this file anymore. Use /etc/containers/registries.conf # from the atomic-registries package. # # On an SELinux system, if you remove the --selinux-enabled option, you # also need to turn on the docker_transition_unconfined boolean. # setsebool -P docker_transition_unconfined 1 # Location used for temporary files, such as those created by # docker load and build operations. Default is /var/lib/docker/tmp # Can be overriden by setting the following environment variable. # DOCKER_TMPDIR=/var/tmp # Controls the /etc/cron.daily/docker-logrotate cron job status. # To disable, uncomment the line below. # LOGROTATE=false # docker-latest daemon can be used by starting the docker-latest unitfile. # To use docker-latest client, uncomment below lines #DOCKERBINARY=/usr/bin/docker-latest #DOCKERDBINARY=/usr/bin/dockerd-latest #DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest #DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest
修改 vi /etc/sysconfig/docker-storage 以下：
DOCKER_STORAGE_OPTIONS="--storage-driver overlay "
修改docker pull源 vi /etc/docker/daemon.json
{
"registry-mirrors":["https://registry.docker-cn.com"] }
修改 vi /usr/lib/systemd/system/docker.service:
[Unit]
Description=Docker Application Container Engine Documentation=http://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify EnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/etc/sysconfig/docker-storage EnvironmentFile=-/etc/sysconfig/docker-network EnvironmentFile=-/run/docker_opts.env EnvironmentFile=-/run/flannel/subnet.env EnvironmentFile=-/run/docker_opts.env EnvironmentFile=-/run/flannel/docker Environment=GOTRACEBACK=crash ExecStart=/usr/bin/dockerd $OPTIONS \ --exec-opt native.cgroupdriver=systemd \ $DOCKER_STORAGE_OPTIONS \ $DOCKER_NETWORK_OPTIONS \ $ADD_REGISTRY \ $BLOCK_REGISTRY \ $INSECURE_REGISTRY ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=1048576 LimitNPROC=1048576 LimitCORE=infinity MountFlags=slave TimeoutStartSec=1min Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target
systemctl reload-daemon systemctl enable docker systemctl restart docker systemctl status docker
注： 重啓了docker後還要重啓kubelet，若是遇到如下問題，kubelet啓動失敗。報錯：
Mar 31 16:44:41 k8s_node1 kubelet[81047]: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
這是kubelet與docker的cgroup driver不一致致使的，kubelet啓動的時候有個—cgroup-driver參數能夠指定爲"cgroupfs"或者「systemd」。
--cgroup-driver string Driver that the kubelet uses to manipulate cgroups on the host. Possible values: 'cgroupfs', 'systemd' (default "cgroupfs")
配置docker的service配置文件 vi /usr/lib/systemd/system/docker.service，設置ExecStart中的 --exec-opt native.cgroupdriver=systemd 再重啓便可。
kubernets1.8不一樣點
相對於kubernetes1.6集羣必須進行的配置有： 對於kuberentes1.8集羣，必須關閉swap，不然kubelet啓動將失敗。 修改/etc/fstab將，swap系統註釋掉。
kubelet 啓動時向 kube-apiserver 發送 TLS bootstrapping 請求，須要先將 bootstrap token 文件中的 kubelet-bootstrap 用戶賦予 system:node-bootstrapper cluster 角色(role)， 而後 kubelet 纔能有權限建立認證請求(certificate signing requests)：
cd /etc/kubernetes
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap
下載最新的kubelet和kube-proxy二進制文件
注意請下載對應的Kubernetes版本的安裝包。
wget https://dl.k8s.io/v1.6.0/kubernetes-server-linux-amd64.tar.gz tar -xzvf kubernetes-server-linux-amd64.tar.gz cp -r kubernetes/server/bin/{kube-proxy,kubelet} /usr/bin/ chmod +x /usr/bin/kube*
建立kubelet的service配置文件
文件位置 vi /usr/lib/systemd/system/kubelet.service。
[Unit]
Description=Kubernetes Kubelet Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/kubelet ExecStart=/usr/bin/kubelet \ $KUBE_LOGTOSTDERR \ $KUBE_LOG_LEVEL \ $KUBELET_API_SERVER \ $KUBELET_ADDRESS \ $KUBELET_PORT \ $KUBELET_HOSTNAME \ $KUBE_ALLOW_PRIV \ $KUBELET_POD_INFRA_CONTAINER \ $KUBELET_ARGS Restart=on-failure [Install] WantedBy=multi-user.target
kubelet的配置文件/etc/kubernetes/kubelet。其中的IP地址更改成你的每臺node節點的IP地址。 注意： 在啓動kubelet以前，須要先手動建立/var/lib/kubelet目錄：mkdir -p /var/lib/kubelet。
kubelet的配置文件 vi /etc/kubernetes/kubelet:
kubernetes1.8不一樣點
相對於kubenrete1.6的配置變更：
### ## kubernetes kubelet (minion) config # ## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces) KUBELET_ADDRESS="--address=192.168.223.206" # ## The port for the info server to serve on #KUBELET_PORT="--port=10250" # ## You may leave this blank to use the actual hostname KUBELET_HOSTNAME="--hostname-override=192.168.223.206" # ## location of the api-server ## COMMENT THIS ON KUBERNETES 1.8+ KUBELET_API_SERVER="--api-servers=http://192.168.223.200:8080" # ## pod infrastructure container KUBELET_POD_INFRA_CONTAINER="--pod_infra_container_image=192.168.223.208:80/k8s/pause-amd64:v3.0" # ## Add your own! KUBELET_ARGS="--cgroup-driver=systemd --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false"
pod-infrastructure 或者 pause 。pod-infrastructure鏡像是Redhat製做的，大小接近80M，下載比較耗時，其實該鏡像並不運行什麼具體進程，推薦使用Google的pause鏡像gcr.io/google_containers/pause-amd64:3.0，這個鏡像只有300多K。systemctl daemon-reload systemctl enable kubelet systemctl start kubelet systemctl status kubelet
kubelet 首次啓動時向 kube-apiserver 發送證書籤名請求，必須經過後 kubernetes 系統纔會將該 Node 加入到集羣。
查看未受權的 CSR 請求
kubectl get csr NAME AGE REQUESTOR CONDITION csr-2b308 4m kubelet-bootstrap Pending kubectl get nodes No resources found.
經過 CSR 請求
kubectl certificate approve csr-2b308 certificatesigningrequest "csr-2b308" approved kubectl get nodes NAME STATUS AGE VERSION 192.168.223.206 Ready 1m v1.6.0
自動生成了 kubelet kubeconfig 文件和公私鑰
ls -l /etc/kubernetes/kubelet.kubeconfig
-rw------- 1 root root 2284 Apr 7 02:07 /etc/kubernetes/kubelet.kubeconfig ls -l /etc/kubernetes/ssl/kubelet* -rw-r--r-- 1 root root 1046 Apr 7 02:07 /etc/kubernetes/ssl/kubelet-client.crt -rw------- 1 root root 227 Apr 7 02:04 /etc/kubernetes/ssl/kubelet-client.key -rw-r--r-- 1 root root 1103 Apr 7 02:07 /etc/kubernetes/ssl/kubelet.crt -rw------- 1 root root 1675 Apr 7 02:07 /etc/kubernetes/ssl/kubelet.key
假如你更新kubernetes的證書，只要沒有更新token.csv，當重啓kubelet後，該node就會自動加入到kuberentes集羣中，而不會從新發送certificaterequest，也不須要在master節點上執行kubectl certificate approve操做。前提是不要刪除node節點上的/etc/kubernetes/ssl/kubelet*和/etc/kubernetes/kubelet.kubeconfig文件。不然kubelet啓動時會提示找不到證書而失敗。
注意： 若是啓動kubelet的時候見到證書相關的報錯，有個trick能夠解決這個問題，能夠將master節點上的~/.kube/config文件（該文件在安裝kubectl命令行工具這一步中將會自動生成）拷貝到node節點的/etc/kubernetes/kubelet.kubeconfig位置，這樣就不須要經過CSR，當kubelet啓動後就會自動加入的集羣中。
安裝conntrack
yum install -y conntrack-tools
建立 kube-proxy 的service配置文件
文件路徑 vi /usr/lib/systemd/system/kube-proxy.service。
[Unit]
Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/proxy ExecStart=/usr/bin/kube-proxy \ $KUBE_LOGTOSTDERR \ $KUBE_LOG_LEVEL \ $KUBE_MASTER \ $KUBE_PROXY_ARGS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target
kube-proxy配置文件 vi /etc/kubernetes/proxy。
### # kubernetes proxy config # default config should be adequate # Add your own! KUBE_PROXY_ARGS="--bind-address=192.168.223.206 --hostname-override=192.168.223.206 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"
systemctl daemon-reload systemctl enable kube-proxy systemctl start kube-proxy systemctl status kube-proxy
192.168.223.207 安裝方式同樣，只須要把相應配置文件裏面的IP改成 192.168.223.207 便可/etc/kubernetes/bootstrap.kubeconfig; /etc/kubernetes/kube-proxy.kubeconfig;/etc/kubernetes/ssl/*.pem ，而後先安裝flanneld，後照本章操做加入集羣便可。咱們建立一個nginx的service試一下集羣是否可用
kubectl run nginx --replicas=2 --labels="run=load-balancer-example" --image=192.168.223.208:80/k8s/nginx:v1.9.4 --port=80
deployment "nginx" created
kubectl expose deployment nginx --type=NodePort --name=example-service
service "example-service" exposed
kubectl describe svc example-service
Name: example-service
Namespace: default
Labels: run=load-balancer-example
Annotations: <none> Selector: run=load-balancer-example Type: NodePort IP: 10.254.62.207 Port: <unset> 80/TCP NodePort: <unset> 32724/TCP Endpoints: 172.30.60.2:80,172.30.94.2:80 Session Affinity: None Events: <none> curl "10.254.62.207:80" <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
注意： 此時可能會出現不一樣node節點上面的pod之間網絡不通，解決方法以下
#設置全部節點iptables yum install iptables-services -y; systemctl disable iptables; systemctl stop iptables; modprobe ip_tables; iptables -P FORWARD ACCEPT;
192.168.223.208:80/k8s/nginx:v1.9.4，你們在測試過程當中請換成本身的nginx鏡像地址。10.254.62.207 爲集羣內部地址，只有在安裝了kube-proxy的節點上可以訪問，訪問這個地址時是作了負載均衡的192.168.223.206:32724 或 192.168.223.207:32724 均可以獲得nginx的頁面。至此kubernets1.6.0集羣基礎環境已經安裝完成，後面將安裝一些經常使用插件
該插件須要使用如下官方鏡像：
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1 gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1
因爲大中華局域網的緣由，這些鏡像是pull不下來的。全部我這裏使用本身搭建的私有鏡像倉庫
192.168.223.208:80/k8s/k8s-dns-kube-dns-amd64:v1.14.1 192.168.223.208:80/k8s/k8s-dns-dnsmasq-nanny-amd64:v1.14.1 192.168.223.208:80/k8s/k8s-dns-sidecar-amd64:v1.14.1
須要使用的yaml配置文件
kubedns-cm.yaml kubedns-sa.yaml kubedns-controller.yaml kubedns-svc.yaml
預約義的 RoleBinding system:kube-dns 將 kube-system 命名空間的 kube-dns ServiceAccount 與 system:kube-dns Role 綁定， 該 Role 具備訪問 kube-apiserver DNS 相關 API 的權限；
kubectl get clusterrolebindings system:kube-dns -o yaml apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" creationTimestamp: 2017-04-11T11:20:42Z labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-dns resourceVersion: "58" selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindingssystem%3Akube-dns uid: e61f4d92-1ea8-11e7-8cd7-f4e9d49f8ed0 roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-dns subjects: - kind: ServiceAccount name: kube-dns namespace: kube-system
yaml文件 vi kubedns-sa.yaml
apiVersion: v1 kind: ServiceAccount metadata: name: kube-dns namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile
yaml文件 vi kubedns-cm.yaml
# Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. apiVersion: v1 kind: ConfigMap metadata: name: kube-dns namespace: kube-system labels: addonmanager.kubernetes.io/mode: EnsureExists
yaml文件 vi kubedns-controller.yaml
# Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Should keep target in cluster/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml # in sync with this file. # __MACHINE_GENERATED_WARNING__ apiVersion: extensions/v1beta1 kind: Deployment metadata: name: kube-dns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: # replicas: not specified here: # 1. In order to make Addon Manager do not reconcile this replicas parameter. # 2. Default is 1. # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on. strategy: rollingUpdate: maxSurge: 10% maxUnavailable: 0 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: tolerations: - key: "CriticalAddonsOnly" operator: "Exists" volumes: - name: kube-dns-config configMap: name: kube-dns optional: true containers: - name: kubedns image: 192.168.223.208:80/k8s/k8s-dns-kube-dns-amd64:v1.14.1 resources: # TODO: Set memory limits when we've profiled the container for large # clusters, then set request = limit to keep this container in # guaranteed class. Currently, this container falls into the # "burstable" category so the kubelet doesn't backoff from restarting it. limits: memory: 170Mi requests: cpu: 100m memory: 70Mi livenessProbe: httpGet: path: /healthcheck/kubedns port: 10054 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /readiness port: 8081 scheme: HTTP # we poll on pod startup for the Kubernetes master service and # only setup the /readiness HTTP server once that's available. initialDelaySeconds: 3 timeoutSeconds: 5 args: - --domain=cluster.local. - --dns-port=10053 - --config-dir=/kube-dns-config - --v=2 #__PILLAR__FEDERATIONS__DOMAIN__MAP__ env: - name: PROMETHEUS_PORT value: "10055" ports: - containerPort: 10053 name: dns-local protocol: UDP - containerPort: 10053 name: dns-tcp-local protocol: TCP - containerPort: 10055 name: metrics protocol: TCP volumeMounts: - name: kube-dns-config mountPath: /kube-dns-config - name: dnsmasq image: 192.168.223.208:80/k8s/k8s-dns-dnsmasq-nanny-amd64:v1.14.1 livenessProbe: httpGet: path: /healthcheck/dnsmasq port: 10054 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 args: - -v=2 - -logtostderr - -configDir=/etc/k8s/dns/dnsmasq-nanny - -restartDnsmasq=true - -- - -k - --cache-size=1000 - --log-facility=- - --server=/cluster.local./127.0.0.1#10053 - --server=/in-addr.arpa/127.0.0.1#10053 - --server=/ip6.arpa/127.0.0.1#10053 ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP # see: https://github.com/kubernetes/kubernetes/issues/29055 for details resources: requests: cpu: 150m memory: 20Mi volumeMounts: - name: kube-dns-config mountPath: /etc/k8s/dns/dnsmasq-nanny - name: sidecar image: 192.168.223.208:80/k8s/k8s-dns-sidecar-amd64:v1.14.1 livenessProbe: httpGet: path: /metrics