kubernetes之手動部署k8s 1.14.1高可用集羣

時間  2019-11-11
標籤 kubernetes 手動 部署 k8s 1.14.1 可用 集羣 欄目 負載均衡 简体版
原文   原文鏈接

1. 架構信息

系統版本：CentOS 7.6
內核：3.10.0-957.el7.x86_64
Kubernetes: v1.14.1
Docker-ce: 18.09.5
推薦硬件配置：4核8G

Keepalived保證apiserever服務器的IP高可用
Haproxy實現apiserver的負載均衡 

2. 節點信息 

目前測試爲 6 臺虛擬機，etcd採用 rpm 安裝、kubernetes 使用二進制安裝，使用 systemd 來作管理，網絡組件採用 flannel，Master 實現了 HA， 集羣開啓 RBAC；master 不負載 pod，在分發證書等階段將在另一臺主機上執行，該主機對集羣內全部節點配置了 ssh 祕鑰登陸，基本環境以下html

hostname ip 組件 內存 cpu
node-01 172.19.8.111 kube-apiserver、kube-controller-manager、etcd、haproxy、keepalived 8G 4c
node-02 172.19.8.112 kube-apiserver、kube-controller-manager、etcd、haproxy、keepalived 8G 4c
node-03 172.19.8.113 kube-apiserver、kube-controller-manager、etcd 8G 4c
node-04 172.19.8.114 node 8G 4c
node-05 172.19.8.115 node 8G 4c
node-06 172.19.8.116 node 8G 4c
VIP 172.19.8.250      

 

 

 

 


3.1  關閉防火牆和selinux3. 部署前準備工做

[root@node-01 ~]# sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
[root@node-01 ~]# setenforce 0
[root@node-01 ~]# systemctl disable firewalld
[root@node-01 ~]# systemctl stop firewalld

3.2 關閉swap

  
  
  
  [root@node-01 ~]# swapoff -a
注：修改/etc/fstab，註銷swap相關信息

3.3 添加host記錄

[root@node-01 ~]# cat >>/etc/hosts<<EOF
172.19.8.111 node-01
172.19.8.112 node-02
172.19.8.113 node-03
172.19.8.114 node-04
172.19.8.115 node-05
172.19.8.116 node-06
EOF

3.4 打通ssh， node-01免密登陸其餘服務器

[root@node-01 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:uckCmzy46SfU6Lq9jRbugn0U8vQsr5H+PtfGBsvrfCA root@node-01
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|                 |
|                 |
|  . o    .       |
|   *.+  S        |
|  +o==E.oo       |
|.=.oBo.o+*       |
|o.**oooo+ *      |
|oBO=++o++=       |
+----[SHA256]-----+

 分發node-01公鑰，用於免密登陸其餘服務器node

 
 
[root@node-01 ~]# for n in `seq -w 01 06`;do ssh-copy-id node-$n;done
 

 

3.5  
  配置內核參數，須要重啓服務器，不然後面初始化的時候會報錯。
 

 
 
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF

sysctl --system
 

 

報錯處理,沒有橋接網絡致使，須要安裝docker，並啓動後纔會出現橋接網絡linux
 

 
 
[root@node-01 ~]# sysctl -p /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: 沒有那個文件或目錄
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: 沒有那個文件或目錄
 

 

3.6 若是kube-proxy使用ipvs模式，須要加載ipvs模塊
 

 
 
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
 

 

3.7  
  添加yum源
 

 
 
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF
 

 

考慮到國內沒法拉取google源，可使用阿里雲源nginx
 

 
 
$ cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

wget http://mirrors.aliyun.com/repo/Centos-7.repo -O /etc/yum.repos.d/CentOS-Base.repo
wget http://mirrors.aliyun.com/repo/epel-7.repo -O /etc/yum.repos.d/epel.repo
 

 

以上部署須要在每一個節點執行。git
 

4.  
  部署keepalived和haproxy
 

4.1 在node-01和node-02上面安裝keepalived和haproxy
 

 
 
$ yum install -y keepalived haproxy
 

 

4.2 配置keepalived
 

node-01 配置信息github
 

 
 
[root@node-01 ~]# cat /etc/keepalived/keepalived.conf
! Configuratile for keepalived
global_defs {
 notification_email {
995958026@qq.com
      }
 notification_email_from keepalived@ptmind.com
 smtp_server 127.0.0.1
 smtp_connect_timeout 30
 router_id node-01
}

vrrp_script check_apiserver {
     script "/workspace/crontab/check_apiserver"
     interval 5
     weight -20
     fall 3
     rise 1
     }

vrrp_instance VIP_250 {
    state MASTER
    interface eth0
    virtual_router_id 250
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 890iop
    }
    track_script {
      check_apiserver
     }
    virtual_ipaddress {
        172.19.8.250
    }
}
 

 

檢查腳本配置redis
 

 
 
$ cat /workspace/crontab/check_apiserver
#!/bin/bash
curl 127.0.0.1:8080 &>/dev/null
if [ $? -eq 0 ];then
    exit 0
else
    #systemctl stop keepalived
    exit 1
fi
$ chmod 755 /workspace/crontab/check_apiserver
 

 

node-02 配置docker
 

 
 
[root@node-02 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
 notification_email {
435002493@qq.com
      }
 notification_email_from keepalived@ptmind.com
 smtp_server 127.0.0.1
 smtp_connect_timeout 30
 router_id node-02
}

vrrp_instance VI_250 {
    state BACKUP
    interface eth0
    virtual_router_id 250
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 890iop
    }
    virtual_ipaddress {
    172.19.8.250
    }
}
 

 

4.3 配置haproxy
 

 
 
 


  node-01和node-02的haproxy配置是同樣的。此處咱們監聽的是172.19.8.250的8443端口，由於haproxy是和k8s apiserver是部署在同一臺服務器上，都用6443會衝突。  

 

 
 
 
  
[root@node-01 ~]# cat /etc/haproxy/haproxy.cfg
global
        chroot  /var/lib/haproxy
        daemon
        group haproxy
        user haproxy
#        log warning
        pidfile /var/lib/haproxy.pid
        maxconn 20000
        spread-checks 3
        nbproc 8

defaults
        log     global
        mode    tcp
        retries 3
        option redispatch

listen https-apiserver
        bind 0.0.0.0:8443
        mode tcp
        balance roundrobin
        timeout server 900s
        timeout connect 15s

        server apiserver01 172.19.8.111:6443 check port 6443 inter 5000 fall 5
        server apiserver02 172.19.8.112:6443 check port 6443 inter 5000 fall 5
        server apiserver03 172.19.8.113:6443 check port 6443 inter 5000 fall 5
 
 
 
 
4.4 啓動服務
 
 
 
  
systemctl enable keepalived && systemctl start keepalived 
systemctl enable haproxy && systemctl start haproxy 
 
 
 
 
5  
   配置證書
 
 
 
   建立集羣 CA 與 Certificatesjson
 
 
 
  
 
 
5.1 證書介紹
 
 
在這部分，將會須要產生 client 與 server 的各組件 certificates，而且替 Kubernetes admin user 產生 client 證書。因爲 Etcd 和 Kubernetes 所有采用 TLS 通信，因此先要生成 TLS 證書，證書生成工具採用 cfssl，生成證書時可在任一節點完成，這裏在宿主機執行，證書列表以下bootstrap
 
 
 
   
   
 
     
      證書名稱 
     
      配置文件 
     
      用途 
   
 
   
 
     
      etcd-root-ca.pem 
     
      etcd-root-ca-csr.json 
     
      etcd 根 CA 證書 
   
 
   
 
     
      etcd.pem 
     
      etcd-gencert.json、etcd-csr.json 
     
      etcd 集羣證書 
   
 
   
 
     
      k8s-root-ca.pem 
     
      k8s-root-ca-csr.json 
     
      k8s 根 CA 證書 
   
 
   
 
     
      kube-proxy.pem 
     
      k8s-gencert.json、kube-proxy-csr.json 
     
      kube-proxy 使用的證書 
   
 
   
 
     
      admin.pem 
     
      k8s-gencert.json、admin-csr.json 
     
      kubectl 使用的證書 
   
 
   
 
     
      kubernetes.pem 
     
      k8s-gencert.json、kubernetes-csr.json 
     
      kube-apiserver 使用的證書 
   
 
   
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5.2 cfssl 工具安裝
 
 
 
   首先下載 cfssl，並給予可執行權限，而後扔到 PATH 目錄下
 
 
 
  
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
 
 
 
 
5.3 生成etcd證書
 
 
 
   注意，全部證書生成只須要在任意一臺主機上生成一遍便可，我這裏在 node-01 上操做的。
 
 
 
   etcd-csr.json   #產生 kube-apiserver certificate 證書
 
 
 
  
cat <<EOF>> etcd-csr.json
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "etcd",
      "OU": "etcd Security",
      "L": "Beijing",
      "ST": "Beijing",
      "C": "CN"
    }
  ],
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "localhost",
    "172.19.8.111",
    "172.19.8.112",
    "172.19.8.113"
  ]
}
EOF
 
 
 
 
 
   etcd-gencert.json    #產生 CA 密鑰
 
 
 
  
cat <<EOF>> etcd-gencert.json
{
  "signing": {
    "default": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "87600h"
    }
  }
}
EOF
 
 
 
 
 
   etcd-root-ca-csr.json     #產生 CA 密鑰
 
 
 
  
cat <<EOF>> etcd-root-ca-csr.json
{
  "key": {
    "algo": "rsa",
    "size": 4096
  },
  "names": [
    {
      "O": "etcd",
      "OU": "etcd Security",
      "L": "Beijing",
      "ST": "Beijing",
      "C": "CN"
    }
  ],
  "CN": "etcd-root-ca"
}
EOF
 
 
 
 
 
   最後生成證書
 
 
 
  
cfssl gencert --initca=true etcd-root-ca-csr.json | cfssljson --bare etcd-root-ca
cfssl gencert --ca etcd-root-ca.pem --ca-key etcd-root-ca-key.pem --config etcd-gencert.json etcd-csr.json | cfssljson --bare etcd
 
 
 
 
 
   完成後刪除沒必要要文件：
 
 
 
  
rm -rf *.json
 
 
 
 
 
   確認/etc/etcd/ssl有如下文件：
 
 
 
  
etcd.csr  etcd-key.pem  etcd.pem  etcd-root-ca.csr  etcd-root-ca-key.pem  etcd-root-ca.pe
 
 
 
 
5.4 生成k8s證書須要藉助kubectl，下面安裝k8s會介紹。 
 
 
6 安裝etcd
 
 
6.1 在node-0一、node-0二、node-03節點安裝etcd
 
 
 
  
$ yum -y install etcd
 
 
 
 
6.2  
   全部節點部署證書
 
 
 
  
$ mkdir /etc/etcd/ssl
$ scp *.pem /etc/etcd/ssl
$ chown -R etcd:etcd /etc/etcd/ssl
$ chmod -R 644 /etc/etcd/ssl/*
$ chmod 755 /etc/etcd/ssl
 
 
 
 
6.3  
   修改etcd數據目錄權限
 
 
 
  
$ chown -R etcd:etcd /var/lib/etcd /etc/etcd
 
 
 
 
6.4 而後修改配置以下(其餘兩個節點相似，只須要改監聽地址和 Etcd Name 便可)
 
 
 
  
[root@node-01 kubernetes]# cat /etc/etcd/etcd.conf
#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
ETCD_LISTEN_PEER_URLS="https://172.19.8.111:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.19.8.111:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.19.8.111:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.19.8.111:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
ETCD_INITIAL_CLUSTER="etcd1=https://172.19.8.111:2380,etcd2=https://172.19.8.112:2380,etcd3=https://172.19.8.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"
#
#[Proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[Security]
#ETCD_CERT_FILE=""
#ETCD_KEY_FILE=""
#ETCD_CLIENT_CERT_AUTH="false"
#ETCD_TRUSTED_CA_FILE=""
#ETCD_AUTO_TLS="false"
#ETCD_PEER_CERT_FILE=""
#ETCD_PEER_KEY_FILE=""
#ETCD_PEER_CLIENT_CERT_AUTH="false"
#ETCD_PEER_TRUSTED_CA_FILE=""
#ETCD_PEER_AUTO_TLS="false"
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_PEER_AUTO_TLS="true"
#
#[Logging]
#ETCD_DEBUG="false"
#ETCD_LOG_PACKAGE_LEVELS=""
#ETCD_LOG_OUTPUT="default"
#
#[Unsafe]
#ETCD_FORCE_NEW_CLUSTER="false"
#
#[Version]
#ETCD_VERSION="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
#
#[Profiling]
#ETCD_ENABLE_PPROF="false"
#ETCD_METRICS="basic"
#
#[Auth]
#ETCD_AUTH_TOKEN="simple"
 
 
 
 
6.5  
   最後啓動集羣並測試以下
 
 
 
  
$ systemctl daemon-reload
$ systemctl start etcd
$ systemctl enable etcd

[root@node-01 ~]# etcdctl --ca-file=/etc/etcd/ssl/etcd-root-ca.pem --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --endpoints=https://172.19.8.111:2379,https://172.19.8.112:2379,https://172.19.8.113:2379 cluster-health
member 20eaf623ac3b75e0 is healthy: got healthy result from https://172.19.8.112:2379
member 49a1ffa1fb4e8c74 is healthy: got healthy result from https://172.19.8.111:2379
member d6afeec04f7054b4 is healthy: got healthy result from https://172.19.8.113:2379
cluster is healthy
 
 
 
 
若是檢查失敗，經過日誌排查，重點檢查etcd節點配置信息。
 
 
7 安裝kubernetes
 
 
7.1 生成kubernetes證書
 
 
下載軟件包
 
 
 
  
# master
wget https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz
cp kubernetes/server/bin/{kube-apiserver,kube-scheduler,kube-controller-manager} /bin/

# node
https://dl.k8s.io/v1.14.0/kubernetes-node-linux-amd64.tar.gz

# 將下載的包解壓，而後將 server/bin/ 和 node/bin/ 中的執行文件拷貝到各自的/bin下
 
 
 
 
  
   在這部分，將會須要生成 client 與 server 的各組件 certificates，而且替 Kubernetes admin user 生成 client 證書。
 
 
 
  
#!/bin/bash
# Deploy the master node.
KUBE_SSL=/etc/kubernetes/ssl
mkdir -p $KUBE_SSL

# Create CA.
cat>$KUBE_SSL/ca-config.json<<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
cat>$KUBE_SSL/ca-csr.json<<EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "IMG"
        }
    ]
}
EOF

# 「10.254.0.1"  注意此處須要添加指定，與cluster-cidr同網段，例如192.168.0.1

cat>$KUBE_SSL/server-csr.json<<EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.254.0.1",    #注意此處須要添加指定，與cluster-cidr同網段，例如192.168.0.1
      "127.0.0.1",
      "172.19.8.111", "172.19.8.112", "172.19.8.113", "172.19.8.114", "172.19.8.115", "172.19.8.116", "172.19.8.250", "localhost",
      "apiclient",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "IMG"
        }
    ]
}
EOF
cd $KUBE_SSL
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

# Create kube-proxy CA.
cat>$KUBE_SSL/kube-proxy-csr.json<<EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
        "C": "CN",
        "L": "Beijing",
        "ST": "Beijing",
        "O": "k8s",
        "OU": "IMG"
    }
  ]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
ls *.pem
cd ~
 
 
 
 
 
  
 
 

   每一個節點須要關閉 swap。 
 
 
 

   Kubernetes 1.8 開始要求關閉系統的 Swap，若是不關閉，默認配置下 kubelet 將沒法啓動。能夠經過 kubelet 的啓動參數–fail-swap-on=false 更改這個限制。 
 
 
 
 
   
 
 
 

     
 
 
 

   生成token 
 
 
 
 
  
# Create a token file.
cat>/etc/kubernetes/token.csv<<EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
 
 
 
 
7.2 node-01安裝apiserver
 
 
 
  
[root@node-01 kubernetes]# cat apiserver.conf
#
KUBE_APISERVER_OPTS="—logtostderr=false \
--etcd-servers=https://172.19.8.111:2379,https://172.19.8.112:2379,https://172.19.8.113:2379 \
--v=4 \
--bind-address=172.19.8.111 \
--secure-port=6443 \
--advertise-address=172.19.8.111 \
--allow-privileged=true \
--service-cluster-ip-range=10.254.0.0/16 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/etc/kubernetes/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/etc/kubernetes/ssl/server.pem  \
--tls-private-key-file=/etc/kubernetes/ssl/server-key.pem \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/etc/etcd/ssl/etcd-root-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem"
 
 
 
 
 
   配置啓動腳本
 
 
 
  
# Create the kube-apiserver service.
$ cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/apiserver.conf
ExecStart=/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target


$ systemctl daemon-reload
$ systemctl enable kube-apiserver.service --now
$ systemctl status kube-apiserver.service
 
 
 
 
7.3  
   node-01 安裝api-controller-manager
 
 
 
  
$ cat /etc/kubernetes/kube-controller-manager.conf

KUBE_CONTROLLER_MANAGER_OPTS="—logtostderr=false \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.254.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem"
 
 
 
 
 
   配置啓動腳本
 
 
 
  
$ cat /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf
ExecStart=/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

-------

$ systemctl daemon-reload
$ systemctl enable kube-controller-manager.service --now
$ systemctl status kube-controller-manager.service
 
 
 
 
7.4  
   node-01安裝schedule
 
 
 
  
[root@node-01 kubernetes]# cat /etc/kubernetes/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="—logtostderr=false \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect"
 
 
 
 
 
   配置啓動腳本
 
 
 
  
$ cat /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf
ExecStart=/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

——————

$ systemctl daemon-reload
$ systemctl enable kube-scheduler.service --now
$ systemctl status kube-scheduler.service
 
 
 
 
 
   檢查節點狀態
 
 
 
  
[root@node-01 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
 
 
 
 
 
   node-0二、node-03 作上述相同配置，須要修改ip地址。
 
 
 8 部署node節點
 
 
8.1  
   建立bootstrap和kube-proxy的kubeconfig文件
 
 
 
  
 
 

   在master 節點上執行下面的腳本建立bootstrap.kubeconfig和kube-proxy.kubeconfig文件 
 
 
 
 
   
  

    因爲經過手動建立 CA 方式太過繁雜，只適合少許機器，由於每次簽證時都須要綁定 Node IP，隨機器增長會帶來不少困擾，所以這邊使用 TLS Bootstrapping 方式進行受權，由 apiserver 自動給符合條件的 Node 發送證書來受權加入集羣。 
  
 
  

    主要作法是 kubelet 啓動時，向 kube-apiserver 傳送 TLS Bootstrapping 請求，而 kube-apiserver 驗證 kubelet 請求的 token 是否與設定的同樣，若同樣就自動產生 kubelet 證書與密鑰。具體做法能夠參考  
   TLS bootstrapping。 
  
 
  

    經過腳本建立kubeconfig文件 
  
 
  
 
   
 
    
$ cat nodeprepare.sh
#!/bin/bash

BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)
KUBE_SSL=/etc/kubernetes/ssl/
KUBE_APISERVER="https://172.19.8.250:8443"

cd $KUBE_SSL
# Set cluster parameters.
kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=/etc/kubernetes/bootstrap.kubeconfig

# Set client parameters.
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=/etc/kubernetes/bootstrap.kubeconfig

# Set context parameters. 
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=/etc/kubernetes/bootstrap.kubeconfig

# Set context.
kubectl config use-context default --kubeconfig=/etc/kubernetes/bootstrap.kubeconfig

# Create kube-proxy kubeconfig file. 
kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
cd ~

# Bind kubelet-bootstrap user to system cluster roles.
kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --user=kubelet-bootstrap
 
   
 
   
8.2  
     配置kube-proxy和kubelet
 
   
 
    
 
   

     將master節點證書、token.csv和bootstrap.kubeconfig,kube-proxy.kubeconfig拷貝到node節點 . 
   
 
   
 
     8.2.1 以node-04爲例,配置kube-proxy節點
 
   
 
    
 
     
[root@node-04 kubernetes]# cat /etc/kubernetes/kube-proxy.conf
KUBE_PROXY_OPTS="--logtostderr=false \
--v=4 \
--hostname-override=172.19.8.114 \
--cluster-cidr=10.254.0.0/16 \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig"
 
    
 
    
 
      配置啓動腳本
 
    
 
     
[root@node-04 kubernetes]# cat /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/kube-proxy.conf
ExecStart=/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
 
    
 
    
啓動
 
    
 
     
[root@node-04 kubernetes]# systemctl daemon-reload 
[root@node-04 kubernetes]# systemctl start kube-proxy.service
[root@node-04 kubernetes]# systemctl enable kube-proxy.service
[root@node-04 kubernetes]# systemctl status kube-proxy.service -l
 
    
 
    
8.2.2  
      配置kubelet
 
    
 
     
[root@node-04 kubernetes]# cat /etc/kubernetes/kubelet.yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 172.19.8.114
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.254.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true
 
    
 
    
 
     
[root@node-04 kubernetes]# cat /etc/kubernetes/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=4 \
--hostname-override=172.19.8.114 \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
--config=/etc/kubernetes/kubelet.yaml \
--cert-dir=/etc/kubernetes/ssl \
--pod-infra-container-image=gcr.io/google-containers/pause-amd64:3.0"
 
    
 
    
 
      配置啓動文件
 
    
 
     
[root@node-04 kubernetes]# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/etc/kubernetes/kubelet.conf
ExecStart=/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target
 
    
 
    
啓動
 
    
 
     
[root@node-04 kubernetes]# systemctl daemon-reload
[root@node-04 kubernetes]# systemctl enable kubelet.service --now
[root@node-04 kubernetes]# systemctl status kubelet.service -l
 
    
 
    
 
      node-05 node-06 作相同配置，注意修改IP地址。
 
   
 
  
 
 
 
 
 
   kubelet啓動後，會向apiserver申請註冊，此時apiserver須要贊成申請，node才能加入進去。
 
 
 
  
[root@node-01 ~]# kubectl get csr
NAME                                                   AGE   REQUESTOR           CONDITION
node-csr-K_cecykFJ_g1cAJf5DTJ_EVFUH3W9QgfulMA-TbFoMo   45s   kubelet-bootstrap   Pending
 
 
 
 
 
  
[root@node-01 ~]# kubectl certificate approve node-csr-K_cecykFJ_g1cAJf5DTJ_EVFUH3W9QgfulMA-TbFoMo
certificatesigningrequest.certificates.k8s.io/node-csr-K_cecykFJ_g1cAJf5DTJ_EVFUH3W9QgfulMA-TbFoMo approved
 
 
 
 
 
  
[root@node-01 ~]# kubectl get node
NAME           STATUS   ROLES    AGE   VERSION
172.19.8.114   Ready    <none>   12s   v1.14.1
 
 
 
 
 
   此時node已經加入。
 
 
9 全部節點部署flannel網絡插件
 
 
 
  9.1 檢查etcd節點狀態
 
 
 
  
[root@node-01 ~]# etcdctl --ca-file=/etc/etcd/ssl/etcd-root-ca.pem --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --endpoints=https://172.19.8.111:2379,https://172.19.8.112:2379,https://172.19.8.113:2379 cluster-health
member 20eaf623ac3b75e0 is healthy: got healthy result from https://172.19.8.112:2379
member 49a1ffa1fb4e8c74 is healthy: got healthy result from https://172.19.8.111:2379
member d6afeec04f7054b4 is healthy: got healthy result from https://172.19.8.113:2379
cluster is healthy
 
 
 
 
9.2 在etcd中建立flannel網絡
 
 
 
  
export ETCDCTL_API=2
etcdctl --ca-file=/etc/etcd/ssl/etcd-root-ca.pem --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --endpoints=https://172.19.8.111:2379,https://172.19.8.112:2379,https://172.19.8.113:2379 set /coreos.com/network/config  '{ "Network": "10.244.0.0/16", "Backend": {"Type": "vxlan"}}'
 
 
 
 
以前使用etcd api3建立flannel建立網絡，在flannel啓動時一直報錯，報錯信息以下，改成api2，問題解決
 
 
 
  
# Writing into a predetermined subnetwork.Master節點設置
#注意 一開始默認使用ETCDCTL_API=3，發現flannel鏈接etcd讀取下列信息失敗，一直保報錯 Couldn't fetch network config: 100: Key not found

 
 
 
 
9.3  
   下載flannel
 
 
 
  
# https://github.com/coreos/flannel/releases
wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz

# 軟件包包含flanneld 和 mk-docker-opts.sh
mv {flanneld,mk-docker-opts.sh} /bin/
 
 
 
 
9.4  
   複製etcd證書到各node節點,目錄爲/etc/etcd/ssl
 
 
 
  
[root@node-04 ~]# cd /etc/etcd/ssl/
[root@node-04 ssl]# ls
etcd-key.pem  etcd.pem  etcd-root-ca-key.pem  etcd-root-ca.pem
 
 
 
 
9.5 配置flannel
 
 
 
  
[root@node-04 kubernetes]# cat /etc/kubernetes/flannel.conf
FLANNEL_ETCD_KEY="/coreos.com/network"
FLANNEL_OPTIONS="--etcd-endpoints=https://172.19.8.111:2379,https://172.19.8.112:2379,https://172.19.8.113:2379 -etcd-cafile=/etc/etcd/ssl/etcd-root-ca.pem -etcd-certfile=/etc/etcd/ssl/etcd.pem -etcd-keyfile=/etc/etcd/ssl/etcd-key.pem"
 
 
 
 
配置啓動文件
 
 
 
  
[root@node-04 kubernetes]# cat /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/etc/kubernetes/flannel.conf
ExecStart=/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target
 
 
 
 
9.6 修改docker啓動文件
 
 
 
  
# Modify the docker service.

cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
 
 
 
 
啓動
 
 
 
  
# Start or restart related services.
systemctl daemon-reload
systemctl enable flanneld --now
systemctl restart docker
systemctl status flanneld
systemctl status docker
 
 
 
 
檢查建立的網絡
 
 
 
  
[root@node-04 kubernetes]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:d7:b7:9e:55:00 brd ff:ff:ff:ff:ff:ff
    inet 172.19.8.114/24 brd 172.19.8.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:88:36:c8:89 brd ff:ff:ff:ff:ff:ff
    inet 10.244.82.1/24 brd 10.244.82.255 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether da:b3:f5:69:bf:de brd ff:ff:ff:ff:ff:ff
    inet 10.244.82.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever

[root@node-04 ~]# cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=10.244.16.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.244.16.1/24 --ip-masq=false --mtu=1450"
 
 
 
 
10 部署nginx小戰一下
 
 
 
  
[root@node-01 ~]# kubectl apply -f nginx-dep.yml

[root@node-01 ~]# kubectl get pods
NAME                               READY   STATUS              RESTARTS   AGE
nginx-deployment-6dd86d77d-txhmg   0/1     ContainerCreating   0          11s
nginx-deployment-6dd86d77d-vc4mx   0/1     ContainerCreating   0          11s

[root@node-01 ~]# kubectl describe pod nginx-deployment-6dd86d77d-txhmg     # 檢查爲何pod啓動失敗

  Warning  FailedCreatePodSandBox  50s    kubelet, 172.19.8.116  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "img.cn/google-containers/pause-amd64:3.0": Error response from daemon: 
Get https://img.cn/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

# 緣由：kubelet.conf 配置的 --pod-infra-container-image 鏡像地址拉取失敗
 
 
 
 
 
 
 
11 配置coredns
 
 
11.1 添加anonymous認證 
 
 
 
  
 
   
kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
 
  
 
  
11.2 配置coredns
 
  
 
   
[root@node-01 ~]# cat coredns.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
  labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: Reconcile
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: EnsureExists
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
  labels:
      addonmanager.kubernetes.io/mode: EnsureExists
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local. in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153 forward . /etc/resolv.conf
        cache 30
    }
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: coredns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "CoreDNS"
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: coredns
  template:
    metadata:
      labels:
        k8s-app: coredns
    spec:
      serviceAccountName: coredns
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      containers:
      - name: coredns
        image: coredns/coredns:1.5.0
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: coredns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: coredns
  clusterIP: 10.254.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
 
  
 
  
部署coredns
 
  
 
   
[root@node-01 ~]# kubectl apply -f coredns.yml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.extensions/coredns created
service/coredns created

[root@node-01 ~]# kubectl get pod -A
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-76458c6948-t5f2b   1/1     Running   0          103s
kube-system   coredns-76458c6948-v56tg   1/1     Running   0          103s
 
  
 
  
12 集成k8s到rancher
 
  
 
    只有上面的部分完成後，才能導入到rancher集羣中。
 
  
 
   
curl --insecure -sfL https://172.19.2.25:8443/v3/import/f6knmcb7pjgtdppckkdgvfz24gbdcfhpg7npjr2xh4bt9xhxlv8j89.yaml | kubectl apply -f -
 
  
 
  
大功告成！
 
 
 



        

            

        	





    

		

            
相關文章

            

                

                

                    

                

            


            
相關標籤/搜索

            
        


    

        

            

        

        

        
        

        

        

    




	

    





    

    	

    

        每日一句
    

    
    
    	每一个你不满意的现在，都有一个你没有努力的曾经。
    







    

    


    



    

        本站公眾號
    

    

           歡迎關注本站公眾號,獲取更多信息

        
    




    

    




	


	







    

        聯繫我們
        最近搜索
        最新文章
        
        沪ICP备13005482号-10

            MyBatis教程
            SQL 教程
            MySQL教程
            Java 教程
            Thymeleaf 教程
            Hibernate教程
            Spring教程 
            Redis教程