K8s手動方式搭建平臺及問題彙總

時間 2021-03-11

標籤 node linux nginx git github golang docker json bootstrap vim 欄目 Linux 简体版

原文原文鏈接

如今換了份工做，新環境使用k8s容器環境管理，心思着把之前的相關文檔彙總起來編寫成博文方便本身複習node

k8s概要

因爲圖片有上一家信息在上面，相關圖片就不放上面了。linux

kubernetes，簡稱K8s，是用8代替8個字符「ubernete」而成的縮寫。是一個開源的，用於管理雲平臺中多個主機上的容器化的應用，Kubernetes的目標是讓部署容器化的應用簡單而且高效（powerful）,Kubernetes提供了應用部署，規劃，更新，維護的一種機制nginx

Kubernetes是Google開源的一個容器編排引擎，它支持自動化部署、大規模可伸縮、應用容器化管理。在生產環境中部署一個應用程序時，一般要部署該應用的多個實例以便對應用請求進行負載均衡。git

在Kubernetes中，咱們能夠建立多個容器，每一個容器裏面運行一個應用實例，而後經過內置的負載均衡策略，實現對這一組應用實例的管理、發現、訪問，而這些細節都不須要運維人員去進行復雜的手工配置和處理。github

kubernetes的特色：

可移植: 支持公有云，私有云，混合雲，多重雲（multi-cloud）
可擴展: 模塊化，插件化，可掛載，可組合
自動化: 自動部署，自動重啓，自動複製，自動伸縮/擴展golang

kubernetes的體系架構

Kubernetes集羣包含有節點代理kubelet和Master組件(APIs, scheduler, etc)，一切都基於分佈式的存儲系統。
docker

kubernetes 組件特色

咱們把服務分爲運行在工做節點上的服務和組成集羣級別控制板的服務。json

Kubernetes節點有運行應用容器必備的服務，而這些都是受Master的控制。bootstrap

每次個節點上固然都要運行Docker。Docker來負責全部具體的映像下載和容器運行。vim

Kubernetes主要由如下幾個核心組件組成：

etcd保存了整個集羣的狀態；
apiserver提供了資源操做的惟一入口，並提供認證、受權、訪問控制、API註冊和發現等機制；
controller manager負責維護集羣的狀態，好比故障檢測、自動擴展、滾動更新等；
scheduler負責資源的調度，按照預約的調度策略將Pod調度到相應的機器上；
kubelet負責維護容器的生命週期，同時也負責Volume（CVI）和網絡（CNI）的管理；
Container runtime負責鏡像管理以及Pod和容器的真正運行（CRI）；
kube-proxy負責爲Service提供cluster內部的服務發現和負載均衡；
除了核心組件，還有一些推薦的Add-ons：

kube-dns負責爲整個集羣提供DNS服務
Ingress Controller爲服務提供外網入口
Heapster提供資源監控
Dashboard提供GUI
Federation提供跨可用區的集羣
Fluentd-elasticsearch提供集羣日誌採集、存儲與查詢
這些官網或者中文網站均可以查閱到資料，後面主要介紹我本身手動部署安裝k8s

手動部署安裝

etcd:3.3.11
kubectl:
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
kubernetes-dashboard:v1.6.3
nginx-ingress-controller:0.9.0
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
calico:v3.2.6

初始化環境

配置hosts地址

這裏我只在test環境進行測試，生產環境同樣配置只是環境地址不一樣。
在每臺服務器上執行
#編輯每臺服務器的 /etc/hosts 文件，配置hostname 通訊

vi /etc/hosts
172.16.16.86 incubator-dc-016
172.16.16.246 incubator-dc-002
172.16.16.250 incubator-dc-003

關閉防火牆

在每臺服務器上執行

systemctl stop firewalld.service               
#中止firewall
systemctl disable firewalld.service            
#禁止firewall開機啓動
firewall-cmd --state                          
#查看默認防火牆狀態（關閉後顯示notrunning，開啓後顯示running）

關閉selinux

在每臺服務器上執行

$ setenforce 0
$ vim /etc/selinux/config
SELINUX=disabled

關閉swap

在每臺服務器上執行
K8s需使用內存，而不用swap

$ swapoff -a   
$ vim /etc/fstab
註釋掉SWAP分區項，便可

安裝go 語言環境（按需）

https://golang.org/dl/
下載 linux版本go，解壓後配置環境變量便可

vi /etc/profile
export GOROOT=/usr/local/go
export PATH=$GOROOT/bin:$PATH

$ source profile

建立K8s集羣驗證

安裝cfssl

這裏使用 CloudFlare 的 PKI 工具集 cfssl 來生成 Certificate Authority (CA) 證書和祕鑰文件。

mkdir /opt/k8s

cd /opt/k8s

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
mv cfssl_linux-amd64 cfssl

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssljson_linux-amd64 cfssljson

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 cfssl-certinfo

chmod +x *

建立CA證書配置

cd /opt/k8s
config.json 文件

vi  config.json

{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

csr.json 文件
vi csr.json

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成CA證書和私鑰

master

cd /opt/k8s

./cfssl gencert -initca csr.json | ./cfssljson -bare ca

會生成3個文件ca.csr、ca-key.pem、ca.pem

分發證書

建立證書目錄master
mkdir -p /etc/kubernetes/ssl

拷貝全部文件到目錄下

cp *.pem /etc/kubernetes/ssl

這裏要將文件拷貝到全部的k8s 機器上

scp -P53742 -P53742 *.pem 172.16.16.246:/etc/kubernetes/ssl/

scp -P53742 -P53742 *.pem 172.16.16.250:/etc/kubernetes/ssl/
chmod 777 /etc/kubernetes/ssl/*.pem

安裝docker

安裝 yum-config-manager
yum -y install yum-utils

導入

yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

更新 repo

yum makecache

刪除已安裝的docker

[root@incubator-dc-016 k8s]# yum remove -y doceker
Loaded plugins: fastestmirror
No Match for argument: doceker
No Packages marked for removal
[root@incubator-dc-016 k8s]# rpm -qa | grep docker
docker-ce-19.03.4-3.el7.x86_64
docker-ce-cli-19.03.4-3.el7.x86_64
[root@incubator-dc-016 k8s]# rpm -e --nodeps docker-ce-19.03.4-3.el7.x86_64
[root@incubator-dc-016 k8s]# rpm -e --nodeps docker-ce-cli-19.03.4-3.el7.x86_64
[root@incubator-dc-016 k8s]# rpm -qa | grep docker

安裝docker

yum install docker-ce –y

更改docker配置

修改配置

前面文件先備份。
mv /usr/lib/systemd/system/docker.service /usr/lib/systemd/system/docker.service.bak

#修改
vi /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS $DOCKER_DNS_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

修改其餘配置

mkdir -p /usr/lib/systemd/system/docker.service.d/
/*註釋
#添加以下 : (注意 environment 必須在同一行，若是出現換行會沒法加載)
#iptables=false 會使 docker run 的容器沒法連網，false 是由於 calico 有一些高級的應用，須要限制容器互通。
#建議通常狀況不添加 --iptables=false,calico須要添加

vi /usr/lib/systemd/system/docker.service.d/docker-options.conf
暫時未加
[Service]
Environment="DOCKER_OPTS=--insecure-registry=10.254.0.0/16 --graph=/opt/docker --registry-mirror=http://b438f72b.m.daocloud.io --disable-legacy-registry --iptables=false"

從新讀取配置，啓動 docker

systemctl daemon-reload
systemctl start docker
systemctl enable docker

安裝etcd集羣

在每臺上服務器上執行
yum -y install etcd

cd /opt/k8s

vi etcd-csr.json

{
  "CN": "etcd",
  "hosts": [
    "172.16.16.86",
    "172.16.16.246",
    "172.16.16.250"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成 etcd 密鑰

/opt/k8s/cfssl gencert -ca=/opt/k8s/ca.pem \
  -ca-key=/opt/k8s/ca-key.pem \
  -config=/opt/k8s/config.json \
  -profile=kubernetes etcd-csr.json | /opt/k8s/cfssljson -bare etcd

查看生成

[root@k8s-master ssl]# ls etcd*
etcd.csr etcd-csr.json etcd-key.pem etcd.pem

拷貝到etcd服務器

etcd-1 
cp etcd*.pem /etc/kubernetes/ssl/

etcd-2
scp -P53742 etcd*.pem 172.16.16.246:/etc/kubernetes/ssl/

etcd-3
scp -P53742 etcd*.pem 172.16.16.250:/etc/kubernetes/ssl/

若是 etcd 非 root 用戶，讀取證書會提示沒權限
chmod 644 /etc/kubernetes/ssl/etcd-key.pem

修改etcd配置

#etcd-1
mv /usr/lib/systemd/system/etcd.service /usr/lib/systemd/system/etcd.service.bak
vi /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
User=etcd
#set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
  --name=etcd1 \
  --cert-file=/etc/kubernetes/ssl/etcd.pem \
  --key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \
  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://172.16.16.86:2380 \
  --listen-peer-urls=https://172.16.16.86:2380 \
  --listen-client-urls=https://172.16.16.86:2379 \
  --advertise-client-urls=https://172.16.16.86:2379 \
  --initial-cluster-token=k8s-etcd-cluster \
  --initial-cluster=etcd1=https://172.16.16.86:2380,etcd2=https://172.16.16.246:2380,etcd3=https://172.16.16.250:2380 \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#etcd-2

vi /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
User=etcd
#set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
  --name=etcd2 \
  --cert-file=/etc/kubernetes/ssl/etcd.pem \
  --key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \
  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://172.16.16.246:2380 \
  --listen-peer-urls=https://172.16.16.246:2380 \
  --listen-client-urls=https://172.16.16.246:2379 \
  --advertise-client-urls=https://172.16.16.246:2379 \
  --initial-cluster-token=k8s-etcd-cluster \
  --initial-cluster=etcd1=https://172.16.16.86:2380,etcd2=https://172.16.16.246:2380,etcd3=https://172.16.16.250:2380 \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#etcd-3

vi /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
User=etcd
#set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
  --name=etcd3 \
  --cert-file=/etc/kubernetes/ssl/etcd.pem \
  --key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \
  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://172.16.16.250:2380 \
  --listen-peer-urls=https://172.16.16.250:2380 \
  --listen-client-urls=https://172.16.16.250:2379 \
  --advertise-client-urls=https://172.16.16.250:2379 \
  --initial-cluster-token=k8s-etcd-cluster \
  --initial-cluster=etcd1=https://172.16.16.86:2380,etcd2=https://172.16.16.246:2380,etcd3=https://172.16.16.250:2380 \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

啓動etcd

分別啓動全部節點的 etcd 服務

systemctl enable etcd
systemctl start etcd
systemctl status etcd

#若是報錯請使用
journalctl -f -t etcd 和 journalctl -u etcd 來定位問題

驗證etcd集羣狀態

查看 etcd 集羣狀態：

etcdctl --endpoints=https://172.16.16.86:2379 \
        --cert-file=/etc/kubernetes/ssl/etcd.pem \
        --ca-file=/etc/kubernetes/ssl/ca.pem \
        --key-file=/etc/kubernetes/ssl/etcd-key.pem \
        cluster-health

查看 etcd 集羣成員：

etcdctl --endpoints=https://172.16.16.86:2379 \
        --cert-file=/etc/kubernetes/ssl/etcd.pem \
        --ca-file=/etc/kubernetes/ssl/ca.pem \
        --key-file=/etc/kubernetes/ssl/etcd-key.pem \
        member list

查看 etcd 集羣成員：
etcdctl --endpoints=https://172.16.16.86:2379 \
--cert-file=/etc/kubernetes/ssl/etcd.pem \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--key-file=/etc/kubernetes/ssl/etcd-key.pem \
member list

安裝kubectl 工具

Master節點 172.16.16.86
#首先安裝 kubectl

wget https://dl.k8s.io/v1.8.0/kubernetes-client-linux-amd64.tar.gz
（若是鏈接不上，直接去git上下載二進制文件）

tar -xzvf kubernetes-client-linux-amd64.tar.gz
cp kubernetes/client/bin/* /usr/local/bin/
cp kubernetes/client/bin/* /usr/bin/
chmod a+x /usr/local/bin/kube*

驗證安裝

$ kubectl version

[root@incubator-dc-016 k8s]# kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

建立 admin 證書

kubectl 與 kube-apiserver 的安全端口通訊，須要爲安全通訊提供 TLS 證書和祕鑰。
cd /opt/k8s
vi admin-csr.json

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

生成 admin 證書和私鑰

cd /opt/k8s

./cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=/opt/k8s/config.json \
  -profile=kubernetes admin-csr.json | ./cfssljson -bare admin

查看生成

#ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem
cp admin*.pem /etc/kubernetes/ssl/
scp -P53742 admin*.pem 172.16.16.246:/etc/kubernetes/ssl/
scp -P53742 admin*.pem 172.16.16.250:/etc/kubernetes/ssl/

配置 kubectl kubeconfig 文件

server 配置爲本機IP 各自鏈接本機的 Api
#配置 kubernetes 集羣

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://172.16.16.86:6443

配置客戶端認證

kubectl config set-credentials admin \
  --client-certificate=/etc/kubernetes/ssl/admin.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/admin-key.pem

kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=admin

kubectl config use-context kubernetes

kubectl config文件

#kubeconfig 文件在以下位置:

/root/.kube

部署 Kubernetes Master 節點

Master 須要部署 kube-apiserver , kube-scheduler , kube-controller-manager 這三個組件。 kube-scheduler 做用是調度pods分配到那個node裏，簡單來講就是資源調度。 kube-controller-manager 做用是對 deployment controller , replication controller, endpoints controller, namespace controller, and serviceaccounts controller等等的循環控制，與kube-apiserver交互。

安裝Master節點組件

#從github 上下載版本
cd /opt/k8s

wget https://dl.k8s.io/v1.8.3/kubernetes-server-linux-amd64.tar.gz

tar -xzvf kubernetes-server-linux-amd64.tar.gz && cd kubernetes

cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/
cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/bin/

建立kubernetes 證書

cd /opt/k8s

vi kubernetes-csr.json

{
  "CN": "kubernetes",
  "hosts": [
    "172.16.16.86",
    "10.254.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

這裏 hosts 字段中三個 IP 分別爲 127.0.0.1 本機， 172.16.16.86爲 Master 的IP，多個Master須要寫多個 10.254.0.1 爲 kubernetes SVC 的 IP，通常是部署網絡的第一個IP , 如: 10.254.0.1 ，在啓動完成後，咱們使用 kubectl get svc ，就能夠查看到。

生成 kubernetes 證書和私鑰

./cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=/opt/k8s/config.json \
  -profile=kubernetes kubernetes-csr.json | ./cfssljson -bare kubernetes

ls -l kubernetes*

拷貝到目錄

cp -r kubernetes*.pem /etc/kubernetes/ssl/
scp -P53742 -r kubernetes*.pem 172.16.16.246:/etc/kubernetes/ssl/
scp -P53742 -r kubernetes*.pem 172.16.16.250:/etc/kubernetes/ssl/

配置 kube-apiserver

kubelet 首次啓動時向 kube-apiserver 發送 TLS Bootstrapping 請求，kube-apiserver 驗證 kubelet 請求中的 token 是否與它配置的 token 一致，若是一致則自動爲 kubelet生成證書和祕鑰。

生成 token

[root@incubator-dc-016 k8s]# head -c 16 /dev/urandom | od -An -t x | tr -d ''
 49d1b983 9aafea9c 90300962 60d51a3d
49d1b9839aafea9c9030096260d51a3d     需記錄下來

建立 token.csv 文件

cd /opt/k8s

vi token.csv
49d1b9839aafea9c9030096260d51a3d,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

#拷貝

cp token.csv /etc/kubernetes/
scp -P53742 token.csv 172.16.16.246:/etc/kubernetes/
scp -P53742 token.csv 172.16.16.250:/etc/kubernetes/

建立 kube-apiserver.service 文件

# 1.8 新增 (Node) --authorization-mode=Node,RBAC
# 自定義 系統 service 文件通常存於 /etc/systemd/system/ 下
# 配置爲 各自的本地 IP
vi /etc/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/kube-apiserver \
  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --advertise-address=172.16.16.86 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/lib/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address=172.16.16.86 \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --enable-swagger-ui=true \
  --etcd-cafile=/etc/kubernetes/ssl/ca.pem \
  --etcd-certfile=/etc/kubernetes/ssl/etcd.pem \
  --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \
  --etcd-servers=https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379 \
  --event-ttl=1h \
  --kubelet-https=true \
  --insecure-bind-address=172.16.16.86 \
  --runtime-config=rbac.authorization.k8s.io/v1alpha1 \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-cluster-ip-range=10.254.0.0/16 \
  --service-node-port-range=30000-32000 \
  --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --enable-bootstrap-token-auth \
  --token-auth-file=/etc/kubernetes/token.csv \
  --v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#這裏面要注意的是 --service-node-port-range=30000-32000
#這個地方是映射外部端口時的端口範圍，隨機映射也在這個範圍內映射，指定映射端口必須也在這個範圍內。

啓動 kube-apiserver

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver

配置 kube-controller-manager

master 配置爲各自本地 IP
建立 kube-controller-manager.service 文件

vi /etc/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --address=127.0.0.1 \
  --master=http://172.16.16.86:8080 \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=10.254.0.0/16 \
  --cluster-cidr=10.233.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  --leader-elect=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

啓動 kube-controller-manager

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl start kube-controller-manager
systemctl status kube-controller-manager

配置 kube-scheduler

master 配置爲各自本地 IP
建立 kube-cheduler.service 文件

vi /etc/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --address=127.0.0.1 \
  --master=http://172.16.16.86:8080 \
  --leader-elect=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

啓動 kube-scheduler

systemctl daemon-reload
systemctl enable kube-scheduler
systemctl start kube-scheduler
systemctl status kube-scheduler

驗證 Master 節點

[root@incubator-dc-016 k8s]# kubectl get componentstatuses
NAME                 STATUS      MESSAGE                                                                                  
scheduler            Healthy     ok                                                                                       
controller-manager   Healthy     ok                                                                                       
etcd-0               Healthy     {"health":"true"}                                                                        
etcd-1               Healthy     {"health":"true"}                                                                        
etcd-2               Healthy     {"health":"true"}

部署 Master節點的 Node 部分
Node 部分須要部署的組件有 docker calico kubectl kubelet kube-proxy 這幾個組件。

配置 kubelet

kubelet 啓動時向 kube-apiserver 發送 TLS bootstrapping 請求，須要先將 bootstrap token 文件中的 kubelet-bootstrap 用戶賦予 system:node-bootstrapper 角色，而後 kubelet 纔有權限建立認證請求(certificatesigningrequests)。

先建立認證請求

#user 爲 master 中 token.csv 文件裏配置的用戶
#只需建立一次就能夠
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

建立 kubelet kubeconfig 文件

server 配置爲 master 本機 IP

#配置集羣

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://172.16.16.86:6443 \
  --kubeconfig=bootstrap.kubeconfig

#配置客戶端認證

kubectl config set-credentials kubelet-bootstrap \
  --token=49d1b9839aafea9c9030096260d51a3d \
  --kubeconfig=bootstrap.kubeconfig

#配置關聯

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

#配置默認關聯
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#拷貝生成的 bootstrap.kubeconfig 文件
mv bootstrap.kubeconfig /etc/kubernetes

建立 kubelet.service 文件

建立 kubelet 目錄

> 配置爲 node 本機 IP

mkdir /var/lib/kubelet

vi /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
  --address=172.16.16.86 \
  --hostname-override=172.16.16.86 \
  --pod-infra-container-image=jicki/pause-amd64:3.0 \
  --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \
  --cluster_dns=10.254.0.2 \
  --cluster_domain=doone.com. \
  --hairpin-mode promiscuous-bridge \
  --allow-privileged=true \
  --fail-swap-on=false \
  --serialize-image-pulls=false \
  --logtostderr=true \
  --max-pods=512 \
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

#如上配置:
172.16.16.86 爲本機的IP
10.254.0.2 預分配的 dns 地址
cluster.local. 爲 kubernetes 集羣的 domain
jicki/pause-amd64:3.0 這個是 pod 的基礎鏡像，既 gcr 的 gcr.io/googlecontainers/pause-amd64:3.0 鏡像，下載下來修改成本身的倉庫中的比較快。

啓動 kubelet

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

#若是報錯請使用
journalctl -f -t kubelet 和 journalctl -u kubelet 來定位問題

配置 TLS 認證

#查看 csr 的名稱

[root@incubator-dc-016 kubelet]# kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-EBjoD_bmtunjaDMTUmlph04kLO9Kz8-jdUhh6GDhb7w   12s       kubelet-bootstrap   Pending

增長認證

kubectl certificate approve node-csr-Sg6CRaxXhdIEJP0hxMHtE2Xoh9fpeFl6OVtocqGeV34

驗證 nodes

[root@incubator-dc-016 kubelet]# kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
172.16.16.86   Ready     <none>    30s       v1.8.3

#成功之後會自動生成配置文件與密鑰

配置文件

ls /etc/kubernetes/kubelet.kubeconfig   
/etc/kubernetes/kubelet.kubeconfig

祕鑰文件

ls /etc/kubernetes/ssl/kubelet*
/etc/kubernetes/ssl/kubelet-client.crt  /etc/kubernetes/ssl/kubelet.crt
/etc/kubernetes/ssl/kubelet-client.key  /etc/kubernetes/ssl/kubelet.key

配置 kube-proxy

建立 kube-proxy 證書

#證書方面因爲咱們node端沒有裝 cfssl
#咱們回到 master 端機器去配置證書，而後拷貝過來

cd /opt/k8s
vi kube-proxy-csr.json

{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成 kube-proxy 證書和私鑰

./cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=/opt/k8s/config.json \
  -profile=kubernetes  kube-proxy-csr.json | ./cfssljson -bare kube-proxy

查看生成

ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

拷貝到目錄

cp kube-proxy*.pem /etc/kubernetes/ssl/
scp -P53742 kube-proxy*.pem 172.16.16.246:/etc/kubernetes/ssl/
scp -P53742 kube-proxy*.pem 172.16.16.250:/etc/kubernetes/ssl/

建立 kube-proxy kubeconfig 文件

server 配置爲各自本機IP

#配置集羣

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://172.16.16.86:6443 \
  --kubeconfig=kube-proxy.kubeconfig

配置客戶端認證

kubectl config set-credentials kube-proxy \
  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

配置關聯

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

配置默認關聯

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

拷貝到目錄

mv kube-proxy.kubeconfig /etc/kubernetes/

建立 kube-proxy.service 文件

配置爲各自的 IP
#建立 kube-proxy 目錄

mkdir -p /var/lib/kube-proxy

vi /etc/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
  --bind-address=172.16.16.86 \
  --hostname-override=172.16.16.86 \
  --cluster-cidr=10.254.0.0/16 \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
  --logtostderr=true \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

啓動 kube-proxy

systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy

#若是報錯請使用
journalctl -f -t kube-proxy 和 journalctl -u kube-proxy 來定位問題

部署Kubernetes Node節點

Node 節點基於 Nginx 負載 API 作 Master HA。172.16.16.246，172.16.16.250。
#master 之間除 api server 之外其餘組件經過 etcd 選舉，api server 默認不做處理；在每一個 node 上啓動一個 nginx，每一個 nginx 反向代理全部 api server，node 上 kubelet、kube-proxy 鏈接本地的 nginx 代理端口，當 nginx 發現沒法鏈接後端時會自動踢掉出問題的 api server，從而實現 api server 的 HA。

安裝Node節點組件

tar -xzvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes
cp -r server/bin/{kube-proxy,kubelet,kubectl} /usr/local/bin/
cp -r server/bin/{kube-proxy,kubelet,kubectl} /usr/bin/

#ALL node

mkdir -p /etc/kubernetes/ssl/

scp -P53742 ca.pem kube-proxy.pem kube-proxy-key.pem  172.16.16.246:/etc/kubernetes/ssl/
scp -P53742 ca.pem kube-proxy.pem kube-proxy-key.pem  172.16.16.250:/etc/kubernetes/ssl/

建立 kubelet kubeconfig 文件

kubectl config set-cluster kubernetes   --certificate-authority=/etc/kubernetes/ssl/ca.pem   --embed-certs=true   --server=https://172.16.16.86:6443 --kubeconfig=bootstrap.kubeconfig
#必須是server-master節點

配置客戶端認證

kubectl config set-credentials kubelet-bootstrap \
  --token=49d1b9839aafea9c9030096260d51a3d \
  --kubeconfig=bootstrap.kubeconfig

配置關聯

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

配置默認關聯

kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

拷貝生成的 bootstrap.kubeconfig 文件

mv bootstrap.kubeconfig /etc/kubernetes/

建立 kubelet.service 文件

mkdir -p /var/lib/kubelet
vi /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
  --address=172.16.16.246 \
  --hostname-override=172.16.16.246 \
  --pod-infra-container-image=jicki/pause-amd64:3.0 \
  --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \
  --cluster_dns=10.254.0.2 \
  --cluster_domain=doone.com. \
  --hairpin-mode promiscuous-bridge \
  --allow-privileged=true \
  --fail-swap-on=false \
  --serialize-image-pulls=false \
  --logtostderr=true \
  --max-pods=512 \
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

啓動 kubelet

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

建立 kube-proxy kubeconfig 文件

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://172.16.16.86:6443 \
  --kubeconfig=kube-proxy.kubeconfig

配置客戶端認證

kubectl config set-credentials kube-proxy \
  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

配置關聯

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

配置默認關聯

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

拷貝到目錄

mv kube-proxy.kubeconfig /etc/kubernetes/

建立 kube-proxy.service 文件

mkdir -p /var/lib/kube-proxy
vi /etc/systemd/system/kube-proxy.service
#node節點ip地址要改
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
  --bind-address=172.16.16.246 \
  --hostname-override=172.16.16.246 \
  --cluster-cidr=10.254.0.0/16 \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
  --logtostderr=true \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

啓動 kube-proxy

systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy

建立Nginx 代理

在每一個 node 都必須建立一個 Nginx 代理，這裏特別注意，當 Master 也作爲 Node 的時候不須要配置 Nginx-proxy

在node上建立

建立配置目錄

mkdir -p /etc/nginx

####寫入代理配置

cat << EOF > /etc/nginx/nginx.conf
error_log stderr notice;

worker_processes auto;
events {
  multi_accept on;
  use epoll;
  worker_connections 1024;
}

stream {
    upstream kube_apiserver {
        least_conn;
        server 172.16.16.86:6443;       
    }

    server {
        listen        0.0.0.0:6443;
        proxy_pass    kube_apiserver;
        proxy_timeout 10m;
        proxy_connect_timeout 1s;
    }
}
EOF

配置 Nginx 基於 docker 進程，而後配置 systemd 來啓動

cat << EOF > /etc/systemd/system/nginx-proxy.service

[Unit]
Description=kubernetes apiserver docker wrapper
Wants=docker.socket
After=docker.service

[Service]
User=root
PermissionsStartOnly=true
ExecStart=/usr/bin/docker run -p 127.0.0.1:6443:6443 \\
                              -v /etc/nginx:/etc/nginx \\
                              --name nginx-proxy \\
                              --net=host \\
                              --restart=on-failure:5 \\
                              --memory=512M \\
                              nginx:1.13.5-alpine
ExecStartPre=-/usr/bin/docker rm -f nginx-proxy
ExecStop=/usr/bin/docker stop nginx-proxy
Restart=always
RestartSec=15s
TimeoutStartSec=30s

[Install]
WantedBy=multi-user.target
EOF

啓動 Nginx

systemctl daemon-reload
systemctl start nginx-proxy
systemctl enable nginx-proxy
systemctl status nginx-proxy

重啓 Node 的 kubelet 與 kube-proxy

systemctl restart kubelet
systemctl status kubelet
systemctl restart kube-proxy
systemctl status kube-proxy

在Master 配置經過 TLS 認證

#查看 csr 的名稱

kubectl get csr

增長認證

kubectl certificate approve NAME

[root@incubator-dc-016 cx]# kubectl certificate approve node-csr-EBjoD_bmtunjaDMTUmlph04kLO9Kz8-jdUhh6GDhb7w
certificatesigningrequest "node-csr-EBjoD_bmtunjaDMTUmlph04kLO9Kz8-jdUhh6GDhb7w" approved
[root@incubator-dc-016 cx]# kubectl certificate approve node-csr-v-UvG2zhPQRMf3hDTMUqSq_wvsurSlNFc7CHjl1v3ss
certificatesigningrequest "node-csr-v-UvG2zhPQRMf3hDTMUqSq_wvsurSlNFc7CHjl1v3ss" approved
[root@incubator-dc-016 cx]# kubectl certificate approve node-csr-Sg6CRaxXhdIEJP0hxMHtE2Xoh9fpeFl6OVtocqGeV34 "node-csr-z2sRlOk0UKbsaB_8J9ZhjtnS7gt886GVZBAYESiuf10" approved
[root@incubator-dc-016 cx]#

部署Calico網絡

修改 kubelet.service

在每一個節點
增長以下配置

vi /etc/systemd/system/kubelet.service
--network-plugin=cni \

從新加載配置

systemctl daemon-reload
systemctl restart kubelet.service
systemctl status kubelet.service

獲取Calico 配置

Calico 部署仍然採用「混搭」方式，即 Systemd 控制 calico node，cni 等由 kubernetes daemonset 安裝。

#獲取 calico.yaml master機器上執行

cat <<EOF > calico-controller.yml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-kube-controllers
  namespace: kube-system
rules:
  - apiGroups:
    - ""
    - extensions
    resources:
      - pods
      - namespaces
      - networkpolicies
    verbs:
      - watch
      - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: calico-policy-controller
  namespace: kube-system
  labels:
    k8s-app: calico-policy
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-policy-controller
      namespace: kube-system
      labels:
        k8s-app: calico-policy
    spec:
      hostNetwork: true
      serviceAccountName: calico-kube-controllers
      containers:
      - name: calico-policy-controller
        image: quay.io/calico/kube-controllers:v1.0.0
        env:
          - name: ETCD_ENDPOINTS
            value: "https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379"
          - name: ETCD_CA_CERT_FILE
            value: "/etc/kubernetes/ssl/ca.pem"
          - name: ETCD_CERT_FILE
            value: "/etc/kubernetes/ssl/etcd.pem"
          - name: ETCD_KEY_FILE
            value: "/etc/kubernetes/ssl/etcd-key.pem"
        volumeMounts:
          - mountPath: /etc/kubernetes/ssl/
            name: etcd-ca-certs
            readOnly: true
      volumes:
        - hostPath:
            path: /etc/kubernetes/ssl/
            type: DirectoryOrCreate
          name: etcd-ca-certs
EOF
kubectl apply -f  calico-controller.yml

kubectl -n kube-system get po -l k8s-app=calico-policy
需修改yaml文件內ETCD集羣的IP地址

在全部節點下載 Calico

cd /usr/local/bin/
curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.2.6/calicoctl

如下操做在三個節點上都須要實現
cd /usr/local/bin/
curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.2.6/calicoctl
chmod +x calicoctl
scp -P53742 calicoctl root@172.16.16.246:/usr/local/bin/
scp -P53742 calicoctl root@172.16.16.250:/usr/local/bin/

下載calico、calico-ipam

wget -N -P /opt/cni/bin/ https://github.com/projectcalico/calico-cni/releases/download/v3.1.6/calico
wget -N -P /opt/cni/bin/ https://github.com/projectcalico/calico-cni/releases/download/v3.1.6/calico-ipam
mkdir -p /opt/cni/bin/
cp -rf /opt/k8s/calico /opt/cni/bin/
cp -rf /opt/k8s/calico-ipam /opt/cni/bin/
scp -P53742 calico root@172.16.16.246:/opt/cni/bin/
scp -P53742 calico root@172.16.16.250:/opt/cni/bin/
scp -P53742 calico-ipam root@172.16.16.246:/opt/cni/bin/
scp -P53742 calico-ipam root@172.16.16.250:/opt/cni/bin/

chmod +x /opt/cni/bin/calico /opt/cni/bin/calico-ipam

在全部節點下載 CNI plugins配置文件

vi /etc/cni/net.d/10-calico.conf

{
    "name": "calico-k8s-network",
    "cniVersion": "0.1.0",
    "type": "calico",
    "etcd_endpoints": "https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379",
    "etcd_ca_cert_file": "/etc/kubernetes/ssl/ca.pem",
    "etcd_cert_file": "/etc/kubernetes/ssl/etcd.pem",
    "etcd_key_file": "/etc/kubernetes/ssl/etcd-key.pem",
    "log_level": "info",
    "ipam": {
        "type": "calico-ipam"
    },
    "policy": {
        "type": "k8s"
    },
    "kubernetes": {
        "kubeconfig": "/etc/kubernetes/kubelet.kubeconfig"
    }
}

部署 KubeDNS

下載kubeDNS鏡像

#官方鏡像

gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7

#國內鏡像

jicki/k8s-dns-sidecar-amd64:1.14.7
jicki/k8s-dns-kube-dns-amd64:1.14.7
jicki/k8s-dns-dnsmasq-nanny-amd64:1.14.7

下載yaml文件

curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kube-dns.yaml.base

#修改後綴

mv kube-dns.yaml.base kube-dns.yaml

系統預約義的 RoleBinding

預約義的 RoleBinding system:kube-dns 將 kube-system 命名空間的 kube-dns ServiceAccount 與 system:kube-dns Role 綁定，該 Role 具備訪問 kube-apiserver DNS 相關 API 的權限；

[root@k8s-master kubedns]# kubectl get clusterrolebindings system:kube-dns -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2017-09-29T04:12:29Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-dns
  resourceVersion: "78"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/system%3Akube-dns
  uid: 688627eb-a4cc-11e7-9f6b-44a8420b9988
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-dns
subjects:
- kind: ServiceAccount
  name: kube-dns
  namespace: kube-system

Kube-dns yaml文件

cat <<EOF > kube-dns.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-dns
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.254.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      dnsPolicy: Default
      serviceAccountName: kube-dns
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      containers:
      - name: kubedns
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-dns-kube-dns-amd64:1.14.7
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        livenessProbe:
          httpGet:
            path: /healthcheck/kubedns
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 3
          timeoutSeconds: 5
        args:
        - "--domain=cluster.local"
        - --dns-port=10053
        - --v=2
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        - containerPort: 10055
          name: metrics
          protocol: TCP
        volumeMounts:
        - name: kube-dns-config
          mountPath: /kube-dns-config
      - name: dnsmasq
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
        livenessProbe:
          httpGet:
            path: /healthcheck/dnsmasq
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - "-v=2"
        - "-logtostderr"
        - "-configDir=/etc/k8s/dns/dnsmasq-nanny"
        - "-restartDnsmasq=true"
        - "--"
        - "-k"
        - "--cache-size=1000"
        - "--log-facility=-"
        - "--server=/cluster.local/127.0.0.1#10053"
        - "--server=/in-addr.arpa/127.0.0.1#10053"
        - "--server=/ip6.arpa/127.0.0.1#10053"
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        resources:
          requests:
            cpu: 150m
            memory: 20Mi
        volumeMounts:
        - name: kube-dns-config
          mountPath: /etc/k8s/dns/dnsmasq-nanny
      - name: sidecar
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-dns-sidecar-amd64:1.14.7
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - "--v=2"
        - "--logtostderr"
        - "--probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A"
        - "--probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A"
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 20Mi
            cpu: 10m
EOF

導入yaml文件

[root@incubator-dc-016 k8s]# kubectl create -f kube-dns.yml
serviceaccount "kube-dns" created
service "kube-dns" created
deployment "kube-dns" created
[root@incubator-dc-016 k8s]#

查看kubedns服務

[root@incubator-dc-016 k8s]# kubectl get all --namespace=kube-system
NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/calico-policy-controller   1         1         1            1           4h
deploy/kube-dns                   1         1         1            0           19s

NAME                                     DESIRED   CURRENT   READY     AGE
rs/calico-policy-controller-5586b678b5   0         0         0         1h
rs/calico-policy-controller-57dd959cc9   0         0         0         4h
rs/calico-policy-controller-6d94579b6b   1         1         1         56m
rs/kube-dns-794845bc6f                   1         1         0         19s

NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/calico-policy-controller   1         1         1            1           4h
deploy/kube-dns                   1         1         1            0           19s

NAME                                     DESIRED   CURRENT   READY     AGE
rs/calico-policy-controller-5586b678b5   0         0         0         1h
rs/calico-policy-controller-57dd959cc9   0         0         0         4h
rs/calico-policy-controller-6d94579b6b   1         1         1         56m
rs/kube-dns-794845bc6f                   1         1         0         19s

NAME                                           READY     STATUS              RESTARTS   AGE
po/calico-policy-controller-6d94579b6b-vksgv   1/1       Running             0          56m
po/kube-dns-794845bc6f-464d8                   0/3       ContainerCreating   0          19s

NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
svc/kube-dns   ClusterIP   10.254.0.2   <none>        53/UDP,53/TCP   19s

建立 calico-node.service 文件

上一步註釋了 calico.yaml 中 Calico Node 相關內容，爲了防止自動獲取 IP 出現問題，將其移動到 Systemd，Systemd service 配置以下，每一個節點都要安裝 calico-node 的 Service，其餘節點請自行修改 ip。

cat <<EOF > /lib/systemd/system/calico-node.service
[Unit]
Description=calico node
After=docker.service
Requires=docker.service

[Service]
User=root
PermissionsStartOnly=true
ExecStart=/usr/bin/docker run --net=host --privileged --name=calico-node \
  -e ETCD_ENDPOINTS=https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379 \
  -e ETCD_CA_CERT_FILE=/etc/kubernetes/ssl/ca.pem \
  -e ETCD_CERT_FILE=/etc/kubernetes/ssl/etcd.pem \
  -e ETCD_KEY_FILE=/etc/kubernetes/ssl/etcd-key.pem \
  -e NODENAME=${HOSTNAME} \
  -e IP= \
  -e NO_DEFAULT_POOLS= \
  -e AS= \
  -e CALICO_LIBNETWORK_ENABLED=true \
  -e IP6= \
  -e CALICO_NETWORKING_BACKEND=bird \
  -e FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT \
  -e FELIX_HEALTHENABLED=true \
  -e CALICO_IPV4POOL_CIDR=10.233.0.0/16 \
  -e CALICO_IPV4POOL_IPIP=always \
  -e IP_AUTODETECTION_METHOD=interface=eth0 \
  -e IP6_AUTODETECTION_METHOD=interface=eth0 \
-v /etc/kubernetes/ssl:/etc/kubernetes/ssl \
  -v /var/run/calico:/var/run/calico \
  -v /lib/modules:/lib/modules \
  -v /run/docker/plugins:/run/docker/plugins \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/log/calico:/var/log/calico \
  quay.io/calico/node:v2.6.2
ExecStop=/usr/bin/docker rm -f calico-node
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

啓動 Calico Node

Calico Node 採用 Systemd 方式啓動，在每一個節點配置好 Systemd service後，每一個節點修改對應的 calico-node.service 中的 IP 和節點名稱，而後啓動便可

systemctl daemon-reload
systemctl restart calico-node
systemctl restart kubelet

在全部節點啓動 Calico-node
systemctl enable calico-node.service && systemctl start calico-node.service

在master查看 Calico nodes

cat <<EOF > ~/calico-rc
export ETCD_ENDPOINTS="https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379"
export ETCD_CA_CERT_FILE="/etc/kubernetes/ssl/ca.pem"
export ETCD_CERT_FILE="/etc/kubernetes/ssl/etcd.pem"
export ETCD_KEY_FILE="/etc/kubernetes/ssl/etcd-key.pem"
EOF

. ~/calico-rc
calicoctl get node -o wide

查看 pending 的 pod 是否已執行

kubectl -n kube-system get po

部署 Ingress

Kubernetes 暴露服務的方式目前只有三種：LoadBlancer Service、NodePort Service、Ingress；什麼是 Ingress ? Ingress 就是利用 Nginx Haproxy 等負載均衡工具來暴露 Kubernetes 服務

配置調度 node

#ingress 有多種方式 1. deployment 自由調度 replicas 2. daemonset 全局調度分配到全部node裏

#deployment 自由調度過程當中，因爲咱們須要約束 controller 調度到指定的 node 中，因此須要對 node 進行 label 標籤

#默認以下:

[root@incubator-dc-016 k8s]# kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
172.16.16.246   Ready     <none>    4h        v1.8.3
172.16.16.250   Ready     <none>    4h        v1.8.3
172.16.16.86    Ready     <none>    5h        v1.8.3
[root@incubator-dc-016 k8s]# 

# 對 86 打上 label
kubectl label nodes 172.16.16.86 ingress=proxy

# 打完標籤之後
NAME            STATUS    ROLES     AGE       VERSION   LABELS
172.16.16.246   Ready     <none>    4h        v1.8.3    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=172.16.16.246
172.16.16.250   Ready     <none>    4h        v1.8.3    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=172.16.16.250
172.16.16.86    Ready     <none>    5h        v1.8.3    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress=proxy,kubernetes.io/hostname=172.16.16.86

下載Ingress鏡像

#官方鏡像
gcr.io/google_containers/defaultbackend:1.0
gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.17

#國內鏡像
jicki/defaultbackend:1.0
jicki/nginx-ingress-controller:0.9.0-beta.17

下載yaml文件

#Ingress yaml 文件模板
#default-backend.yaml
cat <<EOF >default-backend.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: default-http-backend
  namespace: kube-system
spec:
  replicas: 1
  selector:
    k8s-app: default-http-backend
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        image: registry.cn-qingdao.aliyuncs.com/kube8s/defaultbackend:1.0 
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    k8s-app: default-http-backend
EOF
#rbac.yaml

cat <<EOF >rbac.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      hostNetwork: true
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        ingress: proxy
      containers:
        - name: nginx-ingress-controller
          image: jicki/nginx-ingress-controller:0.9.0-beta.17
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --apiserver-host=http://172.16.16.86:8080
#- --configmap=$(POD_NAMESPACE)/nginx-configuration
#- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
#- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: KUBERNETES_MASTER
              value: http://172.16.16.86:8080
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
EOF

#with-rbac.yaml
cat <<EOF >with-rbac.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      hostNetwork: true
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        ingress: proxy
      containers:
        - name: nginx-ingress-controller
          image: jicki/nginx-ingress-controller:0.9.0-beta.17
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --apiserver-host=http://172.16.16.86:8080
#- --configmap=$(POD_NAMESPACE)/nginx-configuration
#- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
# - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: KUBERNETES_MASTER
              value: http://172.16.16.86:8080
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
EOF

導入yaml文件

kubectl apply -f default-backend.yaml 
kubectl apply -f rbac.yaml
kubectl apply -f with-rbac.yaml
[root@incubator-dc-016 Ingress]# curl http://172.16.16.86:8080/healthz

查看ingress服務

kubectl get svc -n kube-system

部署 Dashboard

###下載dashboard鏡像

#官方鏡像
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3

#國內鏡像
jicki/kubernetes-dashboard-amd64:v1.6.3

下載yaml文件

curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dashboard/dashboard-controller.yaml

curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dashboard/dashboard-service.yaml

#由於開啓了 RBAC 因此這裏須要建立一個 RBAC 認證

vi dashboard-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard
  namespace: kube-system

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: dashboard
subjects:
  - kind: ServiceAccount
    name: dashboard
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

#Dashboard yaml文件模板

#dashboard-controller.yaml
cat <<EOF >dashboard-controller.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: dashboard
      containers:
      - name: kubernetes-dashboard
        image: jicki/kubernetes-dashboard-amd64:v1.6.3
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9090
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
EOF

#dashboard-service.yaml
cat <<EOF >dashboard-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    k8s-app: kubernetes-dashboard
  ports:
    - port: 80
      targetPort: 9090
EOF

導入yaml文件

kubectl apply -f .
deployment "kubernetes-dashboard" created
serviceaccount "dashboard" created
clusterrolebinding "dashboard" created
service "kubernetes-dashboard" created

查看Dashboard服務

kubectl get svc -n kube-system

這裏部署結束，看的比較複雜，但也是用心一筆筆操做的來的，如今都是kuberadmin部署k8s,但至少也要手動去部署一遍，這樣才能瞭解裏面的意思和原理，排查錯誤也方便，附上報錯問題及解決方法

報錯彙總

沒法重啓docker

Registered Authentication Agent for unix-process:26237:1270527351
解決：echo 1 > /proc/sys/vm/drop_caches

etcd啓動不了

open /etc/kubernetes/ssl/etcd.pem: permission denied
解決：
chmod +x /etc/kubernetes/ssl/etcd.pem
chmod 755 /etc/kubernetes/ssl/

Etcd 127.0.0.1錯誤

Apr 07 14:44:59 incubator-dc-016 etcd[490]: The scheme of client url http://127.0.0.1:2379 is HTTP while peer key/cert files are presented. Ignored key/cert files.
Apr 07 14:44:59 incubator-dc-016 etcd[490]: listening for client requests on 127.0.0.1:2379
Apr 07 14:44:59 incubator-dc-016 etcd[490]: listening for client requests on 172.16.16.86:2379
Apr 07 14:44:59 incubator-dc-016 etcd[490]: create snapshot directory error: mkdir /var/lib/etcd/member/snap: permission denied

解決：
解決方法和思路:
刪除全部etcd數據,從新初始化.

rm -rf /var/lib/etcd/*
systemctl daemon-reload && systemctl restart etcd
systemctl status etcd.service

Api啓動不了

端口被佔用錯誤
failed to listen on 172.16.16.86:6443: listen tcp 172.16.16.86:6443: bind: address already in use

解決：
發現是docker佔用，中止docker nginx使用
而後重啓api服務

Proxy啓動不了

Failed at step CHDIR spawning /usr/local/bin/kube-proxy: No such file or directory

解決：
須要mkdir -p /var/lib/kube-proxy

calico啓動不了

calico版本以3.2.6版本爲基礎，原則上如今的安裝不能低於3.1，不然會出現各類問題，已踩過相關的坑。
當出現Kubernetes Calico node ‘XXXXXXXXXXX’ already using IPv4 Address XXXXXXXXX, CrashLoopBackOff錯誤時，多是calico的版本太低

ERROR: Error accessing the Calico datastore: open /etc/kubernetes/ssl/etcd.pem: no such file or directoryCalico node failed to start
-v /etc/kubernetes/ssl:/etc/kubernetes/ssl \

Apr 21 16:06:34 incubator-dc-002 docker[22733]: ERROR: Couldn't autodetect a management IPv4 address:
Apr 21 16:06:34 incubator-dc-002 docker[22733]: -  provide an IPv4 address by configuring one in the node resource, or
Apr 21 16:06:34 incubator-dc-002 docker[22733]: -  provide an IPv4 address using the IP environment, or
Apr 21 16:06:34 incubator-dc-002 docker[22733]: -  if auto-detecting, use a different autodetection method.
  -e IP_AUTODETECTION_METHOD=interface=eth0 \
  -e IP6_AUTODETECTION_METHOD=interface=eth0 \

但願你們共同進步，一塊兒學習，往更優秀的方向走！