kubernetes雲平臺管理實戰：集羣部署（一）

時間 2019-11-07

原文原文鏈接

1、環境規劃

一、架構拓撲圖

二、主機規劃

三、軟件版本

[root@k8s-master ~]# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 
[root@k8s-master ~]# uname -r
3.10.0-693.el7.x86_64
 
[root@k8s-master ~]# docker version 
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Mon Dec 11 16:08:42 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Mon Dec 11 16:08:42 2017
 OS/Arch:         linux/amd64
           
[root@k8s-master ~]# kubectl version 
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"269f928217957e7126dc87e6adfa82242bfe5b1e", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"269f928217957e7126dc87e6adfa82242bfe5b1e", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

[root@k8s-master ~]# kubectl cluster-info 
Kubernetes master is running at http://localhost:8080
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

[root@k8s-master ~]# kubectl get node
NAME        STATUS    AGE
k8s-node1   Ready     1d
k8s-node2   Ready     1d

四、修改主機和host解析

一、修改主機node

hostnamectl set-hostname k8s-master
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2

二、hosts解析linux

[root@k8s-master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
k8s-master 10.0.128.0
k8s-node1  10.0.128.1
k8s-node2  10.0.128.2 

scp -rp /etc/hosts 10.0.128.1:/etc/hosts
scp -rp /etc/hosts 10.0.128.2:/etc/hosts

2、全部節點安裝docker-1.12.6-68

一、下載rpm包到本地

http://vault.centos.org/7.4.1708/extras/x86_64/Packages/

二、安裝

yum localinstall docker-common-1.12.6-68.gitec8512b.el7.centos.x86_64.rpm -y
yum localinstall docker-client-1.12.6-68.gitec8512b.el7.centos.x86_64.rpm -y
yum localinstall docker-1.12.6-68.gitec8512b.el7.centos.x86_64.rpm -y

三、啓動

[root@k8s-master ~]# systemctl enable docker.service 
[root@k8s-node1 ~]# systemctl enable docker.service 
[root@k8s-node2 ~]# systemctl enable docker.service

四、遇到的坑

一、docker-common，docker-client，docker-1.12.6-68前後順序必定不能亂，負責會安裝失敗git

二、我安裝的時候就把docker-client和docker-common的順序搞反，耽誤了半天時間docker

3、master節點安裝etcd

k8s數據庫kv類型存儲，原生支持作集羣數據庫

一、安裝

yum install etcd.x86_64 -y

二、配置

vim /etc/etcd/etcd.conf 
修改一下兩行：
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://10.0.128.0:2379"

三、啓動

 systemctl start etcd.service
 systemctl enable etcd.service

四、健康檢查

[root@k8s-master ~]# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      1396/etcd           
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1162/sshd           
tcp6       0      0 :::2379                 :::*                    LISTEN      1396/etcd           
tcp6       0      0 :::22                   :::*                    LISTEN      1162/sshd           
udp        0      0 0.0.0.0:30430           0.0.0.0:*                           1106/dhclient       
udp        0      0 0.0.0.0:68              0.0.0.0:*                           1106/dhclient       
udp        0      0 127.0.0.1:323           0.0.0.0:*                           869/chronyd         
udp6       0      0 :::42997                :::*                                1106/dhclient       
udp6       0      0 ::1:323                 :::*                                869/chronyd         
[root@k8s-master ~]# etcdctl set tesstdir/testkey0 0
0
[root@k8s-master ~]# etcdctl get tesstdir/testkey0
0
[root@k8s-master ~]# etcdctl -C http://10.0.128.0:2379 cluster-health
member 8e9e05c52164694d is healthy: got healthy result from http://10.0.128.0:2379
cluster is healthy
[root@k8s-master ~]# systemctl stop etcd.service
[root@k8s-master ~]# etcdctl -C http://10.0.128.0:2379 cluster-health
cluster may be unhealthy: failed to list members
Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 10.0.128.0:2379: getsockopt: connection refused
error #0: dial tcp 10.0.128.0:2379: getsockopt: connection refused
[root@k8s-master ~]# systemctl start etcd.service
[root@k8s-master ~]# etcdctl -C http://10.0.128.0:2379 cluster-health
member 8e9e05c52164694d is healthy: got healthy result from http://10.0.128.0:2379
cluster is healthy

4、master節點安裝kubernetes，k8s

一、安裝

[root@k8s-master ~]# yum install kubernetes-master.x86_64 -y

二、配置

[root@k8s-master ~]#vim /etc/kubernetes/apiserver
修改以下：
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
KUBE_API_PORT="--port=8080"
KUBELET_PORT="--kubelet-port=10250"
KUBE_ETCD_SERVERS="--etcd-servers=http://10.0.128.0:2379"
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
[root@k8s-master ~]#vim /etc/kubernetes/config 
修改以下：
KUBE_MASTER="--master=http://10.0.128.0:8080"

三、啓動

systemctl start kube-apiserver.service 
systemctl start kube-controller-manager.service 
systemctl start kube-scheduler.service
systemctl enable kube-apiserver.service 
systemctl enable kube-controller-manager.service 
systemctl enable kube-scheduler.service

四、組件做用

api-server：　接受並響應用戶的請求
controller：　控制器管理，保證容器始終存活
scheduler：調度器，選擇啓動容器的node節點vim

5、node節點安裝kubernetes

一、安裝

[root@k8s-node1 ~]#yum install kubernetes-node.x86_64 -y

二、配置

[root@k8s-node1 ~]#  vim /etc/kubernetes/config 
修改以下內容 ：
KUBE_MASTER="--master=http://10.0.128.0:8080"

[root@k8s-node1 ~]# vim /etc/kubernetes/kubelet
修改以下內容：
KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_PORT="--port=10250"
KUBELET_HOSTNAME="--hostname-override=k8s-node1"
KUBELET_API_SERVER="--api-servers=http://10.0.128.0:8080"

三、啓動

[root@k8s-node1 ~]# systemctl enable kubelet.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@k8s-node1 ~]# systemctl start kubelet.service 
[root@k8s-node1 ~]# systemctl enable kube-proxy.service 
[root@k8s-node1 ~]# systemctl start kube-proxy.service

四、組件做用

kubelet 　　　　調用docker，管理容器生命週期
nova-compute 調用libvirt，管理虛擬機的生命週期
kube-poxy 提供容器網絡訪問centos

6、全部節點配置flannel網絡

一、master節點

一、安裝api

[root@k8s-master ~]# yum install flannel -y

二、配置bash

[root@k8s-master ~]# sed -i 's#http://127.0.0.1:2379#http://10.0.128.0:2379#g' /etc/sysconfig/flanneld 
[root@k8s-master ~]# etcdctl mk /atomic.io/network/config '{ "Network":"172.16.0.0/16" }'
{ "Network":"172.16.0.0/16" }

三、啓動網絡

[root@k8s-master ~]# systemctl enable flanneld.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
Created symlink from /etc/systemd/system/docker.service.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
[root@k8s-master ~]# systemctl start flanneld.service

四、檢查是否安裝成功

[root@k8s-master ~]# systemctl restart docker
[root@k8s-master ~]# systemctl restart kube-apiserver.service
[root@k8s-master ~]# systemctl restart kube-controller-manager.service
[root@k8s-master ~]# systemctl restart kube-scheduler.service 
[root@k8s-master ~]# ifconfig flannel0
flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472
        inet 172.16.67.0  netmask 255.255.0.0  destination 172.16.67.0
        inet6 fe80::b9bb:f96a:188d:426e  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 144 (144.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

二、node節點

一、安裝

[root@k8s-node1 ~]# yum install flannel -y
[root@k8s-node2 ~]# yum install flannel -y

二、配置

[root@k8s-node1 ~]#sed -i 's#http://127.0.0.1:2379#http://10.0.128.0:2379#g' /etc/sysconfig/flannel
[root@k8s-node2 ~]#sed -i 's#http://127.0.0.1:2379#http://10.0.128.0:2379#g' /etc/sysconfig/flannel

三、啓動

[root@k8s-node1 ~]# systemctl enable flanneld.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
Created symlink from /etc/systemd/system/docker.service.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
[root@k8s-node1 ~]# systemctl start flanneld.service 
[root@k8s-node2 ~]# systemctl enable flanneld.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
Created symlink from /etc/systemd/system/docker.service.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
[root@k8s-node2 ~]# systemctl start flanneld.service

四、檢查是否安裝成功

[root@k8s-node1 ~]# ifconfig flannel0
flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472
        inet 172.16.10.0  netmask 255.255.0.0  destination 172.16.10.0
        inet6 fe80::6fc:c859:c331:833d  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 144 (144.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@k8s-node2 ~]# ifconfig flannel0
flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472
        inet 172.16.48.0  netmask 255.255.0.0  destination 172.16.48.0
        inet6 fe80::2f4c:6385:45ca:34b8  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 144 (144.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

三、容器網絡互通測試

一、全部節點啓動容器並獲取容器ip地址

[root@k8s-master ~]# kubectl get nodes
NAME        STATUS     AGE
k8s-node1   Ready      10h
k8s-node2   NotReady   9h

[root@k8s-master ~]# docker run -it busybox /bin/sh
Unable to find image 'busybox:latest' locally
Trying to pull repository docker.io/library/busybox ... 
latest: Pulling from docker.io/library/busybox
57c14dd66db0: Pull complete 
Digest: sha256:7964ad52e396a6e045c39b5a44438424ac52e12e4d5a25d94895f2058cb863a0
/ # ifconfig|grep eth0
eth0      Link encap:Ethernet  HWaddr 02:42:AC:10:30:02  
          inet addr:172.16.48.2  Bcast:0.0.0.0  Mask:255.255.255.0

[root@k8s-node1 ~]# docker run -it busybox /bin/sh
Unable to find image 'busybox:latest' locally
Trying to pull repository docker.io/library/busybox ... 
latest: Pulling from docker.io/library/busybox
57c14dd66db0: Pull complete 
Digest: sha256:7964ad52e396a6e045c39b5a44438424ac52e12e4d5a25d94895f2058cb863a0
/ # ifconfig|grep eth0
eth0      Link encap:Ethernet  HWaddr 02:42:AC:10:0A:02  
          inet addr:172.16.10.2  Bcast:0.0.0.0  Mask:255.255.255.0


[root@k8s-node2 ~]# docker run -it busybox /bin/sh
Unable to find image 'busybox:latest' locally
Trying to pull repository docker.io/library/busybox ... 
latest: Pulling from docker.io/library/busybox
57c14dd66db0: Pull complete 
Digest: sha256:7964ad52e396a6e045c39b5a44438424ac52e12e4d5a25d94895f2058cb863a0
/ # ifconfig|grep eth0
eth0      Link encap:Ethernet  HWaddr 02:42:AC:10:30:02  
          inet addr:172.16.48.2  Bcast:0.0.0.0  Mask:255.255.255.0

二、網絡互通性測試

[root@k8s-node2 ~]# docker run -it busybox /bin/sh
Unable to find image 'busybox:latest' locally
Trying to pull repository docker.io/library/busybox ... 
latest: Pulling from docker.io/library/busybox
57c14dd66db0: Pull complete 
Digest: sha256:7964ad52e396a6e045c39b5a44438424ac52e12e4d5a25d94895f2058cb863a0
/ # ifconfig|grep eth0
eth0      Link encap:Ethernet  HWaddr 02:42:AC:10:30:02  
          inet addr:172.16.48.2  Bcast:0.0.0.0  Mask:255.255.255.0
		  
/ # ping 172.16.10.2
PING 172.16.10.2 (172.16.10.2): 56 data bytes
64 bytes from 172.16.10.2: seq=0 ttl=60 time=5.212 ms
64 bytes from 172.16.10.2: seq=1 ttl=60 time=1.076 ms
^C
--- 172.16.10.2 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 1.076/2.118/5.212 ms

/ # ping 172.16.48.2
PING 172.16.48.2 (172.16.48.2): 56 data bytes
64 bytes from 172.16.48.2: seq=0 ttl=60 time=5.717 ms
64 bytes from 172.16.48.2: seq=1 ttl=60 time=1.108 ms
^C
--- 172.16.48.2 ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max = 1.108/1.904/5.717 ms

四、遇到的坑：

若是用docker-1.31可能會有網絡不通的狀況，解決辦法以下

iptables -P FORWARD ACCEPT

7、配置master爲鏡像倉庫

一、master節點

一、配置

 vim /etc/sysconfig/docker

修改內容以下：
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false --registry-mirror=https://registry.docker-cn.com --insecure-registry=10.0.128.0:5000'
systemctl restart docker

二、啓動私有倉庫

[root@k8s-master ~]# docker run -d -p 5000:5000 --restart=always --name registry -v /opt/myregistry:/var/lib/registry registry
Unable to find image 'registry:latest' locally
Trying to pull repository docker.io/library/registry ... 
latest: Pulling from docker.io/library/registry
cd784148e348: Pull complete 
0ecb9b11388e: Pull complete 
918b3ddb9613: Pull complete 
5aa847785533: Pull complete 
adee6f546269: Pull complete 
Digest: sha256:1cd9409a311350c3072fe510b52046f104416376c126a479cef9a4dfe692cf57
ba8d9b958c7c0867d5443ee23825f842a580331c87f6555678709dfc8899ff17
[root@k8s-master ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS                    NAMES
ba8d9b958c7c        registry            "/entrypoint.sh /etc/"   11 seconds ago      Up 9 seconds                0.0.0.0:5000->5000/tcp   registry
15340ee09614        busybox             "/bin/sh"                3 hours ago         Exited (1) 19 minutes ago                            sleepy_mccarthy

三、推送鏡像測試

[root@k8s-master ~]# docker tag docker.io/busybox:latest 10.0.128.0:5000/busybox:latest
[root@k8s-master ~]# docker push 10.0.128.0:5000/busybox:latest
The push refers to a repository [10.0.128.0:5000/busybox]
683f499823be: Pushed 
latest: digest: sha256:bbb143159af9eabdf45511fd5aab4fd2475d4c0e7fd4a5e154b98e838488e510 size: 527

二、node節點

一、配置

 vim /etc/sysconfig/docker

修改內容以下：
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false --registry-mirror=https://registry.docker-cn.com --insecure-registry=10.0.128.0:5000'
systemctl restart docker

二、測試

[root@k8s-node1 ~]# docker pull 10.0.128.0:5000/busybox:latest
Trying to pull repository 10.0.128.0:5000/busybox ... 
latest: Pulling from 10.0.128.0:5000/busybox
Digest: sha256:bbb143159af9eabdf45511fd5aab4fd2475d4c0e7fd4a5e154b98e838488e510