k8s 之kuberadmin部署多節點

系統centos7.6.1801
master: 192.168.60.115
slave1:192.168.60.102
slave2:192.168.60.101
保證全部swap分區是0

全部節點：

[root@k8s-master ~]# cat <<EOF > /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

[root@k8s-master ~]# sysctl --system

• Applying /usr/lib/sysctl.d/00-system.conf ...
• Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
  kernel.yama.ptrace_scope = 0
• Applying /usr/lib/sysctl.d/50-default.conf ...
  kernel.sysrq = 16
  kernel.core_uses_pid = 1
  net.ipv4.conf.default.rp_filter = 1
  net.ipv4.conf.all.rp_filter = 1
  net.ipv4.conf.default.accept_source_route = 0
  net.ipv4.conf.all.accept_source_route = 0
  net.ipv4.conf.default.promote_secondaries = 1
  net.ipv4.conf.all.promote_secondaries = 1
  fs.protected_hardlinks = 1
  fs.protected_symlinks = 1
• Applying /usr/lib/sysctl.d/99-docker.conf ...
  fs.may_detach_mounts = 1
• Applying /etc/sysctl.d/99-sysctl.conf ...
• Applying /etc/sysctl.d/k8s.conf ...
• Applying /etc/sysctl.conf ...

[root@k8s-master ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@k8s-master ~]# yum-config-manager --add-repo https://download.docker.com/linux/cento
s/docker-ce.repo
[root@k8s-master ~]# yum install docker-ce-18.06.3.ce-3.el7

添加k8s阿里源

[root@k8s-master ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/d
oc/rpm-package-key.gpg
EOF
安裝k8s kubeadmin
[root@k8s-master ~]# yum install -y kubelet kubeadm kubectl #不安裝默認1.18.4版本會報如下錯，因kuberadmin，kubectl後續會拉取proxy等鏡像最好同版本這個先這樣
指定版本會報錯：
[root@k8s-master ~]# yum install -y kubelet-1.13.3 kubeadm-1.13.3 kubectl-1.13.3 kubernete
s-cni-0.7.5
已加載插件：fastestmirror, product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.

Loading mirror speeds from cached hostfile

• base: mirrors.bfsu.edu.cn
• epel: mirrors.yun-idc.com
• extras: mirrors.163.com

• updates: mirrors.163.com
  軟件包 kubernetes-cni 已經被 kubelet 取代，改成嘗試安裝 kubelet-1.18.4-0.x86_64
  正在解決依賴關係
  --> 正在檢查事務
  ---> 軟件包 kubeadm.x86_64.0.1.13.3-0 將被 安裝
  --> 正在處理依賴關係 kubernetes-cni >= 0.6.0，它被軟件包 kubeadm-1.13.3-0.x86_64 須要
  軟件包 kubernetes-cni 已經被 kubelet 取代，可是取代的軟件包並未知足需求
  --> 正在處理依賴關係 cri-tools >= 1.11.0，它被軟件包 kubeadm-1.13.3-0.x86_64 須要
  ---> 軟件包 kubectl.x86_64.0.1.13.3-0 將被 安裝
  ---> 軟件包 kubelet.x86_64.0.1.13.3-0 將被 安裝
  --> 正在處理依賴關係 kubernetes-cni = 0.6.0，它被軟件包 kubelet-1.13.3-0.x86_64 須要
  軟件包 kubernetes-cni 已經被 kubelet 取代，可是取代的軟件包並未知足需求
  --> 正在處理依賴關係 socat，它被軟件包 kubelet-1.13.3-0.x86_64 須要
  --> 正在處理依賴關係 ebtables，它被軟件包 kubelet-1.13.3-0.x86_64 須要
  --> 正在處理依賴關係 conntrack，它被軟件包 kubelet-1.13.3-0.x86_64 須要
  ---> 軟件包 kubelet.x86_64.0.1.18.4-0 將被 安裝
  --> 正在檢查事務
  ---> 軟件包 conntrack-tools.x86_64.0.1.4.4-7.el7 將被 安裝
  --> 正在處理依賴關係 libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit)，它被軟件包 conntrack-tool
  s-1.4.4-7.el7.x86_64 須要
  --> 正在處理依賴關係 libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit)，它被軟件包 conntrack-tool
  s-1.4.4-7.el7.x86_64 須要
  --> 正在處理依賴關係 libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit)，它被軟件包 conntrack-tools-
  1.4.4-7.el7.x86_64 須要
  --> 正在處理依賴關係 libnetfilter_queue.so.1()(64bit)，它被軟件包 conntrack-tools-1.4.4-7.el7.x86_64 須要
  --> 正在處理依賴關係 libnetfilter_cttimeout.so.1()(64bit)，它被軟件包 conntrack-tools-1.4.4-7.el7.x86_64 須要
  --> 正在處理依賴關係 libnetfilter_cthelper.so.0()(64bit)，它被軟件包 conntrack-tools-1.4.4-7.el7.x86_64 須要
  ---> 軟件包 cri-tools.x86_64.0.1.13.0-0 將被 安裝
  ---> 軟件包 ebtables.x86_64.0.2.0.10-16.el7 將被 安裝
  ---> 軟件包 kubeadm.x86_64.0.1.13.3-0 將被 安裝
  --> 正在處理依賴關係 kubernetes-cni >= 0.6.0，它被軟件包 kubeadm-1.13.3-0.x86_64 須要
  軟件包 kubernetes-cni 已經被 kubelet 取代，可是取代的軟件包並未知足需求
  ---> 軟件包 kubelet.x86_64.0.1.13.3-0 將被 安裝
  --> 正在處理依賴關係 kubernetes-cni = 0.6.0，它被軟件包 kubelet-1.13.3-0.x86_64 須要
  軟件包 kubernetes-cni 已經被 kubelet 取代，可是取代的軟件包並未知足需求
  ---> 軟件包 socat.x86_64.0.1.7.3.2-2.el7 將被 安裝
  --> 正在檢查事務
  ---> 軟件包 kubeadm.x86_64.0.1.13.3-0 將被 安裝
  --> 正在處理依賴關係 kubernetes-cni >= 0.6.0，它被軟件包 kubeadm-1.13.3-0.x86_64 須要
  軟件包 kubernetes-cni 已經被 kubelet 取代，可是取代的軟件包並未知足需求
  ---> 軟件包 kubelet.x86_64.0.1.13.3-0 將被 安裝
  --> 正在處理依賴關係 kubernetes-cni = 0.6.0，它被軟件包 kubelet-1.13.3-0.x86_64 須要
  軟件包 kubernetes-cni 已經被 kubelet 取代，可是取代的軟件包並未知足需求
  ---> 軟件包 libnetfilter_cthelper.x86_64.0.1.0.0-11.el7 將被 安裝
  ---> 軟件包 libnetfilter_cttimeout.x86_64.0.1.0.0-7.el7 將被 安裝
  ---> 軟件包 libnetfilter_queue.x86_64.0.1.0.2-2.el7_2 將被 安裝
  --> 解決依賴關係完成
  錯誤：軟件包：kubelet-1.13.3-0.x86_64 (kubernetes)
  須要：kubernetes-cni = 0.6.0
  可用: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
  kubernetes-cni = 0.3.0.1-0.07a8a2
  可用: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
  kubernetes-cni = 0.5.1-0
  可用: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
  kubernetes-cni = 0.5.1-1
  可用: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
  kubernetes-cni = 0.6.0-0
  可用: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
  kubernetes-cni = 0.7.5-0
  錯誤：軟件包：kubeadm-1.13.3-0.x86_64 (kubernetes)
  須要：kubernetes-cni >= 0.6.0
  可用: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
  kubernetes-cni = 0.3.0.1-0.07a8a2
  可用: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
  kubernetes-cni = 0.5.1-0
  可用: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
  kubernetes-cni = 0.5.1-1
  可用: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
  kubernetes-cni = 0.6.0-0
  可用: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
  kubernetes-cni = 0.7.5-0
  您能夠嘗試添加 --skip-broken 選項來解決該問題
  您能夠嘗試執行：rpm -Va --nofiles --nodigest
  自定義kuber的版本爲1.18.3也可成功#沒有1.18.4版本如今
  [root@k8s-master ~]# kubeadm init --image-repository=registry.aliyuncs.com/google_containers --pod-network-cidr=172.17.0.0/16 --kubernetes-version=v1.18.3
  To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.60.115:6443 --token 67tkfz.u343ok0fmwhaawop \
--discovery-token-ca-cert-hash sha256:534f975a09ddc465d8f319e5d15cc5ef49a797726dd4e1feba4d1635b9a1e0e0
子節點經過以上黑體字添加到集羣裏

如下錯沒有建立配置文件
[root@k8s-master ~]# kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?
建立配置文件
[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]#
[root@k8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
發現全部節點都是notready緣由是沒有安裝網絡組件
查日誌 journalctl -f -u kubelet
6月 19 16:34:07 k8s-master kubelet[9304]: E0619 16:34:07.402431 9304 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 2m57s v1.18.4
k8s-slave1.novalocal NotReady <none> 109s v1.18.4
k8s-slave2.novalocal NotReady <none> 118s v1.18.4
安裝網路組件
[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
檢查鏡像：
[root@k8s-master ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.18.3 3439b7546f29 4 weeks ago 117MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.18.3 76216c34ed0c 4 weeks ago 95.3MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.18.3 7e28efa976bd 4 weeks ago 173MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.18.3 da26705ccb4b 4 weeks ago 162MB
registry.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 4 months ago 683kB
registry.aliyuncs.com/google_containers/coredns 1.6.7 67da37a9a360 4 months ago 43.8MB
registry.aliyuncs.com/google_containers/kube-proxy v1.17.0 7d54289267dc 6 months ago 116MB
registry.aliyuncs.com/google_containers/etcd 3.4.3-0 303ce5db0e90 7 months ago 288MB
檢查全部k8s 系統組件 發現有拉去失敗的刪除失敗的鏡像從新拉（網絡緣由）
也能夠docker pull 鏡像手動去拉保證flannel組件版本節點一致自動會是running狀態
[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

[root@k8s-master ~]# kubectl get pods -o wide -n kube-systemNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEScoredns-7ff77c879f-844cw 0/1 Pending 0 15m <none> <none> <none> <none>coredns-7ff77c879f-dbhxb 0/1 Pending 0 15m <none> <none> <none> <none>etcd-k8s-master 1/1 Running 0 15m 192.168.60.115 k8s-master <none> <none>kube-apiserver-k8s-master 1/1 Running 0 15m 192.168.60.115 k8s-master <none> <none>kube-controller-manager-k8s-master 1/1 Running 0 15m 192.168.60.115 k8s-master <none> <none>kube-flannel-ds-amd64-5wwws 0/1 Init:0/1 0 8m42s 192.168.60.115 k8s-master <none> <none>kube-flannel-ds-amd64-bv5pf 0/1 Init:ImagePullBackOff 0 8m42s 192.168.60.102 k8s-slave1.novalocal <none> <none>kube-flannel-ds-amd64-sz57p 0/1 Init:ImagePullBackOff 0 8m42s 192.168.60.101 k8s-slave2.novalocal <none> <none>kube-proxy-crwrp 1/1 Running 0 15m 192.168.60.115 k8s-master <none> <none>kube-proxy-njsqf 1/1 Running 0 14m 192.168.60.101 k8s-slave2.novalocal <none> <none>kube-proxy-znh28 1/1 Running 0 14m 192.168.60.102 k8s-slave1.novalocal <none> <none>kube-scheduler-k8s-master 1/1 Running 0 15m 192.168.60.115 k8s-master <none> <none>[root@k8s-master ~]# kubectl get nodesNAME STATUS ROLES AGE VERSIONk8s-master Ready master 83m v1.18.4k8s-slave1.novalocal Ready <none> 82m v1.18.4k8s-slave2.novalocal Ready <none> 82m v1.18.4