k8s集羣部分常見問題處理

時間 2020-05-09

原文原文鏈接

1 Coredns CrashLoopBackOff 致使沒法成功添加工做節點的問題

1.1 問題描述

kubeadm方式安裝的k8s 1.14.1集羣，使用一段時間後k8s-master-15-81機器重啓docker和kubelet服務後，coredns沒法工做了html

[root@k8s-master-15-81 k8s_config]# kubectl get pod -n kube-system
NAME                                       READY   STATUS             RESTARTS   AGE
coredns-8686dcc4fd-4qswq                   0/1     CrashLoopBackOff   15         40d
coredns-8686dcc4fd-769bs                   0/1     CrashLoopBackOff   15         40d
kube-apiserver-k8s-master-15-81            1/1     Running            4          40d
kube-apiserver-k8s-master-15-82            1/1     Running            0          40d
kube-apiserver-k8s-master-15-83            1/1     Running            0          40d
kube-controller-manager-k8s-master-15-81   1/1     Running            5          40d
kube-controller-manager-k8s-master-15-82   1/1     Running            1          40d
kube-controller-manager-k8s-master-15-83   1/1     Running            1          40d
kube-flannel-ds-amd64-4fg7t                1/1     Running            0          40d
kube-flannel-ds-amd64-bcl4j                1/1     Running            0          40d
kube-flannel-ds-amd64-k6vp2                1/1     Running            0          40d
kube-flannel-ds-amd64-lkjlz                1/1     Running            2          40d
kube-flannel-ds-amd64-mb2lg                1/1     Running            0          40d
kube-flannel-ds-amd64-nl9pn                1/1     Running            5          40d
kube-proxy-4sbms                           1/1     Running            2          40d
kube-proxy-9v6fm                           1/1     Running            0          40d
kube-proxy-jsnkk                           1/1     Running            5          40d
kube-proxy-rvkmh                           1/1     Running            0          40d
kube-proxy-s4dfv                           1/1     Running            0          40d
kube-proxy-s8lws                           1/1     Running            0          40d
kube-scheduler-k8s-master-15-81            1/1     Running            5          40d
kube-scheduler-k8s-master-15-82            1/1     Running            1          40d
kube-scheduler-k8s-master-15-83            1/1     Running            1          40d
kubernetes-dashboard-5f7b999d65-d7fpp      0/1     Terminating        0          18m
kubernetes-dashboard-5f7b999d65-k759t      0/1     Terminating        0          21m
kubernetes-dashboard-5f7b999d65-pmvkk      0/1     CrashLoopBackOff   2          43s
[root@k8s-master-15-81 k8s_config]#

此時其餘節點都是notready狀態node

[root@k8s-master-15-81 k8s_config]# kubectl get no
NAME               STATUS     ROLES    AGE   VERSION
k8s-master-15-81   Ready      master   40d   v1.14.1
k8s-master-15-82   NotReady   master   40d   v1.14.1
k8s-master-15-83   NotReady   master   40d   v1.14.1
k8s-node-15-84     NotReady   <none>   40d   v1.14.1
k8s-node-15-85     NotReady   <none>   40d   v1.14.1
k8s-node-15-86     NotReady   <none>   40d   v1.14.1
[root@k8s-master-15-81 k8s_config]#

初步診斷容器崩潰，咱們須要進一步查看日誌，使用「kubectl logs」：linux

此次咱們得到了如下具體錯誤：git

[root@k8s-master-15-81 ~]# kubectl -n kube-system logs coredns-8686dcc4fd-7fwcz 　　#這是主要是日誌
E1028 06:36:35.489403       1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
E1028 06:36:35.489403       1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
log: exiting because of error: log: cannot create log: open /tmp/coredns.coredns-8686dcc4fd-7fwcz.unknownuser.log.ERROR.20191028-063635.1: no such file or directory


[root@k8s-master-15-81 ~]# kubectl -n kube-system describe pod coredns-8686dcc4fd-4j5gv　　#這個日誌沒啥用
Name:               coredns-8686dcc4fd-4j5gv
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               k8s-master-15-81/192.168.15.81
Start Time:         Mon, 28 Oct 2019 14:15:16 +0800
Labels:             k8s-app=kube-dns
                    pod-template-hash=8686dcc4fd
Annotations:        <none>
Status:             Running
IP:                 10.244.0.30
Controlled By:      ReplicaSet/coredns-8686dcc4fd
Containers:
  coredns:
    Container ID:  docker://5473c887d6858f364e8fc4c8001e41b2c5e612ce55d7c409df69788abf6585ed
    Image:         registry.aliyuncs.com/google_containers/coredns:1.3.1
    Image ID:      docker-pullable://registry.aliyuncs.com/google_containers/coredns@sha256:638adb0319813f2479ba3642bbe37136db8cf363b48fb3eb7dc8db634d8d5a5b
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 28 Oct 2019 14:15:39 +0800
      Finished:     Mon, 28 Oct 2019 14:15:40 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 28 Oct 2019 14:15:20 +0800
      Finished:     Mon, 28 Oct 2019 14:15:21 +0800
    Ready:          False
    Restart Count:  2
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-ltkvt (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-ltkvt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-ltkvt
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From                       Message
  ----     ------     ----              ----                       -------
  Normal   Scheduled  27s               default-scheduler          Successfully assigned kube-system/coredns-8686dcc4fd-4j5gv to k8s-master-15-81
  Normal   Pulled     4s (x3 over 26s)  kubelet, k8s-master-15-81  Container image "registry.aliyuncs.com/google_containers/coredns:1.3.1" already present on machine
  Normal   Created    4s (x3 over 26s)  kubelet, k8s-master-15-81  Created container coredns
  Normal   Started    4s (x3 over 25s)  kubelet, k8s-master-15-81  Started container coredns
  Warning  BackOff    1s (x4 over 21s)  kubelet, k8s-master-15-81  Back-off restarting failed container
[root@k8s-master-15-81 ~]# kubectl -n kube-system describe pod coredns-8686dcc4fd-5p6tp
Name:               coredns-8686dcc4fd-5p6tp
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               k8s-master-15-81/192.168.15.81
Start Time:         Mon, 28 Oct 2019 14:15:15 +0800
Labels:             k8s-app=kube-dns
                    pod-template-hash=8686dcc4fd
Annotations:        <none>
Status:             Running
IP:                 10.244.0.29
Controlled By:      ReplicaSet/coredns-8686dcc4fd
Containers:
  coredns:
    Container ID:  docker://4b19e53c68188faa107c310e75c6927bb0e280be042019b2805ef050fcd9aaaf
    Image:         registry.aliyuncs.com/google_containers/coredns:1.3.1
    Image ID:      docker-pullable://registry.aliyuncs.com/google_containers/coredns@sha256:638adb0319813f2479ba3642bbe37136db8cf363b48fb3eb7dc8db634d8d5a5b
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 28 Oct 2019 14:16:09 +0800
      Finished:     Mon, 28 Oct 2019 14:16:10 +0800
    Ready:          False
    Restart Count:  3
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-ltkvt (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-ltkvt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-ltkvt
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                       Message
  ----     ------     ----               ----                       -------
  Normal   Scheduled  90s                default-scheduler          Successfully assigned kube-system/coredns-8686dcc4fd-5p6tp to k8s-master-15-81
  Warning  Unhealthy  85s                kubelet, k8s-master-15-81  Readiness probe failed: HTTP probe failed with statuscode: 503
  Normal   Pulled     36s (x4 over 89s)  kubelet, k8s-master-15-81  Container image "registry.aliyuncs.com/google_containers/coredns:1.3.1" already present on machine
  Normal   Created    36s (x4 over 88s)  kubelet, k8s-master-15-81  Created container coredns
  Normal   Started    36s (x4 over 88s)  kubelet, k8s-master-15-81  Started container coredns
  Warning  BackOff    4s (x11 over 84s)  kubelet, k8s-master-15-81  Back-off restarting failed container
[root@k8s-master-15-81 ~]#

強制刪除coredns pod無效github

[root@k8s-master-15-81 ~]# kubectl delete pod coredns-8686dcc4fd-4j5gv --grace-period=0 --force -n kube-system
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "coredns-8686dcc4fd-4j5gv" force deleted
[root@k8s-master-15-81 ~]# kubectl delete pod coredns-8686dcc4fd-5p6tp --grace-period=0 --force -n kube-system
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "coredns-8686dcc4fd-5p6tp" force deleted
[root@k8s-master-15-81 ~]#

本地dns配置是ok的docker

[root@k8s-master-15-81 k8s_config]# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 10.68.8.65
nameserver 10.68.8.66
[root@k8s-master-15-81 k8s_config]#

1.2 解決方案：

這問題頗有多是防火牆（iptables）規則錯亂或者緩存致使的，能夠依次執行如下命令進行解決：api

systemctl stop kubelet
systemctl stop docker
iptables --flush
iptables -tnat --flush
systemctl start kubelet
systemctl start docker

執行如上命令後問題解決緩存

2 添加工做節點時提示token過時

集羣註冊token的有效時間爲24小時，若是集羣建立完成後沒有及時添加工做節點，那麼咱們須要從新生成token。相關命令以下所示：bash

#生成token
kubeadm token generate
#根據token輸出添加命令
kubeadm token create <token> --print-join-command --ttl=0

而後僅需複製打印出來的命令到工做節點執行便可。服務器

3 kubectl 執行命令報「The connection to the server localhost:8080 was refused」

做爲集羣管理的核心，工做節點上的kubectl可能一上來就跪了，以下圖所示：

出現這個問題的緣由是kubectl命令須要使用kubernetes-admin的身份來運行，在「kubeadm int」啓動集羣的步驟中就生成了「/etc/kubernetes/admin.conf」。

所以，解決方法以下，將主節點中的【/etc/kubernetes/admin.conf】文件拷貝到工做節點相同目錄下：

#複製admin.conf，請在主節點服務器上執行此命令
scp /etc/kubernetes/admin.conf 172.16.2.202:/etc/kubernetes/admin.conf
scp /etc/kubernetes/admin.conf 172.16.2.203:/etc/kubernetes/admin.conf

而後分別在工做節點上配置環境變量：

#設置kubeconfig文件
export KUBECONFIG=/etc/kubernetes/admin.conf
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

接下來，工做節點就正常了，如：

4 網絡組件flannel沒法完成初始化

網絡組件flannel安裝完成後，經過命令查看時一直在初始化狀態，而且經過日誌輸出內容以下所示：

kubectl get pods -n kube-system -o wide
kubectl logs -f kube-flannel-ds-amd64-hl89n -n kube-system

具體錯誤日誌爲：

Error from server: Get https://172.16.2.203:10250/containerLogs/kube-system/kube-flannel-ds-amd64-hl89n/kube-flannel?follow=true: dial tcp 172.16.2.203:10250: connect: no route to host

這時，咱們能夠登陸節點所在的服務器，使用如下命令來查看目標節點上的kubelet日誌：

journalctl -u kubelet -f

注意：journalctl工具能夠查看全部日誌，包括內核日誌和應用日誌。

經過日誌，咱們發現是鏡像拉取的問題。對此，你們能夠參考上文中鏡像拉取的方式以及重命名鏡像標籤來解決此問題，固然也能夠經過設置代理來解決此問題。

5 部分節點沒法啓動pod

有時候，咱們部署了應用以後，發如今部分工做節點上pod沒法啓動（一直處於ContainerCreating的狀態）：

經過排查日誌最終咱們獲得重要信息以下所示：

NetworkPlugin cni failed to set up pod "demo-deployment-675b5f9477-hdcwg_default" network: failed to set bridge addr: "cni0" already has an IP address different from 10.0.2.1/24

這是因爲當前節點以前被反覆註冊，致使flannel網絡出現問題。能夠依次執行如下腳原本重置節點而且刪除flannel網絡來解決：

kubeadm reset    #重置節點
systemctl stop kubelet && systemctl stop docker && rm -rf /var/lib/cni/ && rm -rf /var/lib/kubelet/* && rm -rf /var/lib/etcd && rm -rf /etc/cni/ && ifconfig cni0 down && ifconfig flannel.1 down && ifconfig docker0 down && ip link delete cni0 && ip link delete flannel.1
systemctl start docker

執行完成後，從新生成token並註冊節點便可，具體能夠參考上文內容。

參考：

http://www.javashuo.com/article/p-sehkjbtg-du.html