【解決】Pod 狀態一直 Terminating，Need to kill Pod

偶然查看服務時發現有個單副本的 pod 一直處於Terminating狀態，已達 8 天之久，且手動刪除刪除不掉

[appuser@lyj 2019-09-20 14:35:36 ~]$ kubectl get pods -n sit |grep loan-after-query-service-sit
loan-after-query-service-sit-868758d5c8-7sfcr    1/1       Running   0          14m
loan-after-query-service-sit-868758d5c8-scmfx    1/1     Terminating   0          8d

使用命令kubectl describe pod loan-after-query-service-sit-868758d5c8-scmfx -n NAMESPACE查看該 pod的描述信息, 獲得以下

Events:
  Type    Reason   Age                 From                                        Message
  ----    ------   ----                ----                                        -------
  Normal  Killing  13s (x21 over 36m)  kubelet, cn-beijing.i-2ze0dg85eb6vxi1t0vcr  Killing container with id docker://loan-after-query-service:Need to kill Pod

這個問題是Kubernetes偶發的BUG，使用如下命令強制刪除：

kubectl delete pod xxxxxx --grace-period=0 --force

執行命令後返回一條警告信息---> 警告:當即刪除並不等待正在運行的資源已被終止的確認。資源能夠無限期地繼續在集羣上運行。

[appuser@lyj 2019-09-20 14:24:34 ~]$ kubectl delete pods loan-after-query-service-sit-868758d5c8-scmfx -n NAMESPACE --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "loan-after-query-service-sit-868758d5c8-scmfx" deleted

去到容器所在 Node機器上經過 docker ps 仍看獲得這個已經運行 8 天的容器

[root@iz2ze0dg85eb6vxi1t0vcrz ~]# docker ps -a|grep loan-after-query-service-sit
007eb7071429        registry-vpc.cn-beijing.aliyuncs.com/9fbank/loan-after-query-service    "/bin/sh -c 'sh /u..."   7 minutes ago       Up 7 minutes                                    k8s_loan-after-query-service_loan-after-query-service-sit-868758d5c8-7sfcr_sit_3eb28c46-db6f-11e9-aee8-00163e082a66_0
7b36d58a26d5        registry-vpc.cn-beijing.aliyuncs.com/acs/pause-amd64:3.0                "/pause"                 7 minutes ago       Up 7 minutes                                    k8s_POD_loan-after-query-service-sit-868758d5c8-7sfcr_sit_3eb28c46-db6f-11e9-aee8-00163e082a66_0
21ad6e62a0f2        registry-vpc.cn-beijing.aliyuncs.com/9fbank/loan-after-query-service    "/bin/sh -c 'sh /u..."   8 days ago          Up 8 days                                       k8s_loan-after-query-service_loan-after-query-service-sit-868758d5c8-scmfx_sit_efeb3399-d462-11e9-aee8-00163e082a66_0
7691a8019419        registry-vpc.cn-beijing.aliyuncs.com/acs/pause-amd64:3.0                "/pause"                 8 days ago          Exited (0) 43 minutes ago                       k8s_POD_loan-after-query-service-sit-868758d5c8-scmfx_sit_efeb3399-d462-11e9-aee8-00163e082a66_0

手動刪除該容器

[root@iz2ze0dg85eb6vxi1t0vcrz ~]# docker rm -f k8s_loan-after-query-service_loan-after-query-service-sit-868758d5c8-scmfx_sit_efeb3399-d462-11e9-aee8-00163e082a66_0
k8s_loan-after-query-service_loan-after-query-service-sit-868758d5c8-scmfx_sit_efeb3399-d462-11e9-aee8-00163e082a66_0