kubeadm是Kubernetes官方提供的用於快速安裝Kubernetes集羣的工具,伴隨Kubernetes每一個版本的發佈都會同步更新,kubeadm會對集羣配置方面的一些實踐作調整,經過實驗kubeadm能夠學習到Kubernetes官方在集羣配置上一些新的最佳實踐。node
[root@k8s-master ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.0.246 k8s-master 192.168.0.247 k8s-node1 192.168.0.248 k8s-node2
若是各個主機啓用了防火牆,須要開放Kubernetes各個組件所須要的端口,能夠查看Installing kubeadm中的」Check required ports」一節。 這裏簡單起見在各節點禁用防火牆:linux
systemctl stop firewalld systemctl disable firewalld
禁用SELINUX:nginx
setenforce 0
git
vi /etc/selinux/config SELINUX=disabled
建立/etc/sysctl.d/k8s.conf文件,添加以下內容:github
net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1
執行命令使修改生效。docker
modprobe br_netfilter
json
sysctl -p /etc/sysctl.d/k8s.conf
centos
因爲ipvs已經加入到了內核的主幹,因此爲kube-proxy開啓ipvs的前提須要加載如下的內核模塊:api
ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4
在全部的Kubernetes節點node1和node2上執行如下腳本:bash
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
上面腳本建立了的/etc/sysconfig/modules/ipvs.modules文件,保證在節點重啓後能自動加載所需模塊。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已經正確加載所需的內核模塊。
接下來還須要確保各個節點上已經安裝了ipset軟件包yum install ipset。 爲了便於查看ipvs的代理規則,最好安裝一下管理工具ipvsadm yum install ipvsadm。
若是以上前提條件若是不知足,則即便kube-proxy的配置開啓了ipvs模式,也會退回到iptables模式。
Kubernetes從1.6開始使用CRI(Container Runtime Interface)容器運行時接口。默認的容器運行時仍然是Docker,使用的是kubelet中內置dockershim CRI實現。
安裝docker的yum源:
yum install -y yum-utils device-mapper-persistent-data lvm2 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
查看最新的Docker版本:
yum list docker-ce.x86_64 --showduplicates |sort -r docker-ce.x86_64 3:18.09.7-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.6-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.5-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.4-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.3-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.2-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable docker-ce.x86_64 18.06.3.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.2.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable
Kubernetes 1.15當前支持的docker版本列表是1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09。 這裏在各節點安裝docker的18.09.7版本。
yum makecache fast yum install -y --setopt=obsoletes=0 \ docker-ce-18.09.7-3.el7 systemctl start docker systemctl enable docker
確認一下iptables filter表中FOWARD鏈的默認策略(pllicy)爲ACCEPT。
iptables -nvL Chain INPUT (policy ACCEPT 263 packets, 19209 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
1.4 修改docker cgroup driver爲systemd
根據文檔CRI installation中的內容,對於使用systemd做爲init system的Linux的發行版,使用systemd做爲docker的cgroup driver能夠確保服務器節點在資源緊張的狀況更加穩定,所以這裏修改各個節點上docker的cgroup driver爲systemd。
建立或修改/etc/docker/daemon.json:
{ "exec-opts": ["native.cgroupdriver=systemd"] }
重啓docker:
systemctl restart docker
docker info | grep Cgroup Cgroup Driver: systemd
2.1 安裝kubeadm和kubelet
下面在各節點安裝kubeadm和kubelet:
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
yum makecache fast yum install -y kubelet-1.15.2 kubeadm-1.15.2 kubectl-1.15.2
...
Installed: kubeadm.x86_64 0:1.15.0-0 kubectl.x86_64 0:1.15.0-0 kubelet.x86_64 0:1.15.0-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-4.el7 cri-tools.x86_64 0:1.12.0-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-9.el7 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2
從安裝結果能夠看出還安裝了cri-tools, kubernetes-cni, socat三個依賴:
官方從Kubernetes 1.14開始將cni依賴升級到了0.7.5版本
socat是kubelet的依賴
cri-tools是CRI(Container Runtime Interface)容器運行時接口的命令行工具
運行kubelet –help能夠看到原來kubelet的絕大多數命令行flag參數都被DEPRECATED了,如:
......--address 0.0.0.0 The IP address for the Kubelet to serve on (set to 0.0.0.0 for all IPv4 interfaces and
::for all IPv6 interfaces) (default 0.0.0.0) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
......
而官方推薦咱們使用–config指定配置文件,並在配置文件中指定原來這些flag所配置的內容。具體內容能夠查看這裏Set Kubelet parameters via a config file。這也是Kubernetes爲了支持動態Kubelet配置(Dynamic Kubelet Configuration)才這麼作的,參考Reconfigure a Node’s Kubelet in a Live Cluster。
kubelet的配置文件必須是json或yaml格式,具體可查看這裏。
Kubernetes 1.8開始要求關閉系統的Swap,若是不關閉,默認配置下kubelet將沒法啓動。 關閉系統的Swap方法以下:
swapoff -a
修改 /etc/fstab 文件,註釋掉 SWAP 的自動掛載,使用free -m確認swap已經關閉。 swappiness參數調整,修改/etc/sysctl.d/k8s.conf添加下面一行:
vm.swappiness=0
執行下面命令:
sysctl -p /etc/sysctl.d/k8s.conf
使修改生效。
在各節點開機啓動kubelet服務:
systemctl enable kubelet.service
初始化master以前確認修改/etc/sysconfig/kubelet 中的內容爲:
KUBELET_EXTRA_ARGS=--fail-swap-on=false
kubeadm init \ --apiserver-advertise-address=192.168.0.246 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.13.3 \ --service-cidr=10.1.0.0/16 \ --pod-network-cidr=10.244.0.0/16
結果主要內容以下:
Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.0.246:6443 --token 4qcl2f.gtl3h8e5kjltuo0r \ --discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e
執行如下命令:
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
最後給出了將節點加入集羣的命令:kubeadm join 192.168.0.246:6443 –token 4qcl2f.gtl3h8e5kjltuo0r \ –discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e
查看一下集羣狀態,確認個組件都處於healthy狀態:
kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"}
接下來安裝flannel network add-on:
kdir -p ~/k8s/ cd ~/k8s curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml kubectl apply -f kube-flannel.yml clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.extensions/kube-flannel-ds-amd64 created daemonset.extensions/kube-flannel-ds-arm64 created daemonset.extensions/kube-flannel-ds-arm created daemonset.extensions/kube-flannel-ds-ppc64le created daemonset.extensions/kube-flannel-ds-s390x created
[這裏注意kube-flannel.yml這個文件裏的flannel的鏡像是0.11.0,quay.io/coreos/flannel:v0.11.0-amd64
請添加連接描述]()
若是鏡像拉取失敗請每個node進行手動拉取:
docker pull quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64<br/>docker tag quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64<br/>docker rmi quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
使用kubectl get pod –all-namespaces -o wide確保全部的Pod都處於Running狀態。
kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-5c98db65d4-dr8lf 1/1 Running 0 52m coredns-5c98db65d4-lp8dg 1/1 Running 0 52m etcd-node1 1/1 Running 0 51m kube-apiserver-node1 1/1 Running 0 51m kube-controller-manager-node1 1/1 Running 0 51m kube-flannel-ds-amd64-mm296 1/1 Running 0 44s kube-proxy-kchkf 1/1 Running 0 52m kube-scheduler-node1 1/1 Running 0 51m
下面將node主機添加到Kubernetes集羣中,在node上執行:
kubeadm join 192.168.99.11:6443 --token 4qcl2f.gtl3h8e5kjltuo0r \ --discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e
加入集羣非常順利,下面在master節點上執行命令查看集羣中的節點:
kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready master 57m v1.15.2 k8s-node1 Ready <none> 11s v1.15.2 k8s-node2 Ready <none> 11s v1.15.2
修改ConfigMap的kube-system/kube-proxy中的config.conf,mode: 「ipvs」
kubectl edit cm kube-proxy -n kube-system
以後重啓各個節點上的kube-proxy pod:
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}' kubectl get pod -n kube-system | grep kube-proxy kube-proxy-7fsrg 1/1 Running 0 3s kube-proxy-k8vhm 1/1 Running 0 9s
kubectl logs kube-proxy-7fsrg -n kube-system I0703 04:42:33.308289 1 server_others.go:170] Using ipvs Proxier. W0703 04:42:33.309074 1 proxier.go:401] IPVS scheduler not specified, use rr by default I0703 04:42:33.309831 1 server.go:534] Version: v1.15.0 I0703 04:42:33.320088 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0703 04:42:33.320365 1 config.go:96] Starting endpoints config controller I0703 04:42:33.320393 1 controller_utils.go:1029] Waiting for caches to sync for endpoints config controller I0703 04:42:33.320455 1 config.go:187] Starting service config controller I0703 04:42:33.320470 1 controller_utils.go:1029] Waiting for caches to sync for service config controller I0703 04:42:33.420899 1 controller_utils.go:1036] Caches are synced for endpoints config controller I0703 04:42:33.420969 1 controller_utils.go:1036] Caches are synced for service config controller
愈來愈多的公司和團隊開始使用Helm這個Kubernetes的包管理器,這裏也將使用Helm安裝Kubernetes的經常使用組件。
Helm由客戶端命helm令行工具和服務端tiller組成,Helm的安裝十分簡單。 下載helm命令行工具到master節點node1的/usr/local/bin下,這裏下載的2.14.1版本:
curl -O https://get.helm.sh/helm-v2.14.1-linux-amd64.tar.gz tar -zxvf helm-v2.14.1-linux-amd64.tar.gz cd linux-amd64/ cp helm /usr/local/bin/
爲了安裝服務端tiller,還須要在這臺機器上配置好kubectl工具和kubeconfig文件,確保kubectl工具能夠在這臺機器上訪問apiserver且正常使用。 這裏的master節點已經配置好了kubectl。
由於Kubernetes APIServer開啓了RBAC訪問控制,因此須要建立tiller使用的service account: tiller並分配合適的角色給它。 詳細內容能夠查看helm文檔中的Role-based Access Control。 這裏簡單起見直接分配cluster-admin這個集羣內置的ClusterRole給它。建立helm-rbac.yaml文件:
apiVersion: v1 kind: ServiceAccount metadata: name: tiller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system
kubectl create -f helm-rbac.yaml serviceaccount/tiller created clusterrolebinding.rbac.authorization.k8s.io/tiller created
接下來使用helm部署tiller:
helm init --service-account tiller --skip-refresh Creating /root/.helm Creating /root/.helm/repository Creating /root/.helm/repository/cache Creating /root/.helm/repository/local Creating /root/.helm/plugins Creating /root/.helm/starters Creating /root/.helm/cache/archive Creating /root/.helm/repository/repositories.yaml Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com Adding local repo with URL: http://127.0.0.1:8879/charts $HELM_HOME has been configured at /root/.helm. Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy. To prevent this, run `helm init` with the --tiller-tls-verify flag. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation Happy Helming!
tiller默認被部署在k8s集羣中的kube-system這個namespace下:
kubectl get pod -n kube-system -l app=helm NAME READY STATUS RESTARTS AGE tiller-deploy-c4fd4cd68-dwkhv 1/1 Running 0 83s
helm version Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
若是 tiller拉取失敗請在全部Node節點手動拉取鏡像:
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.14.1 <br/>docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.14.1 gcr.io/kubernetes-helm/tiller:v2.14.1<br/>docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.14.1
最後在master上修改helm chart倉庫的地址爲azure提供的鏡像地址(阿里雲地址:):
helm repo add stable http://mirror.azure.cn/kubernetes/charts "stable" has been added to your repositories helm repo list NAME URL stable http://mirror.azure.cn/kubernetes/charts local http://127.0.0.1:8879/charts
3.3 使用Helm部署dashboard
kubernetes-dashboard.yaml:
image:
repository: k8s.gcr.io/kubernetes-dashboard-amd64
tag: v1.10.1
ingress:
enabled: true
hosts:
helm install stable/kubernetes-dashboard \
-n kubernetes-dashboard \
--namespace kube-system \
-f kubernetes-dashboard.yaml
kubectl -n kube-system get secret | grep kubernetes-dashboard-token
kubernetes-dashboard-token-pkm2s kubernetes.io/service-account-token 3 3m7s
kubectl describe -n kube-system secret/kubernetes-dashboard-token-pkm2s
Name: kubernetes-dashboard-token-pkm2s
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: kubernetes-dashboard
kubernetes.io/service-account.uid: 2f0781dd-156a-11e9-b0f0-080027bb7c43
Type: kubernetes.io/service-account-token
ca.crt: 1025 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1wa20ycyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjJmMDc4MWRkLTE1NmEtMTFlOS1iMGYwLTA4MDAyN2JiN2M0MyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.24ad6ZgZMxdydpwlmYAiMxZ9VSIN7dDR7Q6-RLW0qC81ajXoQKHAyrEGpIonfld3gqbE0xO8nisskpmlkQra72-9X6sBPoByqIKyTsO83BQlME2sfOJemWD0HqzwSCjvSQa0x-bUlq9HgH2vEXzpFuSS6Svi7RbfzLXlEuggNoC4MfA4E2hF1OXml8iAKx-49y1BQQe5FGWyCyBSi1TD-ZpVs44H5gIvsGK2kcvi0JT4oHXtWjjQBKLIWL7xxyRCSE4HmUZT2StIHnOwlX7IEIB0oBX4mPg2_xNGnqwcu-8OERU9IoqAAE2cZa0v3b5O2LMcJPrcxrVOukvRIumA
在dashboard的登陸窗口使用上面的token登陸。
3.4 使用Helm部署metrics-server
從Heapster的github https://github.com/kubernetes/heapster中能夠看到已經,heapster已經DEPRECATED。 這裏是heapster的deprecation timeline。 能夠看出heapster從Kubernetes 1.12開始從Kubernetes各類安裝腳本中移除。
Kubernetes推薦使用metrics-server。咱們這裏也使用helm來部署metrics-server。
metrics-server.yaml:
args:
helm install stable/metrics-server \
-n metrics-server \
--namespace kube-system \
-f metrics-server.yaml
使用下面的命令能夠獲取到關於集羣節點基本的指標信息:
若是拉取鏡像失敗請手動拉取:
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.5 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.5 gcr.io/google_containers/metrics-server-amd64:v0.3.5 docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.5
kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% node1 650m 32% 1276Mi 73% node2 73m 3% 527Mi 30%
kubectl top pod -n kube-system NAME CPU(cores) MEMORY(bytes) coredns-5c98db65d4-dr8lf 8m 7Mi coredns-5c98db65d4-lp8dg 6m 8Mi etcd-node1 44m 46Mi kube-apiserver-node1 74m 295Mi kube-controller-manager-node1 35m 50Mi kube-flannel-ds-amd64-7lwm9 2m 8Mi kube-flannel-ds-amd64-mm296 5m 9Mi kube-proxy-7fsrg 1m 11Mi kube-proxy-k8vhm 3m 11Mi kube-scheduler-node1 8m 15Mi kubernetes-dashboard-848b8dd798-c4sc2 2m 14Mi metrics-server-8456fb6676-fwh2t 10m 19Mi tiller-deploy-7bf78cdbf7-9q94c 1m 16Mi
遺憾的是,當前Kubernetes Dashboard還不支持metrics-server。所以若是使用metrics-server替代了heapster,將沒法在dashboard中以圖形展現Pod的內存和CPU狀況(實際上這也不是很重要,當前咱們是在Prometheus和Grafana中定製的Kubernetes集羣中各個Pod的監控,所以在dashboard中查看Pod內存和CPU也不是很重要)。 Dashboard的github上有不少這方面的討論,如https://github.com/kubernetes/dashboard/issues/2986,Dashboard已經準備在未來的某個時間點支持metrics-server。但因爲metrics-server和metrics pipeline確定是Kubernetes在monitor方面將來的方向,因此推薦使用metrics-server。
docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.15.0 docker tag registry.aliyuncs.com/google_containers/kube-apiserver:v1.15.0 k8s.gcr.io/ube-apiserver:v1.15.0 docker rmi registry.aliyuncs.com/google_containers/kube-apiserver:v1.15.0 docker pull registry.aliyuncs.com/google_containers/pause:3.1 docker tag registry.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1 docker rmi registry.aliyuncs.com/google_containers/pause:3.1 docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.15.0 docker tag registry.aliyuncs.com/google_containers/kube-controller-manager:v1.15.0 k8s.gcr.io/kube-controller-manager:v1.15.0 docker rmi registry.aliyuncs.com/google_containers/kube-controller-manager:v1.15.0 docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.15.0 docker tag registry.aliyuncs.com/google_containers/kube-scheduler:v1.15.0 k8s.gcr.io/kube-scheduler:v1.15.0 docker rmi registry.aliyuncs.com/google_containers/kube-scheduler:v1.15.0 docker pull registry.aliyuncs.com/google_containers/coredns:1.3.1 docker tag registry.aliyuncs.com/google_containers/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1 docker rmi registry.aliyuncs.com/google_containers/coredns:1.3.1 docker pull registry.aliyuncs.com/google_containers/etcd:3.3.10 docker tag registry.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10 docker rmi registry.aliyuncs.com/google_containers/etcd:3.3.10 docker pull quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64 docker tag quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64 docker rmi quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.14.1 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.14.1 gcr.io/kubernetes-helm/tiller:v2.14.1 docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.14.1 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1 docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1 docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.15.0 docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.15.0 k8s.gcr.io/kube-proxy:v1.15.0 docker rmi registry.aliyuncs.com/google_containers/kube-proxy:v1.15.0 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.5 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.5 k8s.gcr.io/metrics-server-amd64:v0.3.5 docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.5