[k8s]kubespray(ansible)自動化安裝k8s集羣

時間 2021-04-12

標籤 node python nginx git github docker bootstrap vim centos api 欄目測試自動化简体版

原文原文鏈接

kubespray(ansible)自動化安裝k8s集羣

https://github.com/kubernetes-incubator/kubespraynode

https://kubernetes.io/docs/setup/pick-right-solution/python

kubespray本質是一堆ansible的role文件,經過這種方式,即ansible方式能夠自動化的安裝高可用k8s集羣,目前支持1.9.
nginx

安裝完成後,k8s全部組件都是經過hyperkube的容器化來運行的.git

最佳安裝centos7github
安裝docker(kubespray默認給docker添加的啓動參數)docker

/usr/bin/dockerd
--insecure-registry=10.233.0.0/18
--graph=/var/lib/docker
--log-opt max-size=50m
--log-opt max-file=5
--iptables=false
--dns 10.233.0.3
--dns 114.114.114.114
--dns-search default.svc.cluster.local
--dns-search svc.cluster.local
--dns-opt ndots:2
--dns-opt timeout:2
--dns-opt attempts:2

- 規劃3(n1  n2 n3)主2從(n4 n5)

- hosts

192.168.2.11 n1.ma.com n1
192.168.2.12 n2.ma.com n2
192.168.2.13 n3.ma.com n3
192.168.2.14 n4.ma.com n4
192.168.2.15 n5.ma.com n5
192.168.2.16 n6.ma.com n6


- 1.9的ansible-playbook裏涉及到這些鏡像
gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1
gcr.io/google_containers/pause-amd64:3.0
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
gcr.io/google_containers/elasticsearch:v2.4.1
gcr.io/google_containers/fluentd-elasticsearch:1.22
gcr.io/google_containers/kibana:v4.6.1
gcr.io/kubernetes-helm/tiller:v2.7.2
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.0.1
gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.7.1
quay.io/l23network/k8s-netchecker-agent:v1.0
quay.io/l23network/k8s-netchecker-server:v1.0
quay.io/coreos/etcd:v3.2.4
quay.io/coreos/flannel:v0.9.1
quay.io/coreos/flannel-cni:v0.3.0
quay.io/calico/ctl:v1.6.1
quay.io/calico/node:v2.6.2
quay.io/calico/cni:v1.11.0
quay.io/calico/kube-controllers:v1.0.0
quay.io/calico/routereflector:v0.4.0
quay.io/coreos/hyperkube:v1.9.0_coreos.0
quay.io/ant31/kargo:master
quay.io/external_storage/local-volume-provisioner-bootstrap:v1.0.0
quay.io/external_storage/local-volume-provisioner:v1.0.0

- 1.9鏡像dockerhub版
lanny/gcr.io_google_containers_cluster-proportional-autoscaler-amd64:1.1.1
lanny/gcr.io_google_containers_pause-amd64:3.0
lanny/gcr.io_google_containers_k8s-dns-kube-dns-amd64:1.14.7
lanny/gcr.io_google_containers_k8s-dns-dnsmasq-nanny-amd64:1.14.7
lanny/gcr.io_google_containers_k8s-dns-sidecar-amd64:1.14.7
lanny/gcr.io_google_containers_elasticsearch:v2.4.1
lanny/gcr.io_google_containers_fluentd-elasticsearch:1.22
lanny/gcr.io_google_containers_kibana:v4.6.1
lanny/gcr.io_kubernetes-helm_tiller:v2.7.2
lanny/gcr.io_google_containers_kubernetes-dashboard-init-amd64:v1.0.1
lanny/gcr.io_google_containers_kubernetes-dashboard-amd64:v1.7.1
lanny/quay.io_l23network_k8s-netchecker-agent:v1.0
lanny/quay.io_l23network_k8s-netchecker-server:v1.0
lanny/quay.io_coreos_etcd:v3.2.4
lanny/quay.io_coreos_flannel:v0.9.1
lanny/quay.io_coreos_flannel-cni:v0.3.0
lanny/quay.io_calico_ctl:v1.6.1
lanny/quay.io_calico_node:v2.6.2
lanny/quay.io_calico_cni:v1.11.0
lanny/quay.io_calico_kube-controllers:v1.0.0
lanny/quay.io_calico_routereflector:v0.4.0
lanny/quay.io_coreos_hyperkube:v1.9.0_coreos.0
lanny/quay.io_ant31_kargo:master
lanny/quay.io_external_storage_local-volume-provisioner-bootstrap:v1.0.0
lanny/quay.io_external_storage_local-volume-provisioner:v1.0.0



- 配置文件
kubespray/inventory/group_vars/k8s-cluster.yml  爲控制一些基礎信息的配置文件。
修改成flannel

kubespray/inventory/group_vars/all.yml           控制一些須要詳細配置的信息 
修改成centos

- 替換flannel的vxlan爲host-gw
roles/network_plugin/flannel/defaults/main.yml
能夠經過grep -r 'vxlan' .  這種方式來找到

- 修改kube_api_pwd
vi roles/kubespray-defaults/defaults/main.yaml
kube_api_pwd: xxxx

- 證書時間
kubespray/roles/kubernetes/secrets/files/make-ssl.sh



- 修改image爲我的倉庫地址
能夠經過grep -r 'gcr.io' .  這種方式來找到
sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/download/defaults/main.yml
sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/dnsmasq/templates/dnsmasq-autoscaler.yml.j2
sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/kubernetes-apps/ansible/defaults/main.yml
sed -i 's#gcr\.io\/kubernetes-helm\/#lanny/gcr\.io_kubernetes-helm_#g' roles/download/defaults/main.yml

sed -i 's#quay\.io\/coreos\/#lanny/quay\.io_coreos_#g' roles/download/defaults/main.yml
sed -i 's#quay\.io\/calico\/#lanny/quay\.io_calico_#g' roles/download/defaults/main.yml
sed -i 's#quay\.io\/l23network\/#lanny/quay\.io_l23network_#g' roles/download/defaults/main.yml
sed -i 's#quay\.io\/l23network\/#lanny/quay\.io_l23network_#g' docs/netcheck.md
sed -i 's#quay\.io\/external_storage\/#lanny/quay\.io_external_storage_#g' roles/kubernetes-apps/local_volume_provisioner/defaults/main.yml
sed -i 's#quay\.io\/ant31\/kargo#lanny/quay\.io_ant31_kargo_#g' .gitlab-ci.yml





- 安裝完後查看鏡像(用到的鏡像(calico))
nginx:1.13
quay.io/coreos/hyperkube:v1.9.0_coreos.0
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
quay.io/calico/node:v2.6.2
gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.0.1
quay.io/calico/cniv:1.11.0
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.7.1
quay.io/calico/ctlv:1.6.1
quay.io/calico/routereflectorv:0.4.0
quay.io/coreos/etcdv:3.2.4
gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1
gcr.io/google_containers/pause-amd64:3.0


- kubespray生成配置所需的環境(python3 ansible) Ansible v2.4 (or newer) Jinja 2.9 (or newer) 
yum install python34 python34-pip python-pip  python-netaddr -y
cd
mkdir .pip
cd .pip
cat > pip.conf <<EOF
[global]
index-url = http://mirrors.aliyun.com/pypi/simple/

[install]
trusted-host=mirrors.aliyun.com
EOF

yum install gcc libffi-devel python-devel openssl-devel -y
pip install  Jinja2-2.10-py2.py3-none-any.whl # https://pypi.python.org/pypi/Jinja2
pip install cryptography
pip install ansible



- 克隆
git clone https://github.com/kubernetes-incubator/kubespray.git


- 修改配置

1. 使docker能pull gcr的鏡像
vim inventory/group_vars/all.yml
2 bootstrap_os: centos
95 http_proxy: "http://192.168.1.88:1080/"

2. 若是vm內存<=1G,若是>=3G,則無需修改
vim roles/kubernetes/preinstall/tasks/verify-settings.yml 
52 - name: Stop if memory is too small for masters
53   assert:
54     that: ansible_memtotal_mb <= 1500
55   ignore_errors: "{{ ignore_assert_errors }}"
56   when: inventory_hostname in groups['kube-master']
57 
58 - name: Stop if memory is too small for nodes
59   assert:
60     that: ansible_memtotal_mb <= 1024              
61   ignore_errors: "{{ ignore_assert_errors }}"


3. 修改swap,
vim roles/download/tasks/download_container.yml
 75 - name: Stop if swap enabled
 76   assert:
 77     that: ansible_swaptotal_mb == 0         
 78   when: kubelet_fail_swap_on|default(false)

全部機器執行: 關閉swap
swapoff -a
[root@n1 kubespray]# free -m
              total        used        free      shared  buff/cache   available
Mem:           2796         297        1861           8         637        2206
Swap:             0           0           0  #這欄爲0,表示關閉

- 生成 kubespray配置,開始ansible安裝k8s之旅(很是用時間,大到1h,小到20min)
cd kubespray
IPS=(192.168.2.11 192.168.2.12 192.168.2.13 192.168.2.14 192.168.2.15)
CONFIG_FILE=inventory/inventory.cfg python3 contrib/inventory_builder/inventory.py ${IPS[@]}
ansible-playbook -i inventory/inventory.cfg cluster.yml -b -v --private-key=~/.ssh/id_rsa


- 修改好的配置
git clone https://github.com/lannyMa/kubespray
run.sh 有啓動命令


- 個人inventory.cfg, 我不想讓node和master混在一塊兒手動改了下
[root@n1 kubespray]# cat inventory/inventory.cfg 
[all]
node1 	 ansible_host=192.168.2.11 ip=192.168.2.11
node2 	 ansible_host=192.168.2.12 ip=192.168.2.12
node3 	 ansible_host=192.168.2.13 ip=192.168.2.13
node4 	 ansible_host=192.168.2.14 ip=192.168.2.14
node5 	 ansible_host=192.168.2.15 ip=192.168.2.15

[kube-master]
node1 	 
node2 	 
node3

[kube-node] 	 
node4 	 
node5 	 

[etcd]
node1 	 
node2 	 
node3 	 

[k8s-cluster:children]
kube-node 	 
kube-master 	 

[calico-rr]

遇到的問題wait for the apiserver to be running

ansible-playbook -i inventory/inventory.ini cluster.yml -b -v --private-key=~/.ssh/id_rsa
...
<!-- We recommend using snippets services like https://gist.github.com/ etc. -->
RUNNING HANDLER [kubernetes/master : Master | wait for the apiserver to be running] ***
Thursday 23 March 2017  10:46:16 +0800 (0:00:00.468)       0:08:32.094 ******** 
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (10 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (10 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (9 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (9 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (8 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (8 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (7 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (7 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (6 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (6 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (5 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (5 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (4 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (4 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (3 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (3 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (2 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (2 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (1 retries left).
FAILED - RETRYING: HANDLER: kubernetes/master : Master | wait for the apiserver to be running (1 retries left).
fatal: [node1]: FAILED! => {"attempts": 10, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://localhost:8080/healthz"}
fatal: [node2]: FAILED! => {"attempts": 10, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://localhost:8080/healthz"}
        to retry, use: --limit @/home/dev_dean/kargo/cluster.retry

解決: 全部節點關閉swapbootstrap

swapoff -a

快捷命令

alias kk='kubectl get pod --all-namespaces -o wide --show-labels'
alias ks='kubectl get svc --all-namespaces -o wide'
alias kss='kubectl get svc --all-namespaces -o wide --show-labels'
alias kd='kubectl get deploy --all-namespaces -o wide'
alias wk='watch kubectl get pod --all-namespaces -o wide --show-labels'
alias kv='kubectl get pv -o wide'
alias kvc='kubectl get pvc -o wide --all-namespaces --show-labels'
alias kbb='kubectl run -it --rm --restart=Never busybox --image=busybox sh'
alias kbbc='kubectl run -it --rm --restart=Never curl --image=appropriate/curl sh'
alias kd='kubectl get deployment --all-namespaces --show-labels'
alias kcm='kubectl get cm --all-namespaces -o wide'
alias kin='kubectl get ingress --all-namespaces -o wide'

kubespray的默認啓動參數

ps -ef|egrep  "apiserver|controller-manager|scheduler"


/hyperkube apiserver \
--advertise-address=192.168.2.11 \
--etcd-servers=https://192.168.2.11:2379,https://192.168.2.12:2379,https://192.168.2.13:2379 \
--etcd-quorum-read=true \
--etcd-cafile=/etc/ssl/etcd/ssl/ca.pem \
--etcd-certfile=/etc/ssl/etcd/ssl/node-node1.pem \
--etcd-keyfile=/etc/ssl/etcd/ssl/node-node1-key.pem \
--insecure-bind-address=127.0.0.1 \
--bind-address=0.0.0.0 \
--apiserver-count=3 \
--admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ValidatingAdmissionWebhook,ResourceQuota \
--service-cluster-ip-range=10.233.0.0/18 \
--service-node-port-range=30000-32767 \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--profiling=false \
--repair-malformed-updates=false \
--kubelet-client-certificate=/etc/kubernetes/ssl/node-node1.pem \
--kubelet-client-key=/etc/kubernetes/ssl/node-node1-key.pem \
--service-account-lookup=true \
--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem \
--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem \
--proxy-client-cert-file=/etc/kubernetes/ssl/apiserver.pem \
--proxy-client-key-file=/etc/kubernetes/ssl/apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem \
--secure-port=6443 \
--insecure-port=8080 
--storage-backend=etcd3 \
--runtime-config=admissionregistration.k8s.io/v1alpha1 --v=2 \
--allow-privileged=true \
--anonymous-auth=False \
--authorization-mode=Node,RBAC \
--feature-gates=Initializers=False \
PersistentLocalVolumes=False


/hyperkube controller-manager \
--kubeconfig=/etc/kubernetes/kube-controller-manager-kubeconfig.yaml \
--leader-elect=true \
--service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--enable-hostpath-provisioner=false \
--node-monitor-grace-period=40s \
--node-monitor-period=5s \
--pod-eviction-timeout=5m0s \
--profiling=false \
--terminated-pod-gc-threshold=12500 \
--v=2 \
--use-service-account-credentials=true \
--feature-gates=Initializers=False \
PersistentLocalVolumes=False


/hyperkube scheduler \
--leader-elect=true \
--kubeconfig=/etc/kubernetes/kube-scheduler-kubeconfig.yaml \
--profiling=false --v=2 \
--feature-gates=Initializers=False \
PersistentLocalVolumes=False

/usr/local/bin/kubelet \
--logtostderr=true --v=2 \
--address=192.168.2.14 \
--node-ip=192.168.2.14 \
--hostname-override=node4 \
--allow-privileged=true \
--pod-manifest-path=/etc/kubernetes/manifests \
--cadvisor-port=0 \
--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \
--node-status-update-frequency=10s \
--docker-disable-shared-pid=True \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--tls-cert-file=/etc/kubernetes/ssl/node-node4.pem \
--tls-private-key-file=/etc/kubernetes/ssl/node-node4-key.pem \
--anonymous-auth=false \
--cgroup-driver=cgroupfs \
--cgroups-per-qos=True \
--fail-swap-on=True \
--enforce-node-allocatable= \
--cluster-dns=10.233.0.3 \
--cluster-domain=cluster.local \
--resolv-conf=/etc/resolv.conf \
--kubeconfig=/etc/kubernetes/node-kubeconfig.yaml \
--require-kubeconfig \
--kube-reserved cpu=100m,memory=256M \
--node-labels=node-role.kubernetes.io/node=true \
--feature-gates=Initializers=False,PersistentLocalVolumes=False \
--network-plugin=cni --cni-conf-dir=/etc/cni/net.d \
--cni-bin-dir=/opt/cni/bin


ps -ef|grep kube-porxy
hyperkube proxy --v=2 \
    --kubeconfig=/etc/kubernetes/kube-proxy-kubeconfig.yaml\
    --bind-address=192.168.2.14 \
    --cluster-cidr=10.233.64.0/18 \
    --proxy-mode=iptables

我把所須要的鏡像v1.9推送到dockerhub了

lanny/gcr.io_google_containers_cluster-proportional-autoscaler-amd64:1.1.1
lanny/gcr.io_google_containers_pause-amd64:3.0
lanny/gcr.io_google_containers_k8s-dns-kube-dns-amd64:1.14.7
lanny/gcr.io_google_containers_k8s-dns-dnsmasq-nanny-amd64:1.14.7
lanny/gcr.io_google_containers_k8s-dns-sidecar-amd64:1.14.7
lanny/gcr.io_google_containers_elasticsearch:v2.4.1
lanny/gcr.io_google_containers_fluentd-elasticsearch:1.22
lanny/gcr.io_google_containers_kibana:v4.6.1
lanny/gcr.io_kubernetes-helm_tiller:v2.7.2
lanny/gcr.io_google_containers_kubernetes-dashboard-init-amd64:v1.0.1
lanny/gcr.io_google_containers_kubernetes-dashboard-amd64:v1.7.1
lanny/quay.io_l23network_k8s-netchecker-agent:v1.0
lanny/quay.io_l23network_k8s-netchecker-server:v1.0
lanny/quay.io_coreos_etcd:v3.2.4
lanny/quay.io_coreos_flannel:v0.9.1
lanny/quay.io_coreos_flannel-cni:v0.3.0
lanny/quay.io_calico_ctl:v1.6.1
lanny/quay.io_calico_node:v2.6.2
lanny/quay.io_calico_cni:v1.11.0
lanny/quay.io_calico_kube-controllers:v1.0.0
lanny/quay.io_calico_routereflector:v0.4.0
lanny/quay.io_coreos_hyperkube:v1.9.0_coreos.0
lanny/quay.io_ant31_kargo:master
lanny/quay.io_external_storage_local-volume-provisioner-bootstrap:v1.0.0
lanny/quay.io_external_storage_local-volume-provisioner:v1.0.0




gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1
gcr.io/google_containers/pause-amd64:3.0
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
gcr.io/google_containers/elasticsearch:v2.4.1
gcr.io/google_containers/fluentd-elasticsearch:1.22
gcr.io/google_containers/kibana:v4.6.1
gcr.io/kubernetes-helm/tiller:v2.7.2
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.0.1
gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.7.1
quay.io/l23network/k8s-netchecker-agent:v1.0
quay.io/l23network/k8s-netchecker-server:v1.0
quay.io/coreos/etcd:v3.2.4
quay.io/coreos/flannel:v0.9.1
quay.io/coreos/flannel-cni:v0.3.0
quay.io/calico/ctl:v1.6.1
quay.io/calico/node:v2.6.2
quay.io/calico/cni:v1.11.0
quay.io/calico/kube-controllers:v1.0.0
quay.io/calico/routereflector:v0.4.0
quay.io/coreos/hyperkube:v1.9.0_coreos.0
quay.io/ant31/kargo:master
quay.io/external_storage/local-volume-provisioner-bootstrap:v1.0.0
quay.io/external_storage/local-volume-provisioner:v1.0.0


sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/download/defaults/main.yml
sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/dnsmasq/templates/dnsmasq-autoscaler.yml.j2
sed -i 's#gcr\.io\/google_containers\/#lanny/gcr\.io_google_containers_#g' roles/kubernetes-apps/ansible/defaults/main.yml
sed -i 's#gcr\.io\/kubernetes-helm\/#lanny/gcr\.io_kubernetes-helm_#g' roles/download/defaults/main.yml

sed -i 's#quay\.io\/coreos\/#lanny/quay\.io_coreos_#g' roles/download/defaults/main.yml
sed -i 's#quay\.io\/calico\/#lanny/quay\.io_calico_#g' roles/download/defaults/main.yml
sed -i 's#quay\.io\/l23network\/#lanny/quay\.io_l23network_#g' roles/download/defaults/main.yml
sed -i 's#quay\.io\/l23network\/#lanny/quay\.io_l23network_#g' docs/netcheck.md
sed -i 's#quay\.io\/external_storage\/#lanny/quay\.io_external_storage_#g' roles/kubernetes-apps/local_volume_provisioner/defaults/main.yml
sed -i 's#quay\.io\/ant31\/kargo#lanny/quay\.io_ant31_kargo_#g' .gitlab-ci.yml

參考: https://jicki.me/2017/12/08/kubernetes-kubespray-1.8.4/vim

構建etcd容器集羣

/usr/bin/docker run
    --restart=on-failure:5 
    --env-file=/etc/etcd.env
    --net=host
    -v /etc/ssl/certs:/etc/ssl/certs:ro
    -v /etc/ssl/etcd/ssl:/etc/ssl/etcd/ssl:ro
    -v /var/lib/etcd:/var/lib/etcd:rw
    --memory=512M
    --oom-kill-disable
    --blkio-weight=1000
    --name=etcd1
    lanny/quay.io_coreos_etcd:v3.2.4
    /usr/local/bin/etcd

[root@node1 ~]# cat /etc/etcd.env 
ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.2.11:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.2.11:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_METRICS=basic
ETCD_LISTEN_CLIENT_URLS=https://192.168.2.11:2379,https://127.0.0.1:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
ETCD_LISTEN_PEER_URLS=https://192.168.2.11:2380
ETCD_NAME=etcd1
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.2.11:2380,etcd2=https://192.168.2.12:2380,etcd3=https://192.168.2.13:2380
ETCD_AUTO_COMPACTION_RETENTION=8

# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-node1.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-node1-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-node1.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-node1-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true



[root@node1 ssl]# docker exec b1159a1c6209 env|grep -i etcd
ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.2.11:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.2.11:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_METRICS=basic
ETCD_LISTEN_CLIENT_URLS=https://192.168.2.11:2379,https://127.0.0.1:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
ETCD_LISTEN_PEER_URLS=https://192.168.2.11:2380
ETCD_NAME=etcd1
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.2.11:2380,etcd2=https://192.168.2.12:2380,etcd3=https://192.168.2.13:2380
ETCD_AUTO_COMPACTION_RETENTION=8
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-node1.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-node1-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-node1.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-node1-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true


[root@node1 ssl]# tree /etc/ssl/etcd/ssl
/etc/ssl/etcd/ssl
├── admin-node1-key.pem
├── admin-node1.pem
├── admin-node2-key.pem
├── admin-node2.pem
├── admin-node3-key.pem
├── admin-node3.pem
├── ca-key.pem
├── ca.pem
├── member-node1-key.pem
├── member-node1.pem
├── member-node2-key.pem
├── member-node2.pem
├── member-node3-key.pem
├── member-node3.pem
├── node-node1-key.pem
├── node-node1.pem
├── node-node2-key.pem
├── node-node2.pem
├── node-node3-key.pem
├── node-node3.pem
├── node-node4-key.pem
├── node-node4.pem
├── node-node5-key.pem
└── node-node5.pem

[root@n2 ~]# tree /etc/ssl/etcd/ssl
/etc/ssl/etcd/ssl
├── admin-node2-key.pem
├── admin-node2.pem
├── ca-key.pem
├── ca.pem
├── member-node2-key.pem
├── member-node2.pem
├── node-node1-key.pem
├── node-node1.pem
├── node-node2-key.pem
├── node-node2.pem
├── node-node3-key.pem
├── node-node3.pem
├── node-node4-key.pem
├── node-node4.pem
├── node-node5-key.pem
└── node-node5.pem

admin-node1.crt
CN = etcd-admin-node1

DNS Name=localhost
DNS Name=node1
DNS Name=node2
DNS Name=node3
IP Address=192.168.2.11
IP Address=192.168.2.11
IP Address=192.168.2.12
IP Address=192.168.2.12
IP Address=192.168.2.13
IP Address=192.168.2.13
IP Address=127.0.0.1

member-node1.crt
CN = etcd-member-node1

DNS Name=localhost
DNS Name=node1
DNS Name=node2
DNS Name=node3
IP Address=192.168.2.11
IP Address=192.168.2.11
IP Address=192.168.2.12
IP Address=192.168.2.12
IP Address=192.168.2.13
IP Address=192.168.2.13
IP Address=127.0.0.1

node-node1.crt
CN = etcd-node-node1

DNS Name=localhost
DNS Name=node1
DNS Name=node2
DNS Name=node3
IP Address=192.168.2.11
IP Address=192.168.2.11
IP Address=192.168.2.12
IP Address=192.168.2.12
IP Address=192.168.2.13
IP Address=192.168.2.13
IP Address=127.0.0.1


[root@node1 bin]# tree 
.
├── etcd
├── etcdctl
├── etcd-scripts
│   └── make-ssl-etcd.sh
├── kubectl
├── kubelet
└── kubernetes-scripts
    ├── kube-gen-token.sh
    └── make-ssl.sh