使用Kubespray 部署kubernetes 高可用集羣

Requirements

• Ansible v2.4 及以上版本，安裝python-netaddr到運行Ansible commands的機器
• Jinja 2.9 及以上版本，運行Ansible Playbooks
• 目標servers 必須能夠訪問外網，能夠pull docker images
• 目標servers 配置容許 IPv4 forwarding
• 將公鑰複製到全部機器
• 關閉防火牆
• 請提早安裝好docker,由於k8s不支持最新版docker,具體適配哪些docker版本，請看k8s上的changelog

0、環境

主機名	IP
master1	172.16.105.21
master2	172.16.105.22
master3	172.16.105.23
node1	172.16.105.24
node2	172.16.105.25
ansible-client	172.16.105.20

一、安裝ansible和依賴

在172.16.105.20安裝ansible

# 安裝 python 及 epel
yum install -y epel-release python-pip python34 python34-pip
# 安裝 ansible
yum install -y ansible
pip install netaddr
pip install --upgrade jinja2

二、創建公私鑰，分發各服務器

在ansible-clinet機器生成免密密鑰對

ssh-keygen -t rsa -P ''

將生成的公鑰(id_rsa.pub)傳到其餘節點，這樣ansible-client能夠免密登錄其餘機器

cat id_rsa.pub >> ~/.ssh/authorized_keys

三、下載kuberspay源碼

cd /usr/local/src/
wget https://github.com/kubernetes-incubator/kubespray/archive/v2.3.0.tar.gz

本版本所包含的組件版本

• Kubernetes v1.8.1
• Docker 1.13.1
• etcd v3.2.4
• Rkt v1.21.0 (optional)
• Calico v2.5.0
• Weave 2.0.4
• Flannel v0.8.0

3.1 禁用docker yum倉和docker安裝

vim roles/docker/tasks/main.yml

---
- name: gather os specific variables
  include_vars: "{{ item }}"
  with_first_found:
    - files:
        - "{{ ansible_distribution|lower }}-{{ ansible_distribution_version|lower|replace('/', '_') }}.yml"
        - "{{ ansible_distribution|lower }}-{{ ansible_distribution_release }}.yml"
        - "{{ ansible_distribution|lower }}-{{ ansible_distribution_major_version|lower|replace('/', '_') }}.yml"
        - "{{ ansible_distribution|lower }}.yml"
        - "{{ ansible_os_family|lower }}.yml"
        - defaults.yml
      paths:
        - ../vars
      skip: true
  tags:
    - facts

- include: set_facts_dns.yml
  when: dns_mode != 'none' and resolvconf_mode == 'docker_dns'
  tags:
    - facts

- name: check for minimum kernel version
  fail:
    msg: >
          docker requires a minimum kernel version of
          {{ docker_kernel_min_version }} on
          {{ ansible_distribution }}-{{ ansible_distribution_version }}
  when: (not ansible_os_family in ["CoreOS", "Container Linux by CoreOS"]) and (ansible_kernel|version_compare(docker_kernel_min_version, "<"))
  tags:
    - facts

#禁用docker倉庫，已經使用清華源
#- name: ensure docker repository public key is installed
#  action: "{{ docker_repo_key_info.pkg_key }}"
#  args:
#    id: "{{item}}"
#    keyserver: "{{docker_repo_key_info.keyserver}}"
#    state: present
#  register: keyserver_task_result
#  until: keyserver_task_result|succeeded
#  retries: 4
#  delay: "{{ retry_stagger | random + 3 }}"
#  environment: "{{ proxy_env }}"
#  with_items: "{{ docker_repo_key_info.repo_keys }}"
#  when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic)

#- name: ensure docker repository is enabled
#  action: "{{ docker_repo_info.pkg_repo }}"
#  args:
#    repo: "{{item}}"
#    state: present
#  with_items: "{{ docker_repo_info.repos }}"
#  when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic) and(docker_repo_info.repos|length > 0)

#- name: Configure docker repository on RedHat/CentOS
#  template:
#    src: "rh_docker.repo.j2"
#    dest: "/etc/yum.repos.d/docker.repo"
#  when: ansible_distribution in ["CentOS","RedHat"] and not is_atomic

#- name: ensure docker packages are installed
#  action: "{{ docker_package_info.pkg_mgr }}"
#  args:
#    pkg: "{{item.name}}"
#    force: "{{item.force|default(omit)}}"
#    state: present
#  register: docker_task_result
#  until: docker_task_result|succeeded
#  retries: 4
#  delay: "{{ retry_stagger | random + 3 }}"
#  environment: "{{ proxy_env }}"
#  with_items: "{{ docker_package_info.pkgs }}"
#  notify: restart docker
#  when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic) and (docker_package_info.pkgs|length > 0)

#對於docker的版本檢測進行了保留

- name: check minimum docker version for docker_dns mode. You need at least docker version >= 1.12 for resolvconf_mode=docker_dns
  command: "docker version -f '{{ '{{' }}.Client.Version{{ '}}' }}'"
  register: docker_version
  failed_when: docker_version.stdout|version_compare('1.12', '<')
  changed_when: false
  when: dns_mode != 'none' and resolvconf_mode == 'docker_dns'

#對於docker的systemd配置，能夠根據本身需求修改，可是注意會覆蓋原來的
- name: Set docker systemd config
  include: systemd.yml

- name: ensure docker service is started and enabled
  service:
    name: "{{ item }}"
    enabled: yes
    state: started
  with_items:
    - docker

四、替換鏡像

由於長城的緣由，須要的鏡像在安裝的時候沒法獲取，因此須要改下源碼，下載本身私有倉的鏡像
腳本內容以下：

gcr_image_files=(
./kubespray/roles/download/defaults/main.yml
./kubespray/roles/dnsmasq/templates/dnsmasq-autoscaler.yml.j2
./kubespray/roles/kubernetes-apps/ansible/defaults/main.yml
)

for file in ${gcr_image_files[@]} ; do
    sed -i 's/gcr.io/docker.emarbox.com/g' $file
done

鏡像列表，最好提早下載到私有倉,畢竟下載會很慢

gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5
gcr.io/google_containers/pause-amd64:3.0
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3

nginx:1.11.4-alpine
busybox:latest

quay.io/coreos/hyperkube:v1.8.1_coreos.0
quay.io/coreos/etcd:v3.2.4
quay.io/calico/ctl:v1.5.0
quay.io/calico/node:v2.5.0
quay.io/calico/routereflector:v0.4.0
quay.io/calico/cni:v1.10.0

五、配置文件內容

能夠對auth的密碼進行修改，網絡插件默認calico，可替換成weave或flannel，還能夠配置是否安裝helm和efk,以及修改安裝路徑

more kubespray/kubespray-2.3.0/inventory/group_vars/k8s-cluster.yml

#六、生成本身的集羣配置

由於kubespray自帶的python腳本是Python3 ，因此要安裝Python3

yum install -y python-pip python34 python34-pip

# 定義集羣IP
IP=(
172.16.105.21
172.16.105.22
172.16.105.23
)

# 利用kubespray自帶的python腳本生成配置
CONFIG_FILE=./kubespray/inventory/inventory.cfg python3 ./kubespray/contrib/inventory_builder/inventory.py ${IP[*]}

查看配置

cat ./kubespray/inventory/inventory.cfg

[all]
node1    ansible_host=172.16.105.21 ip=172.16.105.21
node2    ansible_host=172.16.105.22 ip=172.16.105.22
node3    ansible_host=172.16.105.23 ip=172.16.105.23

[kube-master]
node1    
node2    
node3

[kube-node]
node1    
node2    
node3    

[etcd]
node1    
node2    
node3    

[k8s-cluster:children]
kube-node        
kube-master      

[calico-rr]

[vault]
node1    
node2    
node3

七、安裝集羣

ansible-playbook -i inventory/inventory.cfg cluster.yml -b -v

鏡像地址
kubespray/roles/download/tasks/download_container.yml

八、問題

8.1

從1.8 開始，kubelet 會檢測機器是否有swap,若是啓用swap，kubelet會沒法啓動，須要手動添加參數。
去以下目錄，修改kubelet參數

/usr/local/src/kubespray/kubespray-2.3.0/roles/kubernetes/node/defaults

### fail with swap on (default true)
kubelet_fail_swap_on: false

8.2

注意機器主機名，要符合k8s的規範

九、安裝失敗如何清理

rm -rf /etc/kubernetes/
rm -rf /var/lib/kubelet
rm -rf /var/lib/etcd
rm -rf /usr/local/bin/kubectl
rm -rf /etc/systemd/system/calico-node.service
rm -rf /etc/systemd/system/kubelet.service
systemctl stop etcd.service
systemctl disable etcd.service
systemctl stop calico-node.service
systemctl disable calico-node.service
docker stop $(docker ps -q)
docker rm $(docker ps -a -q)
systemctl restart docker

十、安裝完成

[root@node2 .kube]# kubectl get nodes
NAME      STATUS    ROLES         AGE       VERSION
node1     Ready     master,node   9m        v1.8.1+coreos.0
node2     Ready     master,node   9m        v1.8.1+coreos.0
node3     Ready     master,node   9m        v1.8.1+coreos.0

十一、擴展集羣node

把須要添加的node 寫入配置文件，而後執行ansible
以添加node4舉例
修改 inventory.cfg

[all]
node1    ansible_host=172.16.105.21 ip=172.16.105.21
node2    ansible_host=172.16.105.22 ip=172.16.105.22
node3    ansible_host=172.16.105.23 ip=172.16.105.23
node4    ansible_host=172.16.105.37 ip=172.16.105.37

[kube-master]
node1    
node2    
node3

[kube-node]
node1    
node2    
node3
node4    

[etcd]
node1    
node2    
node3    

[k8s-cluster:children]
kube-node        
kube-master      

[calico-rr]

[vault]
node1    
node2    
node3

ansible-playbook -i inventory/inventory.cfg scale.yml -b -v \
  --private-key=~/.ssh/private_key

後感

瞭解ansible，也就能本身隨便搞kubespray,這個東西比較透明，不像kubeadm,封裝很嚴，不知道具體流程，出問題也不知道怎麼處理。