centos7.2 安裝k8s v1.11.0

時間 2019-11-10

標籤 centos7.2 centos 安裝 k8s v1.11.0 欄目 CentOS 简体版

原文原文鏈接

前言

最近因爲公司業務發展到了瓶頸，原有的技術架構已經逐漸沒法知足業務開發和測試的需求，出現了應用測試環境搭建複雜，有許多套（真的不少不少）應用環境，應用在持續集成/持續交付也遇到了很大的困難，通過討論研究決定對應用和微服務進行容器化，這就是我首次直面docker和k8s的契機（好吧，我是菜鳥）

Kubernetes 介紹

Kubernetes 是 Google 團隊發起的開源項目，它的目標是管理跨多個主機的容器，提供基本的部署，維護以及運用伸縮，主要實現語言爲
Go 語言。
Kubernetes的特色：node

易學：輕量級，簡單，容易理解

便攜：支持公有云，私有云，混合雲，以及多種雲平臺

可拓展：模塊化，可插拔，支持鉤子，可任意組合

自修復：自動重調度，自動重啓，自動複製

好吧，這是從別人ppt上拷下來的=。=
下面就正式開始部署咱們本身的k8s吧linux

準備工做

注：如下操做都是在root權限下執行的docker

安裝docker-ce，這裏使用docker-ce-17.09.0.c版本，安裝方法見以前的教程
安裝Kubeadm
安裝 Kubeadm 首先咱們要配置好阿里雲的國內源，執行以下命令：json
```
cat <<EOF > /etc/yum.repos.d/kubernetes.repo 
[kubernetes] 
name=Kubernetes 
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 
enabled=1 
gpgcheck=0 
EOF
```
以後，執行如下命令來重建yum緩存：bootstrap
```
yum -y install epel-releaseyum 
clean all 
yum makecache
```
接下來須要安裝指定版本的Kubeadm（這裏要安裝指定版本，由於後續依賴的鏡像因爲有牆沒法拉取，這裏咱們只有指定版本的鏡像），注意：這裏是安裝指定版本的Kubeadm，k8s的版本更新之快徹底超出你的想象！segmentfault
```
yum -y install kubelet-1.11.0-0
yum -y install kubeadm-1.11.0-0
yum -y install kubectl-1.11.0-0
yum -y install kubernetes-cni
```
執行命令啓動Kubeadm服務：centos
```
systemctl enable kubelet && systemctl start kubelet
```

配置 Kubeadm 所用到的鏡像
這裏是重中之重，由於在國內的緣由，沒法訪問到 Google 的鏡像庫，因此咱們須要執行如下腳原本從 Docker Hub 倉庫中獲取相同的鏡像，而且更改 TAG 讓其變成與 Google 拉去鏡像一致。
新建一個 Shell 腳本，填入如下代碼以後保存。api

#docker.sh
#!/bin/bash
images=(kube-proxy-amd64:v1.11.0 kube-scheduler-amd64:v1.11.0 kube-controller-manager-amd64:v1.11.0 kube-apiserver-amd64:v1.11.0 etcd-amd64:3.2.18 coredns:1.1.3 pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3 k8s-dns-sidecar-amd64:1.14.9 k8s-dns-kube-dns-amd64:1.14.9 k8s-dns-dnsmasq-nanny-amd64:1.14.9 ) 
for imageName in ${images[@]} ; do
docker pull keveon/$imageName
docker tag keveon/$imageName k8s.gcr.io/$imageName
docker rmi keveon/$imageName
done
# 我的新加的一句，V 1.11.0 必加
docker tag da86e6ba6ca1 k8s.gcr.io/pause:3.1

保存後使用chmod命令賦予腳本執行權限緩存

chmod -R 777 ./docker.sh

執行腳本拉取鏡像bash

sh docker.sh
#這裏就開始了漫長的拉取鏡像之路

關閉swap

sudo swapoff -a
#要永久禁掉swap分區，打開以下文件註釋掉swap那一行 
# sudo vi /etc/stab

關閉selinux

# 臨時禁用selinux
# 永久關閉 修改/etc/sysconfig/selinux文件設置
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux 
# 這裏按回車，下面是第二條命令
setenforce 0

關閉防火牆

systemctl disable firewalld.service && systemctl stop firewalld.service

配置轉發參數

# 配置轉發相關參數，不然可能會出錯 
cat <<EOF > /etc/sysctl.d/k8s.conf 
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF 
# 這裏按回車，下面是第二條命令 
sysctl --system

這裏就完成了k8s集羣搭建的準備工做，集羣搭建的話以上操做結束後將操做完的系統製做成系統鏡像，方便集羣搭建

正式安裝

如下的操做都只在主節點上進行：

正式開始安裝k8s
初始化鏡像，執行如下命令：

kubeadm init --kubernetes-version=v1.11.0 #這裏是以前所安裝K8S的版本號 --pod-network-cidr=10.10.0.0/16  #這裏填寫集羣所在網段

以後的輸出會是這樣：

I0712 10:46:30.938979   13461 feature_gate.go:230] feature gates: &{map[]}
[init] using Kubernetes version: v1.11.0
[preflight] running pre-flight checks
I0712 10:46:30.961005   13461 kernel_validator.go:81] Validating kernel version
I0712 10:46:30.961061   13461 kernel_validator.go:96] Validating kernel config
    [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.03.1-ce. Max validated version: 17.03
    [WARNING Hostname]: hostname "g2-apigateway" could not be reached
    [WARNING Hostname]: hostname "g2-apigateway" lookup g2-apigateway on 100.100.2.138:53: no such host
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [g2-apigateway kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.8.62]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [g2-apigateway localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [g2-apigateway localhost] and IPs [172.16.8.62 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" 
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 41.001672 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node g2-apigateway as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node g2-apigateway as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "g2-apigateway" as an annotation
[bootstraptoken] using token: o337m9.ceq32wg9g2gro7gx
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

kubeadm join 10.10.207.253:6443 --token t69z6h.lr2etdbg9mfx5r15 --discovery-token-ca-cert-hash sha256:90e3a748c0eb4cb7058f3d0ee8870ee5d746214ab0589b5e841fd5d68fec8f00

這裏注意最後一行：

kubeadm join 10.10.207.253:6443 --token t69z6h.lr2etdbg9mfx5r15 --discovery-token-ca-cert-hash sha256:90e3a748c0eb4cb7058f3d0ee8870ee5d746214ab0589b5e841fd5d68fec8f00

證實集羣主節點安裝成功，這裏要記得保存這條命令，以便以後各個節點加入集羣

配置kubetl認證信息

export KUBECONFIG=/etc/kubernetes/admin.conf 
# 若是你想持久化的話，直接執行如下命令【推薦】
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

安裝flanel網絡
依次執行如下命令：

mkdir -p /etc/cni/net.d/

cat <<EOF> /etc/cni/net.d/10-flannel.conf 
{ 
「name」: 「cbr0」,
「type」: 「flannel」,
「delegate」: { 
「isDefaultGateway」: true
} 
} 
EOF

mkdir /usr/share/oci-umount/oci-umount.d -p

mkdir /run/flannel/

cat <<EOF> /run/flannel/subnet.env 
FLANNEL_NETWORK=10.244.0.0/16 
FLANNEL_SUBNET=10.244.1.0/24 
FLANNEL_MTU=1450 
FLANNEL_IPMASQ=true 
EOF

最後須要新建一個flannel.yml文件：

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.10.0.0/16",    #這裏換成集羣所在的網段
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.9.1-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conf
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.9.1-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

執行：

kubectl create -f ./flannel.yml

默認狀況下，master節點不參與工做負載，但若是但願安裝出一個all-in-one的k8s環境，則能夠執行如下命令，讓master節點成爲一個node節點：

kubectl taint nodes --all node-role.kubernetes.io/master-

執行以後，運行如下命令，查看節點信息：

kubectl get nodes

會看到以下的輸出：

NAME            STATUS     ROLES     AGE       VERSION
k8s-master      Ready      master    18h       v1.11.0

如下是節點配置

在配置好主節點以後，就能夠配置集羣的其餘節點了，這裏建議直接安裝以前作好準備工做的系統鏡像
進入節點機器以後，直接執行以前保存好的命令

kubeadm join 10.10.207.253:6443 --token t69z6h.lr2etdbg9mfx5r15 --discovery-token-ca-cert-hash sha256:90e3a748c0eb4cb7058f3d0ee8870ee5d746214ab0589b5e841fd5d68fec8f00

執行完後會看到：

[preflight] running pre-flight checks
        [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr] or no builtin kernel ipvs support: map[ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0725 09:59:27.929247   10196 kernel_validator.go:81] Validating kernel version
I0725 09:59:27.929356   10196 kernel_validator.go:96] Validating kernel config
[discovery] Trying to connect to API Server "10.10.207.253:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.10.207.253:6443"
[discovery] Requesting info from "https://10.10.207.253:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.10.207.253:6443"
[discovery] Successfully established connection with API Server "10.10.207.253:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-node1" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

這裏就表示執行完畢了，能夠去主節點執行命令：

kubectl get nodes

能夠看到節點已加入集羣：

NAME        STATUS    ROLES     AGE       VERSION
k8s-master  Ready     master    20h       v1.11.0
k8s-node1   Ready     <none>    20h       v1.11.0
k8s-node2   Ready     <none>    20h       v1.11.0

這期間可能須要等待一段時間，狀態纔會所有變爲ready