k8s單機部署1.11.5

時間 2019-12-01

標籤 k8s 單機部署 1.11.5 简体版

原文原文鏈接

1、概述

因爲服務器有限，所以只能用虛擬機搭建 k8s。可是開3個節點，電腦卡的不行。html

k8s中文社區封裝了一個 Minikube，用來搭建單機版，連接以下：node

https://yq.aliyun.com/articles/221687python

我測試過了，是能夠運行。可是，沒有calico網絡插件。根據其餘文檔來部署calico，始終沒法使用！linux

所以，靠人不如靠本身。本身搭建的，內心有數！git

2、環境依賴

環境說明

配置	操做系統	主機名	IP地址	功能
2核4G 40G磁盤	ubuntu-16.04.5-server-amd64	ubuntu	192.168.91.128	主節點，etcd，docker registry

請確保核心至少有2核，內存不要過小！github

本文所使用的操做系統爲：ubuntu:16.04，注意：切勿使用18.04，會致使安裝失敗！docker

k8s依賴於etcd和docker私有庫，下面會一一介紹。json

文件準備

k8s-1.11.5 目錄下載連接以下：flask

連接：https://pan.baidu.com/s/152ECxRuq3aX4HQco4GH6cA
提取碼：bz5h

ubuntu

目錄結構以下：

./
├── calico-cni-1.11.8.tar.gz
├── calico-kube-controllers-1.0.5.tar.gz
├── calico-node-2.6.12.tar.gz
├── calico.yaml
├── coredns-1.1.3.tar.gz
├── cri-tools_1.12.0-00_amd64.deb
├── kubeadm_1.11.5-00_amd64.deb
├── kube-apiserver-amd64-v1.11.5.tar.gz
├── kube-controller-manager-amd64-v1.11.5.tar.gz
├── kubectl_1.11.5-00_amd64.deb
├── kubelet_1.11.5-00_amd64.deb
├── kube-proxy-amd64-v1.11.5.tar.gz
├── kubernetes-cni_0.6.0-00_amd64.deb
├── kube-scheduler-amd64-v1.11.5.tar.gz
└── pause3.1.tar.gz

必須將 k8s-1.11.5 目錄放置到 /repo/k8s-1.11.5 目錄下

etcd

etcd 目錄結構以下：

./
├── etcd-v3.3.10-linux-amd64.tar.gz
└── etcd_v3.3.10.sh

etcd_v3.3.10.sh

#/bin/bash
# 單擊版etcd安裝腳本
# 本腳本,只能在本地服務器安裝。
# 請確保etcd-v3.3.10-linux-amd64.tar.gz文件和當前腳本在同一目錄下。
# 務必使用root用戶執行此腳本!
# 確保能夠直接執行python3,由於倒數第4行,有一個json格式化輸出。若是不須要能夠忽略
#set -e

# 輸入本機ip
while true
do
    echo '請輸入本機ip'
    echo 'Example: 192.168.0.1'
    echo -e "etcd server ip=\c"
    read ETCD_Server
    if [ "$ETCD_Server" == "" ];then
        echo 'No input etcd server IP'
    else
            #echo 'No input etcd server IP'
            break
    fi
done

# etcd啓動服務
cat > /lib/systemd/system/etcd.service <<EOF
[Unit]
Description=etcd - highly-available key value store
Documentation=https://github.com/coreos/etcd
Documentation=man:etcd
After=network.target
Wants=network-online.target

[Service]
Environment=DAEMON_ARGS=
Environment=ETCD_NAME=%H
Environment=ETCD_DATA_DIR=/var/lib/etcd/default
EnvironmentFile=-/etc/default/%p
Type=notify
User=etcd
PermissionsStartOnly=true
#ExecStart=/bin/sh -c "GOMAXPROCS=\$(nproc) /usr/bin/etcd \$DAEMON_ARGS"
ExecStart=/usr/bin/etcd \$DAEMON_ARGS
Restart=on-abnormal
#RestartSec=10s
#LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF


# 主機名
name=`hostname`
# etcd的http鏈接地址
initial_cluster="http://$ETCD_Server:2380"


# 判斷進程是否啓動
A=`ps -ef|grep /usr/bin/etcd|grep -v grep|wc -l`
if [ $A -ne 0 ];then
    # 殺掉進程
    killall etcd
fi 

# 刪除etcd相關文件
rm -rf /var/lib/etcd/*
rm -rf /etc/default/etcd

# 設置時區
ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

# 判斷壓縮文件
if [ ! -f "etcd-v3.3.10-linux-amd64.tar.gz" ];then
    echo "當前目錄etcd-v3.3.10-linux-amd64.tar.gz文件不存在"
    exit
fi

# 安裝etcd
tar zxf etcd-v3.3.10-linux-amd64.tar.gz -C /tmp/
cp -f /tmp/etcd-v3.3.10-linux-amd64/etcd /usr/bin/
cp -f /tmp/etcd-v3.3.10-linux-amd64/etcdctl /usr/bin/


# etcd配置文件
cat > /etc/default/etcd <<EOF
ETCD_NAME=$name
ETCD_DATA_DIR="/var/lib/etcd/"
ETCD_LISTEN_PEER_URLS="http://$ETCD_Server:2380"
ETCD_LISTEN_CLIENT_URLS="http://$ETCD_Server:2379,http://127.0.0.1:4001"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://$ETCD_Server:2380"
ETCD_INITIAL_CLUSTER="$ETCD_Servernitial_cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-sdn"
ETCD_ADVERTISE_CLIENT_URLS="http://$ETCD_Server:2379"
EOF


# 臨時腳本,添加用戶和組
cat  > /tmp/foruser <<EOF
#!/bin/bash
if [ \`cat /etc/group|grep etcd|wc -l\` -eq 0 ];then groupadd -g 217 etcd;fi
if [ \`cat /etc/passwd|grep etcd|wc -l\` -eq 0 ];then mkdir -p /var/lib/etcd && useradd -g 217 -u 111 etcd -d /var/lib/etcd/ -s /bin/false;fi
if [ \`cat /etc/profile|grep ETCDCTL_API|wc -l\` -eq 0 ];then bash -c "echo 'export ETCDCTL_API=3' >> /etc/profile" && bash -c "source /etc/profile";fi
EOF

# 執行腳本
bash /tmp/foruser

# 啓動服務
systemctl daemon-reload
systemctl enable etcd.service
chown -R etcd:etcd /var/lib/etcd
systemctl restart etcd.service
#netstat -anpt | grep 2379
# 查看版本
etcdctl -v
# 訪問API, -s 去掉curl的統計信息. python3 -m json.tool 表示json格式化
curl $initial_cluster/version -s | python3 -m json.tool

# 刪除臨時文件
rm -rf /tmp/foruser /tmp/etcd-v3.3.10-linux-amd64

View Code

etcd目錄能夠任意放置，這裏我直接放到/root目錄中

執行腳本

bash etcd_v3.3.10.sh

腳本執行輸出：

請輸入本機ip
Example: 192.168.0.1
etcd server ip=192.168.0.162
etcdctl version: 3.3.10
API version: 2
{
    "etcdserver": "3.3.10",
    "etcdcluster": "3.3.0"
}

docker registry

安裝docker

apt-get install -y docker.io

注意：千萬不要安裝 docker-ce，不然安裝k8s報錯

[preflight] Some fatal errors occurred:
    [ERROR SystemVerification]: unsupported docker version: 18.09.0
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

由於k8s不支持最新版本的docker

啓動docker registry

docker run -d --name docker-registry --restart=always -p 5000:5000 registry

3、一鍵安裝

在root目錄下新建腳本

kube-v1.11.5.sh

#/bin/bash
# k8s 1.11.5版本安裝,單機版
# 基於ubuntu 16.04

REPO=192.168.0.162
K8S_MASTER_IP=$REPO
dockerREG=$REPO:5000
EXTERNAL_ETCD_ENDPOINTS=http://$REPO:2379

curl $EXTERNAL_ETCD_ENDPOINTS/version
if [ `echo $?` != 0 ];then
    echo "etcd沒有運行,請執行腳本"
    echo "bash etcd_v3.3.10.sh"
    exit
fi

curl http://$dockerREG/v2/_catalog
if [ `echo $?` != 0 ];then
    echo "docker私有倉庫沒有運行,請執行命令："
    echo "docker run -d --name docker-registry --restart=always -p 5000:5000 registry"
    exit
fi

if [ ! -d "/repo/k8s-1.11.5/" ];then
    echo "/repo/k8s-1.11.5目錄不存在,請上傳文件夾到指定位置!"
    exit
fi

######################################################################################
#安裝相應的組件
sudo apt-get update
sudo apt-get install -y ipvsadm --allow-unauthenticated
sudo apt-get install -y ebtables socat --allow-unauthenticated
Dline=`sudo grep -n LimitNOFILE /lib/systemd/system/docker.service|cut -f 1 -d ":" `
sudo sed -i "$Dline c\LimitNOFILE=1048576" /lib/systemd/system/docker.service 

# 配置docker
cat > daemon.json <<EOF
{ "insecure-registries":["$dockerREG"],"storage-driver":"overlay2" }
EOF

# 重啓docker
sudo cp -f daemon.json /etc/docker/
sudo service docker restart

#安裝kubernetes 服務器
if [ `dpkg -l|grep kube|wc -l` -ne 0 ];then
    sudo apt purge -y `dpkg -l|grep kube|awk '{print $2}'`
fi
#sudo apt-get install -y kubelet kubeadm --allow-unauthenticated
dpkg -i /repo/k8s-1.11.5/*.deb
#開啓cadvisor
sudo sed -i 's?config.yaml?config.yaml --cadvisor-port=4194 --eviction-hard=memory.available<512Mi,nodefs.available<13Gi,imagefs.available<100Mi?g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
#add cgroup drivers
if [ `sudo cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf|grep cgroup-driver|wc -l` -eq 0 ];then
    sudo sed -i 8i'Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
    sudo sed -i "s?`sudo tail -n1 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf`?& $KUBELET_CGROUP_ARGS?g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
fi
sudo systemctl daemon-reload
IMAGES="calico-cni-1.11.8.tar.gz
calico-kube-controllers-1.0.5.tar.gz
calico-node-2.6.12.tar.gz
coredns-1.1.3.tar.gz
kube-apiserver-amd64-v1.11.5.tar.gz
kube-scheduler-amd64-v1.11.5.tar.gz
kube-controller-manager-amd64-v1.11.5.tar.gz
kube-proxy-amd64-v1.11.5.tar.gz
pause3.1.tar.gz"
for i in $IMAGES;do
        #wget -O /tmp/$i http://$repo/k8s-1.11.5/$i
        sudo docker load</repo/k8s-1.11.5/$i
done

CalicoPro=$(sudo docker images|grep quay.io|awk -F 'quay.io/' '{print $2}'|awk '{print $1}'|sort|uniq)
for i in $CalicoPro;do
    Proversion=$(sudo docker images|grep quay.io|grep $i|awk '{print $2}')
    for j in $Proversion;do
    sudo docker tag quay.io/$i:$j $dockerREG/$i:$j
    sudo docker push $dockerREG/$i:$j
    done
done

sudo systemctl enable kubelet.service
sudo systemctl stop kubelet.service

#close swap
sudo swapoff -a
fswap=`cat /etc/fstab |grep swap|awk '{print $1}'`
for i in $fswap;do
    sudo sed -i "s?$i?#$i?g" /etc/fstab
done

######################################################################################

sudo kubeadm reset -f
sudo rm -rf /etc/kubernetes/*
sudo rm -rf /var/lib/kubelet/*

sudo rm -f /tmp/kubeadm-conf.yaml
cat > /tmp/kubeadm-conf.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
networking:
  podSubnet: 192.138.0.0/16
#apiServerCertSANs:
#- master01
#- master02
#- master03
#- 172.16.2.1
#- 172.16.2.2
#- 172.16.2.3
#- 172.16.2.100
etcd:
  endpoints:
#token: 67e411.zc3617bb21ad7ee3
kubernetesVersion: v1.11.5
#imageRepository: $dockerREG
api:
  advertiseAddress: $K8S_MASTER_IP
EOF


if [ -z "$EXTERNAL_ETCD_ENDPOINTS" ]; then
    sudo sed -i 's?etcd:?#etcd?g' /tmp/kubeadm-conf.yaml
    sudo sed -i 's?endpoints:?#endpoints:?g' /tmp/kubeadm-conf.yaml
    sudo kubeadm init --config=/tmp/kubeadm-conf.yaml | sudo tee /etc/kube-server-key
else
    for i in `echo $EXTERNAL_ETCD_ENDPOINTS|sed 's?,? ?g'`;do
        sudo sed -i "15i\  - $i" /tmp/kubeadm-conf.yaml    
    done
    sudo kubeadm init --config=/tmp/kubeadm-conf.yaml| sudo tee /etc/kube-server-key
fi


#添加Nodrport端口範圍"--service-node-port-range=1000-62000"到/etc/kubernetes/manifests/kube-apiserver.yaml
if [ `sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml|grep service-node-port-range|wc -l` -eq 0 ];then
    line_conf=$(sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml|grep -n "allow-privileged=true"|cut -f 1 -d ":")
    sudo sed -i -e "$line_conf"i'\    - --service-node-port-range=1000-62000' /etc/kubernetes/manifests/kube-apiserver.yaml
    line_conf=$(sudo grep -n "insecure-port" /etc/kubernetes/manifests/kube-apiserver.yaml|awk -F ":" '{print $1}')
    sudo sed -i -e "$line_conf"i"\    - --insecure-bind-address=$K8S_MASTER_IP" /etc/kubernetes/manifests/kube-apiserver.yaml
    sudo sed -i -e 's?insecure-port=0?insecure-port=8080?g' /etc/kubernetes/manifests/kube-apiserver.yaml
fi

#設置kubectl權限
mkdir -p $HOME/.kube
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

######################################################################################
#                                 Deploy etcd as a master pod
######################################################################################
# We need: (1) modify it to listen on K8S_MASTER_IP so that minion nodes can access.
# (2) modify kube-apiserver configuration.
# (3) restart the service kubelet.
if [ -z "$EXTERNAL_ETCD_ENDPOINTS" ]; then
    sudo sed -i -e "s/127.0.0.1/$K8S_MASTER_IP/g" /etc/kubernetes/manifests/etcd.yaml
    sudo sed -i "s?127.0.0.1?$K8S_MASTER_IP?g" /etc/kubernetes/manifests/kube-apiserver.yaml

    sudo service docker restart
    sudo service kubelet restart
fi
######################################################################################



######################################################################################
#                                 Deploy calico network plugin
######################################################################################

cp /repo/k8s-1.11.5/calico.yaml /tmp/calico.yaml
line=`grep "etcd_endpoints:" -n /tmp/calico.yaml | cut -f 1 -d ":"`
if [ -z "$EXTERNAL_ETCD_ENDPOINTS" ]; then
    sudo sed -i "$line c \ \ etcd_endpoints: \"http://$K8S_MASTER_IP:2379\"" /tmp/calico.yaml
else
    # TODO: multiple instances
    sudo sed -i "$line c \ \ etcd_endpoints: \"$EXTERNAL_ETCD_ENDPOINTS\"" /tmp/calico.yaml
fi

# Change CIDR
PODNET=$(cat /tmp/kubeadm-conf.yaml|grep podSubnet|awk '{print $NF}'|cut -d "/" -f1)
sudo sed -i -e "s/192.168.0.0/$PODNET/g" /tmp/calico.yaml
sudo sed -i -e "s?quay.io?$dockerREG?g" /tmp/calico.yaml
sudo sed -i -e "s?192.168.0.179?$K8S_MASTER_IP?g" /tmp/calico.yaml

# Wait for everything is up except kube-dns, which needs a network plugin
sudo sleep 1m
mkdir -p k8s-1.11.5
cp /tmp/calico.yaml k8s-1.11.5/
sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f k8s-1.11.5/calico.yaml

sudo sleep 10
# 默認狀況下，爲了保證 master 的安全，master 是不會被調度到 app 的。你能夠取消這個限制經過輸入
sudo kubectl taint nodes --all node-role.kubernetes.io/master-

# 查看集羣中的節點
sudo kubectl get nodes

sudo sleep 20
# 查看k8s服務狀態
kubectl get po -n kube-system

View Code

注意：運行以前，請修改腳本最開頭部分的IP地址，爲本身的本機IP

運行腳本

bash kube-v1.11.5.sh

執行輸出：

...
NAME      STATUS     ROLES     AGE       VERSION
ubuntu    NotReady   master    1m        v1.11.5
NAME                                       READY     STATUS    RESTARTS   AGE
calico-kube-controllers-55445fbcb6-8znzw   1/1       Running   0          25s
calico-node-49w2t                          2/2       Running   0          25s
coredns-78fcdf6894-9t8wd                   1/1       Running   0          25s
coredns-78fcdf6894-ztxm6                   1/1       Running   0          25s
kube-apiserver-ubuntu                      1/1       Running   0          59s
kube-controller-manager-ubuntu             1/1       Running   1          59s
kube-proxy-6xw6j                           1/1       Running   0          25s
kube-scheduler-ubuntu                      1/1       Running   1          59s

4、驗證

flask.yaml

新建yaml文件 flask.yaml

apiVersion: extensions/v1beta1
kind: Deployment 
metadata: 
  name: flaskapp-1
spec: 
  replicas: 1
  template: 
    metadata: 
      labels: 
        name: flaskapp-1
    spec: 
      containers: 
        - name: flaskapp-1
          image: jcdemo/flaskapp
          ports:
          - containerPort: 5000

---
apiVersion: v1
kind: Service
metadata:
  name: flaskapp-1
  labels:
    name: flaskapp-1
spec:
  type: NodePort
  ports:
  - port: 5000
    name: flaskapp-port
    targetPort: 5000
    protocol: TCP
    nodePort: 5000
  selector:
    name: flaskapp-1

View Code

建立pod

kubectl create -f flask.yaml --validate

查看Pod狀態

root@ubuntu:~# kubectl get pods -o wide
NAME                          READY     STATUS    RESTARTS   AGE       IP                NODE      NOMINAT
flaskapp-1-84b7f79cdf-dgctk   1/1       Running   0          6m        192.138.243.195   ubuntu    <none>

訪問頁面

root@ubuntu:~# curl http://192.138.243.195:5000
<html><head><title>Docker + Flask Demo</title></head><body><table><tr><td> Start Time </td> <td>2019-Feb-01 10:05:34</td> </tr><tr><td> Hostname </td> <td>flaskapp-1-84b7f79cdf-dgctk</td> </tr><tr><td> Local Address </td> <td>192.138.243.195</td> </tr><tr><td> Remote Address </td> <td>192.168.0.162</td> </tr><tr><td> Server Hit </td> <td>1</td> </tr></table></body></html>

有輸出 Hostname 表示成功了

我想直接使用windows 10訪問flask，可不可呢？

不能夠！爲何？由於個人電腦不能直接訪問 192.138.0.0/16 的網絡。這個是k8s的pod 網絡。

添加路由

首先須要在windows 10 中添加路由，好比：k8s主控端的地址爲 192.168.0.102

確保cmd打開時，以管理員身份運行，不然提示沒有權限！

在cmd中輸入以下命令：

route add -p 192.138.0.0 MASK 255.255.0.0 192.168.0.162

設置nat規則

iptables -t nat -I POSTROUTING -s 192.168.0.0/24 -d 192.138.0.0/16 -o tunl0 -j MASQUERADE

使用windos 10 訪問flask頁面，效果以下：

將iptables規則加入到開機啓動項中

echo 'iptables -t nat -I POSTROUTING -s 192.168.0.0/24 -d 192.138.0.0/16 -o tunl0 -j MASQUERADE' >> /etc/rc.local

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。