k8s集羣部署v1.15實踐1:基礎環境準備

1.基礎環境準備

參考文檔

1.節點基礎環境

系統版本

[root@k8s-node1 ~]# cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)
[root@k8s-node1 ~]#

三個主機:

k8s-node1   192.168.174.128
k8s-node2   192.168.174.129
k8s-node3   192.168.174.130

2.節點系統的基本設置,全部節點都要作

關閉selinux

sed -i 's/'SELINUX=enforcing'/'SELINUX=disabled'/' /etc/selinux/config

關閉防火牆

systemctl stop firewalld.service&&systemctl disable firewalld.service

關閉swap

swapoff -a && sed -i 's/.*swap.*/#&/' /etc/fstab

配置Host解析

echo -e "192.168.174.128 k8s-node1\n192.168.174.129 k8s-node2\n192.168.174.130 k8s-node3" >>/etc/hosts

配置系統參數,容許數據轉發

/etc/sysctl.d/目錄下，新建個k8s.conf
容許路由轉發，不對bridge的數據進行處理
kubernetes1在/etc/sysctl.d/目錄下，新建個k8s.conf
內容以下:

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100

建好k8s.conf後執行
加載內核模塊

modprobe br_netfilter
modprobe ip_vs

再執行

sysctl -p /etc/sysctl.d/k8s.conf

3.配置節點ssh免密鑰登錄

節點1,k8s-node1

[root@k8s-node1 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:2tyWwPZfUpYUBLa1uc06xDNRMMdKJa+uHwM37yQDY5M root@k8s-node1
The key's randomart image is:
+---[RSA 2048]----+
|            ooB++|
|           . o.X.|
|            ..=..|
|       .     +.B |
|        S   E & o|
|       = + o @ * |
|      . o = . X o|
|         . . + O |
|            o.. .|
+----[SHA256]-----+
[root@k8s-node1 ~]#

ssh-copy-id root@k8s-node2
ssh-copy-id root@k8s-node3

節點2,k8s-node2

[root@k8s-node2 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:jvZH/8xJbBTkiBcHWZlphbeN2Chw+e8MmoiNkgb+UlM root@k8s-node2
The key's randomart image is:
+---[RSA 2048]----+
|           .o++*.|
|        . o..** .|
|         o..o=ooo|
|      E   ..+ oo.|
|     .  S  . ..  |
|  . o  o  . .o.  |
| . o oo+.o + ++  |
|  o +.o.o + .=o. |
|   +..  ..   .=  |
+----[SHA256]-----+
[root@k8s-node2 ~]#

ssh-copy-id root@k8s-node1
ssh-copy-id root@k8s-node3

節點3,k8s-node3

[root@k8s-node3 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:1ZamMCf1DhkjrVUIx4OfrJWTxr0sh4+S2U3y43YGeVo root@k8s-node3
The key's randomart image is:
+---[RSA 2048]----+
|        o==o.    |
|        .+** .   |
|        +*=**    |
|        .*@*.    |
|        S+.+.o   |
|        . + B E  |
|         + X =   |
|        + o B o  |
|         . o.+   |
+----[SHA256]-----+
[root@k8s-node3 ~]#

ssh-copy-id root@k8s-node1
ssh-copy-id root@k8s-node2

4.安裝docker

參考前面docker文檔,略

5.添加k8s用戶並加入docker羣組和配置sudo權限,三個節點作一樣的操做

添加用戶k8s並配置密碼爲123456

[root@k8s-node1 ~]# useradd -m k8s
[root@k8s-node1 ~]# sh -c 'echo 123456 | passwd k8s --stdin'
Changing password for user k8s.
passwd: all authentication tokens updated successfully.

加入wheel羣組,這個羣組具備sudo權限

[root@k8s-node1 ~]# gpasswd -a k8s wheel
Adding user k8s to group wheel

加入docker羣組

[root@k8s-node1 ~]# gpasswd -a k8s docker
Adding user k8s to group docker
[root@k8s-node1 ~]#

6.安裝依賴包,三個節點都須要安裝

ipvs依賴ipset

yum install -y epel-release
yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp

7.建立k8s集羣使用的目錄,三個節點作同樣的操做

注意這裏使用的用戶是咱們前面建立好的k8s

[root@k8s-node1 ~]# mkdir -p /opt/k8s/bin
[root@k8s-node1 ~]# chown -R k8s /opt/k8s/
[root@k8s-node1 ~]# mkdir -p /etc/kubernetes/cert
[root@k8s-node1 ~]# chown -R k8s /etc/kubernetes
[root@k8s-node1 ~]# mkdir -p /etc/etcd/cert
[root@k8s-node1 ~]# chown -R k8s /etc/etcd/
[root@k8s-node1 ~]# mkdir -p /var/lib/etcd && chown -R k8s /var/lib/etcd

8.集羣環境變量

#!/usr/bin/bash
# 生成 EncryptionConfig 所需的加密 key
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
# 最好使用 當前未用的網段 來定義服務網段和 Pod 網段
# 服務網段，部署前路由不可達，部署後集羣內路由可達(kube-proxy 和 ipvs 保證)
SERVICE_CIDR="10.254.0.0/16"
# Pod 網段，建議 /16 段地址，部署前路由不可達，部署後集羣內路由可達(flanneld 保證)
CLUSTER_CIDR="172.30.0.0/16"
# 服務端口範圍 (NodePort Range)
export NODE_PORT_RANGE="8400-9000"
# 集羣各機器 IP 數組
export NODE_IPS=(192.168.174.128 192.168.174.129 192.168.174.130)
# 集羣各 IP 對應的 主機名數組
export NODE_NAMES=(k8s-node1 k8s-node2 k8s-node3)
# kube-apiserver 的 VIP（HA 組件 keepalived 發佈的 IP）
export MASTER_VIP=192.168.174.127
# kube-apiserver VIP 地址（HA 組件 haproxy 監聽 8443 端口）
export KUBE_APISERVER="https://${MASTER_VIP}:8443"
# HA 節點，VIP 所在的網絡接口名稱
export VIP_IF="ens33"
# etcd 集羣服務地址列表
export ETCD_ENDPOINTS="https://192.168.174.128:2379,https://192.168.174.129:2379,https://192.168.174.130:2379"
# etcd 集羣間通訊的 IP 和端口
export ETCD_NODES="k8s-node1=https://192.168.174.128:2380,k8s-node2=https://192.168.174.129:2380,k8s-node3=https://192.168.74.130:2380"
# flanneld 網絡配置前綴
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
# kubernetes 服務 IP (通常是 SERVICE_CIDR 中第一個IP)
export CLUSTER_KUBERNETES_SVC_IP="10.254.0.1"
# 集羣 DNS 服務 IP (從 SERVICE_CIDR 中預分配)
export CLUSTER_DNS_SVC_IP="10.254.0.2"
# 集羣 DNS 域名
export CLUSTER_DNS_DOMAIN="cluster.local."
# 將二進制目錄 /opt/k8s/bin 加到 PATH 中
export PATH=/opt/k8s/bin:$PATH

打包後的變量定義見 environment.sh，後續部署時會提示導入該腳本；
把全局變量定義腳本拷貝到全部節點的 /opt/k8s/bin目錄.

[root@k8s-node1 ~]# cp environment.sh /opt/k8s/bin/
[root@k8s-node1 ~]# scp environment.sh root@k8s-node2:/opt/k8s/bin/
environment.sh                                                                                        100% 1749     1.6MB/s   00:00    
[root@k8s-node1 ~]# scp environment.sh root@k8s-node3:/opt/k8s/bin/
environment.sh                                                                                        100% 1749     1.8MB/s   00:00    
[root@k8s-node1 ~]#

給與執行權限

[root@k8s-node1 ~]# chmod +x /opt/k8s/bin/*
[root@k8s-node1 ~]# ssh k8s-node2 "chmod +x /opt/k8s/bin/*"
[root@k8s-node1 ~]# ssh k8s-node3 "chmod +x /opt/k8s/bin/*"

9.可能遇到的問題記錄

基礎環境部署中遇到的錯誤記錄：
可能會報錯，見下:

sysctl -p /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory

sysctl -p /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory

解決方法,加載br_betfilter模塊.

modprobe br_netfilter

lsmod |grep bridge
bridge                107106  1 br_netfilter
stp                    12976  1 bridge
llc                    14552  2 stp,bridge