K8S二進制部署
第1章 基礎架構
1.1 集羣規劃
| 主機名 | 角色 | ip |
|---|---|---|
| HDSS7-11.host.com | k8s代理節點1 | 10.4.7.11 |
| HDSS7-12.host.com | k8s代理節點2 | 10.4.7.12 |
| HDSS7-21.host.com | k8s運算節點1 | 10.4.7.21 |
| HDSS7-22.host.com | k8s運算節點2 | 10.4.7.22 |
| HDSS7-200.host.com | k8s運維節點(docker倉庫) | 10.4.7.200 |
1.2 架構圖
1.3 部署前的準備工做
1.3.1 檢查系統內核版本
uname -r #內核不能低於3.10
1.3.2 關閉firewalled和selinux
1. 關閉firewalled systemctl stop firewalld systemctl status firewalld 2. 關閉selinux
1.3.3 安裝epel源
替換爲aliyun的epel源 1. 備份原有的 mv /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.backup 2. 下載阿里雲的epel源 wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
1.3.4 安裝必要的工具
yum install -y tree vim wget bash-completion bash-completion-extras lrzsz net-tools sysstat iotop iftop htop unzip telnet dos2unix bind-utils -y
1.3.5 開啓內核轉發功能
echo 'net.ipv4.ip_forward=1 ' >>/etc/sysctl.conf sysctl -p
1.3.6 安裝時間同步服務
選擇一個節點做爲服務端，剩下的所有做爲客戶端 1. yum install chrony -y (5臺節點所有執行) 2. 修改服務端配置文件(這裏選擇10.4.7.11做爲服務端) vim /etc/chrony.conf 修改以下三項 server time1.aliyun.com iburst allow 10.4.7.0/24 local stratum 10 #這一行把註釋打開便可 3. 啓動服務 systemctl start chronyd systemctl enable chronyd 4. 檢查 [root@hdss7-11 ~]# ss -unl | grep 123 UNCONN 0 0 *:123 *:* 5. 修改客戶端配置文件 vim /etc/chrony.conf 修改一行便可 server 10.4.7.11 iburst 6. 客戶端檢查 [root@hdss7-12 ~]# chronyc sources 210 Number of sources = 1 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 10.4.7.11 3 6 37 3 +27us[+6434us] +/- 89ms #^* 10.4.7.11表示成功 7. 全部節點執行date明明看時間是否一致
第2章 安裝bind9服務(hdss7-11上)
2.1 安裝bind軟件
[root@hdss7-11 ~]# yum install bind -y
2.2 檢查是否安裝
[root@hdss7-11 ~]# rpm -qa bind bind-9.11.4-16.P2.el7_8.6.x86_64
2.3 配置bind主配置文件
vi /etc/named.conf
...修改以下幾行的內容
listen-on port 53 { 10.4.7.11; };
allow-query { any; };
forwarders { 10.4.7.254; }; #這一行是在原有的配置文件增長的
dnssec-enable no;
dnssec-validation no;
#檢查配置是否正確
[root@hdss7-11 ~]# named-checkconf
沒有提示表示配置的沒有問題
第2章 安裝bind9服務(hdss7-11上)
2.1 安裝bind軟件
yum install bind -y
2.2 檢查是否安裝
[root@hdss7-11 ~]# rpm -qa bind bind-9.11.4-16.P2.el7_8.6.x86_64
2.3 配置bind主配置文件
vi /etc/named.conf
...修改以下幾行的內容
listen-on port 53 { 10.4.7.11; };
allow-query { any; };
forwarders { 10.4.7.254; }; #這一行是在原有的配置文件增長的
dnssec-enable no;
dnssec-validation no;
#檢查配置是否正確
[root@hdss7-11 ~]# named-checkconf
沒有提示表示配置的沒有問題
2.4 配置區域配置文件
im /etc/named.rfc1912.zones
....在文件的最後加上以下內容
zone "host.com" IN {
type master;
file "host.com.zone";
allow-update { 10.4.7.11; };
};
zone "od.com" IN {
type master;
file "od.com.zone";
allow-update { 10.4.7.11; };
};
2.5 配置區域數據文件
2.5.1 配置主機域數據文件
vim /var/named/host.com.zone $ORIGIN host.com. $TTL 600 ; 10 minutes @ IN SOA dns.host.com. dnsadmin.host.com. ( 2019111001 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.host.com. $TTL 60 ; 1 minute dns A 10.4.7.11 HDSS7-11 A 10.4.7.11 HDSS7-12 A 10.4.7.12 HDSS7-21 A 10.4.7.21 HDSS7-22 A 10.4.7.22 HDSS7-200 A 10.4.7.200
2.5.2 配置業務域數據文件
vim /var/named/od.com.zone $ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2019111001 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11
2.5.3 檢查配置是否正確
named-checkconf
2.6 啓動bind
ystemctl start named 1. 檢查端口是否存在 [root@hdss7-11 ~]# netstat -lntup |grep named tcp 0 0 10.4.7.11:53 0.0.0.0:* LISTEN 15642/named tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 15642/named tcp6 0 0 ::1:953 :::* LISTEN 15642/named udp 0 0 10.4.7.11:53 0.0.0.0:* 15642/named
2.7 檢查服務是否正常
[root@hdss7-11 ~]# dig -t A hdss7-21.host.com @10.4.7.11 +short 10.4.7.21
2.8 配置DNS客戶端
2.8.1 Linux上(5臺機器都要以下操做)
vim /etc/sysconfig/network-scripts/ifcfg-eth0 ...修改DNS1=10.4.7.11 [root@hdss7-11 ~]# vim /etc/resolv.conf # Generated by NetworkManager search host.com nameserver 10.4.7.11 #重啓網卡 systemctl restart network #若是啓動網卡報錯 1.在 /etc/sysconfig/network-scripts目錄下刪除掉別的ifcfg-xxx(只保留一個你網卡同名的文檔) cd /etc/sysconfig/network-scripts rm -fr ifcfg-eth1 2. 再把NetworkManager停用 systemctl stop NetworkManager systemctl disable NetworkManager 3. 使用ip addr查看mac地址 (ether後面的就是) ether xx:xx:xx:xx:xx:xx txqueuelen 1000 (Ethernet) 4. 把mac地址添加進 ifcfg-xxx 的HWADDR中(若是沒有就再新添加一項) HWADDR=xx:xx:xx:xx:xx:xx 5. 重啓網卡 systemctl restart network #測試 [root@hdss7-11 ~]# ping baidu.com PING baidu.com (220.181.38.148) 56(84) bytes of data. 64 bytes from 220.181.38.148 (220.181.38.148): icmp_seq=1 ttl=128 time=136 ms 64 bytes from 220.181.38.148 (220.181.38.148): icmp_seq=2 ttl=128 time=115 ms [root@hdss7-11 ~]# ping hdss7-11 PING HDSS7-11.host.com (10.4.7.11) 56(84) bytes of data. 64 bytes from hdss7-11.host.com (10.4.7.11): icmp_seq=1 ttl=64 time=0.007 ms 64 bytes from hdss7-11.host.com (10.4.7.11): icmp_seq=2 ttl=64 time=0.028 ms
2.8.2 windows上
控制面板---》網絡和共享中心---》更改適配器設置-----》vmnet8----》屬性
在window上cmd窗口ping hdss7-200看是否成功
node
第3章 準備證書籤發環境(在hdss7-200上)
3.1 安裝CFSLL
1. 上傳軟件包 下載地址以下 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/bin/cfssl-json wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/bin/cfssl-certinfo 2. 把下載的軟件包移動到/usr/bin下 mv cfssl /usr/bin mv cfssl-certinfo /usr/bin mv cfssl-json /usr/bin 3. 授予權限 chmod +x /usr/bin/cfssl*
3.2 建立存放證書的目錄
cd /opt mkdir certs cd certs
3.3 建立生成CA證書籤名請求（csr）的JSON配置文件
vim /opt/certs/ca-csr.json
{
"CN": "OldboyEdu",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
],
"ca": {
"expiry": "175200h"
}
}
CN: Common Name，瀏覽器使用該字段驗證網站是否合法，通常寫的是域名。很是重要。瀏覽器使用該字段驗證網站是否合法
C: Country， 國家
ST: State，州，省
L: Locality，地區，城市
O: Organization Name，組織名稱，公司名稱
OU: Organization Unit Name，組織單位名稱，公司部門
3.4 生成CA證書和私鑰
cd /opt/certs cfssl gencert -initca ca-csr.json | cfssl-json -bare ca [root@hdss7-200 certs]# ll total 16 -rw-r--r-- 1 root root 1001 Jun 20 16:23 ca.csr -rw-r--r-- 1 root root 332 Jun 20 16:13 ca-csr.json -rw------- 1 root root 1679 Jun 20 16:23 ca-key.pem -rw-r--r-- 1 root root 1354 Jun 20 16:23 ca.pem #注意 生成ca.pem、ca.csr、ca-key.pem(CA私鑰,需妥善保管)
第4章 部署docker環境(HDSS7-200.host.com,HDSS7-21.host.com,HDSS7-22.host.com上）
4.1 安裝docker
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
4.2 配置
hdss7-21上linux
mkdir /etc/docker
mkdir /data/docker -p
vim /etc/docker/daemon.json
{
"graph": "/data/docker",
"storage-driver": "overlay",
"insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
"bip": "172.7.21.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
systemctl start docker
hdss7-22上nginx
mkdir /etc/docker
mkdir /data/docker -p
vim /etc/docker/daemon.json
{
"graph": "/data/docker",
"storage-driver": "overlay",
"insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
"bip": "172.7.22.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
systemctl start docker
hdss7-200上git
mkdir /etc/docker
mkdir /data/docker -p
vim /etc/docker/daemon.json
{
"graph": "/data/docker",
"storage-driver": "overlay",
"insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
"bip": "172.7.200.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
systemctl start docker
第5章 部署docker鏡像私有倉庫harbor(HDSS7-200上)
5.1 下載軟件包
1. 上傳軟件包 下載地址https://github.com/goharbor/harbor 建議選擇1.7.6以上的版本，由於1.7.5如下的版本有漏洞
5.2 解壓並作軟鏈接
mkdir /opt/src -p cd src tar xf harbor-offline-installer-v1.8.3.tgz -C /opt cd /opt mv harbor harbor-v1.8.3 ln -s harbor-v1.8.3 harbor
5.3 配置harbor
cd /opt/harbor vim harbor.yml ...修改以下幾行 hostname: harbor.od.com http: port: 180 data_volume: /data/harbor location: /data/harbor/logs #建立目錄 mkdir -p /data/harbor/logs
5.4 安裝docker-compose
yum install docker-compose -y [root@hdss7-200 ~]# rpm -qa docker-compose docker-compose-1.18.0-4.el7.noarch
5.4 安裝harbor
cd /opt/harbor ./install.sh
5.5 檢查harbor的啓動狀況
[root@hdss7-200 harbor]# docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------
harbor-core /harbor/start.sh Up
harbor-db /entrypoint.sh postgres Up 5432/tcp
harbor-jobservice /harbor/start.sh Up
harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp
harbor-portal nginx -g daemon off; Up 80/tcp
nginx nginx -g daemon off; Up 0.0.0.0:180->80/tcp
redis docker-entrypoint.sh redis ... Up 6379/tcp
registry /entrypoint.sh /etc/regist ... Up 5000/tcp
registryctl /harbor/start.sh Up
5.6 配置harbor的dns內網解析(在hdss7-11上)
vi /var/named/od.com.zone ...添加以下一行 harbor A 10.4.7.200 #注意serial前滾一個序號 1. 重啓named服務 systemc restart named 2. 檢查是否能解析到 dig -t A harbor.od.com +short
5.7 安裝nginx並配置(在hdss7-200上)
yum install nginx -y
#修改配置文件
vim /etc/nginx/conf.d/harbor.od.com.conf
server {
listen 80;
server_name harbor.od.com;
client_max_body_size 1000m;
location / {
proxy_pass http://127.0.0.1:180;
}
}
瀏覽器打開http://harbor.od.comgithub
這裏登陸harbor的時候若是登陸不進去 1. 先關閉dokcer-compose docker-compose down 2. 再啓動dokcer-compose docker-compose up -d
登陸harbor新建項目
在linux上登陸harborweb
docker login harbor.od.com 用戶名admin 密碼:Harbor12345
pull鏡像並推送鏡像(hdss7-200上)redis
1. pull nginx鏡像 docker pull nginx:1.7.9 2. 從新打上標籤 docker tag 84581e99d807 harbor.od.com/public/nginx:v1.7.9 3. 推送鏡像到harbor倉庫 docker push harbor.od.com/public/nginx:v1.7.9
在web界面檢查是否有了nginx鏡像docker
第6章 部署Master節點服務
6.1 部署etcd集羣
| 主機名 | 角色 | ip |
|---|---|---|
| HDSS7-12.host.com | etcd lead | 10.4.7.12 |
| HDSS7-21.host.com | etcd follow | 10.4.7.21 |
| HDSS7-22.host.com | etcd follow | 10.4.7.22 |
6.2 建立生成證書籤名請求(csr)的JSON配置文件(在hdss7-200上)
vim /opt/certs/etcd-peer-csr.json
{
"CN": "etcd-peer",
"hosts": [
"10.4.7.11",
"10.4.7.12",
"10.4.7.21",
"10.4.7.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
6.3 建立基於根證書的config配置文件
cd /opt/certs
vim ca-config.json
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
6.4 生成etcd證書和私鑰
cd /opt/certs cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer
6.5 檢查生成的證書、私鑰
cd /opt/certs [root@hdss7-200 certs]# ll total 36 -rw-r--r-- 1 root root 836 Jun 20 19:14 ca-config.json -rw-r--r-- 1 root root 1001 Jun 20 16:23 ca.csr -rw-r--r-- 1 root root 332 Jun 20 16:13 ca-csr.json -rw------- 1 root root 1679 Jun 20 16:23 ca-key.pem -rw-r--r-- 1 root root 1354 Jun 20 16:23 ca.pem -rw-r--r-- 1 root root 1066 Jun 20 19:17 etcd-peer.csr -rw-r--r-- 1 root root 364 Jun 20 19:14 etcd-peer-csr.json -rw------- 1 root root 1675 Jun 20 19:17 etcd-peer-key.pem -rw-r--r-- 1 root root 1436 Jun 20 19:17 etcd-peer.pem
6.6 安裝etcd(hdss7-12,hdss7-21,hdss7-22)
6.6.1 說明
etcd的集羣都要安裝，這裏先在hdss7-12上安裝，其他的2臺主機上按照同樣的方法安裝便可
6.6.2 hdss7-12部署流程以下
第一個里程: 建立etcd用戶
useradd -s /sbin/nologin -M etcd
第二個里程：下載軟件，解壓，作軟鏈接
1. 下載地址 https://github.com/etcd-io/etcd/tags 2. 這裏已經下載好了上傳救能夠了 mkdir /opt/src 上傳到此目錄 3. 解壓軟件 tar xf etcd-v3.1.20-linux-amd64.tar.gz -C /opt 4. 作軟鏈接 cd /opt mv etcd-v3.1.20-linux-amd64 etcd-v3.1.20 ln -s etcd-v3.1.20 etcd
第三個里程: 建立目錄
kdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
將運維主機上生成的ca.pem、etcd-peer-key.pem、etcd-peer.pem拷貝到/opt/etcd/certs目錄中，注意私鑰文件權限600json
cd /opt/etcd/certs scp hdss7-200:/opt/certs/ca.pem . scp hdss7-200:/opt/certs/etcd-peer-key.pem . scp hdss7-200:/opt/certs/etcd-peer.pem .
第四個里程: 建立etcd服務啓動腳本並給予權限
vim /opt/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-12 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://10.4.7.12:2380 \
--listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://10.4.7.12:2380 \
--advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
#給予腳本權限
chmod +x /opt/etcd/etcd-server-startup.sh
第五個里程：給予建立的目錄的權限
chown -R etcd.etcd /data/etcd /data/logs/etcd-server/ /opt/etcd-v3.1.20/
第六個里程: 安裝supervisor軟件
yum install supervisor -y
第七個里程: 啓動supervisor並設置爲開機自啓動
systemctl start supervisord systemctl enable supervisord
第八個里程: 建立etcd-server的啓動配置
vim /etc/supervisord.d/etcd-server.ini [program:etcd-server-7-12] command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/etcd ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=22 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=etcd ; setuid to this UNIX account to run the program redirect_stderr=false ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false) stderr_logfile=/data/logs/etcd-server/etcd.stderr.log ; stderr log path, NONE for none; default AUTO stderr_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stderr_logfile_backups=4 ; # of stderr logfile backups (default 10) stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stderr_events_enabled=false ; emit events on stderr writes (default false)
第九個里程: 啓動etcd服務並檢查
[root@hdss7-12 etcd]# supervisorctl update etcd-server-7-12: added process group [root@hdss7-12 etcd]# netstat -luntp|grep etcd tcp 0 0 10.4.7.12:2379 0.0.0.0:* LISTEN 1499/./etcd tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1499/./etcd tcp 0 0 10.4.7.12:2380 0.0.0.0:* LISTEN 1499/./etcd
6.6.3 hdss7-21部署流程以下
第一個里程：建立etcd用戶
useradd -s /sbin/nologin -M etcd
第二個里程: 下載軟件，解壓，作軟鏈接
1. 下載地址 https://github.com/etcd-io/etcd/tags 2. 這裏已經下載好了上傳救能夠了 mkdir /opt/src -p 上傳到此目錄 3. 解壓軟件 tar xf etcd-v3.1.20-linux-amd64.tar.gz -C /opt 4. 作軟鏈接 cd /opt mv etcd-v3.1.20-linux-amd64 etcd-v3.1.20 ln -s etcd-v3.1.20 etcd
第三個里程: 建立目錄
mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
將運維主機上生成的ca.pem、etcd-peer-key.pem、etcd-peer.pem拷貝到/opt/etcd/certs目錄中，注意私鑰文件權限600vim
cd /opt/etcd/certs scp hdss7-200:/opt/certs/ca.pem . scp hdss7-200:/opt/certs/etcd-peer-key.pem . scp hdss7-200:/opt/certs/etcd-peer.pem .
第四個里程: 建立etcd服務啓動腳本並給予權限
vim /opt/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-21 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://10.4.7.21:2380 \
--listen-client-urls https://10.4.7.21:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://10.4.7.21:2380 \
--advertise-client-urls https://10.4.7.21:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
#給予腳本權限
chmod +x /opt/etcd/etcd-server-startup.sh
第五個里程：給予建立的目錄的權限
chown -R etcd.etcd /data/etcd /data/logs/etcd-server/ /opt/etcd-v3.1.20/
第六個里程: 安裝supervisor軟件
yum install supervisor -y
第七個里程: 啓動supervisor並設置爲開機自啓動
systemctl start supervisord systemctl enable supervisord
第八個里程: 建立etcd-server的啓動配置
vim /etc/supervisord.d/etcd-server.ini [program:etcd-server-7-21] command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/etcd ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=22 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=etcd ; setuid to this UNIX account to run the program redirect_stderr=false ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false) stderr_logfile=/data/logs/etcd-server/etcd.stderr.log ; stderr log path, NONE for none; default AUTO stderr_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stderr_logfile_backups=4 ; # of stderr logfile backups (default 10) stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stderr_events_enabled=false ; emit events on stderr writes (default false)
第九個里程: 啓動etcd服務並檢查
[root@hdss7-21 etcd]# supervisorctl update etcd-server-7-21: added process group [root@hdss7-21 etcd]# netstat -luntp|grep etcd tcp 0 0 10.4.7.12:2379 0.0.0.0:* LISTEN 1499/./etcd tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1499/./etcd tcp 0 0 10.4.7.12:2380 0.0.0.0:* LISTEN 1499/./etcd
6.6.4 hdss7-22部署流程以下
第一個里程：建立etcd用戶
useradd -s /sbin/nologin -M etcd
第二個里程: 下載軟件,解壓,作軟鏈接
1. 下載地址 https://github.com/etcd-io/etcd/tags 2. 這裏已經下載好了上傳救能夠了 mkdir /opt/src 上傳到此目錄 3. 解壓軟件 tar xf etcd-v3.1.20-linux-amd64.tar.gz -C /opt 4. 作軟鏈接 cd /opt mv etcd-v3.1.20-linux-amd64 etcd-v3.1.20 ln -s etcd-v3.1.20 etcd
第三個里程: 建立目錄
mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
將運維主機上生成的ca.pem、etcd-peer-key.pem、etcd-peer.pem拷貝到/opt/etcd/certs目錄中，注意私鑰文件權限600
cd /opt/etcd/certs scp hdss7-200:/opt/certs/ca.pem . scp hdss7-200:/opt/certs/etcd-peer-key.pem . scp hdss7-200:/opt/certs/etcd-peer.pem .
第四個里程: 建立etcd服務啓動腳本並給予權限
vim /opt/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-22 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://10.4.7.22:2380 \
--listen-client-urls https://10.4.7.22:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://10.4.7.22:2380 \
--advertise-client-urls https://10.4.7.22:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
#給予腳本權限
chmod +x /opt/etcd/etcd-server-startup.sh
第五個里程：給予建立的目錄的權限
chown -R etcd.etcd /data/etcd /data/logs/etcd-server/ /opt/etcd-v3.1.20/
第六個里程: 安裝supervisor軟件
yum install supervisor -y
第七個里程: 啓動supervisor並設置爲開機自啓動
systemctl start supervisord systemctl enable supervisord
第八個里程: 建立etcd-server的啓動配置
vim /etc/supervisord.d/etcd-server.ini [program:etcd-server-7-22] command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/etcd ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=22 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=etcd ; setuid to this UNIX account to run the program redirect_stderr=false ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false) stderr_logfile=/data/logs/etcd-server/etcd.stderr.log ; stderr log path, NONE for none; default AUTO stderr_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stderr_logfile_backups=4 ; # of stderr logfile backups (default 10) stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stderr_events_enabled=false ; emit events on stderr writes (default false)
第九個里程: 啓動etcd服務並檢查
[root@hdss7-22 etcd]# supervisorctl update etcd-server-7-22: added process group [root@hdss7-22 etcd]# netstat -luntp|grep etcd tcp 0 0 10.4.7.12:2379 0.0.0.0:* LISTEN 1499/./etcd tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1499/./etcd tcp 0 0 10.4.7.12:2380 0.0.0.0:* LISTEN 1499/./etcd
6.7 檢查集羣狀態
三個etcd節點都起來後
隨便哪個節點上執行 cd /opt/etcd ./etcdctl cluster-health [root@hdss7-22 etcd]# ./etcdctl cluster-health member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379 member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379 member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379 cluster is healthy
第7章 部署Master節點上的kube-apiserver集羣
7.1 集羣規劃
| 主機名 | 角色 | ip |
|---|---|---|
| HDSS7-21.host.com | kube-apiserver | 10.4.7.21 |
| HDSS7-22.host.com | kube-apiserver | 10.4.7.22 |
| HDSS7-11.host.com | 4層負載均衡 | 10.4.7.11 |
| HDSS7-12.host.com | 4層負載均衡 | 10.4.7.12 |
注意：這裏10.4.7.11和10.4.7.12使用nginx作4層負載均衡器，用keepalived跑一個vip：10.4.7.10，代理兩個kube-apiserver，實現高可用
7.2 hdss7-21部署流程以下
第一個里程: 下載軟件,解壓,作軟鏈接
1. githud下載地址 https://github.com/kubernetes/kubernetes/releases/tag/v1.15.2 2. 上傳軟件包到/opt/src 3. 解壓軟件包 tar xf kubernetes-server-linux-amd64-v1.15.2.tar.gz -C /opt 4. 作軟鏈接 cd /opt mv kubernetes kubernetes-v1.15.2 ln -s kubernetes-v1.15.2 kubernetes
第二個里程: 刪除沒必要要的一些文件
cd kubernetes rm -fr kubernetes-src.tar.gz #這個是go語言寫的源碼包 cd /opt/kubernetes/server/bin rm -fr *.tar #這些tar包是用kubeadm的方式安裝須要的tar包 rm -fr *_tag
第三個里程: 簽發client證書(apiserver去找etcd通訊須要的證書)
hdss7-200上
建立生成證書籤名請求（csr）的JSON配置文件
cd /opt/certs
vi client-csr.json
{
"CN": "k8s-node",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成client證書和私鑰
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json |cfssl-json -bare client
檢查生成的證書、私鑰
[root@hdss7-200 certs]# ls -l|grep client -rw-r--r-- 1 root root 993 Jun 20 21:38 client.csr -rw-r--r-- 1 root root 280 Jun 20 21:38 client-csr.json -rw------- 1 root root 1675 Jun 20 21:38 client-key.pem -rw-r--r-- 1 root root 1371 Jun 20 21:38 client.pem
第四個里程：簽發kube-apiserver證書(要來找apiserver須要的證書)
vim /opt/certs/apiserver-csr.json
{
"CN": "apiserver",
"hosts": [
"127.0.0.1",
"192.168.0.1",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成kube-apiserver證書和私鑰
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json |cfssl-json -bare apiserver
檢查生成的證書、私鑰
[root@hdss7-200 certs]# ls -l|grep apiserver -rw-r--r-- 1 root root 1245 Jun 20 21:45 apiserver.csr -rw-r--r-- 1 root root 562 Jun 20 21:44 apiserver-csr.json -rw------- 1 root root 1679 Jun 20 21:45 apiserver-key.pem -rw-r--r-- 1 root root 1598 Jun 20 21:45 apiserver.pem
第五個里程: 拷貝證書至各運算節點，並建立配置
HDSS7-21.host.com
cd /opt/kubernetes/server/bin mkdir certs cd certs scp hdss7-200:/opt/certs/ca.pem . scp hdss7-200:/opt/certs/ca-key.pem . scp hdss7-200:/opt/certs/client-key.pem . scp hdss7-200:/opt/certs/client.pem . scp hdss7-200:/opt/certs/apiserver-key.pem . scp hdss7-200:/opt/certs/apiserver.pem .
建立配置
cd /opt/kubernetes/server/bin
mkdir conf
cd conf
vi audit.yaml
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
- level: RequestResponse
resources:
- group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
# Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
# Don't log authenticated requests to certain non-resource URL paths.
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
# Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
namespaces: ["kube-system"]
# Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]
# Log all other resources in core and extensions at the Request level.
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
# A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
# Long-running requests like watches that fall under this rule will not
# generate an audit event in RequestReceived.
omitStages:
- "RequestReceived"
第六個里程：建立啓動腳本並給予權限
vim /opt/kubernetes/server/bin/kube-apiserver.sh #!/bin/bash ./kube-apiserver \ --apiserver-count 2 \ --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \ --audit-policy-file ./conf/audit.yaml \ --authorization-mode RBAC \ --client-ca-file ./certs/ca.pem \ --requestheader-client-ca-file ./certs/ca.pem \ --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \ --etcd-cafile ./certs/ca.pem \ --etcd-certfile ./certs/client.pem \ --etcd-keyfile ./certs/client-key.pem \ --etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \ --service-account-key-file ./certs/ca-key.pem \ --service-cluster-ip-range 192.168.0.0/16 \ --service-node-port-range 3000-29999 \ --target-ram-mb=1024 \ --kubelet-client-certificate ./certs/client.pem \ --kubelet-client-key ./certs/client-key.pem \ --log-dir /data/logs/kubernetes/kube-apiserver \ --tls-cert-file ./certs/apiserver.pem \ --tls-private-key-file ./certs/apiserver-key.pem \ --v 2 #給予權限 chmod +x /opt/kubernetes/server/bin/kube-apiserver.sh
第七個里程: 建立supervisor配置
vim /etc/supervisord.d/kube-apiserver.ini [program:kube-apiserver-7-21] command=/opt/kubernetes/server/bin/kube-apiserver.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=22 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=false ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log ; stdout log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false) stderr_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stderr.log ; stderr log path, NONE for none; default AUTO stderr_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stderr_logfile_backups=4 ; # of stderr logfile backups (default 10) stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stderr_events_enabled=false ; emit events on stderr writes (default false)
第八個里程: 建立相應的目錄
mkdir -p /data/logs/kubernetes/kube-apiserver
第九個里程: 啓動服務並檢查
[root@hdss7-21 bin]# supervisorctl update kube-apiserverr: added process group
7.3 hdss7-22部署流程以下
第一個里程: 下載軟件,解壓,作軟鏈接
1. githud下載地址 https://github.com/kubernetes/kubernetes/releases/tag/v1.15.2 2. 上傳軟件包到/opt/src 3. 解壓軟件包 tar xf kubernetes-server-linux-amd64-v1.15.2.tar.gz -C /opt 4. 作軟鏈接 cd /opt mv kubernetes kubernetes-v1.15.2 ln -s kubernetes-v1.15.2 kubernetes
第二個里程: 刪除沒必要要的一些文件
cd kubernetes rm -fr kubernetes-src.tar.gz #這個是go語言寫的源碼包 cd /opt/kubernetes/server/bin rm -fr *.tar #這些tar包是用kubeadm的方式安裝須要的tar包 rm -fr *_tag
第三個里程: 拷貝證書至各運算節點，並建立配置
HDSS7-22.host.com
cd /opt/kubernetes/server/bin mkdir certs cd certs scp hdss7-200:/opt/certs/ca.pem . scp hdss7-200:/opt/certs/ca-key.pem . scp hdss7-200:/opt/certs/client-key.pem . scp hdss7-200:/opt/certs/client.pem . scp hdss7-200:/opt/certs/apiserver-key.pem . scp hdss7-200:/opt/certs/apiserver.pem .
建立配置
cd /opt/kubernetes/server/bin
mkdir conf
cd conf
vi audit.yaml
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
- level: RequestResponse
resources:
- group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
# Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
# Don't log authenticated requests to certain non-resource URL paths.
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
# Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
namespaces: ["kube-system"]
# Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]
# Log all other resources in core and extensions at the Request level.
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
# A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
# Long-running requests like watches that fall under this rule will not
# generate an audit event in RequestReceived.
omitStages:
- "RequestReceived"
第四個里程：建立啓動腳本並給予權限
vim /opt/kubernetes/server/bin/kube-apiserver.sh #!/bin/bash ./kube-apiserver \ --apiserver-count 2 \ --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \ --audit-policy-file ./conf/audit.yaml \ --authorization-mode RBAC \ --client-ca-file ./certs/ca.pem \ --requestheader-client-ca-file ./certs/ca.pem \ --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \ --etcd-cafile ./certs/ca.pem \ --etcd-certfile ./certs/client.pem \ --etcd-keyfile ./certs/client-key.pem \ --etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \ --service-account-key-file ./certs/ca-key.pem \ --service-cluster-ip-range 192.168.0.0/16 \ --service-node-port-range 3000-29999 \ --target-ram-mb=1024 \ --kubelet-client-certificate ./certs/client.pem \ --kubelet-client-key ./certs/client-key.pem \ --log-dir /data/logs/kubernetes/kube-apiserver \ --tls-cert-file ./certs/apiserver.pem \ --tls-private-key-file ./certs/apiserver-key.pem \ --v 2 #給予權限 chmod +x /opt/kubernetes/server/bin/kube-apiserver.sh
第五個里程: 建立supervisor配置
vim /etc/supervisord.d/kube-apiserver.ini [program:kube-apiserver-7-22] command=/opt/kubernetes/server/bin/kube-apiserver.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=22 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=false ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log ; stdout log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false) stderr_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stderr.log ; stderr log path, NONE for none; default AUTO stderr_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stderr_logfile_backups=4 ; # of stderr logfile backups (default 10) stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stderr_events_enabled=false ; emit events on stderr writes (default false)
第六個里程: 建立相應的目錄
mkdir -p /data/logs/kubernetes/kube-apiserver
第七個里程: 啓動服務並檢查
[root@hdss7-22 bin]# supervisorctl update kube-apiserverr: added process group
第8章 配4層反向代理(HDSS7-11.host.com,HDSS7-12.host.com)
8.1 安裝nginx
yum install nginx -y
8.2 配置nginx
vim /etc/nginx/nginx.conf
stream {
upstream kube-apiserver {
server 10.4.7.21:6443 max_fails=3 fail_timeout=30s;
server 10.4.7.22:6443 max_fails=3 fail_timeout=30s;
}
server {
listen 7443;
proxy_connect_timeout 2s;
proxy_timeout 900s;
proxy_pass kube-apiserver;
}
}
8.3 啓動nginx並設置開機自啓動
systemctl start nginx systemctl enable nginx
8.4 下載keepalived
yum install keepalived -y
8.5 keepalived監控腳本
vim /etc/keepalived/check_port.sh
#!/bin/bash
#keepalived 監控端口腳本
#使用方法：
#在keepalived的配置文件中
#vrrp_script check_port {#建立一個vrrp_script腳本,檢查配置
# script "/etc/keepalived/check_port.sh 6379" #配置監聽的端口
# interval 2 #檢查腳本的頻率,單位（秒）
#}
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l`
if [ $PORT_PROCESS -eq 0 ];then
echo "Port $CHK_PORT Is Not Used,End."
exit 1
fi
else
echo "Check Port Cant Be Empty!"
fi
#給予腳本權限
chmod +x /etc/keepalived/check_port.sh
8.6 編寫keepalived配置文件
8.6.1 keepalived主配置文件(hdss7-11)
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 10.4.7.11
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 251
priority 100
advert_int 1
mcast_src_ip 10.4.7.11
nopreempt
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.4.7.10
}
}
8.6.2 keepalived備配置文件(hdss7-12)
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 10.4.7.12
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 251
mcast_src_ip 10.4.7.12
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.4.7.10
}
}
8.7 啓動代理並檢查
啓動
[root@hdss7-11 ~]# systemctl start keepalived [root@hdss7-11 ~]# systemctl enable keepalived [root@hdss7-12 ~]# systemctl start keepalived [root@hdss7-12 ~]# systemctl enable keepalived [root@hdss7-12 ~]# nginx -s reload
檢查
[root@hdss7-11 ~]# netstat -luntp|grep 7443
tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 17970/nginx: master
[root@hdss7-12 ~]# netstat -luntp|grep 7443
tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 17970/nginx: master
[root@hdss7-11 ~]# ip add|grep 10.4.7.10
inet 10.4.7.10/32 scope global vir0
[root@hdss7-12 ~]# ip add|grep 10.4.7.10
（空）
第9章 部署Master節點controller-manager
9.1 集羣規劃
| 主機名 | 角色 | ip |
|---|---|---|
| HDSS7-21.host.com | controller-manager | 10.4.7.21 |
| HDSS7-22.host.com | controller-manager | 10.4.7.22 |
9.2 hdss7-21部署流程
第一個里程: 建立啓動腳本並給與權限
vim /opt/kubernetes/server/bin/kube-controller-manager.sh #!/bin/sh ./kube-controller-manager \ --cluster-cidr 172.7.0.0/16 \ --leader-elect true \ --log-dir /data/logs/kubernetes/kube-controller-manager \ --master http://127.0.0.1:8080 \ --service-account-private-key-file ./certs/ca-key.pem \ --service-cluster-ip-range 192.168.0.0/16 \ --root-ca-file ./certs/ca.pem \ --v 2 #給予權限 chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh
第二個里程: 建立腳本中須要的目錄
mkdir -p /data/logs/kubernetes/kube-controller-manager
第三個里程: 建立supervisor配置
vim /etc/supervisord.d/kube-conntroller-manager.ini [program:kube-controller-manager-7-21] command=/opt/kubernetes/server/bin/kube-controller-manager.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
第四個里程: 啓動服務並檢查
[root@hdss7-21 bin]# supervisorctl update kube-controller-manager: added process group [root@hdss7-21 bin]# supervisorctl status etcd-server-7-21 RUNNING pid 6661, uptime 1 day, 8:41:13 kube-apiserver RUNNING pid 43765, uptime 2:09:41 kube-controller-manager RUNNING pid 44230, uptime 2:05:01
9.3 hdss7-22部署流程
第一個里程: 建立啓動腳本並給與權限
vim /opt/kubernetes/server/bin/kube-controller-manager.sh #!/bin/sh ./kube-controller-manager \ --cluster-cidr 172.7.0.0/16 \ --leader-elect true \ --log-dir /data/logs/kubernetes/kube-controller-manager \ --master http://127.0.0.1:8080 \ --service-account-private-key-file ./certs/ca-key.pem \ --service-cluster-ip-range 192.168.0.0/16 \ --root-ca-file ./certs/ca.pem \ --v 2 #給予權限 chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh
第二個里程: 建立腳本中須要的目錄
mkdir -p /data/logs/kubernetes/kube-controller-manager
第三個里程: 建立supervisor配置
vim /etc/supervisord.d/kube-conntroller-manager.ini [program:kube-controller-manager-7-22] command=/opt/kubernetes/server/bin/kube-controller-manager.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
第四個里程: 啓動服務並檢查
[root@hdss7-22 bin]# supervisorctl update kube-controller-manager: added process group [root@hdss7-22 bin]# supervisorctl status etcd-server-7-22 RUNNING pid 6661, uptime 1 day, 8:41:13 kube-apiserver RUNNING pid 43765, uptime 2:09:41 kube-controller-manager RUNNING pid 44230, uptime 2:05:01
第10章 部署Master節點的kube-scheduler
10.1 集羣規劃
| 主機名 | 角色 | ip |
|---|---|---|
| HDSS7-21.host.com | kube-scheduler | 10.4.7.21 |
| HDSS7-22.host.com | kube-scheduler | 10.4.7.22 |
注意：這裏部署文檔以HDSS7-21.host.com主機爲例，另一臺運算節點安裝部署方法相似
10.2 10.2 hdss7-21部署流程
第一個里程: 建立啓動腳本並給予權限
vim /opt/kubernetes/server/bin/kube-scheduler.sh #!/bin/sh ./kube-scheduler \ --leader-elect \ --log-dir /data/logs/kubernetes/kube-scheduler \ --master http://127.0.0.1:8080 \ --v 2 #給予權限 chmod +x /opt/kubernetes/server/bin/kube-scheduler.sh
第二個里程: 建立腳本中須要的目錄
mkdir -p /data/logs/kubernetes/kube-scheduler
第三個里程：建立supervisor配置
vim /etc/supervisord.d/kube-scheduler.ini [program:kube-scheduler-7-21] command=/opt/kubernetes/server/bin/kube-scheduler.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
第四個里程：啓動服務並檢查
[root@hdss7-21 bin]# supervisorctl update kube-scheduler: added process group [root@hdss7-21 bin]# supervisorctl status etcd-server-7-21 RUNNING pid 6661, uptime 1 day, 8:41:13 kube-apiserver RUNNING pid 43765, uptime 2:09:41 kube-controller-manager RUNNING pid 44230, uptime 2:05:01 kube-scheduler RUNNING pid 44779, uptime 2:02:27
10.3 hdss7-22部署流程
第一個里程: 建立啓動腳本並給予權限
vim /opt/kubernetes/server/bin/kube-scheduler.sh #!/bin/sh ./kube-scheduler \ --leader-elect \ --log-dir /data/logs/kubernetes/kube-scheduler \ --master http://127.0.0.1:8080 \ --v 2 #給予權限 chmod +x /opt/kubernetes/server/bin/kube-scheduler.sh
第二個里程: 建立腳本中須要的目錄
mkdir -p /data/logs/kubernetes/kube-scheduler
第三個里程：建立supervisor配置
vim /etc/supervisord.d/kube-scheduler.ini [program:kube-scheduler-7-22] command=/opt/kubernetes/server/bin/kube-scheduler.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
第四個里程：啓動服務並檢查
[root@hdss7-22 bin]# supervisorctl update kube-scheduler: added process group [root@hdss7-22 bin]# supervisorctl status etcd-server-7-22 RUNNING pid 6661, uptime 1 day, 8:41:13 kube-apiserver RUNNING pid 43765, uptime 2:09:41 kube-controller-manager RUNNING pid 44230, uptime 2:05:01 kube-scheduler RUNNING pid 44779, uptime 2:02:27
10.4 檢查集羣的健康狀態
10.4.1 說明
檢查集羣健康狀態須要用到kubectl這個工具，因此咱們要把它作軟鏈接到/usr/bin/下方便咱們調用
10.4.2 作軟連接(2個節點都須要作)
ln -s /opt/kubernetes/server/bin/kubectl /usr/bin/kubectl
10.4.3 檢查集羣健康狀態
[root@hdss7-21 src]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
第11章 部署Node節點服務
部署kubelet
11.1 集羣規劃
| 主機名 | 角色 | ip |
|---|---|---|
| HDSS7-21.host.com | kubelet | 10.4.7.21 |
| HDSS7-22.host.com | kubelet | 10.4.7.22 |
注意：這裏部署文檔以HDSS7-21.host.com主機爲例，另一臺運算節點安裝部署方法相似
11.2 簽發kubelet證書(hdss7-200)
建立生成證書籤名請求（csr）的JSON配置文件
cd /opt/certs
vi kubelet-csr.json
{
"CN": "k8s-kubelet",
"hosts": [
"127.0.0.1",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23",
"10.4.7.24",
"10.4.7.25",
"10.4.7.26",
"10.4.7.27",
"10.4.7.28"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成kubelet證書和私鑰
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet
檢查生成的證書、私鑰
[root@hdss7-200 certs]# ls -l|grep kubelet -rw-r--r-- 1 root root 1115 Jun 21 00:53 kubelet.csr -rw-r--r-- 1 root root 452 Jun 21 00:52 kubelet-csr.json -rw------- 1 root root 1675 Jun 21 00:53 kubelet-key.pem -rw-r--r-- 1 root root 1472 Jun 21 00:53 kubelet.pem
11.3 hdss7-21部署流程以下
第一個里程: 拷貝hdss7-200剛剛建立的證書到/opt/kubernetes/server/bin/certs
cd /opt/kubernetes/server/bin/certs scp hdss7-200:/opt/certs/kubelet.pem . scp hdss7-200:/opt/certs/kubelet-key.pem .
第二個里程: 建立配置(只能在conf目錄下面)
set-cluster
cd /opt/kubernetes/server/bin/conf kubectl config set-cluster myk8s \ --certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \ --embed-certs=true \ --server=https://10.4.7.10:7443 \ --kubeconfig=kubelet.kubeconfig
set-credentials
kubectl config set-credentials k8s-node \ --client-certificate=/opt/kubernetes/server/bin/certs/client.pem \ --client-key=/opt/kubernetes/server/bin/certs/client-key.pem \ --embed-certs=true \ --kubeconfig=kubelet.kubeconfig
set-contex
kubectl config set-context myk8s-context \ --cluster=myk8s \ --user=k8s-node \ --kubeconfig=kubelet.kubeconfig
use-context
[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=kubelet.kubeconfig
k8s-node.yaml
vim /opt/kubernetes/server/bin/conf/k8s-node.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: k8s-node roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: k8s-node
應用資源配置文件
[root@hdss7-21 conf]# kubectl create -f k8s-node.yaml clusterrolebinding.rbac.authorization.k8s.io/k8s-node created
檢查
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node -o yaml
11.4 hdss7-22部署流程以下
第一個里程: 拷貝kubelet.kubeconfig
cd /opt/kubernetes/server/bin/conf scp hdss7-21:/opt/kubernetes/server/bin/conf/kubelet.kubeconfig .
第二個里程: 拷貝hdss7-200剛剛建立的證書到/opt/kubernetes/server/bin/certs
cd /opt/kubernetes/server/bin/certs scp hdss7-200:/opt/certs/kubelet.pem . scp hdss7-200:/opt/certs/kubelet-key.pem .
11.5 準備pause基礎鏡像(hdss7-200)
[root@hdss7-200 ~]# docker pull kubernetes/pause [root@hdss7-200 ~]# docker tag f9d5de079539 harbor.od.com/public/pause:latest
11.6 提交至私有倉庫（harbor）中
[root@hdss7-200 ~]# docker push harbor.od.com/public/pause:latest
11.7 建立kubelet啓動腳本並給予權限
11.7.1 hdss7-21流程以下
第一個里程：編寫腳本並給予權限
vim /opt/kubernetes/server/bin/kubelet.sh #!/bin/sh ./kubelet \ --anonymous-auth=false \ --cgroup-driver systemd \ --cluster-dns 192.168.0.2 \ --cluster-domain cluster.local \ --runtime-cgroups=/systemd/system.slice \ --kubelet-cgroups=/systemd/system.slice \ --fail-swap-on="false" \ --client-ca-file ./certs/ca.pem \ --tls-cert-file ./certs/kubelet.pem \ --tls-private-key-file ./certs/kubelet-key.pem \ --hostname-override hdss7-21.host.com \ --image-gc-high-threshold 20 \ --image-gc-low-threshold 10 \ --kubeconfig ./conf/kubelet.kubeconfig \ --log-dir /data/logs/kubernetes/kube-kubelet \ --pod-infra-container-image harbor.od.com/public/pause:latest \ --root-dir /data/kubelet #給予權限 chmod +x /opt/kubernetes/server/bin/kubelet.sh
第二個里程：建立腳本中須要的目錄
mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet
第三個里程：建立supervisor配置
vim /etc/supervisord.d/kube-kubelet.ini [program:kube-kubelet-7-21] command=/opt/kubernetes/server/bin/kubelet.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
第四個里程：啓動服務並檢查
[root@hdss7-21 bin]# supervisorctl update kube-kubelet: added process group [root@hdss7-21 bin]# supervisorctl status etcd-server-7-21 RUNNING pid 9507, uptime 22:44:48 kube-apiserver RUNNING pid 9770, uptime 21:10:49 kube-controller-manager RUNNING pid 10048, uptime 18:22:10 kube-kubelet STARTING kube-scheduler RUNNING pid 10041, uptime 18:22:13
第五個里程：檢查運算節點
[root@hdss7-21 bin]# kubectl get node NAME STATUS ROLES AGE VERSION 10.4.7.21 Ready <none> 3m v1.13.2
11.7.1 hdss7-22流程以下
第一個里程：編寫腳本並給予權限
vim /opt/kubernetes/server/bin/kubelet.sh #!/bin/sh ./kubelet \ --anonymous-auth=false \ --cgroup-driver systemd \ --cluster-dns 192.168.0.2 \ --cluster-domain cluster.local \ --runtime-cgroups=/systemd/system.slice \ --kubelet-cgroups=/systemd/system.slice \ --fail-swap-on="false" \ --client-ca-file ./certs/ca.pem \ --tls-cert-file ./certs/kubelet.pem \ --tls-private-key-file ./certs/kubelet-key.pem \ --hostname-override hdss7-22.host.com \ --image-gc-high-threshold 20 \ --image-gc-low-threshold 10 \ --kubeconfig ./conf/kubelet.kubeconfig \ --log-dir /data/logs/kubernetes/kube-kubelet \ --pod-infra-container-image harbor.od.com/public/pause:latest \ --root-dir /data/kubelet #給予權限 chmod +x /opt/kubernetes/server/bin/kubelet.sh
第二個里程：建立腳本中須要的目錄
mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet
第三個里程：建立supervisor配置
vim /etc/supervisord.d/kube-kubelet.ini [program:kube-kubelet-7-22] command=/opt/kubernetes/server/bin/kubelet.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
第四個里程：啓動服務並檢查
[root@hdss7-22 bin]# supervisorctl update kube-kubelet: added process group [root@hdss7-22 bin]# supervisorctl status etcd-server-7-22 RUNNING pid 9507, uptime 22:44:48 kube-apiserver RUNNING pid 9770, uptime 21:10:49 kube-controller-manager RUNNING pid 10048, uptime 18:22:10 kube-kubelet STARTING kube-scheduler RUNNING pid 10041, uptime 18:22:13
第五個里程：檢查運算節點
[root@hdss7-22 bin]# kubectl get node NAME STATUS ROLES AGE VERSION 10.4.7.22 Ready <none> 3m v1.13.2
11.8 若是要給節點信息打上標籤
kubectl label node hdss7-21.host.com node-role.kubernetes.io/master= kubectl label node hdss7-21.host.com node-role.kubernetes.io/node= kubectl label node hdss7-22.host.com node-role.kubernetes.io/master= kubectl label node hdss7-22.host.com node-role.kubernetes.io/node= [root@hdss7-22 certs]# kubectl get nodes NAME STATUS ROLES AGE VERSION hdss7-21.host.com Ready master,node 11m v1.15.2 hdss7-22.host.com Ready master,node 4m10s v1.15.2
第12章 部署Node節點上kube-proxy
12.1 集羣規劃
| 主機名 | 角色 | ip |
|---|---|---|
| HDSS7-21.host.com | kube-proxy | 10.4.7.21 |
| HDSS7-22.host.com | kube-proxy | 10.4.7.22 |
注意：這裏部署文檔以HDSS7-21.host.com主機爲例，另一臺運算節點安裝部署方法相似
12.2 簽發kube-proxy證書(hdss7-200)
建立生成證書籤名請求（csr）的JSON配置文件
vim /opt/certs/kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成kube-proxy證書和私鑰
cd /opt/certs [root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json |cfssl-json -bare kube-proxy-client
檢查生成的證書、私鑰
[root@hdss7-200 certs]# ls -l|grep kube-proxy -rw-r--r-- 1 root root 1005 Jun 21 12:08 kube-proxy-client.csr -rw------- 1 root root 1679 Jun 21 12:08 kube-proxy-client-key.pem -rw-r--r-- 1 root root 1383 Jun 21 12:08 kube-proxy-client.pem -rw-r--r-- 1 root root 267 Jun 21 12:07 kube-proxy-csr.json
12.3 hdss7-21部署流程
第一個里程：拷貝證書到/opt/kubernetes/server/bin/certs
cd /opt/kubernetes/server/bin/certs scp hdss7-200:/opt/certs/kube-proxy-client.pem . scp hdss7-200:/opt/certs/kube-proxy-client-key.pem .
第二個里程: 建立配置(在conf目錄下面)
set-cluster
cd /opt/kubernetes/server/bin/conf kubectl config set-cluster myk8s \ --certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \ --embed-certs=true \ --server=https://10.4.7.10:7443 \ --kubeconfig=kube-proxy.kubeconfig
set-credentials
cd /opt/kubernetes/server/bin/conf kubectl config set-credentials kube-proxy \ --client-certificate=/opt/kubernetes/server/bin/certs/kube-proxy-client.pem \ --client-key=/opt/kubernetes/server/bin/certs/kube-proxy-client-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig
set-context
cd /opt/kubernetes/server/bin/conf kubectl config set-context myk8s-context \ --cluster=myk8s \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig
use-context
[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=kube-proxy.kubeconfig
第三個里程: 加載ipvs模塊
cd ~
vim ipvs.sh
#!/bin/bash
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"
for i in $(ls $ipvs_mods_dir|grep -o "^[^.]*")
do
/sbin/modinfo -F filename $i &>/dev/null
if [ $? -eq 0 ];then
/sbin/modprobe $i
fi
done
#給予腳本權限
chmod +x ipvs.sh
#執行腳本
./ipvs.sh
#檢查是否加載了模塊
lsmod | grep ip_vs
第四個里程: 建立kube-proxy啓動腳本並給予權限
vim /opt/kubernetes/server/bin/kube-proxy.sh #!/bin/sh ./kube-proxy \ --cluster-cidr 172.7.0.0/16 \ --hostname-override 10.4.7.21 \ --kubeconfig ./conf/kube-proxy.kubeconfig #給予腳本權限 chmod +x /opt/kubernetes/server/bin/kube-proxy.sh
第五個里程: 建立日誌目錄
mkdir -p /data/logs/kubernetes/kube-proxy
第六個里程: 建立supervisor配置
vim /etc/supervisord.d/kube-proxy.ini [program:kube-proxy-7-21] command=/opt/kubernetes/server/bin/kube-proxy.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
第七個里程: 啓動服務並檢查
[root@hdss7-21 bin]# supervisorctl update kube-proxy: added process group [root@hdss7-21 bin]# supervisorctl status etcd-server-7-21 RUNNING pid 9507, uptime 22:44:48 kube-apiserver RUNNING pid 9770, uptime 21:10:49 kube-controller-manager RUNNING pid 10048, uptime 18:22:10 kube-kubelet RUNNING pid 14597, uptime 0:32:59 kube-proxy STARTING kube-scheduler RUNNING pid 10041, uptime 18:22:13
12.4 hdss7-22部署流程
第一個里程: 拷貝證書/opt/kubernetes/server/bin/certs
cd /opt/kubernetes/server/bin/certs scp hdss7-200:/opt/certs/kube-proxy-client.pem . scp hdss7-200:/opt/certs/kube-proxy-client-key.pem .
第二個里程: 拷貝hdss7-21上建立的kube-proxy.kubeconfig文件到/opt/kubernetes/server/bin/conf
cd /opt/kubernetes/server/bin/conf scp hdss7-21:/opt/kubernetes/server/bin/conf/kube-proxy.kubeconfig .
第三個里程: 加載ipvs模塊
cd ~
vim ipvs.sh
#!/bin/bash
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"
for i in $(ls $ipvs_mods_dir|grep -o "^[^.]*")
do
/sbin/modinfo -F filename $i &>/dev/null
if [ $? -eq 0 ];then
/sbin/modprobe $i
fi
done
#給予腳本權限
chmod +x ipvs.sh
#執行腳本
./ipvs.sh
#檢查是否加載了模塊
lsmod | grep ip_vs
第四個里程: 建立kube-proxy啓動腳本並給予權限
vim /opt/kubernetes/server/bin/kube-proxy.sh #!/bin/sh ./kube-proxy \ --cluster-cidr 172.7.0.0/16 \ --hostname-override 10.4.7.22 \ --kubeconfig ./conf/kube-proxy.kubeconfig #給予腳本權限 chmod +x /opt/kubernetes/server/bin/kube-proxy.sh
第五個里程: 建立日誌目錄
mkdir -p /data/logs/kubernetes/kube-proxy
第六個里程: 建立supervisor配置
vim /etc/supervisord.d/kube-proxy.ini [program:kube-proxy-7-22] command=/opt/kubernetes/server/bin/kube-proxy.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
第七個里程: 啓動服務並檢查
[root@hdss7-22 bin]# supervisorctl update kube-proxy: added process group [root@hdss7-22 bin]# supervisorctl status etcd-server-7-22 RUNNING pid 9507, uptime 22:44:48 kube-apiserver RUNNING pid 9770, uptime 21:10:49 kube-controller-manager RUNNING pid 10048, uptime 18:22:10 kube-kubelet RUNNING pid 14597, uptime 0:32:59 kube-proxy STARTING kube-scheduler RUNNING pid 10041, uptime 18:22:13
第13章 驗證kubernetes集羣
13.1 在任意一個運算節點，建立一個資源配置清單
這裏咱們選擇HDSS7-21.host.com主機
vim /root/nginx-ds.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: nginx-ds
spec:
template:
metadata:
labels:
app: nginx-ds
spec:
containers:
- name: my-nginx
image: harbor.od.com/public/nginx:v1.7.9
ports:
- containerPort: 80
應用資源配置，並檢查
[root@hdss7-21 ~]# kubectl create -f nginx-ds.yaml
daemonset.extensions/nginx-ds created
#檢查
[root@hdss7-21 bin]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
etcd-1 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
controller-manager Healthy ok
[root@hdss7-21 bin]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready master,node 35m v1.15.2
hdss7-22.host.com Ready master,node 35m v1.15.2
[root@hdss7-21 bin]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ds-qz7nk 1/1 Running 0 113s
nginx-ds-rvplm 1/1 Running 0 113s
相關文章
- 1. 二進制部署k8s
- 2. K8S 二進制部署
- 3. 二進制部署K8s集羣
- 4. Kubernetes - - k8s - v1.12.4 二進制部署
- 5. K8S二進制部署之ETCD小結
- 6. K8S二進制部署node節點
- 7. Kubernetes V1.15 二進制部署k8s集羣
- 8. k8s單master節點部署【一】（二進制部署）--etcd集羣部署（理論）
- 9. 二進制部署K8s集羣第17節pod控制器
- 10. Azure 部署K8S(二)
- 更多相關文章...
- • MySQL BIT、BINARY、VARBINARY、BLOB（二進制類型） - MySQL教程
- • C# 二進制文件的讀寫 - C#教程
- • 使用阿里雲OSS+CDN部署前端頁面與加速靜態資源
- • 漫談MySQL的鎖機制
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
