搭建docker和rancher的挖坑、踩坑以及填坑

時間 2020-07-21

原文原文鏈接

服務器環境linux

服務器版本：CentOS Linux release 7.2.1511 (Core)docker

內核版本：3.10.0-327.el7.x86_64json

一、搭建dockercentos

（1）編輯/etc/sysctl.conf，添加以下內容bash

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

　執行下面命令服務器

modprobe br_netfilter sysctl -p

（2）安裝docker的yum源網絡

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

（3）查看docker的版本架構

yum list docker-ce.x86_64  --showduplicates |sort -r
 * updates: mirrors.aliyun.com
Loading mirror speeds from cached hostfile
Loaded plugins: fastestmirror
Installed Packages
 * extras: mirrors.aliyun.com
 * epel: mirrors.aliyun.com
docker-ce.x86_64            3:18.09.0-3.el7                    docker-ce-stable 
docker-ce.x86_64            18.06.1.ce-3.el7                   docker-ce-stable 
docker-ce.x86_64            18.06.1.ce-3.el7                   @docker-ce-stable
docker-ce.x86_64            18.06.0.ce-3.el7                   docker-ce-stable 
docker-ce.x86_64            18.03.1.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            18.03.0.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.12.1.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.12.0.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.09.1.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.09.0.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.06.2.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.06.1.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.06.0.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.03.3.ce-1.el7                   docker-ce-stable 
docker-ce.x86_64            17.03.2.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.03.1.ce-1.el7.centos            docker-ce-stable 
docker-ce.x86_64            17.03.0.ce-1.el7.centos            docker-ce-stable 
 * base: mirrors.aliyun.com
Available Packages

（4）安裝最新版本dockerapp

yum makecache fast

yum install -y --setopt=obsoletes=0 \
  docker-ce-18.06.1.ce-3.el7

systemctl start docker
systemctl enable docker

（5）確認下iptables filter表中FOWARD鏈的默認策略(pllicy)爲ACCEPTcurl

iptables -nvL
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 255K   14M CATTLE_NETWORK_POLICY  all  --  *      *       10.42.0.0/16         10.42.0.0/16        
1300K 1914M CATTLE_FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 786K  406M DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 786K  406M DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 433K  109M ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
 2130  113K DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
 351K  296M ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
  716 37220 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

二、搭建rancher的server端

（1）用docker搭建rancher

docker run -d --restart=always -p 8080:8080 rancher/server

（2）出現下面的錯誤，下面爲docker錯誤日誌

Dec 10 15:42:31 iZbp10cnscbfblnh5buomfZ dockerd: time="2018-12-10T15:42:31.530843247+08:00" level=error msg="Handler for POST /v1.38/containers/create returned error: mkdir /var/lib/docker/overlay2/6f1ba4b09228e52acc14d12f40f9af6754781f6b253a878d9e98e292c6fc41fb-init/merged/dev/shm: invalid argument"

查了不少資料，確認緣由爲存儲驅動的問題，改成devicemapper就搞定了

修改文件/etc/docker/daemon.json

cat /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://quv6i4g3.mirror.aliyuncs.com"],
  "storage-driver": "devicemapper"
}

去掉啓動選擇存儲驅動的啓動參數,修改完成以下

cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

（3）從新啓動docker，就能夠了

systemctl restart docker

（4）server端搭建完成，用ip+端口直接能夠訪問rancher

（5）本身配置下用戶權限

三、配置rancher的client端

（1）基礎架構->主機->添加主機

（2）複製上面的命令在要添加client端的服務器運行，出現以下的錯誤：

INFO: Running Agent Registration Process, CATTLE_URL=http://xx.xx.xx.xx:8080/v1
INFO: Attempting to connect to: http://xx.xx.xx.xx:8080/v1
ERROR: http://xx.xx.xx.xx:8080/v1 is not accessible (Failed to connect to xx.xx.xx.xx port 8080: Connection timed out)

curl地址看下：

curl -k -i http://xx.xx.xx.xx:8080/v1
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Date: Tue, 11 Dec 2018 02:47:40 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: PL=rancher;Path=/
Www-Authenticate: Basic realm="Enter API access key and secret key as username and password"
X-Api-Schemas: http://xx.xx.xx.xx:8080/v1/schemas
X-Rancher-Version: v1.6.25
Content-Length: 177

{"id":"6df18d62-d509-499c-8805-bdbf0b4e5783","type":"error","links":{},"actions":{},"status":401,"code":"Unauthorized","message":"Unauthorized","detail":null,"baseType":"error"}

curl -k -i http://xx.xx.xx.xx:8080/v1/scripts/2ACEE7A659A39604D5B0:1514678400000:rnrITfbHWobU2tjD3SNfleY8ePQ
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Tue, 11 Dec 2018 02:48:09 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: PL=rancher;Path=/
Vary: Accept-Encoding, User-Agent
Www-Authenticate: Basic realm="Enter API access key and secret key as username and password"
X-Api-Schemas: http://xx.xx.xx.xx:8080/v1/schemas
X-Rancher-Version: v1.6.25
Content-Length: 268

#!/bin/sh

export CATTLE_REGISTRATION_ACCESS_KEY="registrationToken"
export CATTLE_REGISTRATION_SECRET_KEY="2ACEE7A659A39604D5B0:1514678400000:rnrITfbHWobU2tjD3SNfleY8ePQ"
export CATTLE_URL="http://xx.xx.xx.xx:8080/v1"
export DETECTED_CATTLE_AGENT_IP="192.168.0.1"

查了大量資料，開放與其餘全部主機之間的 UDP 端口 500 和 4500，也嘗試關閉了防火牆仍是不行

最後找到了緣由，進入rancher的server的容器內，發現容器的網絡不通