OpenShift 4.2 離線安裝補充記錄

時間 2019-11-06

標籤 openshift 4.2 離線安裝補充記錄简体版

原文原文鏈接

OpenShift4.2詳細安裝參考同事王徵的安裝手冊(感謝王徵大師的研究和答疑解惑, 大坑文章都已經搞定了，我這裏是一些小坑)node

https://github.com/wangzheng422/docker_env/blob/master/redhat/ocp4/4.2.disconnect.install.mdgit

由於我這邊的環境有些不一樣，因此這裏只是本身的補充記錄，詳細的須要對照來看.github

1.架構

啓動的虛擬機經過bridge和主機網絡在同一個網段，ip規劃保持和文檔一致web

Bootstrap nodesdocker	192.168.7.12json
master-0.ocp4.redhat.renbootstrap	192.168.7.13api
master-1.ocp4.redhat.ren網絡	192.168.7.14架構
master-2.ocp4.redhat.ren	192.168.7.15
worker-0.ocp4.redhat.ren	192.168.7.16
worker-1.ocp4.redhat.ren	192.168.7.17
worker-2.ocp4.redhat.ren	192.168.7.18

2.網絡

我手頭的機器是4臺NUC,每臺4CPU,32G內存，而4.2OCP集羣最少須要3臺master, 1個bootstrap,1臺做爲負載均衡，dns解析等工做，再配上幾個worker節點，所以須要的機器在6+以上，採用虛擬機後,OpenShift節點啓動之後的跨主機網絡鏈接就是一個問題。

通過嘗試，採用的是KVM的Bridge模式，具體設置以下。

在每臺機器上設置

添加一個br0

[root@base ocp4]# cat /etc/sysconfig/network-scripts/ifcfg-br0 
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.7.1
NETMASK=255.255.255.0
GATEWAY=192.168.7.1
ONBOOT=yes
DEFROUTE=yes
NAME=br0
DEVICE=br0
PREFIX=25

修改現有的網卡加入br0

[root@base ocp4]# cat /etc/sysconfig/network-scripts/ifcfg-eno1 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
#IPADDR=192.168.7.1 #NETMASK=255.255.255.0 #GATEWAY=192.168.7.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eno1
UUID=4e9504c6-a5c4-4093-88b8-89a153dd66de
DEVICE=eno1
ONBOOT=yes
BRIDGE=br0

重啓網絡

systemctl restart network

啓動之後驗證筆記本還能繼續鏈接

[root@base ocp4]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000 link/ether 00:1f:c6:9c:56:60 brd ff:ff:ff:ff:ff:ff 3: wlp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:c2:c6:f0:c8:78 brd ff:ff:ff:ff:ff:ff
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:1f:c6:9c:56:60 brd ff:ff:ff:ff:ff:ff inet 192.168.7.1/25 brd 192.168.7.127 scope global noprefixroute br0 valid_lft forever preferred_lft forever inet6 fe80::e458:f6ff:fea8:b655/64 scope link valid_lft forever preferred_lft forever 5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:6d:9d:9f brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:6d:9d:9f brd ff:ff:ff:ff:ff:ff
12: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:9c:66:29 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe9c:6629/64 scope link 
       valid_lft forever preferred_lft forever
20: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:88:62:de brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe88:62de/64 scope link 
       valid_lft forever preferred_lft forever

網絡生效後，應該生成的虛擬機能夠訪問主機網絡。

創建虛擬機採用的網絡

[root@base data]# cat virt-net.xml
<network>
  <name>br0</name>
  <forward  mode='bridge'>
    <bridge name ='br0'/>
  </forward>
</network>

virsh net-define --file virt-net.xml
virsh net-autostart br0
virsh net-start br0

查看一下

[root@base data]# virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 br0                  active     yes           yes
 default              active     yes           yes

3. Yum源設置

參照3.11來設置yum源，但有一點要注意，儘可能用3.11的高版本，以前用3.11.16來設置，結果致使podman build鏡像到本地有問題，後來替換成3.11.146版本的yum.

[root@base ocp4]# cat /etc/yum.repos.d/base.repo 
[base]
name=base
baseurl=http://192.168.7.1:8080/repo/rhel-7-server-rpms/
enabled=1
gpgcheck=0
[ansible]
name=ansible
baseurl=http://192.168.7.1:8080/repo/rhel-7-server-ansible-2.6-rpms/
enabled=1
gpgcheck=0
[extra]
name=extra
baseurl=http://192.168.7.1:8080/repo/rhel-7-server-extras-rpms/
enabled=1
gpgcheck=0
[ose]
name=ose
baseurl=http://192.168.7.1:8080/repo/rhel-7-server-ose-3.11-rpms/
enabled=1
gpgcheck=0

4.啓動虛擬機和安裝過程

指定bridge網絡啓動，調整了網絡和ram的大小

virt-install --name=ocp4-bootstrap --vcpus=4 --ram=8192 \
--disk path=/data/kvm/ocp4-bootstrap.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network bridge=br0,model=virtio \
--boot menu=on --cdrom /data/ocp4/bootstrap-static.iso   

virt-install --name=ocp4-master0 --vcpus=4 --ram=16384 \
--disk path=/data/kvm/ocp4-master0.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network bridge=br0,model=virtio \
--boot menu=on --cdrom /data/ocp4/master-0.iso 

virt-install --name=ocp4-master1 --vcpus=4 --ram=16384 \
--disk path=/data/kvm/ocp4-master1.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network bridge=br0,model=virtio \
--boot menu=on --cdrom /data/ocp4/master-1.iso 

virt-install --name=ocp4-master2 --vcpus=4 --ram=16384 \
--disk path=/data/kvm/ocp4-master2.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network bridge=br0,model=virtio \
--boot menu=on --cdrom /data/ocp4/master-2.iso 

virt-install --name=ocp4-worker0 --vcpus=4 --ram=8192 \
--disk path=/data/kvm/ocp4-worker0.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network bridge=br0,model=virtio \
--boot menu=on --cdrom /data/ocp4/worker-0.iso 

virt-install --name=ocp4-worker1 --vcpus=4 --ram=8192 \
--disk path=/data/kvm/ocp4-worker1.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network bridge=br0,model=virtio \
--boot menu=on --cdrom /data/ocp4/worker-1.iso 

virt-install --name=ocp4-worker2 --vcpus=4 --ram=8192 \
--disk path=/data/kvm/ocp4-worker2.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network bridge=br0,model=virtio \
--boot menu=on --cdrom /data/ocp4/worker-2.iso

等大概5分鐘時間bootstrap會ready, 能夠ready之後在創建其餘虛擬機

等待一段時間後

在helper節點上經過命令查看安裝進度

openshift-install wait-for bootstrap-complete --log-level debug

處理完存儲後，仍是在helper節點

[root@helper ocp4]# openshift-install wait-for install-complete
INFO Waiting up to 30m0s for the cluster at https://api.ocp4.redhat.ren:6443 to initialize... 
INFO Waiting up to 10m0s for the openshift-console route to be created... 
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/ocp4/auth/kubeconfig' 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp4.redhat.ren 
INFO Login to the console with user: kubeadmin, password: WRTp9-avPVu-IMWLX-KiIQ2

5. 關於bootstrap節點不ready問題

剛開始的時候，haproxy界面中bootstrap一直不ready,登陸到helper上去 sudo -i, podman images看到鏡像爲空。

查看192.168.7.1的registry服務，發現shake hand error.

須要更新install-config.yaml中的additionalTrustBundle,和/etc/crts/redhat.ren.crt 一致

install-config.yaml中須要修改的部分用粗體標出

apiVersion: v1
baseDomain: redhat.ren
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 3
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp4
networking:
  clusterNetworks:
  - cidr: 10.254.0.0/16
    hostPrefix: 24
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: '{"auths":{"registry.redhat.ren": {"auth": "ZHVtbXk6ZHVtbXk=","email": "noemail@localhost"}}}' sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnejC+QkKXqEOj7lSKxpHnnIxPli2iwNveE9apd0QUFgc3xTyaQWyOqbFEsUzR2MnXV36a89DiOVnecVgXZqVDFrDZDRkMLKJTm2U85AExWE0Lmtkxpmyg5OdpFmTBCutpNy2LigG8LTkMPXIgDrfNF+37/BvKzvWdrhR6/dQwqfMGqfRi+PYscD6nUJG5kAzVugalyw8+Sv9CzS+4BMRCZ4EVKu5bB2wl1bw7KCJc+D0nhnc87qGswJquleT7CGi7N2k6/Q1iK80l1KymmwWcwvh+Yf4Nhdk4cxbeSZmPGBQIQMmOUzK0Q4xs3XZd2WvZd/NYj0D83sSCQGXEUkGL root@helper' additionalTrustBundle: |
  -----BEGIN CERTIFICATE----- MIIDszCCApugAwIBAgIJAPRFC4yzZOpxMA0GCSqGSIb3DQEBCwUAMHAxCzAJBgNV BAYTAkNOMQswCQYDVQQIDAJHRDELMAkGA1UEBwwCU1oxGDAWBgNVBAoMD0dsb2Jh bCBTZWN1cml0eTEWMBQGA1UECwwNSVQgRGVwYXJ0bWVudDEVMBMGA1UEAwwMKi5y ZWRoYXQucmVuMB4XDTE5MTAxODEwMTAzMFoXDTI5MTAxNTEwMTAzMFowcDELMAkG A1UEBhMCQ04xCzAJBgNVBAgMAkdEMQswCQYDVQQHDAJTWjEYMBYGA1UECgwPR2xv YmFsIFNlY3VyaXR5MRYwFAYDVQQLDA1JVCBEZXBhcnRtZW50MRUwEwYDVQQDDAwq LnJlZGhhdC5yZW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDA1Mgq hGebtpCx93KtaaRw5jDRbxrTdJkZvV6Wyq1BYFRQDKZ3QOcFFOMrLbN7g8Nrw1dl zgvKLLc1l4god12RgOiM1fOVODoLIk2Z0x2VFbQ7ZIx0jKdKmaNex/fGd/MoLhij dYtAmZokjs7sw0VNkZLlHzPgR9AXYtJp07zUUL1eRWNTOhO8LxDUviOg2eVy31yW TrYla1ze7+meTvZs3edr5/dLncZ2PCiyaF6hOEf/t7ev4vA33p6SUY6prgaPaKlb PiB8+7ZKsucgXd/ikKoCP/0rMcqRSIrpYuudM8Dff8OGxhfL0ChUx3VkKd2t5T3l N3717qj+siuUb7OLAgMBAAGjUDBOMB0GA1UdDgQWBBTwuyzX5stt+Pyrs7VIr508 1VMR8zAfBgNVHSMEGDAWgBTwuyzX5stt+Pyrs7VIr5081VMR8zAMBgNVHRMEBTAD AQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBhhicfn9fY+PAxnVNn7R0PscxbYof4DVv3 lqkkO6BCLkHUivljxjU7OYpxkva34vSuK1WVZf74Mbif7NkzVS3EG0+b0h+8EcQ+ Fnv4qyKBfs8LG/V/A0ukAD5AYP098jsj5tmREbnFbMy7UojVEK54w6262iefvg0b uT5I0Y3jLljIlsxSbX4tTXjX0X/KHXK4PJ7hqdRLXnD4CgWKHjU6yNQS+sZg83VC jsZpKl5eSBqOdXB1CFteZm571/AXlagcyGf9hvK4fV2ybQoOxgkZt9zyUvtm3myb S5FAo4B5IvEhkge+jvolj31AWnB4v6GX0TgWotJd52GUpWDJDr5T -----END CERTIFICATE-----
imageContentSources:
- mirrors:
  - registry.redhat.ren/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - registry.redhat.ren/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

6.證書過時問題

登陸helper經過命令行去查安裝進度

須要從新刪除openshift-install create ignition-configs 生成的部分。從下面這段從新開始

/bin/rm -rf *.ign .openshift_install_state.json auth bootstrap master0 master1 master2 worker0 worker1 worker2

openshift-install create ignition-configs --dir=/root/ocp4

固然個人問題是各機器的時間不一樣步，設置完時間同步後問題從新作問題解決。

安裝完後訪問

https://console-openshift-console.apps.ocp4.redhat.ren

美中不足是Operatorhub沒有內容，也須要離線安裝

在Helper機器上

cd ~/ocp4
export KUBECONFIG=auth/kubeconfig

[root@helper ocp4]# oc get nodes
NAME                       STATUS   ROLES    AGE   VERSION
master-0.ocp4.redhat.ren   Ready    master   71m   v1.14.6+c07e432da
master-1.ocp4.redhat.ren   Ready    master   71m   v1.14.6+c07e432da
master-2.ocp4.redhat.ren   Ready    master   71m   v1.14.6+c07e432da
worker-0.ocp4.redhat.ren   Ready    worker   71m   v1.14.6+c07e432da
worker-1.ocp4.redhat.ren   Ready    worker   71m   v1.14.6+c07e432da
worker-2.ocp4.redhat.ren   Ready    worker   71m   v1.14.6+c07e432da

裝機現場

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。