Harbor 私有鏡像倉庫

時間 2019-12-18

標籤 harbor 私有鏡像倉庫简体版

原文原文鏈接

Harbor 簡介
使用 DNSMASQ 快速搭建簡單 DNS 服務
爲 Harbor 簽發域名證書
信任自簽發的域名證書
Harbor 1.8 版本配置與安裝
鏡像管理與安全: 漏洞掃描和鏡像簽名
鏡像複製與同步
Harbor HA: 環境與準備
Harbor HA: 修改配置
Harbor HA: 啓動 Harbor
Harbor HA: keepalived 安裝配置與測試
OpenLDAP 安裝與配置
OpenLDAP 建立組織架構與用戶和 Harbor 配置

Harbor 簡介html

簡介
- VMware 的開源項目 https://github.com/vmware/harbor
- Harbor 可幫助用戶迅速搭建企業級的 Registry 服務, 它提供了管理圖形界面, 基於角色的訪問控制 ( Role Based Access Control), 鏡像遠程複製 (同步), AD/LDAP 集成, 以及審計日誌等企業用戶需求的功能, 同時還原生支持中文, 深受中國用戶的喜好;
- 該項目自退出以來, 在 GitHub 得到了超過 3300 多個 star 和 900 多個 forks (截至 2017-12-25)
主要功能
- 基於角色的訪問控制: 用戶與 Docker 鏡像倉庫經過 "項目" 進行組織管理, 一個用戶能夠對多個鏡像倉庫在同一命名空間(project)裏有不一樣的權限
- 圖形化用戶界面: 用戶能夠經過瀏覽器來瀏覽, 檢索當前 Docker 鏡像倉庫, 管理項目和命名空間
- 審計管理: 全部針對鏡像倉庫的操做均可以被記錄追溯, 用於審計管理
- 國際化: 基於英文與中文語言進行了本地化, 能夠增長更多的語言信息
- RESTful API: 提供給管理員對於 Harbor 更多的操控, 使得與其它管理軟件集成變得更容易
- LDAP 認證
- 基於策略的 Docker 鏡像複製功能, 可在不一樣的數據中心, 不一樣的運行環境之間同步鏡像, 並提供友好的管理界面, 大大簡化了實際運維中的鏡像管理工做
- 與 Clair 集成, 添加漏洞掃描功能(能夠設置漏洞級別, 當漏洞級別達到必定程度時, 限制鏡像的拉取) . Clair 是 coreos 開源的容器漏洞掃描工具, 在容器逐漸普及的幾天, 容器鏡像安全問題日益嚴重, Clair 是目前少數的開源安全掃描工具
- Notary 是 Docker 鏡像的簽名工具, 用來保證鏡像在 pull , push 和傳輸過程當中的一致性和完整性, 避免中間人攻擊, 避免非法的鏡像更新和運行

使用 DNSMASQ 快速搭建簡單 DNS 服務java

環境 DNS Server(DNS服務器) / Demo Server (測試DNS服務器);

CPU 1核↑ CPU 1核↑node

內存 512M↑ 內存 512M↑python

磁盤 10G↑ 磁盤 10G↑mysql

IP:192.168.9.28(自定義) IP: 192.168.9.30(自定義)linux

目標
- 快速安裝本地 DNS 服務器;
- 使用 DNS 來自動設置主機名;

步驟
- 安裝 dnsmasq
- 修改配置
- 啓動 DNS 服務
- 測試

// DNS 服務器安裝 dnsmasq
[root@localhost ~]# yum -y install dnsmasq

// 修改配置文件, 添加一行指定文件(存放dns地址)位置
[root@localhost ~]# vim /etc/dnsmasq.conf 
... ...
110 #addn-hosts=/etc/banner_add_hosts
111 addn-hosts=/etc/domains
... ...

// 修改 /etc/domains 文件, 格式同 hosts 文件相同, IP 地址對應主機域名
[root@localhost ~]# vim /etc/domains
192.168.9.30    test-dns.mine.com

// 啓動 dnsmasq 服務, 並設置開機啓動
[root@localhost ~]# /etc/init.d/dnsmasq start
Starting dnsmasq:                                          [  OK  ]
[root@localhost ~]# chkconfig --level 35 dnsmasq on

// 修改測試機的 dns 
[root@tian ~]# vim /etc/resolv.conf 
nameserver 192.168.9.28

// 肯定測試機能夠ping 通, 及防火牆
[root@rong ~]# vim /etc/sysctl.conf net.ipv4.icmp_echo_ignore_all = 0
[root@rong ~]# sysctl -p
[root@rong ~]# echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_all // 臨時開啓
[root@rong ~]# ping test-dns.mine.com
PING test-dns.mine.com (192.168.9.30) 56(84) bytes of data.
64 bytes from test-dns.mine.com (192.168.9.30): icmp_seq=1 ttl=64 time=0.016 ms

// 使用 dig 或者 nslookup 跟蹤 test-dns.mine.com; dig中可看到 A 記錄 test-dns.mine.com 解析192.168.9.30
[root@rong ~]# dig test-dns.mine.com

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> test-dns.mine.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20131
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;test-dns.mine.com.        IN    A

;; ANSWER SECTION:
test-dns.mine.com.    0    IN    A    192.168.9.30

;; Query time: 0 msec
;; SERVER: 192.168.9.28#53(192.168.9.28)
;; WHEN: Sun Jun 30 23:41:14 EDT 2019
;; MSG SIZE  rcvd: 51

[root@rong ~]# nslookup test-dns.mine.com
Server:        192.168.9.28
Address:    192.168.9.28#53

Name:    test-dns.mine.com
Address: 192.168.9.30

爲 Harbor 簽發域名證書nginx

// Linux中本身手動簽發 ssl 證書
// 建立一個存放 證書 的目錄
mkdir -p /data/ssl && cd /data/ssl

// 使用 RSA 方式加密, 生成 ca.key , 長度爲 3072
[root@localhost ssl]# openssl genrsa -out ca.key 3072
Generating RSA private key, 3072 bit long modulus
...............................................................................++
.........................................................................................................................++
e is 65537 (0x10001)

// 經過 key 生成證書, 有效期爲 1095 = 三年
[root@localhost ssl]# openssl req -new -x509 -days 1095 -key ca.key -out ca.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN                 // 國家
State or Province Name (full name) []:BJ             // 州或者省名
Locality Name (eg, city) [Default City]:BJ           // 城市名稱
Organization Name (eg, company) [Default Company Ltd]:zxjr      // 組織名稱
Organizational Unit Name (eg, section) []:           // 組織單位名稱(空)
Common Name (eg, your name or your server's hostname) []:       // 你的名稱或服務器主機名
Email Address []:                                               // 郵箱

// 生成域名的證書
[root@localhost ssl]# openssl genrsa -out wap.zxjr.com.key 3072
Generating RSA private key, 3072 bit long modulus
........++
.......++
e is 65537 (0x10001)

// 生成證書請求,, 是後面簽發證書時所須要的
[root@localhost ssl]# openssl req -new -key wap.zxjr.com.key -out wap.zxjr.com.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:BJ
Locality Name (eg, city) [Default City]:BJ
Organization Name (eg, company) [Default Company Ltd]:zxjr
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:wap.zxjr.com              // 填寫域名,
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

// 生成域名證書
[root@localhost ssl]# openssl x509 -req -in wap.zxjr.com.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out wap.zxjr.com.pem -days 1095
Signature ok
subject=/C=CN/ST=BJ/L=BJ/O=zxjr/CN=wap.zxjr.com
Getting CA Private Key

// 查看生成的證書
[root@localhost ssl]# ll
total 24
-rw-r--r-- 1 root root 2455 Jul  1 02:06 ca.key
-rw-r--r-- 1 root root 1533 Jul  1 02:09 ca.pem
-rw-r--r-- 1 root root   17 Jul  1 02:26 ca.srl
-rw-r--r-- 1 root root 1314 Jul  1 02:21 wap.zxjr.com.csr
-rw-r--r-- 1 root root 2455 Jul  1 02:15 wap.zxjr.com.key
-rw-r--r-- 1 root root 1448 Jul  1 02:26 wap.zxjr.com.pem

// 確認一下 wap.zxjr.com.pem 證書
[root@localhost ssl]# openssl x509 -noout -text -in wap.zxjr.com.pem
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            c8:19:22:49:3d:f8:f1:b3
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=BJ, L=BJ, O=zxjr
        Validity           // 有效期從 2019年7月1號到 2022年6月30號.
            Not Before: Jul  1 06:26:03 2019 GMT
            Not After : Jun 30 06:26:03 2022 GMT
        Subject: C=CN, ST=BJ, L=BJ, O=zxjr, CN=wap.zxjr.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (3072 bit)
                Modulus:
                    00:ac:11:ce:8e:e6:f9:a4:18:2a:24:f2:e3:d8:af:
                    60:8d:8f:ef:b1:ef:47:de:07:29:7c:3e:cd:f1:b2:
                    80:1c:d6:11:2c:41:1d:35:13:9b:8a:8b:c6:9b:ac:
                    44:34:fa:fd:c0:24:f9:7f:b1:77:d7:bb:59:2b:99:
                    64:6a:25:8e:0f:77:f1:38:eb:bd:dc:da:a9:70:40:
                    bd:6a:ca:0e:bc:3d:0a:08:d2:77:55:fa:06:31:df:
                    eb:26:23:fe:fd:ba:5f:d2:4c:41:3f:f8:bd:b9:36:
                    ab:64:51:63:e4:5b:a8:ed:1c:da:8c:a6:6f:93:9b:
                    57:dd:fc:03:b5:62:67:ce:f5:a9:39:a3:da:0a:03:
                    7b:18:f7:95:aa:1b:f9:6e:80:88:f8:44:8b:58:91:
                    47:66:ff:a0:af:d7:b8:6d:d7:fa:23:e3:7a:54:4d:
                    61:2c:8c:26:5f:d4:d4:14:71:75:69:a7:68:f9:7b:
                    e8:08:96:6a:5c:6f:f2:8f:0f:b1:88:b2:fc:db:67:
                    2f:2f:c7:30:05:ba:ed:b0:a5:de:77:55:45:5d:7f:
                    36:b0:93:a8:06:f3:96:86:ee:b6:db:ef:8e:54:b0:
                    ee:97:85:5f:45:29:e9:c7:a5:9a:ce:a9:ef:3a:b0:
                    f5:31:9e:b5:d8:3d:87:f1:d0:18:92:d9:74:a4:29:
                    d2:6a:ab:63:d6:17:d1:b2:2f:ec:04:6c:d0:cc:de:
                    6b:54:ab:b4:56:5c:71:36:43:6e:37:03:2b:db:de:
                    ab:fe:5c:0c:de:fd:c9:60:cd:58:f1:86:1b:fe:39:
                    df:f7:06:2e:d3:5a:26:13:78:fa:7d:33:61:8d:59:
                    5d:ee:32:9d:74:9c:94:1e:a1:c3:22:61:05:6b:20:
                    a7:87:28:fb:7f:55:04:9d:c2:00:b4:30:59:32:89:
                    3a:4b:7e:27:df:5f:f7:ed:a5:23:32:ec:b6:7f:e0:
                    3c:3b:b4:a0:ae:cf:4a:89:8f:b9:41:d3:d9:78:4f:
                    df:ce:ed:82:b0:cd:1d:86:e7:11
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         23:90:b7:cc:cf:a5:ee:10:f2:37:a0:42:42:50:cf:30:59:71:
         46:e5:53:1e:13:9b:a5:5e:95:7c:ea:c5:44:0c:6d:71:55:e1:
         53:ac:ae:f2:66:2d:4e:76:ea:00:7d:7b:f7:93:6d:77:e6:0e:
         59:3c:33:7e:91:17:75:c8:49:e3:14:65:07:51:f2:69:cd:7f:
         ee:34:a3:97:5f:4e:1b:27:f6:93:7b:95:7f:2f:30:83:58:35:
         c6:74:86:f8:86:05:c9:53:38:e3:71:06:07:a4:8b:57:6c:b4:
         69:19:30:93:3a:82:04:d6:93:fd:64:3c:5c:23:56:61:73:8d:
         d1:93:7b:2c:ab:69:60:72:32:20:82:33:f7:af:71:fb:7e:48:
         aa:df:e0:dc:14:f1:e8:9d:64:b8:89:92:e4:f2:5b:ed:f8:ca:
         a6:1b:ce:ea:20:8a:73:9e:5d:73:fa:e5:4f:fe:c6:60:43:ff:
         23:c2:5b:79:db:61:a1:f9:47:48:a5:cf:e5:ee:cc:bb:6c:f8:
         3f:31:bd:70:a3:55:22:2b:c1:f9:9d:29:67:89:74:bd:d5:af:
         2f:4c:df:72:87:20:d4:9c:2b:1f:f9:33:f2:04:d2:06:56:9b:
         63:74:01:16:6f:66:32:e1:f3:56:d1:51:94:ad:09:6c:a5:a5:
         40:17:2f:42:a4:31:28:a4:cb:07:05:86:ca:6d:e5:02:84:57:
         ac:75:06:78:11:96:3c:b2:e1:76:a1:91:c7:7a:ce:eb:84:cc:
         55:e8:5b:f5:2e:fa:da:af:38:8d:63:9e:74:e0:79:56:7c:e9:
         de:c7:96:c2:44:91:9e:95:4c:dd:2d:d5:3d:bf:99:6a:61:22:
         21:78:8b:a1:49:8b:d7:74:3c:69:57:88:f1:f0:3a:81:16:62:
         c3:29:22:e8:0c:16:55:5c:cd:81:63:71:40:0c:11:a4:28:e4:
         9b:6c:d2:a5:2e:6d:99:59:24:34:9c:83:9f:e9:1e:71:27:ca:
         40:bd:e1:5b:25:0c

信任自簽發的域名證書git

// 證書是咱們在服務器上自簽發的, 服務器不信任, 須要添加信任
[root@localhost ssl]# cp ca.pem /etc/pki/ca-trust/source/anchors/
[root@localhost ssl]# update-ca-trust enable
[root@localhost ssl]# update-ca-trust extract   // 更新版本

Harbor 1.8 版本配置與安裝github

// Harbor 對文件系統, 分區沒有什麼要求, 首先建立目錄,並進入
[root@localhost ~]# mkdir -p /data/harbor
[root@localhost ~]# cd /data/harbor

// 安裝 docker , 前面有安裝部署docker-ce , 並啓動
https://www.cnblogs.com/haorong/p/11008652.html

 // 將證書文件放置到 /etc/ssl/harbor 目錄下
[root@localhost harbor]# ll /etc/ssl/harbor/
total 8
-rw-r--r-- 1 root root 2455 Jul  1 03:11 wap.zxjr.com.key
-rw-r--r-- 1 root root 1448 Jul  1 03:11 wap.zxjr.com.pem

// 使用最新的穩定版本 Latest release, 使用離線版本, 如果下載慢的話, 有一個鏡像網站 http://harbor.orientsoft.cn/; 可下載對應版本的包;
// 搜索地址: https://github.com/goharbor/harbor/releases
[root@localhost harbor]# curl -LO https://storage.googleapis.com/harbor-releases/release-1.8.0/harbor-offline-installer-v1.8.0.tgz

// 解壓
tar xf harbor-offline-installer-v1.8.0.tgz

// harbor的配置文件從1.8.0 之後, 由 harbor.cfg 改成 harbor.yml; 修改配置文件
[root@localhost harbor]# vim harbor.yml 

  1 # Configuration file of Harbor
  2 
  3 # The IP address or hostname to access admin UI and registry service.
  4 # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
  5 hostname: wap.zxjr.com
  6 
  7 # http related config
  8 # http:
  9   # port for http, default is 80. If https enabled, this port will redirect to https port
 10   # port: 80
 11 
 12 # https related config
 13 https:          // 使用 https
 14 #   # https port for harbor, default is 443
 15     port: 443
 16 #   # The path of cert and key files for nginx
 17 #   certificate: /your/certificate/path
 18 #   private_key: /your/private/key/path
 19     certificate: /etc/ssl/harbor/wap.zxjr.com.pem        // ssl證書
 20     private_key: /etc/ssl/harbor/wap.zxjr.com.key
... ...

// 安裝docker-compose, 須要epel源;
[root@localhost yum.repos.d]# yum -y install epel-release
[root@localhost yum.repos.d]# yum -y install python-pip
[root@localhost yum.repos.d]# pip install docker-compose
　　如有報錯'''
　　　　You are using pip version 8.1.2, however version 19.2.2 is available.
　　　　You should consider upgrading via the 'pip install --upgrade pip' command
　　''' 須要先升級pip
[root@localhost yum.repos.d]# pip install --upgrade pip
[root@localhost yum.repos.d]# yum -y install docker-compose

// 執行install.sh, 啓動鏡像簽名 --with-notary , 啓動漏洞掃描 --with-clair; 如果有docker容器,須要清空後再運行下面命令;
[root@localhost harbor]# ./install.sh --with-notary --with-clair

// 查看運行的容器
[root@wap harbor]# docker ps -a
CONTAINER ID        IMAGE                                               COMMAND                  CREATED             STATUS                    PORTS                                                              NAMES
e3ea64a9c609        goharbor/nginx-photon:v1.8.0                        "nginx -g 'daemon of…"   51 seconds ago      Up 49 seconds (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp   nginx
dd989d38885b        goharbor/notary-server-photon:v0.6.1-v1.8.0         "/bin/server-start.sh"   52 seconds ago      Up 50 seconds                                                                                notary-server
3a8b664a60bb        goharbor/harbor-jobservice:v1.8.0                   "/harbor/start.sh"       53 seconds ago      Up 51 seconds                                                                                harbor-jobservice
26e23dd78065        goharbor/harbor-portal:v1.8.0                       "nginx -g 'daemon of…"   53 seconds ago      Up 51 seconds (healthy)   80/tcp                                                             harbor-portal
... ...

// 下面使用瀏覽器訪問,須要在本機上對 hosts文件中 wap.zxjr.com 作解析

鏡像管理與安全: 漏洞掃描和鏡像簽名redis

上傳鏡像 --> 鏡像掃描

// 倉庫的使用, 首先將鏡像從新打標籤
[root@wap ~]# docker images |grep centos
centos                          7                          9f38484d220f        3 months ago        202MB
[root@wap ~]# docker tag centos:7 wap.zxjr.com/library/centos:7

// 須要在本機中的hosts文件添加解析
192.168.9.29     wap.zxjr.com

// 推送以前, 須要提早登陸, 不然requested access to the resource is denied
[root@wap ~]# docker login -u admin wap.zxjr.com
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[root@wap ~]# docker push wap.zxjr.com/library/centos:7
The push refers to repository [wap.zxjr.com/library/centos]
d69483a6face: Pushed 
7: digest: sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66 size: 529

// 刷新 harbor 倉庫, centos:7 鏡像已經上傳, 進行漏洞掃描;

// 再上傳一個鏡像
[root@wap ~]# docker tag alpine wap.zxjr.com/library/alpine:latest
[root@wap ~]# docker push wap.zxjr.com/library/alpine:latest
The push refers to repository [wap.zxjr.com/library/alpine]
256a7af3acb1: Pushed 
latest: digest: sha256:97a042bf09f1bf78c8cf3dcebef94614f2b95fa2f988a5c07314031bc2570c7a size: 528

// 再次查看 harbor 倉庫; 掃描那仍是須要手動掃描...

鏡像複製與同步

Harbor的主從複製, 一對多.
再起一個 Harbor; 按上面步驟走, 簽發並信任域名證書, 安裝 Harbor
應用場景: 公司存在多個機房, 跨機房harbor下載鏡像慢, 多個機房存在多個 Harbor
上面倉庫配置時勾選 "阻止潛在漏洞鏡像" 後, 鏡像存在漏洞, 匹配到嚴重/中等/較低(本身設置的) 後, 不會將此鏡像同步到 harbor2 中;

隨後再生成鏡像後,推送到harbor上, 就會自動同步到 harbor2上

Harbor HA: 環境與準備

Harbor Server 2 臺 -- 9.29 9.30
運行組件: (無狀態的組件, 它們的數據都保存在數據庫中的)
- AdminServer
- Proxy
- Registry
- UI
- Logs
- Clair
- JobService
Harbor HA: 文件存儲(儘可能也使用高可用, NFS是便於使用, 生產中不要用 NFS)
- 共享存儲使用 NFS
  - 192.168.9.31:/data/nfs
- 支持: (可使用下面標示的文件系統, 可高可用)
  - Swift
  - NFS
  - S3
  - Azure
  - GCS
  - ClusterFS
  - Ceph
  - OSS
Harbor HA: 數據庫
- Harbor 數據庫( MySQL ) 192.168.9.32:3306
- Clair 數據庫 ( PostgreSQL ) 192.168.9.32:5432
- Session ( Redis ) 192.168.9.32:6379
Harbor HA: Keepalived
- VIP: 192.168.9.33
  - Harbor1: 192.168.9.29
  - Harbor2: 192.168.9.30

// 9.31 上安裝 nfs
mkdir /data/nfs                                   // 建立儲存目錄
yum -y install nfs-utils                          // 安裝nfs
vim /etc/exports                                  // 編輯配置文件, 將目錄發不出去;9網段可讀可寫,
/data/nfs       192.168.9.0/24(rw,no_root_squash)
    // (1) Ro 該主機對該共享目錄有隻讀權限 
    // (2) Rw 該主機對該共享目錄有讀寫權限 
    // (3) Root_squash 客戶機用root用戶訪問該共享文件夾時，將root用戶映射成匿名用戶 
    // (4) No_root_squash 客戶機用root訪問該共享文件夾時，不映射root用戶 
    // (5) All_squash 客戶機上的任何用戶訪問該共享目錄時都映射成匿名用戶 
    // (6) Anonuid 將客戶機上的用戶映射成指定的本地用戶ID的用戶 
    // (7) Anongid 將客戶機上的用戶映射成屬於指定的本地用戶組ID 
    // (8) Sync 資料同步寫入到內存與硬盤中 
    // (9) Async 資料會先暫存於內存中，而非直接寫入硬盤 
    // (10) Insecure 容許從這臺機器過來的非受權訪問  
    // (11) subtree_check           若是共享/usr/bin之類的子目錄時，強制NFS檢查父目錄的權限（默認） 
    // (12) no_subtree_check        和上面相對，不檢查父目錄權限 
    // (13) wdelay                  若是多個用戶要寫入NFS目錄，則歸組寫入（默認） 
    // (14 )no_wdelay               若是多個用戶要寫入NFS目錄，則當即寫入，當使用async時，無需此設置。 
    // (15) hide                       在NFS共享目錄中不共享其子目錄 
    // (16) no_hide                 共享NFS目錄的子目錄 
    // (17) secure                  NFS經過1024如下的安全TCP/IP端口發送 
    // (18) insecure                NFS經過1024以上的端口發送 

// 啓動 nfs 並設置開機啓動
systemctl start nfs
systemctl enable nfs

// 兩個 harbor 節點掛載目錄; 掛載的客戶端上也須要安裝 nfs
yum -y install nfs-utils
mkdir /data/public
mount -t nfs 192.168.9.31:/data/nfs /data/public

// 查看掛載
df -hT |grep 192
192.168.9.31:/data/nfs nfs4       10G   32M   10G   1% /data/public

// 測試, harbor1 上建立測試文件, 在harbor2和nfs上查看, 存在...
echo "12345 test nfs" > /data/public/test
cat /data/public/test                          // harbor2 上
12345 test nfs
cat /data/nfs/test                             // nfs 上
12345 test nfs

// 9.32 上安裝 docker, 並使用docker啓動 redis
yum -y install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo=http://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl start docker
systemctl enable docker
docker run -d -p 6379:6379 redis:alpine
[root@localhost ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
99e7f00fddb6        redis:alpine        "docker-entrypoint.s…"   16 minutes ago      Up 16 minutes       0.0.0.0:6379->6379/tcp   elegant_bartik
[root@localhost ~]# docker inspect -f {{.NetworkSettings.Networks.bridge.IPAddress}} 99e
172.17.0.2

// 9.32 上使用 docker 啓動 PostgreSQL 數據庫; -e 參數後面這裏指定的是數據庫的密碼
docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=harborp postgres
[root@localhost ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
0d94c22fd68d        postgres            "docker-entrypoint.s…"   11 minutes ago      Up 11 minutes       0.0.0.0:5432->5432/tcp   loving_morse
99e7f00fddb6        redis:alpine        "docker-entrypoint.s…"   About an hour ago   Up About an hour    0.0.0.0:6379->6379/tcp   elegant_bartik
[root@localhost ~]# docker inspect -f {{.NetworkSettings.Networks.bridge.IPAddress}} 0d9
172.17.0.3

// 使用 docker 運行 mysql
docker run -d -e MYSQL_ROOT_PASSWORD=harbor -p 3306:3306 mysql:5.6
[root@localhost ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
d2c1db976e0a        mysql:5.6           "docker-entrypoint.s…"   5 minutes ago       Up 5 minutes        0.0.0.0:3306->3306/tcp   affectionate_tesla
0d94c22fd68d        postgres            "docker-entrypoint.s…"   2 hours ago         Up 2 hours          0.0.0.0:5432->5432/tcp   loving_morse
99e7f00fddb6        redis:alpine        "docker-entrypoint.s…"   2 hours ago         Up 2 hours          0.0.0.0:6379->6379/tcp   elegant_bartik

// 爲三個容器更更名稱
[root@localhost ~]# docker rename d2c mysql
[root@localhost ~]# docker rename 0d9 clair
[root@localhost ~]# docker rename 99e session
[root@localhost ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
d2c1db976e0a        mysql:5.6           "docker-entrypoint.s…"   6 minutes ago       Up 6 minutes        0.0.0.0:3306->3306/tcp   mysql
0d94c22fd68d        postgres            "docker-entrypoint.s…"   2 hours ago         Up 2 hours          0.0.0.0:5432->5432/tcp   clair
99e7f00fddb6        redis:alpine        "docker-entrypoint.s…"   2 hours ago         Up 2 hours          0.0.0.0:6379->6379/tcp   session

Harbor HA: 修改配置

// 此處操做的是 harbor 1.4 版本的操做, /usr/local/harbor/ 目錄下有 ha/ 目錄;
cd /usr/local/harbor
ll ha/
total 24
-rw-r--r-- 1 root root  603 Feb  6  2018 docker-compose.clair.tpl
-rw-r--r-- 1 root root  599 Feb  6  2018 docker-compose.clair.yml
-rw-r--r-- 1 root root 2959 Feb  6  2018 docker-compose.tpl
-rw-r--r-- 1 root root 2926 Feb  6  2018 docker-compose.yml
-rw-r--r-- 1 root root 7630 Feb  6  2018 registry.sql
drwxr-xr-x 4 root root   49 Feb  6  2018 sample

// 鏈接 mysql ; 並將 registry.sql 導入, 沒有 registry 數據庫的話, 直接導入就行; 藥是存在 registry 數據庫的話, 先將庫裏的數據導出來, 再導入新的;
mysql -uroot -pharbor -h 192.168.9.32 -P 3306
MySQL [(none)]> source ha/registry.sql
MySQL [registry]> show databases;                                    // 查看數據庫, registry已建立好;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| registry           |
+--------------------+
4 rows in set (0.00 sec)                        
MySQL [registry]> use registry;                                      // 進入 registry 庫
Database changed
MySQL [registry]> show tables;                                       // 查看庫中的表
+-------------------------------+
| Tables_in_registry            |
+-------------------------------+
| access                        |
| access_log                    |
| alembic_version               |
| clair_vuln_timestamp          |
| img_scan_job                  |
| img_scan_overview             |
| project                       |
| project_member                |
| project_metadata              |
| properties                    |
| replication_immediate_trigger |
| replication_job               |
| replication_policy            |
| replication_target            |
| repository                    |
| role                          |
| user                          |
+-------------------------------+
17 rows in set (0.00 sec)                                                                         

// 修改 docker-compose.yml 配置文件中數據存放的路徑, /data/public 爲掛載目錄;
vim ha/docker-compose.yml 
  1 version: '2'
  2 services:
  3   log:
  4     image: vmware/harbor-log:v1.4.0
  5     container_name: harbor-log
  6     restart: always
  7     volumes:
  8       - /data/public/:/var/log/docker/:z
  ... ...

// 修改 harbor.cfg; 數據庫 redis Postgres 鏈接信息
vim harbor.cfg 
  1 ## Configuration file of Harbor
  2 
  3 #The IP address or hostname to access admin UI and registry service.
  4 #DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
  5 hostname = wap.zxjr.com                    // 修改成harbor服務器的域名;
  6 
  7 #The protocol for accessing the UI and token/notification service, by default it is http.
  8 #It can be set to https if ssl is enabled on nginx.
  9 ui_url_protocol = https                    // 使用https;
 10 
 11 #Maximum number of job workers in job service  
 12 max_job_workers = 3 
 13 
 14 #Determine whether or not to generate certificate for the registry's token.
 15 #If the value is on, the prepare script creates new root cert and private key 
 16 #for generating token to access the registry. If the value is off the default key/cert will b    e used.
 17 #This flag also controls the creation of the notary signer's cert.
 18 customize_crt = on
 19 
 20 #The path of cert and key files for nginx, they are applied only the protocol is set to https
 21 ssl_cert = /etc/ssl/harbor/wap.zxjr.com.pem                    // 使用生成的簽名證書;
 22 ssl_cert_key = /etc/ssl/harbor/wap.zxjr.com.key
 ... ...
 110 #######Harbor DB configuration section#######
111 
112 #The address of the Harbor database. Only need to change when using external db.
113 db_host = 192.168.9.32               // 配置數據庫的ip地址;
114 
115 #The password for the root user of Harbor DB. Change this before any production use.
116 db_password = harbor                 // 更改成docker啓動時使用的密碼;
117 
118 #The port of Harbor database host
119 db_port = 3306
120 
121 #The user name of Harbor database
122 db_user = root
123 
124 ##### End of Harbor DB configuration#######
125 
126 #The redis server address. Only needed in HA installation.
127 redis_url = 192.168.9.32:6379        // 配置 redis 的ip + 端口
128 
129 ##########Clair DB configuration############
130 
131 #Clair DB host address. Only change it when using an exteral DB.
132 clair_db_host = 192.168.9.32         // postgres 數據庫的地址;
133 
134 #The password of the Clair's postgres database. Only effective when Harbor is deployed with C    lair.
135 #Please update it before deployment. Subsequent update will cause Clair's API server and Harb    or unable to access Clair's database.
136 clair_db_password = harborp
137 
138 #Clair DB connect port
139 clair_db_port = 5432
140 
141 #Clair DB username
142 clair_db_username = postgres
143 
144 #Clair default database
145 clair_db = postgres

// 修改common/templates/registry/config_ha.yml
vim common/templates/registry/config_ha.yml
version: 0.1
log:
  level: debug
  fields:
    service: registry
storage:
  cache:
    layerinfo: redis                      // 緩存使用 redis
  $storage_provider_info
  maintenance:
    uploadpurging:
      enabled: false
  delete:
    enabled: true
redis:
  addr: $redis_url                        // redis地址
  db: 0

Harbor HA: 啓動 Harbor

// 啓動 harbor
cd /usr/local/harbor 
./install.sh --with-clair --ha

// 第一臺啓動成功後, 配置第二臺harbor, 相同的配置, 能夠將第一臺的harbor目錄直接拷貝到第二臺上, 並將生成的證書也拷貝
cd /usr/local
scp -r harbor/ 192.168.9.30:/usr/local/
cd /etc/ssl/harbor
scp wap.zxjr.com.* 192.168.9.30:$PWD            // $PWD表示當前目錄

// 在第二臺 harbor 服務器上查看,harbor目錄及證書都存在後, 啓動 harbor
cd /usr/local/harbor
./install.sh --with-clair --ha

Harbor HA: keepalived 安裝配置與測試

// 第一臺harbor安裝 keepalived
yum -y install keepalived

// 修改keepalived配置文件; 可參考 https://github.com/goharbor/harbor/blob/release-1.4.0/make/ha/sample/active_active/keepalived_active_active.conf
cd /etc/keepalived
cp keepalived.conf keepalived.conf.original                // 備份原配置文件
vim keepalived.conf
! Configuration File for keepalived

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.9.33
    }
}

virtual_server 192.168.9.33 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    persistence_timeout 50
    protocol TCP

    real_server 192.168.9.29 80 {
        weight 1
    }
}

// 配置健康檢查腳本 check.sh ; 官方文檔中 https://github.com/goharbor/harbor/blob/release-1.4.0/make/ha/sample/active_active/check.sh
vim /usr/local/bin/check.sh
#!/bin/bash

set -e
#get protocol

#LOG=/var/log/keepalived_check.log
nodeip=$1
nodeaddress="http://${nodeip}"
http_code=`curl -s -o /dev/null -w "%{http_code}" ${nodeaddress}`

if [ $http_code == 200 ] ; then
  protocol="http"
elif [ $http_code == 301 ]
then
  protocol="https"
else
#  echo "`date +"%Y-%m-%d %H:%M:%S"` $1, CHECK_CODE=$http_code" >> $LOG
  exit 1
fi

systeminfo=`curl -k -o - -s ${protocol}://${nodeip}/api/systeminfo`

echo $systeminfo | grep "registry_url"
if [ $? != 0 ] ; then
  exit 1
fi
#TODO need to check Clair, but currently Clair status api is unreachable from LB.
# echo $systeminfo | grep "with_clair" | grep "true"
# if [ $? == 0 ] ; then
# clair is enabled
# do some clair check
# else
# clair is disabled
# fi

#check top api

http_code=`curl -k -s -o /dev/null -w "%{http_code}\n" ${protocol}://${nodeip}/api/repositories/top`
set +e
if [ $http_code == 200 ] ; then
  exit 0
else
  exit 1
fi

// 添加可執行權限
chmod +x /usr/local/bin/check.sh

// 啓動 keepalived ; 並查看ip
systemctl start keepalived
ip a |grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    inet 192.168.9.29/24 brd 192.168.9.255 scope global eth0
    inet 192.168.9.33/32 scope global eth0
    
// 在第二臺 harbor 上安裝 keepalived; 並從第一臺harbor 上拷貝 keepalived配置文件及check.sh 到第二臺harbor 服務器上;
yum -y install keepalived
scp keepalived.conf 192.168.9.30:/etc/keepalived/
scp /usr/local/bin/check.sh 192.168.9.30:/usr/local/bin/

// 修改 keepalived.conf 文件
vim keepalived.conf 

! Configuration File for keepalived

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.9.33
    }
}

virtual_server 192.168.9.33 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    persistence_timeout 50
    protocol TCP

    real_server 192.168.9.30 80 {
        weight 1
    }
}

// 啓動 keepalived 
systemctl start keepalived

// 訪問第二臺的IP地址, 如果報鏈接超時, 能夠嘗試重啓 docker;
// 而後使用 vip 訪問; 192.168.9.33
    // 正常訪問, 將第一臺harbor停掉keepalived或者停掉nginx; 再使用 vip 訪問, 仍是能訪問到 harbor, vip飄到第二臺上; 再啓動第一臺上關掉的服務, vip又飄到第一臺上; 都正常訪問;
    // 向第一臺harbor中, 上傳鏡像, 在第二臺上查看存在;

OpenLDAP 安裝與配置

環境

　　Centos 7 x86_64 minimal

安裝:

　　使用 yum 安裝 OpenLDAP :

yum -y install openldap-servers openldap openldap-clients openldap-devel

SSL 證書

生成 ca 證書及 KEY

openssl genrsa -out ca.key 3072
openssl req -new -x509 -days 1095 -key ca.key -out ca.pem

爲域名 ldap.linge.io 生成長度爲 3072bit 的 KEY :

openssl genrsa -out ldap.linge.io.key 3072

爲域名 ldap.linge.io 生成證書請求

openssl req -new -key ldap.linge.io.key -out ldap.linge.io.csr -subj "/CN=ldap.linge.io/OU=LDAP/C=CN/ST=Shanghai/L=Shanghai"

爲域名 ldap.linge.io 簽發證書

openssl x509 -req -in ldap.linge.io.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out ldap.linge.io.pem -days 1095

配置

　　如下文檔使用 ldap.linge.io 做爲域名, 實際使用時請注意自行替換;

修改 slapd.conf, 使用 slapd-without-ssl.conf
rootpw 加密的密碼生成使用 : slappasswd 生成 DN 密碼, 並將輸出的密碼字符串複製到 rootpw 字段後 , 保存退出便可
注意: 配置文件中不要有多餘的空格, 尤爲是空行的時候( 5b003ece /etc/openldap/slapd.conf: line 9: bad config line.)

// 編輯ldap配置文件
vim /etc/openldap/slapd.conf
// 加載不一樣的 schema, 屬性;
include    /etc/openldap/schema/core.schema
include    /etc/openldap/schema/cosine.schema
include    /etc/openldap/schema/inetorgperson.schema
include    /etc/openldap/schema/openldap.schema
include    /etc/openldap/schema/nis.schema
include    /etc/openldap/schema/dyngroup.schema
include    /etc/openldap/schema/misc.schema
include    /etc/openldap/schema/ppolicy.schema

pidfile    /run/openldap/slapd.pid            // pid 文件;
argsfile   /run/openldap/slapd.args            // 參數;

TLSCACertificateFile    /etc/ssl/ca.pem                // 配置 CA 證書;
TLSCertificateFile      /etc/ssl/ldap.linge.io.pem        // 配置域名證書;
TLSCertificateKeyFile   /etc/ssl/ldap.linge.io.key        // 配置域名證書的 KEY;

// 配置證書使用的算法,版本;
TLSCipherSuite    EECDH:EDH:CAMELLIA:ECDH:RSA:!eNULL:!SSLv2:!RC4:!DES:!EXP:!SEED:!IDEA:!3DES
TLSProtocolMin    3.2
TLSVerifyClient   allow

database    bdb                    // 使用 Berkeley DB;
cachesize   10000                // 緩存大小;        
suffix      "dc=ldap,dc=linge,dc=io"                // 根據本身的域名替換;
rootdn      "cn=Manager,dc=ldap,dc=linge,dc=io"        // 同上
rootpw      {SSHA}EGUgLpu5rFOzbLKMij83pphTKppxv94v    // 指定用來管理的密碼, 使用 slappasswd 生成;
directory   /var/lib/ldap

# access control policy:
# Restrict password access to change by owner and authentication.
# Allow read access by everyone to all other attributes.

access to attrs=shadowLastChange,userPassword        // 配置哪些人能夠修改用戶密碼;
    by self write                    // 能夠修改本身的密碼;
    by anonymous auth                // 匿名用戶能夠登陸;

access to *                            // 提供給客戶端使用(jenkins,harbor,zabbix等),須要ldap認證的時候使用;
    by dn.exact="cn=admin,ou=Users,dc=ldap,dc=linge,dc=io" read        // Users下面的admin用戶能夠讀取;
    by * none                        // 其餘用戶沒有權限

# Indicees to maintain for this database
index objectClass                     eq,pres
index ou,cn,mail,surname,givenname    eq,pres,sub
index uidNumber,gidNumber,loginShell  eq,pres
index uid,memberUid                   eq,pres,sub
index nisMapName,nisMapEntry          eq,pres,sub

生成數據

清空 /etc/openldap/slapd.d 目錄 (根據默認的配置生成的一些內容, 與咱們的配置文件生成的不相同, 因此先清空)

rm -rf /etc/openldap/slapd.d/*

複製 db2 配置文件至 /var/lib/ldap/DB_CONFIG

cp -rf /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG

從新生成數據

// 文件沒找到的報錯能夠忽略, 由於咱們尚未啓動
slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d/
5d3569b3 bdb_db_open: database "dc=ldap,dc=linge,dc=io": db_open(/var/lib/ldap/id2entry.bdb) failed: No such file or directory (2).
5d3569b3 backend_startup_one (type=bdb, suffix="dc=ldap,dc=linge,dc=io"): bi_db_open failed! (2)
slap_startup failed (test would succeed using the -u switch)

更改文件權限

chown -R ldap.ldap /etc/openldap/slapd.d/ /var/lib/ldap/

查看新生成的內容

ls /etc/openldap/slapd.d/
cn=config  cn=config.ldif

啓動 ldap 服務

// 啓動 slapd 服務,設置開機啓動, 並查看進程
systemctl start slapd
systemctl enable slapd
ps -ef |grep slapd
ldap      8354     1  0 04:53 ?        00:00:00 /usr/sbin/slapd -u ldap -h ldapi:/// ldap:///
root      8492  5816  0 04:56 pts/1    00:00:00 grep --color=auto slapd

// 查看進程時,可看到 ldap:/// , 可添加 ldaps:///; 或者直接將 ldap:/// 替換成 ldaps:///; ldap:///開啓的 389 端口; ldaps啓用的 636 端口;
vim /etc/sysconfig/slapd 
# SLAPD_URLS="ldapi:/// ldap:///"          // 註釋此行;
SLAPD_URLS="ldapi:/// ldaps:///"           // 改成 ldaps,;

// 重啓 slapd
systemctl restart slapd

// 查看服務端口號
ss -tnl |grep 636
LISTEN     0      128          *:636                      *:*                  
LISTEN     0      128         :::636                     :::*

// 服務器開啓防火牆的狀況下,添加防火牆規則; 自行替換 192.168.0.0/16
iptables -I INPUT -m conntrack --ctstate NEW -p tcp -s 192.168.0.0/16 --dport 636 -j ACCEPT

OpenLDAP 建立組織架構與用戶和 Harbor 配置

鏈接工具的下載及使用

下載地址:http://directory.apache.org/studio

下載完成後, 進行安裝;
依賴 1.8 版本以上的 java
apache directory studio 的使用

在 Apache directory Studio 中建立新條目

右鍵 → New → New Context Entry

選擇 Create entry from scratch → Next

搜索 domain → Add

新建立好的條目 → 右鍵 New → New Context Entry → 同上操做搜索 or → 選擇 organizationalUnit → 補充ou=Users,

建立的ou=Users條目右鍵 → New → New Context Entry → Create entry from scratch → 搜索 openldap 選擇 OpenLDAPperson → 按如下填寫

如果一樣建立用戶條目的時候, 能夠點擊 cn = admin 右鍵, 複製; 而後在 ou=Users 上右鍵粘貼, 粘貼的時候提示 cn=admin 條目已經存在, 選擇 Rename entry and continue; 最後手動修改相關內容[ sn/uid/mail/userPasswd ];

密碼修改,雙擊 SSHA hashed password 框,選擇 New Password 進行設置;

Harbor 配置及認證測試 [admin 及 tian 用戶鏈接 LDAP 服務器正常 ]

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。