Docker私有倉庫Harbor介紹和部署記錄

時間  2019-11-11
標籤 docker 私有 倉庫 harbor 介紹 部署 記錄 欄目 Docker 简体版
原文   原文鏈接

Docker容器應用的開發和運行離不開可靠的鏡像管理，雖然Docker官方也提供了公共的鏡像倉庫，可是從安全和效率等方面考慮，部署咱們私有環境內的Registry
也是很是必要的。以前介紹了Docker私有倉庫Registry，這裏介紹另外一款企業級Docker鏡像倉庫Harbor的部署和使用，在Kubernetes集羣中，推薦使用Harbor倉庫環境。html

1、Harbor倉庫介紹前端

咱們在平常Docker容器使用和管理過程當中，漸漸發現部署企業私有倉庫每每是頗有必要的, 它能夠幫助你管理企業的一些敏感鏡像, 同時因爲Docker Hub的下載速度和GFW的緣由, 每每須要將一些沒法直接下載的鏡像導入本地私有倉庫. 而Harbor就是部署企業私有倉庫的一個不二之選。Harbor是由VMware公司開源的企業級的Docker Registry管理項目，Harbor主要提供Dcoker Registry管理UI，提供的功能包括：基於角色訪問的控制權限管理(RBAC)、AD/LDAP集成、日誌審覈、管理界面、自我註冊、鏡像複製和中文支持等。Harbor的目標是幫助用戶迅速搭建一個企業級的Docker registry服務。它以Docker公司開源的registry爲基礎，額外提供了以下功能:
->  基於角色的訪問控制(Role Based Access Control)
->  基於策略的鏡像複製(Policy based image replication)
->  鏡像的漏洞掃描(Vulnerability Scanning)
->  AD/LDAP集成(LDAP/AD support)
->  鏡像的刪除和空間清理(Image deletion & garbage collection)
->  友好的管理UI(Graphical user portal)
->  審計日誌(Audit logging)
->  RESTful API
->  部署簡單(Easy deployment)node

Harbor的全部組件都在Dcoker中部署，因此Harbor可以使用Docker Compose快速部署。須要特別注意：因爲Harbor是基於Docker Registry V2版本，因此docker必須大於等於1.10.0版本，docker-compose必需要大於1.6.0版本！python

2、Harbor倉庫結構mysql

Harbor的每一個組件都是以Docker容器的形式構建的，可使用Docker Compose來進行部署。若是環境中使用了kubernetes，Harbor也提供了kubernetes的配置文件。Harbor大概須要如下幾個容器組成：ui(Harbor的核心服務)、log(運行着rsyslog的容器，進行日誌收集)、mysql(由官方mysql鏡像構成的數據庫容器)、Nginx(使用Nginx作反向代理)、registry(官方的Docker registry)、adminserver(Harbor的配置數據管理器)、jobservice(Harbor的任務管理服務)、redis(用於存儲session)。linux

Harbor是一個用於存儲和分發Docker鏡像的企業級Registry服務器，總體架構仍是很清晰的。下面借用了網上的架構圖:nginx

===========================================================================git

Harbor依賴的外部組件
-> Nginx(即Proxy代理層): Nginx前端代理，主要用於分發前端頁面ui訪問和鏡像上傳和下載流量; Harbor的registry,UI,token等服務，經過一個前置的反向代理統一接收瀏覽器、Docker客戶端的請求，並將請求轉發給後端不一樣的服務。
-> Registry v2: 鏡像倉庫，負責存儲鏡像文件; Docker官方鏡像倉庫, 負責儲存Docker鏡像，並處理docker push/pull命令。因爲咱們要對用戶進行訪問控制，即不一樣用戶對Docker image有不一樣的讀寫權限，Registry會指向一個token服務，強制用戶的每次docker pull/push請求都要攜帶一個合法的token, Registry會經過公鑰對token進行解密驗證。
-> Database(MySQL或Postgresql)：爲core services提供數據庫服務，負責儲存用戶權限、審計日誌、Docker image分組信息等數據。github

Harbor自有組件
-> Core services(Admin Server): 這是Harbor的核心功能，主要提供如下服務： 
    -> UI：提供圖形化界面，幫助用戶管理registry上的鏡像（image）, 並對用戶進行受權。
    -> webhook：爲了及時獲取registry 上image狀態變化的狀況， 在Registry上配置webhook，把狀態變化傳遞給UI模塊。
    -> Auth服務：負責根據用戶權限給每一個docker push/pull命令簽發token. Docker 客戶端向Regiøstry服務發起的請求,若是不包含token，會被重定向到這裏，得到token後再從新向Registry進行請求。
    -> API: 提供Harbor RESTful API
-> Replication Job Service：提供多個 Harbor 實例之間的鏡像同步功能。
-> Log collector：爲了幫助監控Harbor運行，負責收集其餘組件的log，供往後進行分析。web

再來仔細看下Harbor主要組件和數據流走向：

 

-> proxy，它是一個nginx前端代理，主要是分發前端頁面ui訪問和鏡像上傳和下載流量，上圖中經過深藍色先標識；
-> ui提供了一個web管理頁面，固然還包括了一個前端頁面和後端API，底層使用mysql數據庫；
-> registry是鏡像倉庫，負責存儲鏡像文件，當鏡像上傳完畢後經過hook通知ui建立repository，上圖經過紅色線標識，固然registry的token認證也是經過ui組件完成；
-> adminserver是系統的配置管理中心附帶檢查存儲用量，ui和jobserver啓動時候回須要加載adminserver的配置，經過灰色線標識；
-> jobsevice是負責鏡像複製工做的，他和registry通訊，從一個registry pull鏡像而後push到另外一個registry，並記錄job_log，上圖經過紫色線標識；
-> log是日誌彙總組件，經過docker的log-driver把日誌彙總到一塊兒，經過淺藍色線條標識。

Harbor的誤區
誤區一: Harbor是負責存儲容器鏡像的 (Harbor是鏡像倉庫，那麼它就應當是存儲鏡像的)
其實關於鏡像的存儲，Harbor使用的是官方的docker registry服務去完成，至於registry是用本地存儲或者s3都是能夠的，Harbor的功能是在此之上提供用戶權限管理、鏡像複製等功能，提升使用的registry的效率。

誤區二：Harbor鏡像複製是存儲直接複製 (鏡像的複製，不少人覺得應該是鏡像分層文件的直接拷貝)
其實Harbor鏡像複製採用了一個更加通用、高屋建瓴的作法，經過docker registry 的API去拷貝，這不是省事，這種作法屏蔽了繁瑣的底層文件操做、不只能夠利用現有docker registry功能沒必要重複造輪子，並且能夠解決衝突和一致性的問題。

Harbor的部署
這裏不建議使用kubernetes來部署, 緣由是鏡像倉庫很是重要, 儘可能保證部署和維護的簡潔性, 所以這裏直接使用compose的方式進行部署。官方提供3種部署Harbor的方式:
1）在線安裝: 從Docker Hub下載Harbor的鏡像來安裝, 因爲Docker Hub比較慢, 建議Docker配置好加速器。
2）離線安裝: 這種方式應對與部署主機沒聯網的狀況使用。須要提早下載離線安裝包: harbor-offline-installer-.tgz 到本地
3）OVA安裝: 這個主要用vCentor環境是使用

後面部署時會爲Docker配置鏡像加速器, 所以會採用在線部署的方式, 部署步驟以下：
-> 下載Harbor最新的在線安裝包
-> 配置Harbor (harbor.cfg)
-> 運行install.sh來安裝和啓動Harbor
-> Harbor的日誌路徑:/var/log/harbor

Harbor倉庫部署的官方要求的最小系統配置
-> 2個cpu
-> 4g內存
-> 40g硬盤，由於是存儲鏡像的因此推薦硬盤大點。

參考
Harbor GitHub
Harbor 安裝手冊
Harbor 用戶手冊

3、Harbor倉庫環境部署記錄

3.1) 環境要求和準備工做

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Harbor以容器的形式進行部署, 所以能夠被部署到任何支持Docker的Linux發行版, 要使用Harbor，須要安裝docker和docker-compose編排工具，而且具有以下環境:
Python2.7+
Docker Engine 1.10+
Docker Compose 1.6.0+
 
這裏測試環境部署到Centos7.5機器上，以下：
[root@harbor-node ~] # cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
 
[root@harbor-node ~] # setenforce 0
[root@harbor-node ~] # cat /etc/sysconfig/selinux
...........
SELINUX=disabled
 
[root@harbor-node ~] # systemctl stop firewalld
[root@harbor-node ~] # systemctl disable firewalld
[root@harbor-node ~] # firewall-cmd --state
not running
 
centos7自帶的python版本就是2.7.5
[root@harbor-node ~] # python --version
Python 2.7.5

3.2) 安裝Docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
更新yum包
[root@harbor-node ~] # yum update
   
卸載舊版本 Docker
[root@harbor-node ~] # yum remove docker docker-common docker-selinux docker-engine
   
安裝軟件包
[root@harbor-node ~] # yum install -y yum-utils device-mapper-persistent-data lvm2
   
添加 Docker yum源
[root@harbor-node ~] # yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror
adding repo from: https: //download .docker.com /linux/centos/docker-ce .repo
grabbing  file  https: //download .docker.com /linux/centos/docker-ce .repo to  /etc/yum .repos.d /docker-ce .repo
repo saved to  /etc/yum .repos.d /docker-ce .repo
   
安裝 Docker （直接yum按照docker-ce便可）
隨着Docker的不斷流行與發展，docker公司（或稱爲組織）也開啓了商業化之路，Docker 從 17.03版本以後分爲 CE（Community Edition） 和 EE（Enterprise Edition）:
1) Docker EE由公司支持，可在通過認證的操做系統和雲提供商中使用，並可運行來自Docker Store的、通過認證的容器和插件。
2) Docker CE是免費的Docker產品的新名稱，Docker CE包含了完整的Docker平臺，很是適合開發人員和運維團隊構建容器APP。
    事實上，Docker CE 17.03，可理解爲Docker 1.13.1的Bug修復版本。所以，從Docker 1.13升級到Docker CE 17.03風險相對是較小的。
   
[root@harbor-node ~] # yum -y install docker-ce
   
啓動 Docker
[root@harbor-node ~] # systemctl start docker
[root@harbor-node ~] # systemctl enable docker
[root@harbor-node ~] # systemctl status docker
● docker.service - Docker Application Container Engine
    Loaded: loaded ( /usr/lib/systemd/system/docker .service; enabled; vendor preset: disabled)
    Active: active (running) since Sun 2019-05-26 22:15:34 CST; 27min ago
      Docs: https: //docs .docker.com
  Main PID: 15260 (dockerd)
    CGroup:  /system .slice /docker .service
            └─15260  /usr/bin/dockerd  -H fd: //  --containerd= /run/containerd/containerd .sock
 
May 26 22:15:33 harbor-node dockerd[15260]:  time = "2019-05-26T22:15:33.570826805+08:00"  level=info msg= "pickfirstBalancer: HandleSubConnStateChange: 0xc420175490, READY"  module=grpc
May 26 22:15:33 harbor-node dockerd[15260]:  time = "2019-05-26T22:15:33.570899114+08:00"  level=info msg= "pickfirstBalancer: HandleSubConnStateChange: 0xc42006de20, READY"  module=grpc
May 26 22:15:33 harbor-node dockerd[15260]:  time = "2019-05-26T22:15:33.665440742+08:00"  level=info msg= "Graph migration to content-addressability took 0.00 seconds"
May 26 22:15:33 harbor-node dockerd[15260]:  time = "2019-05-26T22:15:33.666111994+08:00"  level=info msg= "Loading containers: start."
May 26 22:15:33 harbor-node dockerd[15260]:  time = "2019-05-26T22:15:33.913110547+08:00"  level=info msg= "Default bridge (docker0) is assigned with an IP address 172.17.0.0/16... IP address"
May 26 22:15:34 harbor-node dockerd[15260]:  time = "2019-05-26T22:15:34.088687650+08:00"  level=info msg= "Loading containers: done."
May 26 22:15:34 harbor-node dockerd[15260]:  time = "2019-05-26T22:15:34.128885651+08:00"  level=info msg= "Docker daemon"  commit=481bc77 graphdriver(s)=overlay2 version=18.09.6
May 26 22:15:34 harbor-node dockerd[15260]:  time = "2019-05-26T22:15:34.129073367+08:00"  level=info msg= "Daemon has completed initialization"
May 26 22:15:34 harbor-node dockerd[15260]:  time = "2019-05-26T22:15:34.223886566+08:00"  level=info msg= "API listen on /var/run/docker.sock"
May 26 22:15:34 harbor-node systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show  in  full.
   
查看 Docker 版本號
[root@harbor-node ~] # docker --version
Docker version 18.09.6, build 481bc77156
  
[root@harbor-node ~] # docker version
Client:
  Version:           18.09.6
  API version:       1.39
  Go version:        go1.10.8
  Git commit:        481bc77156
  Built:             Sat May  4 02:34:58 2019
  OS /Arch :           linux /amd64
  Experimental:       false
  
Server: Docker Engine - Community
  Engine:
   Version:          18.09.6
   API version:      1.39 (minimum version 1.12)
   Go version:       go1.10.8
   Git commit:       481bc77
   Built:            Sat May  4 02:02:43 2019
   OS /Arch :          linux /amd64
   Experimental:      false

3.3) 安裝Docker Compose

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Docker Compose 是 Docker 容器進行編排的工具，定義和運行多容器的應用，能夠一條命令啓動多個容器。
 
安裝 epel-release
[root@harbor-node ~] # yum install epel-release
   
安裝 python-pip
[root@harbor-node ~] # yum install -y python-pip
   
安裝 docker-compose
[root@harbor-node ~] # pip install docker-compose
.........
Successfully installed asn1crypto-0.24.0 bcrypt-3.1.6 cached-property-1.5.1 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 cryptography-2.6.1 docker-3.7.2 docker-compose-1.24.0 docker-pycreds-0.4.0 dockerpty-0.4.1 docopt-0.6.2 enum34-1.1.6 functools32-3.2.3.post2 idna-2.7 jsonschema-2.6.0 paramiko-2.4.2 pyasn1-0.4.5 pycparser-2.19 pynacl-1.3.0 requests-2.20.1 texttable-0.9.1 urllib3-1.24.3 websocket-client-0.56.0
You are using pip version 8.1.2, however version 19.1.1 is available.
You should consider upgrading via the  'pip install --upgrade pip'  command .
   
查看 docker-compose 版本號
[root@harbor-node ~] # docker-compose -version
docker-compose version 1.24.0, build 0aa5906
  
[root@harbor-node ~] # pip freeze | grep compose
You are using pip version 8.1.2, however version 19.1.1 is available.
You should consider upgrading via the  'pip install --upgrade pip'  command .
docker-compose==1.24.0
   
安裝 git
[root@harbor-node ~] # yum install git

3.4) 爲Docker配置加速器, 方便經過國內鏡像服務器快速拉取Docker Hub提供的鏡像

1
2
3
4
5
[root@harbor-node ~] # mkdir -p /etc/docker
[root@harbor-node ~] # cat /etc/docker/daemon.json
{
   "registry-mirrors" : [ "https://v5d7kh0f.mirror.aliyuncs.com" ]
}

3.5) 下載Harbor安裝包，配置Harbor
到Harbor的GitHub倉庫的Release頁面, 下載最新的在線安裝包(若是下載不了, 請從這裏下載百度網盤Harbor   提取碼：xu3j）

這裏分爲在線和離線的版本，我下載的是1.8.0在線的版本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
[root@harbor-node ~] # ll harbor-online-installer-v1.8.0.tgz
-rw-r--r-- 1 root root 7954 May 26 22:45 harbor-online-installer-v1.8.0.tgz
 
[root@harbor-node ~] # tar -zvxf harbor-online-installer-v1.8.0.tgz
[root@harbor-node ~] # cd harbor
[root@harbor-node harbor] # ls
harbor.yml   install .sh  LICENSE  prepare
 
下載下來以後解壓縮，目錄下會有harbor.yaml (新版本是.yaml文件，以前版本是.conf 或者 .cfg文件)，就是Harbor的配置文件了。
[root@harbor-node harbor] # cp harbor.yml harbor.yml.bak
[root@harbor-node harbor] # vim harbor.yml
[root@harbor-node harbor] # cat harbor.yml |grep -v "#"|grep -v "^$"
hostname : 172.16.60.213
http:
   port: 80
harbor_admin_password: kevin@BO123             
database:
   password: root123
data_volume:  /data
clair:
   updaters_interval: 12
   http_proxy:
   https_proxy:
   no_proxy: 127.0.0.1,localhost,core,registry
jobservice:
   max_job_workers: 10
chart:
   absolute_url: disabled
log:
   level: info
   rotate_count: 50
   rotate_size: 200M
   location:  /var/log/harbor
_version: 1.8.0
 
配置解釋
hostname : 修改爲Harbao部署機自身的ip地址
db_password: 這是postgresql數據庫root密碼
harbor_admin_password: harbor初始管理員密碼爲Harbor12345, 這裏最好修改爲本身的密碼，默認密碼至少8位，最好是大小寫、數字和特殊字符。
 
配置完Harbor以後，接着進行安裝啓動Harbor，Harbor目錄下有一個 install .sh, 執行它來進行安裝
[root@harbor-node harbor] # ./install.sh
...........
...........
✔ ----Harbor has been installed and started successfully.----
 
Now you should be able to visit the admin portal at http: //172 .16.60.213.
For  more  details, please visit https: //github .com /goharbor/harbor  .
 
安裝完成後，會發現解壓目錄harbor下面多了一個docker-compose.yml文件，裏面包含了harbor依賴的鏡像和對應容器建立的信息
 
查看harbor對應容器信息（還能夠執行 "docker images" 和 "docker ps" 查看harbor的鏡像和容器狀況）
[root@harbor-node harbor] # docker-compose ps       #"注意docker-compose"命令只能在當前harbor目錄下使用（由於該目錄下有harbor配置文件）
       Name                     Command                  State                 Ports         
---------------------------------------------------------------------------------------------
harbor-core          /harbor/start .sh                 Up (healthy)                           
harbor-db            /entrypoint .sh postgres          Up (healthy)   5432 /tcp                
harbor-jobservice    /harbor/start .sh                 Up                                     
harbor-log           /bin/sh  -c  /usr/local/bin/  ...   Up (healthy)   127.0.0.1:1514->10514 /tcp
harbor-portal       nginx -g daemon off;             Up (healthy)   80 /tcp                  
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->80 /tcp      
redis               docker-entrypoint.sh redis ...   Up             6379 /tcp                
registry             /entrypoint .sh  /etc/regist  ...   Up (healthy)   5000 /tcp                
registryctl          /harbor/start .sh                 Up (healthy) 
 
而後就能夠訪問harbor了,訪問地址爲：http: //172 .16.60.213
用戶名爲admin，密碼爲配置文件中定義的 "kevin@BO123"

Harbor 服務的關閉和啓動

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
1) Harbor的日誌路徑：var /log/harbor
[root@harbor-node harbor] # cat harbor.yml|grep log
log:
   # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
   # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
   # The directory on your host that store log
   location:  /var/log/harbor
    
[root@harbor-node harbor] # ls /var/log/harbor/
core.log  jobservice.log  portal.log  postgresql.log  proxy.log  redis.log  registryctl.log  registry.log
    
2) 中止和關閉harbor命令:  "docker-compose down -v"
[root@harbor-node harbor] # docker-compose down -v
Stopping nginx             ...  done
Stopping harbor-jobservice ...  done
Stopping harbor-portal     ...  done
Stopping harbor-core       ...  done
Stopping redis             ...  done
Stopping registryctl       ...  done
Stopping registry          ...  done
Stopping harbor-db         ...  done
Stopping harbor-log        ...  done
Removing nginx             ...  done
Removing harbor-jobservice ...  done
Removing harbor-portal     ...  done
Removing harbor-core       ...  done
Removing redis             ...  done
Removing registryctl       ...  done
Removing registry          ...  done
Removing harbor-db         ...  done
Removing harbor-log        ...  done
Removing network harbor_harbor
    
[root@harbor-node harbor] # docker-compose ps
Name   Command   State   Ports
------------------------------
    
[root@harbor-node harbor] # docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
    
能夠修改harbor配置文件，好比這裏修改harbor的web登陸端口，由80端口修改成8080端口
[root@harbor-node harbor] # vim harbor.yml
.........
http:
   # port for http, default is 80. If https enabled, this port will redirect to https port
   port: 8080
    
而後將harbor修改的配置更新到 docker-compose.yml 文件
[root@harbor-node harbor] # ./prepare
prepare base  dir  is  set  to  /root/harbor
Clearing the configuration  file :  /config/log/logrotate .conf
Clearing the configuration  file :  /config/nginx/nginx .conf
Clearing the configuration  file :  /config/core/env
Clearing the configuration  file :  /config/core/app .conf
Clearing the configuration  file :  /config/registry/config .yml
Clearing the configuration  file :  /config/registry/root .crt
Clearing the configuration  file :  /config/registryctl/env
Clearing the configuration  file :  /config/registryctl/config .yml
Clearing the configuration  file :  /config/db/env
Clearing the configuration  file :  /config/jobservice/env
Clearing the configuration  file :  /config/jobservice/config .yml
Generated configuration  file :  /config/log/logrotate .conf
Generated configuration  file :  /config/nginx/nginx .conf
Generated configuration  file :  /config/core/env
Generated configuration  file :  /config/core/app .conf
Generated configuration  file :  /config/registry/config .yml
Generated configuration  file :  /config/registryctl/env
Generated configuration  file :  /config/db/env
Generated configuration  file :  /config/jobservice/env
Generated configuration  file :  /config/jobservice/config .yml
loaded secret from  file :  /secret/keys/secretkey
Generated configuration  file :  /compose_location/docker-compose .yml
Clean up the input  dir
    
防止容器進程沒有權限讀取生成的配置
[root@harbor-node harbor] # ll common/
total 0
drwxr-xr-x 9 root root 105 May 26 23:10 config
    
[root@harbor-node harbor] # chmod -R 777 common
    
[root@harbor-node harbor] # ll common/
total 0
drwxrwxrwx 9 root root 105 May 27 00:41 config
    
=======================================================================================================================================
特別注意： 這裏的common權限若是設置過小，可能會致使harbor啓動後，報下面的錯
發現啓動harbor後，如上有些服務，如nginx，registry狀態一直是 "Restarting" ，這時須要查看日誌：
[root@harbor-node harbor] # tail -100 /var/log/harbor/registry.log |grep error
May 27 01:01:18 172.19.0.1 registry[2960]: configuration error:  open  /etc/registry/config .yml: permission denied
May 27 01:01:21 172.19.0.1 registry[2960]: configuration error:  open  /etc/registry/config .yml: permission denied
May 27 01:01:23 172.19.0.1 registry[2960]: configuration error:  open  /etc/registry/config .yml: permission denied
May 27 01:01:27 172.19.0.1 registry[2960]: configuration error:  open  /etc/registry/config .yml: permission denied
=======================================================================================================================================
    
最後再次啓動 harbor
[root@harbor-node harbor] # docker-compose up -d
Creating network  "harbor_harbor"  with the default driver
Creating harbor-log ...  done
Creating registryctl ...  done
Creating registry    ...  done
Creating redis       ...  done
Creating harbor-db   ...  done
Creating harbor-core ...  done
Creating harbor-jobservice ...  done
Creating harbor-portal     ...  done
Creating nginx             ...  done
    
查看服務
[root@harbor-node harbor] # docker-compose  ps
       Name                     Command                       State                     Ports      
------------------------------------------------------------------------------------------------------
harbor-core          /harbor/start .sh                 Up (health: starting)                        
harbor-db            /entrypoint .sh postgres          Up (health: starting)   5432 /tcp             
harbor-jobservice    /harbor/start .sh                 Up                                           
harbor-log           /bin/sh  -c  /usr/local/bin/  ...   Up (health: starting)   127.0.0.1:1514->10514 /tcp
harbor-portal       nginx -g daemon off;             Up (health: starting)   80 /tcp               
nginx               nginx -g daemon off;             Up (health: starting)   0.0.0.0:8080->80 /tcp 
redis               docker-entrypoint.sh redis ...   Up                      6379 /tcp             
registry             /entrypoint .sh  /etc/regist  ...   Up (health: starting)   5000 /tcp             
registryctl          /harbor/start .sh                 Up (health: starting) 
    
而後訪問http: //172 .16.60.213:8080，便可訪問harbor的web界面
 
###############################################################################################
要是想修改harbor的登錄用戶密碼，則最好在harbor web界面裏直接修改，這樣是最保險的！
 
若是是想經過修改harbar.yaml文件來重置harbor用戶密碼，則不能單純的修改後就執行 "./prepare" 和重啓docker-compose，這樣是不能修改harbor用戶密碼的！
這時由於harbor在這裏用的是postgresql數據庫，以pdkdf2算法保存的祕文密碼！須要先進入 "harbor-db" 容器內部,執行相關postgresql命令行。
並且postgresql的用戶密碼採用的是pbkdf2算法，須要提早計算好新密碼的密鑰值，pdkdf2算法須要 "鹽值" ， "迭代次數" 和密鑰長度int型等，特別麻煩!!
 
因此若是忘記harbor的web密碼或者是admin密碼須要重置，而且對於postgresql數據庫 或者 pbkdf2算法操做不熟悉的話，建議刪除data源數據的database，從新部署！
作法以下：
# docker-compose down -v
# rm -rf /data/database
# vim harbor.yaml           #在這裏重置或修改密碼
# docker-compose up -d
 
這樣就可使用重置後的新密碼登錄harbor web界面了，可是以前建立的用戶和項目就都刪除了。
這種狀況最好適用於剛建立初期。
 
###############################################################################################
docker-compose up -d           # 後臺啓動，若是容器不存在根據鏡像自動建立
docker-compose down - v         # 中止容器並刪除容器
docker-compose start           # 啓動容器，容器不存在就沒法啓動，不會自動建立鏡像
docker-compose stop            # 中止容器
   
須要注意：
其實上面是中止docker-compose.yml中定義的全部容器，默認狀況下docker-compose就是操做同目錄下的docker-compose.yml文件。
若是使用其餘yml文件，可使用-f本身指定。

->  登陸Harbor web界面，在"系統管理"->"配置管理"->"認證模式"->"容許自注冊"這一項的對勾去掉，則登陸的時候就不會有"用戶註冊"這一功能了。
->  能夠在"配置管理"這一項進行認證模式，郵箱，標籤等設置。

3.6） 使用Harbor私有倉庫　　

3.6.1）harbor的login登陸

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
1）在harbor遠程別的客戶機上登陸
[root@docker-client ~] # docker login 172.16.60.213
Username: admin
Password:
Error response from daemon: Get https: //172 .16.60.213 /v1/users/ : dial tcp 172.16.60.213:443: connect: connection refused
  
在進行harbor登陸或上傳代碼時，會報出上面錯誤！
這是由於docker1.3.2版本開始默認docker registry使用的是https，而Harbor默認設置的是http方式而不是https，因此當執行用docker login、pull、push等
命令操做非https的docker regsitry的時就會報錯。
  
解決辦法：
以下，在 /etc/docker/daemon .json文件裏添加 "insecure-registries" 配置
[root@docker-client ~] # vim /etc/docker/daemon.json
{
     "insecure-registries" : [
         "172.16.60.213"
       ]
}
  
而後重啓docker服務
[root@docker-client ~] # systemctl restart docker
  
接着再次驗證harbor登陸，發現就能登陸上了
[root@docker-client ~] # docker login 172.16.60.213      #或者直接執行"docker login -u admin -p kevin@BO123 172.16.60.213"命令登錄
Username: admin
Password:
Login Succeeded
[root@docker-client ~] #
  
2）若是是在harbor本機登陸，出現上面的報錯：
[root@harbor-node harbor] # docker login 172.16.60.213
Username: admin
Password:
Error response from daemon: Get https: //172 .16.60.213 /v1/users/ : dial tcp 172.16.60.213:443: connect: connection refused
  
解決辦法：
在 /etc/docker/daemon .json 文件裏添加 "insecure-registries" 配置 （第一行是以前添加的docker加速配置），注意兩行之間有一個 "," 逗號隔開
[root@harbor-node harbor] # vim /etc/docker/daemon.json
{
   "registry-mirrors" : [ "https://v5d7kh0f.mirror.aliyuncs.com" ],
   "insecure-registries" : [ "172.16.60.213" ]
}
  
修改事後重啓docker, 而後重啓Harbor服務
[root@harbor-node harbor] # systemctl restart docker
[root@harbor-node harbor] # docker-compose stop
[root@harbor-node harbor] # docker-compose start
  
而後再測試再harbor本機登陸
[root@harbor-node harbor] # docker login 172.16.60.213
Username: admin
Password:
WARNING! Your password will be stored unencrypted  in  /root/ .docker /config .json.
Configure a credential helper to remove this warning. See
https: //docs .docker.com /engine/reference/commandline/login/ #credentials-store
  
Login Succeeded
[root@harbor-node harbor] #
  
登陸的帳號信息都保存到 /root/ .docker /config .json文件裏了
[root@harbor-node harbor] # cat /root/.docker/config.json
{
         "auths" : {
                 "172.16.60.213" : {
                         "auth" :  "YWRtaW46a2V2aW5AQk8xOTg3"
                 }
         },
         "HttpHeaders" : {
                 "User-Agent" :  "Docker-Client/18.09.6 (linux)"
         }
  
只要 /root/ .docker /config .json裏的信息不刪除，後續再次登陸的時候，就不用輸入用戶名和密碼了
[root@harbor-node ~] # docker login 172.16.60.213
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted  in  /root/ .docker /config .json.
Configure a credential helper to remove this warning. See
https: //docs .docker.com /engine/reference/commandline/login/ #credentials-store
  
Login Succeeded

===============================================================
注意事項總結：
harbor支持http和https，但若是使用http的話，在拉取鏡像的時候，會拋出倉庫不受信任的異常。
須要在全部的docker客戶端的docker配置文件/etc/docker/daemon.json中添加以下配置:

1
2
3
{
     "insecure-registries" : [ "https://*.*.*.*" ]
}

若是使用自簽名的https證書，仍然會提示證書不受信任的問題。須要將自簽名的ca證書發送到全部的docker客戶端的指定目錄。
關於使用自簽名證書配置harbor的具體過程能夠參考: https://github.com/goharbor/harbor/blob/master/docs/configure_https.md

3.6.2）harbor倉庫的使用

1
2
3
4
5
6
7
8
#鏡像打標籤的命令
# docker tag 鏡像名:標籤 私服地址/倉庫項目名/鏡像名:標籤
 
#推送到私服的命令
#docker push 私服地址/倉庫項目名/鏡像名:標籤
 
#從私服拉取鏡像的命令
#docker pull 私服地址/倉庫項目名/鏡像名:標籤

首先在Harbor web界面裏最好建立一個本身須要的"項目" (或者使用默認的"library"項目),項目公開和私有：
-  Public: 全部用戶對於公開項目都有讀權限,這種方式對於你想把一些倉庫分享給其餘人的時候,是很是方便的.
-  Private: 私有項目只能被有特定用戶權限的人去訪問。這種方式對於內部團隊來講共享也是比較方便的

好比建立一個公開項目"kevin_bo"，點擊進去能夠看到推送鏡像的信息提示

而後就能夠在Harbor服務器的終端命令行裏進行鏡像推送到Harbor倉庫的操做了:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
在進行harbor鏡像推送和拉取操做前，須要事先login登陸到harbor倉庫裏，這樣纔有harbor鏡像的推送和拉取的權限！！
[root@harbor-node ~] # docker login 172.16.60.213
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted  in  /root/ .docker /config .json.
Configure a credential helper to remove this warning. See
https: //docs .docker.com /engine/reference/commandline/login/ #credentials-store
 
Login Succeeded
 
先查看本機有哪些鏡像
[root@harbor-node ~] # docker images
REPOSITORY                    TAG                        IMAGE ID            CREATED             SIZE
goharbor /redis-photon          v1.8.0                     66d7402d2770        10 days ago         103MB
goharbor /harbor-registryctl    v1.8.0                     0ca3e2b624f5        10 days ago         96.2MB
goharbor /registry-photon       v2.7.1-patch-2819-v1.8.0   1e7d99ccba24        10 days ago         81.3MB
goharbor /nginx-photon          v1.8.0                     4a4b48b32ae4        10 days ago         36MB
goharbor /harbor-log            v1.8.0                     e718bdc405a3        10 days ago         81.5MB
goharbor /harbor-jobservice     v1.8.0                     d47940dd883f        10 days ago         118MB
goharbor /harbor-core           v1.8.0                     b07a1a4be17f        10 days ago         135MB
goharbor /harbor-portal         v1.8.0                     76298a1ef089        10 days ago         42.9MB
goharbor /harbor-db             v1.8.0                     d1e0b3df3e95        10 days ago         140MB
goharbor /prepare               v1.8.0                     769ca785dab0        10 days ago         139MB
 
好比推送其中的goharbor /redis-photon :v1.8.0鏡像到Harbor倉庫的 "kevin_bo" 項目裏
[root@harbor-node ~] # docker tag goharbor/redis-photon:v1.8.0 172.16.60.213/kevin_bo/redis-photon:v1.0
[root@harbor-node ~] # docker push 172.16.60.213/kevin_bo/redis-photon:v1.0
The push refers to repository [172.16.60.213 /kevin_bo/redis-photon ]
8864c4b9ac3d: Pushed
420b26399278: Pushed
4433bcd802e7: Pushed
268091c30a67: Pushed
23d9f72a5270: Pushed
v1.0: digest: sha256:1e2ce8e6a852713d789c6315642d1483d1efdb4acee4699817810bef219ec93d size: 1366
 
查看本機的images，發現多了一個上面製做的鏡像，就是原來goharbor /redis-photon :v1.8.0的tag，能夠選擇刪除
[root@harbor-node ~] # docker rmi 172.16.60.213/kevin_bo/redis-photon:v1.0
Untagged: 172.16.60.213 /kevin_bo/redis-photon :v1.0
Untagged: 172.16.60.213 /kevin_bo/redis-photon @sha256:1e2ce8e6a852713d789c6315642d1483d1efdb4acee4699817810bef219ec93d
 
[root@harbor-node ~] # docker images
REPOSITORY                    TAG                        IMAGE ID            CREATED             SIZE
goharbor /redis-photon          v1.8.0                     66d7402d2770        10 days ago         103MB
goharbor /harbor-registryctl    v1.8.0                     0ca3e2b624f5        10 days ago         96.2MB
goharbor /registry-photon       v2.7.1-patch-2819-v1.8.0   1e7d99ccba24        10 days ago         81.3MB
goharbor /nginx-photon          v1.8.0                     4a4b48b32ae4        10 days ago         36MB
goharbor /harbor-log            v1.8.0                     e718bdc405a3        10 days ago         81.5MB
goharbor /harbor-jobservice     v1.8.0                     d47940dd883f        10 days ago         118MB
goharbor /harbor-core           v1.8.0                     b07a1a4be17f        10 days ago         135MB
goharbor /harbor-portal         v1.8.0                     76298a1ef089        10 days ago         42.9MB
goharbor /harbor-db             v1.8.0                     d1e0b3df3e95        10 days ago         140MB
goharbor /prepare               v1.8.0                     769ca785dab0        10 days ago         139MB
 
同理，推送其餘鏡像是一樣的操做，好比再推送goharbor /harbor-core :v1.8.0到harbor倉庫的 "kevin_bo" 項目裏
[root@harbor-node ~] # docker tag goharbor/harbor-core:v1.8.0 172.16.60.213/kevin_bo/goharbor/harbor-core:v1.0
[root@harbor-node ~] # docker push 172.16.60.213/kevin_bo/goharbor/harbor-core:v1.0
The push refers to repository [172.16.60.213 /kevin_bo/goharbor/harbor-core ]
5385ffb8451e: Pushed
36e1cb2d6ffa: Pushed
452d238b3e48: Pushed
af3a6f89469a: Pushed
05bc5efb1724: Pushed
23d9f72a5270: Mounted from kevin_bo /redis-photon
v1.0: digest: sha256:7899f284617bb051180adf6c3aedd140a519d9092b8986dd9058d4dcec0d31de size: 1580
 
[root@harbor-node ~] # docker images
REPOSITORY                                    TAG                        IMAGE ID            CREATED             SIZE
goharbor /redis-photon                          v1.8.0                     66d7402d2770        10 days ago         103MB
goharbor /harbor-registryctl                    v1.8.0                     0ca3e2b624f5        10 days ago         96.2MB
goharbor /registry-photon                       v2.7.1-patch-2819-v1.8.0   1e7d99ccba24        10 days ago         81.3MB
goharbor /nginx-photon                          v1.8.0                     4a4b48b32ae4        10 days ago         36MB
goharbor /harbor-log                            v1.8.0                     e718bdc405a3        10 days ago         81.5MB
goharbor /harbor-jobservice                     v1.8.0                     d47940dd883f        10 days ago         118MB
goharbor /harbor-core                           v1.8.0                     b07a1a4be17f        10 days ago         135MB
172.16.60.213 /kevin_bo/goharbor/harbor-core    v1.0                       b07a1a4be17f        10 days ago         135MB
goharbor /harbor-portal                         v1.8.0                     76298a1ef089        10 days ago         42.9MB
goharbor /harbor-db                             v1.8.0                     d1e0b3df3e95        10 days ago         140MB
goharbor /prepare                               v1.8.0                     769ca785dab0        10 days ago         139MB
[root@harbor-node ~] # docker rmi 172.16.60.213/kevin_bo/goharbor/harbor-core:v1.0
Untagged: 172.16.60.213 /kevin_bo/goharbor/harbor-core :v1.0
Untagged: 172.16.60.213 /kevin_bo/goharbor/harbor-core @sha256:7899f284617bb051180adf6c3aedd140a519d9092b8986dd9058d4dcec0d31de
 
[root@harbor-node ~] # docker images
REPOSITORY                    TAG                        IMAGE ID            CREATED             SIZE
goharbor /redis-photon          v1.8.0                     66d7402d2770        10 days ago         103MB
goharbor /harbor-registryctl    v1.8.0                     0ca3e2b624f5        10 days ago         96.2MB
goharbor /registry-photon       v2.7.1-patch-2819-v1.8.0   1e7d99ccba24        10 days ago         81.3MB
goharbor /nginx-photon          v1.8.0                     4a4b48b32ae4        10 days ago         36MB
goharbor /harbor-log            v1.8.0                     e718bdc405a3        10 days ago         81.5MB
goharbor /harbor-jobservice     v1.8.0                     d47940dd883f        10 days ago         118MB
goharbor /harbor-core           v1.8.0                     b07a1a4be17f        10 days ago         135MB
goharbor /harbor-portal         v1.8.0                     76298a1ef089        10 days ago         42.9MB
goharbor /harbor-db             v1.8.0                     d1e0b3df3e95        10 days ago         140MB
goharbor /prepare               v1.8.0                     769ca785dab0        10 days ago         139MB

而後登陸到Harbor web 界面裏，就能夠看到"kevin_bo"項目裏就有了上面推送的兩個鏡像了，點擊到對應的鏡像了，還能夠對鏡像進行打標籤，複製鏡像等操做：

好比將"kevin_bo"項目裏上面的kevin_bo/goharbor/harbor-core鏡像複製到"library"項目裏

而後到"library"項目裏就能看到上面從"kevin_bo"項目裏複製過來的鏡像了

注意： harbor私倉的相關容器映射到主機的volumes數據卷的空間要有保證，最好是單獨的分區空間。
上面測試harbor容器經過volumes映射到主機的目錄是/data, 能夠到這裏查看harbor推送的鏡像:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@harbor-node repositories] # pwd
/data/registry/docker/registry/v2/repositories
 
能夠查看兩個 "項目"
[root@harbor-node repositories] # ll
total 0
drwxr-xr-x 4 10000 10000 42 May 27 14:01 kevin_bo
drwxr-xr-x 3 10000 10000 27 May 27 14:08 library
 
[root@harbor-node repositories] # ll kevin_bo/
total 0
drwxr-xr-x 3 10000 10000 25 May 27 14:01 goharbor
drwxr-xr-x 5 10000 10000 55 May 27 13:58 redis-photon
 
[root@harbor-node repositories] # ll library/
total 0
drwxr-xr-x 3 10000 10000 21 May 27 14:08 172.16.60.213
 
[root@harbor-node repositories] # ll library/172.16.60.213/
total 0
drwxr-xr-x 3 10000 10000 25 May 27 14:08 library

========測試下在harbor客戶端下載harbor倉庫裏的鏡像=======

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@client ~] # docker login 172.16.60.213
Username: admin
Password:
Login Succeeded
 
[root@client ~] # docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
 
從harbor倉庫拉取鏡像
[root@client ~] # docker pull 172.16.60.213/kevin_bo/goharbor/harbor-core:v1.0
Trying to pull repository 172.16.60.213 /kevin_bo/goharbor/harbor-core  ...
v1.0: Pulling from 172.16.60.213 /kevin_bo/goharbor/harbor-core
4e360eca2e60: Pull complete
c066267eb2b9: Pull complete
932afda2a169: Pull complete
7ed16fb7e79a: Pull complete
d09137d80617: Pull complete
588769341947: Pull complete
Digest: sha256:7899f284617bb051180adf6c3aedd140a519d9092b8986dd9058d4dcec0d31de
 
Status: Downloaded newer image  for  172.16.60.213 /kevin_bo/goharbor/harbor-core :v1.0
[root@client ~] # docker pull 172.16.60.213/kevin_bo/redis-photon:v1.0
Trying to pull repository 172.16.60.213 /kevin_bo/redis-photon  ...
v1.0: Pulling from 172.16.60.213 /kevin_bo/redis-photon
4e360eca2e60: Already exists
b08cc3be5c43: Pull complete
a750a309c85d: Pull complete
49b2d8335a1a: Pull complete
31e8f89dc042: Pull complete
Digest: sha256:1e2ce8e6a852713d789c6315642d1483d1efdb4acee4699817810bef219ec93d
Status: Downloaded newer image  for  172.16.60.213 /kevin_bo/redis-photon :v1.0
 
[root@client ~] # docker images
REPOSITORY                                    TAG                 IMAGE ID            CREATED             SIZE
172.16.60.213 /kevin_bo/redis-photon            v1.0                66d7402d2770        10 days ago         103 MB
172.16.60.213 /kevin_bo/goharbor/harbor-core    v1.0                b07a1a4be17f        10 days ago         135 MB

能夠在登陸Harbor web界面以後，修改相關用戶的密碼。在不一樣用戶帳號下建立項目，以及推送和拉取harbor鏡像操做。

3.6.3）Harbor的https證書啓用
經過上面可知，harbor默認安裝後採用的是http方式，後面使用的時候可能會發現不少不方面。由於Docker客戶端登陸harbor進行鏡像推送或拉取時默認是https方式！因此http方式下，須要在每一臺harbor客戶端機器上都要設置"insecure-registries", 感受很麻煩！因此最好仍是將harbor默認的http方式改成https方式！另外，從安全角度考慮，容器的倉庫在生產環境中每每也是須要被設定爲https的方式，而harbor將這些證書的建立和設定都進行了簡單的集成，下面來看一下在harbor下如何使用https的方式。配置記錄以下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
在建立證書以前，爲了方面驗證，須要將前面在客戶端機器上 /etc/docker/daemon .json文件裏添加 "insecure-registries" 配置刪除
[root@docker-client ~] # vim /etc/docker/daemon.json
{}
  
[root@docker-client ~] # rm -rf /root/.docker
    
重啓docker服務
[root@docker-client ~] # systemctl restart docker
  
將harbor部署機自身的 /etc/docker/daemon .json文件裏添加 "insecure-registries" 配置也刪除
[root@harbor-node ~] # vim /etc/docker/daemon.json
{
   "registry-mirrors" : [ "https://v5d7kh0f.mirror.aliyuncs.com" ]
}
  
[root@harbor-node ~] # rm -rf /root/.docker
  
而後重啓docker和docker-compose
[root@harbor-node ~] # systemctl restart docker
[root@harbor-node ~] # docker-compose down -t
[root@harbor-node ~] # docker-compose up -d
  
1）建立CA
[root@harbor-node harbor] # pwd
/root/harbor
[root@harbor-node harbor] # mkdir ssl
[root@harbor-node harbor] # cd ssl/
[root@harbor-node ssl] # pwd
/root/harbor/ssl
  
[root@harbor-node ssl] # openssl req  -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 365 -out ca.crt
Generating a 4096 bit RSA private key
......................................++
...................................................................................................................................................................................................................................................................++
writing new private key to  'ca.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter  '.' , the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:beijing
Locality Name (eg, city) [Default City]:beijing
Organization Name (eg, company) [Default Company Ltd]:DevOps               
Organizational Unit Name (eg, section) []:Tec
Common Name (eg, your name or your server's  hostname ) []:172.16.60.213
Email Address []:wangshibo@kevin.com
  
2) 建立證書請求文件csr
[root@harbor-node ssl] # openssl req -newkey rsa:4096 -nodes -sha256 -keyout 172.16.60.213.key  -out 172.16.60.213.csr
Generating a 4096 bit RSA private key
.++
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................++
writing new private key to  '172.16.60.213.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter  '.' , the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:beijing
Locality Name (eg, city) [Default City]:beijing
Organization Name (eg, company) [Default Company Ltd]:DevOps
Organizational Unit Name (eg, section) []:Tec
Common Name (eg, your name or your server's  hostname ) []:172.16.60.213
Email Address []:wangshibo@kevin.com
  
Please enter the following  'extra'  attributes
to be sent with your certificate request
A challenge password []:123456
An optional company name []:DevOps
  
3) 建立證書
[root@harbor-node ssl] # echo subjectAltName = IP:172.16.60.213 > extfile.cnf
[root@harbor-node ssl] # openssl x509 -req -days 365 -in 172.16.60.213.csr -CA ca.crt -CAkey ca.key -CAcreateserial -extfile extfile.cnf -out 172.16.60.213.crt
Signature ok
subject= /C =CN /ST =beijing /L =beijing /O =DevOps /OU =Tec /CN =172.16.60.213 /emailAddress =wangshibo@kevin.com
Getting CA Private Key
  
4) 設定證書 & 修改
查看證書所在路徑, 後面將harbor.yaml文件中的路徑也一樣設定
[root@harbor-node ssl] # pwd
/root/harbor/ssl
[root@harbor-node ssl] # ll
total 28
-rw-r--r-- 1 root root 2033 May 28 01:16 172.16.60.213.crt
-rw-r--r-- 1 root root 1809 May 28 01:15 172.16.60.213.csr
-rw-r--r-- 1 root root 3272 May 28 01:15 172.16.60.213.key
-rw-r--r-- 1 root root 2114 May 28 01:13 ca.crt
-rw-r--r-- 1 root root 3268 May 28 01:13 ca.key
-rw-r--r-- 1 root root   17 May 28 01:16 ca.srl
-rw-r--r-- 1 root root   34 May 28 01:16 extfile.cnf
  
5) 修改harbor.yaml文件
先關閉docker-compose
[root@harbor-node harbor] # pwd
/root/harbor
[root@harbor-node harbor] # docker-compose down -v
Stopping nginx             ...  done
Stopping harbor-jobservice ...  done
Stopping harbor-portal     ...  done
Stopping harbor-core       ...  done
Stopping harbor-db         ...  done
Stopping registryctl       ...  done
Stopping redis             ...  done
Stopping registry          ...  done
Stopping harbor-log        ...  done
Removing nginx             ...  done
Removing harbor-jobservice ...  done
Removing harbor-portal     ...  done
Removing harbor-core       ...  done
Removing harbor-db         ...  done
Removing registryctl       ...  done
Removing redis             ...  done
Removing registry          ...  done
Removing harbor-log        ...  done
Removing network harbor_harbor
  
[root@harbor-node harbor] # docker-compose ps
Name   Command   State   Ports
------------------------------
  
[root@harbor-node harbor] # vim harbor.yml
.................
.................
  
# http related config
#http:
   # port for http, default is 80. If https enabled, this port will redirect to https port
   #port: 80
  
# https related config
https:
#   # https port for harbor, default is 443
     port: 443
#   # The path of cert and key files for nginx
     certificate:  /root/harbor/ssl/172 .16.60.213.crt
     private_key:  /root/harbor/ssl/172 .16.60.213.key
.................
.................
  
================================================================================================================
特別注意：
上面harbor.yaml文件中修改的配置格式必定要正確！ "https" 要頂格寫， "port:443"  和  "certificate" 、 "private_key" 保持縮進一致！
不然在下面執行 "./prepare" 更新命令時，會報錯：
[root@harbor-node harbor] # ./prepare
..........
   File  "/usr/lib/python3.6/site-packages/yaml/composer.py" , line 84,  in  compose_node
     node = self.compose_mapping_node(anchor)
   File  "/usr/lib/python3.6/site-packages/yaml/composer.py" , line 127,  in  compose_mapping_node
     while  not self.check_event(MappingEndEvent):
   File  "/usr/lib/python3.6/site-packages/yaml/parser.py" , line 98,  in  check_event
     self.current_event = self.state()
   File  "/usr/lib/python3.6/site-packages/yaml/parser.py" , line 439,  in  parse_block_mapping_key
     "expected <block end>, but found %r"  % token. id , token.start_mark)
yaml.parser.ParserError:  while  parsing a block mapping
   in  "/input/harbor.yml" , line 15, column 4
expected <block end>, but found  '<block mapping start>'
   in  "/input/harbor.yml" , line 17, column 5
   
上面的報錯，就是因爲harbor.yaml文件配置格式不正確致使的！！！！
================================================================================================================
  
接着執行prepare腳本，將harbor修改的配置更新到 docker-compose.yml 文件
[root@harbor-node harbor] # ./prepare
prepare base  dir  is  set  to  /root/harbor
Clearing the configuration  file :  /config/log/logrotate .conf
Clearing the configuration  file :  /config/nginx/nginx .conf
Clearing the configuration  file :  /config/core/env
Clearing the configuration  file :  /config/core/app .conf
Clearing the configuration  file :  /config/registry/config .yml
Clearing the configuration  file :  /config/registry/root .crt
Clearing the configuration  file :  /config/registryctl/env
Clearing the configuration  file :  /config/registryctl/config .yml
Clearing the configuration  file :  /config/db/env
Clearing the configuration  file :  /config/jobservice/env
Clearing the configuration  file :  /config/jobservice/config .yml
Generated configuration  file :  /config/log/logrotate .conf
Generated configuration  file :  /config/nginx/nginx .conf
Generated configuration  file :  /config/core/env
Generated configuration  file :  /config/core/app .conf
Generated configuration  file :  /config/registry/config .yml
Generated configuration  file :  /config/registryctl/env
Generated configuration  file :  /config/db/env
Generated configuration  file :  /config/jobservice/env
Generated configuration  file :  /config/jobservice/config .yml
loaded secret from  file :  /secret/keys/secretkey
Generated configuration  file :  /compose_location/docker-compose .yml
Clean up the input  dir
  
查看一下docker-compose.yml文件，發現已經將新配置的443端口的https信息更新到docker-compose.yml文件裏了
以下80端口和443端口都配置了，因此harbor訪問時是http強轉到https的
[root@harbor-node harbor] # cat docker-compose.yml |grep 443 -C3
     dns_search: .
     ports:
       - 80:80
       - 443:443
     depends_on:
       - postgresql
       - registry
  
重啓docker-compose
[root@harbor-node harbor] # docker-compose up -d
Creating network  "harbor_harbor"  with the default driver
Creating harbor-log ...  done
Creating registry    ...  done
Creating harbor-db   ...  done
Creating registryctl ...  done
Creating redis       ...  done
Creating harbor-core ...  done
Creating harbor-jobservice ...  done
Creating harbor-portal     ...  done
Creating nginx             ...  done
  
[root@harbor-node harbor] # docker-compose ps
       Name                     Command                  State                        Ports                
------------------------------------------------------------------------------------------------------------
harbor-core          /harbor/start .sh                 Up (healthy)                                         
harbor-db            /entrypoint .sh postgres          Up (healthy)   5432 /tcp                              
harbor-jobservice    /harbor/start .sh                 Up                                                   
harbor-log           /bin/sh  -c  /usr/local/bin/  ...   Up (healthy)   127.0.0.1:1514->10514 /tcp             
harbor-portal       nginx -g daemon off;             Up (healthy)   80 /tcp                                
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:443->443 /tcp , 0.0.0.0:80->80 /tcp
redis               docker-entrypoint.sh redis ...   Up             6379 /tcp                              
registry             /entrypoint .sh  /etc/regist  ...   Up (healthy)   5000 /tcp                              
registryctl          /harbor/start .sh                 Up (healthy)
  
在harbor部署機本機確認login登錄 (使用80端口或443端口均可以，自動跳轉的)
[root@harbor-node harbor] # docker login -u admin -p kevin@Bo1987 172.16.60.213
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https: //172 .16.60.213 /v2/ : x509: certificate signed by unknown authority
  
[root@harbor-node harbor] # docker login -u admin -p kevin@Bo1987 172.16.60.213:443
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https: //172 .16.60.213:443 /v2/ : x509: certificate signed by unknown authority
  
以上出現報錯，解決辦法：
此種狀況多發生在自簽名的證書，報錯含義是簽發證書機構未經認證，沒法識別。
 
解決辦法：
[root@harbor-node harbor] # chmod 644 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
[root@harbor-node harbor] # cat /root/harbor/ssl/172.16.60.213.crt >> /etc/pki/tls/certs/ca-bundle.crt
[root@harbor-node harbor] # chmod 444 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
 
因爲證書是docker的daemon須要用到的，因此須要重啓docker服務，進而也要重啓docker-compose
[root@harbor-node harbor] # systemctl restart docker
[root@harbor-node harbor] # docker-compose down -v
[root@harbor-node harbor] # docker-compose up -d
 
而後再次嘗試在harbor本機登錄, 發現就能夠正常登錄了！！
[root@harbor-node harbor] # docker login -u admin -p kevin@Bo1987 172.16.60.213
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted  in  /root/ .docker /config .json.
Configure a credential helper to remove this warning. See
https: //docs .docker.com /engine/reference/commandline/login/ #credentials-store
 
Login Succeeded
 
登錄的權限信息保存到 /root/ .docker /config .json 文件裏了，只要這個文件不刪除，下次就能夠不須要輸入用戶名和密碼直接登錄了！
[root@harbor-node harbor] # cat /root/.docker/config.json
{
         "auths" : {
                 "172.16.60.213" : {
                         "auth" :  "YWRtaW46a2V2aW5AQk9CTzEyMw=="
                 }
         },
         "HttpHeaders" : {
                 "User-Agent" :  "Docker-Client/18.09.6 (linux)"
         }
}
 
[root@harbor-node harbor] # docker login 172.16.60.213
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted  in  /root/ .docker /config .json.
Configure a credential helper to remove this warning. See
https: //docs .docker.com /engine/reference/commandline/login/ #credentials-store
 
Login Succeeded
 
上面是使用80端口登錄的，後面加上443端口也是能夠登錄的
[root@harbor-node harbor] # docker login 172.16.60.213:443
Username: admin
Password:
WARNING! Your password will be stored unencrypted  in  /root/ .docker /config .json.
Configure a credential helper to remove this warning. See
https: //docs .docker.com /engine/reference/commandline/login/ #credentials-store
 
Login Succeeded
 
[root@harbor-node harbor] # docker login 172.16.60.213:443
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted  in  /root/ .docker /config .json.
Configure a credential helper to remove this warning. See
https: //docs .docker.com /engine/reference/commandline/login/ #credentials-store
 
Login Succeeded
 
[root@harbor-node ssl] # docker login -u admin -p kevin@Bo1987 172.16.60.213:443
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted  in  /root/ .docker /config .json.
Configure a credential helper to remove this warning. See
https: //docs .docker.com /engine/reference/commandline/login/ #credentials-store
 
Login Succeeded
 
==========================================================================================
上面是在harbor本機嘗試的登錄，如今在遠程客戶機上(這裏客戶機爲172.16.60.214)測試harbor登錄：
 
首先很重要的一步,這一步極其關鍵！必定不要忘記操做！！
就是須要將harbor服務端生成的CA證書拷貝到每一個遠程客戶機的 "/etc/docker/certs.d/harbor服務器的域名或ip/"  目錄下
[root@client ~] # mkdir /etc/docker/certs.d/172.16.60.213/
 
接着在harbor服務器將CA證書拷貝過來
[root@harbor-node ssl] # rsync -e "ssh -p22" -avpgolr ./* root@172.16.60.214:/etc/docker/certs.d/172.16.60.213/
 
而後在客戶機上查看是否拷貝過來了harbor服務端的CA證書
[root@client 172.16.60.213] # pwd
/etc/docker/certs .d /172 .16.60.213
[root@client 172.16.60.213] # ls
172.16.60.213.crt  172.16.60.213.csr  172.16.60.213.key  ca.crt  ca.key  ca.srl  extfile.cnf
 
進行一樣的受權操做，
[root@client 172.16.60.213] # chmod 644 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
[root@client 172.16.60.213] # cat /etc/docker/certs.d/172.16.60.213/172.16.60.213.crt >> /etc/pki/tls/certs/ca-bundle.crt
[root@client 172.16.60.213] # chmod 444 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
 
重啓docker服務
[root@client 172.16.60.213] # systemctl restart docker
 
最後進行harbor登錄，就能夠正常登錄了！
[root@client 172.16.60.213] # docker login -u admin -p kevin@Bo1987 172.16.60.213:443
Login Succeeded
 
[root@client 172.16.60.213] # cat /root/.docker/config.json                          
{
         "auths" : {
                 "172.16.60.213:443" : {
                         "auth" :  "YWRtaW46a2V2aW5AQk9CTzEyMw=="
                 }
         }
}
 
上面是使用443端口登錄harbor倉庫是正常的，若是此時使用80端口登錄，則出現以下報錯：
[root@client 172.16.60.213] # docker login -u admin -p kevin@Bo1987 172.16.60.213
Error response from daemon: Missing client certificate 172.16.60.213.cert  for  key 172.16.60.213.key
 
因此在客戶端就使用443端口來登錄harbor倉庫了！

3.6.4) Harbor私倉的高可用
經過三個harbor完成高可用部署，前面經過負載均衡器對外提供服務。共享數據庫與緩存。結構以下 ：

還有一種比較簡單的方法：部署2臺harbor服務器，配置部署同樣，實現原理:

建立docker harbor主主複製就是在兩個harbor私倉的web界面裏建立相互之間的鏡像同步關係，同步關係能夠選擇相同用戶或不一樣用戶之間。這樣就能夠保證harbor私倉的雙機熱備關係了。即：1) "系統管理"->"倉庫管理"->"新建目標", 填寫對端harbor信息2) "系統管理"->"同步管理"->"新建規則", 規則裏會引用目的Registry，也就是上面一步建立的目標。同步模式有Push-based，Pull-based；觸發模式有自動和定時。

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程