10.部署私有 docker registry

10.部署私有 docker registry

注意：本文檔介紹使用 docker 官方的 registry v2 鏡像部署私有倉庫的步驟，你也能夠部署 Harbor 私有倉庫（部署 Harbor 私有倉庫）。

本文檔講解部署一個 TLS 加密、HTTP Basic 認證、用 ceph rgw 作後端存儲的私有 docker registry 步驟，若是使用其它類型的後端存儲，則能夠從 「建立 docker registry」 節開始；

示例兩臺機器 IP 以下：

• ceph rgw: 192.168.1.66
• docker registry: 192.168.1.67

部署 ceph RGW 節點

$ ceph-deploy rgw create 192.168.1.66 # rgw 默認監聽7480端口 $ 

建立測試帳號 demo

$ radosgw-admin user create --uid=demo --display-name="ceph rgw demo user" $ 

建立 demo 帳號的子帳號 swift

當前 registry 只支持使用 swift 協議訪問 ceph rgw 存儲，暫時不支持 s3 協議；

$ radosgw-admin subuser create --uid demo --subuser=demo:swift --access=full --secret=secretkey --key-type=swift $ 

建立 demo:swift 子帳號的 sercret key

$ radosgw-admin key create --subuser=demo:swift --key-type=swift --gen-secret { "user_id": "demo", "display_name": "ceph rgw demo user", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [ { "id": "demo:swift", "permissions": "full-control" } ], "keys": [ { "user": "demo", "access_key": "5Y1B1SIJ2YHKEHO5U36B", "secret_key": "nrIvtPqUj7pUlccLYPuR3ntVzIa50DToIpe7xFjT" } ], "swift_keys": [ { "user": "demo:swift", "secret_key": "ttQcU1O17DFQ4I9xzKqwgUe7WIYYX99zhcIfU9vb" } ], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] } 

• ttQcU1O17DFQ4I9xzKqwgUe7WIYYX99zhcIfU9vb 爲子帳號 demo:swift 的 secret key；

建立 docker registry

建立 registry 使用的 x509 證書

$ mkdir -p registry/{auth,certs} $ cat > registry-csr.json <<EOF { "CN": "registry", "hosts": [ "127.0.0.1", "192.168.1.67" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "4Paradigm" } ] } EOF $ cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \ -ca-key=/etc/kubernetes/cert/ca-key.pem \ -config=/etc/kubernetes/cert/ca-config.json \ -profile=kubernetes registry-csr.json | cfssljson -bare registry $ cp registry.pem registry-key.pem registry/certs $ 

• 這裏複用之前建立的 CA 證書和祕鑰文件；
• hosts 字段指定 registry 的 NodeIP；

建立 HTTP Baisc 認證文件

$ docker run --entrypoint htpasswd registry:2 -Bbn foo foo123 > registry/auth/htpasswd $ cat registry/auth/htpasswd foo:$2y$05$iZaM45Jxlcg0DJKXZMggLOibAsHLGybyU.CgU9AHqWcVDyBjiScN. 

配置 registry 參數

export RGW_AUTH_URL="https://192.168.1.66:7480/auth/v1" export RGW_USER="demo:swift" export RGW_SECRET_KEY="ttQcU1O17DFQ4I9xzKqwgUe7WIYYX99zhcIfU9vb" cat > config.yml << EOF # https://docs.docker.com/registry/configuration/#list-of-configuration-options version: 0.1 log: level: info fromatter: text fields: service: registry storage: cache: blobdescriptor: inmemory delete: enabled: true swift: authurl: ${RGW_AUTH_URL} username: ${RGW_USER} password: ${RGW_SECRET_KEY} container: registry auth: htpasswd: realm: basic-realm path: /auth/htpasswd https: addr: 0.0.0.0:8000 headers: X-Content-Type-Options: [nosniff] tls: certificate: /certs/registry.pem key: /certs/registry-key.pem health: storagedriver: enabled: true interval: 10s threshold: 3 EOF [k8s@kube-node1 cert]$ cp config.yml registry [k8s@kube-node1 cert]$ scp -r registry 192.168.1.67:/opt/k8s 

• storage.swift 指定後端使用 swfit 接口協議的存儲，這裏配置的是 ceph rgw 存儲參數；
• auth.htpasswd 指定了 HTTP Basic 認證的 token 文件路徑；
• http.tls 指定了 registry http 服務器的證書和祕鑰文件路徑；

建立 docker registry

ssh k8s@192.168.1.67
$ docker run -d -p 8000:8000 --privileged \
    -v /opt/k8s/registry/auth/:/auth \ -v /opt/k8s/registry/certs:/certs \ -v /opt/k8s/registry/config.yml:/etc/docker/registry/config.yml \ --name registry registry:2 

• 執行該 docker run 命令的機器 IP 爲 192.168.1.67；

向 registry push image

將簽署 registry 證書的 CA 證書拷貝到 /etc/docker/certs.d/192.168.1.67:8000 目錄下

[k8s@kube-node1 cert]$ sudo mkdir -p /etc/docker/certs.d/192.168.1.67:8000 [k8s@kube-node1 cert]$ sudo cp /etc/kubernetes/cert/ca.pem /etc/docker/certs.d/192.168.1.67:8000/ca.crt 

登錄私有 registry

$ docker login 192.168.1.67:8000 Username: foo Password: Login Succeeded 

登錄信息被寫入 ~/.docker/config.json 文件

$ cat ~/.docker/config.json { "auths": { "192.168.1.67:8000": { "auth": "Zm9vOmZvbzEyMw==" } } } 

將本地的 image 打上私有 registry 的 tag

$ docker tag prom/node-exporter:v0.16.0 192.168.1.67:8000/prom/node-exporter:v0.16.0 $ docker images |grep pause prom/node-exporter:v0.16.0 latest f9d5de079539 2 years ago 239.8 kB 192.168.1.67:8000/prom/node-exporter:v0.16.0 latest f9d5de079539 2 years ago 239.8 kB 

將 image push 到私有 registry

$ docker push 192.168.1.67:8000/prom/node-exporter:v0.16.0 The push refers to a repository [192.168.1.67:8000/prom/node-exporter:v0.16.0] 5f70bf18a086: Pushed e16a89738269: Pushed latest: digest: sha256:9a6b437e896acad3f5a2a8084625fdd4177b2e7124ee943af642259f2f283359 size: 916 

查看 ceph 上是否已經有 push 的 pause 容器文件

[k8s@kube-node1 ~]$ rados lspools
rbd
cephfs_data
cephfs_metadata
.rgw.root
k8s
default.rgw.control default.rgw.meta default.rgw.log default.rgw.buckets.index default.rgw.buckets.data [k8s@kube-node1 ~]$ rados --pool default.rgw.buckets.data ls|grep node-exporter 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/cdb7590af5f064887f3d6008d46be65e929c74250d747813d85199e04fc70463/link 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_manifests/revisions/sha256/55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086/link 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_manifests/tags/v0.16.0/current/link 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_manifests/tags/v0.16.0/index/sha256/55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086/link 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c/link 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/528dda9cf23d0fad80347749d6d06229b9a19903e49b7177d5f4f58736538d4e/link 1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/188af75e2de0203eac7c6e982feff45f9c340eaac4c7a0f59129712524fa2984/link 

私有 registry 的運維操做

查詢私有鏡像中的 images

[k8s@kube-node1 ~]$ curl --user foo:foo123 --cacert /etc/docker/certs.d/192.168.1.67\:8000/ca.crt https://192.168.1.67:8000/v2/_catalog {"repositories":["prom/node-exporter"]} 

查詢某個鏡像的 tags 列表

[k8s@kube-node1 ~]$ curl --user foo:foo123 --cacert /etc/docker/certs.d/192.168.1.67\:8000/ca.crt https://192.168.1.67:8000/v2/prom/node-exporter/tags/list {"name":"prom/node-exporter","tags":["v0.16.0"]} 

獲取 image 或 layer 的 digest

向 v2/<repoName>/manifests/<tagName> 發 GET 請求，從響應的頭部 Docker-Content-Digest 獲取 image digest，從響應的 body 的 fsLayers.blobSum 中獲取 layDigests;

注意，必須包含請求頭：Accept: application/vnd.docker.distribution.manifest.v2+json：

[k8s@kube-node1 ~]$ curl -v -H "Accept: application/vnd.docker.distribution.manifest.v2+json" --user foo:foo123 --cacert /etc/docker/certs.d/192.168.1.67\:8000/ca.crt https://192.168.1.67:8000/v2/prom/node-exporter/manifests/v0.16.0 * About to connect() to 192.168.1.67 port 8000 (#0) * Trying 192.168.1.67... * Connected to 192.168.1.67 (192.168.1.67) port 8000 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/docker/certs.d/192.168.1.67:8000/ca.crt CApath: none * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=registry,OU=4Paradigm,O=k8s,L=BeiJing,ST=BeiJing,C=CN * start date: Jul 05 12:52:00 2018 GMT * expire date: Jul 02 12:52:00 2028 GMT * common name: registry * issuer: CN=kubernetes,OU=4Paradigm,O=k8s,L=BeiJing,ST=BeiJing,C=CN * Server auth using Basic with user 'foo' > GET /v2/prom/node-exporter/manifests/v0.16.0 HTTP/1.1 > Authorization: Basic Zm9vOmZvbzEyMw== > User-Agent: curl/7.29.0 > Host: 192.168.1.67:8000 > Accept: application/vnd.docker.distribution.manifest.v2+json > < HTTP/1.1 200 OK < Content-Length: 949 < Content-Type: application/vnd.docker.distribution.manifest.v2+json < Docker-Content-Digest: sha256:55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086 < Docker-Distribution-Api-Version: registry/2.0 < Etag: "sha256:55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086" < X-Content-Type-Options: nosniff < Date: Fri, 06 Jul 2018 06:18:41 GMT < { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 3511, "digest": "sha256:188af75e2de0203eac7c6e982feff45f9c340eaac4c7a0f59129712524fa2984" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 2392417, "digest": "sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 560703, "digest": "sha256:cdb7590af5f064887f3d6008d46be65e929c74250d747813d85199e04fc70463" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 5332460, "digest": "sha256:528dda9cf23d0fad80347749d6d06229b9a19903e49b7177d5f4f58736538d4e" } ] 

刪除 image

向 /v2/<name>/manifests/<reference> 發送 DELETE 請求，reference 爲上一步返回的 Docker-Content-Digest 字段內容：

$ curl -X DELETE --user foo:foo123 --cacert /etc/docker/certs.d/192.168.1.67\:8000/ca.crt https://192.168.1.67:8000/v2/prom/node-exporter/manifests/sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5 $ 

刪除 layer

向 /v2/<name>/blobs/<digest>發送 DELETE 請求，其中 digest 是上一步返回的 fsLayers.blobSum 字段內容：

$ curl -X DELETE --user foo:foo123 --cacert /etc/docker/certs.d/192.168.1.67\:8000/ca.crt https://192.168.1.67:8000/v2/prom/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 $ curl -X DELETE --cacert /etc/docker/certs.d/192.168.1.67\:8000/ca.crt https://192.168.1.67:8000/v2/prom/node-exporter/blobs/sha256:04176c8b224aa0eb9942af765f66dae866f436e75acef028fe44b8a98e045515 $ 

常見問題

login 失敗 416

執行 https://docs.ceph.com/docs/master/install/install-ceph-gateway/ 裏面的 s3 test.py 程序失敗：

[k8s@kube-node1 cert]$ python s3test.py
Traceback (most recent call last):
File "s3test.py", line 12, in
bucket = conn.create_bucket('my-new-bucket')
File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 625, in create_bucket
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 416 Requested Range Not Satisfiable

解決版辦法：

1. 在管理節點上修改 ceph.conf
2. ceph-deploy config push kube-node1 kube-node2 kube-node3
3. systemctl restart 'ceph-mds@kube-node3.service'
   systemctl restart ceph-osd@0
   systemctl restart 'ceph-mon@kube-node1.service'
   systemctl restart 'ceph-mgr@kube-node1.service'

For anyone who is hitting this issue
set default pg_num and pgp_num to lower value(8 for example), or set mon_max_pg_per_osd to a high value in ceph.conf
radosgw-admin doesn' throw proper error when internal pool creation fails, hence the upper level error which is very confusing.

https://tracker.ceph.com/issues/21497

login 失敗 503

[root@kube-node1 ~]# docker login 192.168.1.67:8000
Username: foo
Password:
Error response from daemon: login attempt to https://192.168.1.67:8000/v2/ failed with status: 503 Service Unavailable

緣由： docker run 缺乏 --privileged 參數；

連接：https://www.orchome.com/663 著做權歸做者全部。商業轉載請聯繫做者得到受權，非商業轉載請註明出處。