Harbor-hlem鏡像庫從新部署後PV數據恢復

時間 2021-01-19

標籤 node git github redis api app elasticsearch ide this 欄目 Git 简体版

原文原文鏈接

原由node

開發反饋habor鏡像庫登錄不了，初步查看是證書過時了。git

解決方案
以前Harbor-helm部署鏡像庫文檔能夠回顧連接Kubernetes1.13.1集羣集成Harbor-helm
1.首先新建新證書的secretgithub

[root@elasticsearch01 harbor-helm]#  kubectl  create secret tls ingress-secret2021 --key minminmsnauto.key --cert minminmsnauto.crt

2.而後修改harbor-helm的value.yaml，把secretName替換下redis

[root@elasticsearch01 harbor-helm]# head -n 20 values.yaml
expose:
  # Set the way how to expose the service. Set the type as "ingress", 
  # "clusterIP" or "nodePort" and fill the information in the corresponding 
  # section
  type: ingress
  tls:
    # Enable the tls or not. Note: if the type is "ingress" and the tls 
    # is disabled, the port must be included in the command when pull/push
    # images. Refer to https://github.com/goharbor/harbor/issues/5291 
    # for the detail.
    enabled: true
    # Fill the name of secret if you want to use your own TLS certificate
    # and private key. The secret must contain keys named tls.crt and 
    # tls.key that contain the certificate and private key to use for TLS
    # The certificate and private key will be generated automatically if 
    # it is not set
    secretName: "ingress-secret2021"
    # By default, the Notary service will use the same cert and key as
    # described above. Fill the name of secret if you want to use a 
    # separated one. Only needed when the type is "ingress".

3.最後使用helm upgrade更新版本api

[root@elasticsearch01 harbor-helm]#  helm upgrade  minminmsn . -f values.yaml

到這個時候應該能解決需求，但是事與願違，不知道哪兒除了問題，這時登錄Harbor證書問題是解決了，可是項目及庫訪問不了提示內部錯誤，看Pod的運行狀態也都是Running。
最後打算使用helm先delete掉再install，可是這樣建立的harbor看起來一切正常，其實是個初始化環境，是自動生成的新PV並無原來的數據。此時發現原來的PV還在，下面就開始找PV恢復的方案。app

調整PV狀態elasticsearch

1.查詢此時PV與PVC狀態ide

[root@elasticsearch01 harbor-helm]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                            STORAGECLASS   REASON   AGE                    9h
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Released   default/minminmsn-harbor-chartmuseum                rbd                     417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/minminmsn-harbor-jobservice                 rbd                     417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Released   default/minminmsn-harbor-registry                   rbd                     417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/database-data-minminmsn-harbor-database-0   rbd                     417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/data-minminmsn-harbor-redis-0               rbd                     417d
[root@elasticsearch01 harbor-helm]# kubectl get pvc
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-minminmsn-harbor-redis-0               Bound    pvc-6cd422e4-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9h
database-data-minminmsn-harbor-database-0   Bound    pvc-6ccda00b-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9h
minminmsn-harbor-chartmuseum                Bound    pvc-6c903857-c5f0-11ea-9386-52540089b2b6   50Gi       RWO            rbd            9h
minminmsn-harbor-jobservice                 Bound    pvc-6c91d1a4-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9h
minminmsn-harbor-registry                   Bound    pvc-6c92bfc0-c5f0-11ea-9386-52540089b2b6   500Gi      RWO            rbd            9h

2.修改PV狀態
先把PV的狀態由Released改變成
備註：默認建立的PV的回收策略是Delete就是用完就刪除，以前特地把RECLAIM POLICY改成了Retain，在線修改PV回收策略能夠參考文檔在線修改PV的回收策略，不然這裏Helm Delete後就會自動刪除PV，就沒有後來這篇PV數據恢復操做了。
在線編輯PV,須要把其中claimRef這段刪除，這樣狀態就能夠變成Available了。ui

claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: minminmsn-harbor-chartmuseum
    namespace: default
    resourceVersion: "91736092"
    uid: b31ec8ca-c649-11ea-9386-52540089b2b6
  persistentVolumeReclaimPolicy: Retain

具體以下修改 this

[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: ceph.com/rbd
    rbdProvisionerIdentity: ceph.com/rbd
  creationTimestamp: "2019-05-24T06:33:55Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6
  resourceVersion: "91736100"
  selfLink: /api/v1/persistentvolumes/pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6
  uid: e7ade7f7-7ded-11e9-a09d-52540089b2b6
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: minminmsn-harbor-chartmuseum
    namespace: default
    resourceVersion: "91736092"
    uid: b31ec8ca-c649-11ea-9386-52540089b2b6
  persistentVolumeReclaimPolicy: Retain
  rbd:
    image: kubernetes-dynamic-pvc-e79b34d3-7ded-11e9-ac1b-02420afe4905
    keyring: /etc/ceph/keyring
    monitors:
    - 10.0.4.8:6789
    pool: rbd-k8s
    secretRef:
      name: ceph-secret
      namespace: default
    user: admin
  storageClassName: rbd
  volumeMode: Filesystem
status:
  phase: Released

3.其餘四個PV一樣操做

[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7985b55-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7d38097-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6

4.查看效果
如今看PV的STATUS已經變成了Available，而後CLAIM也變空了，這樣就能夠在後面綁定使用了

[root@elasticsearch01 harbor-helm]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Available                               rbd                     417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Available                               rbd                     417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d

建立PVC

1.先設置好PVC及PV對應關係

[root@elasticsearch01 yaml]# cat minminmsn.pvc 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minminmsn-harbor-registry
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 2000Gi
  volumeName: "pvc-e7985b55-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minminmsn-harbor-jobservice
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 20Gi
  volumeName: "pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minminmsn-harbor-chartmuseum
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 50Gi
  volumeName: "pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-data-minminmsn-harbor-database-0
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 20Gi
  volumeName: "pvc-e7d38097-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-minminmsn-harbor-redis-0
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 20Gi
  volumeName: "pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6"

2.建立PVC

[root@elasticsearch01 yaml]# kubectl apply -f minminmsn.pvc 
persistentvolumeclaim/minminmsn-harbor-registry created
persistentvolumeclaim/minminmsn-harbor-jobservice created
persistentvolumeclaim/minminmsn-harbor-chartmuseum created
persistentvolumeclaim/database-data-minminmsn-harbor-database-0 created
persistentvolumeclaim/data-minminmsn-harbor-redis-0 created

3.檢查PV與PVC

[root@elasticsearch01 yaml]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                            STORAGECLASS   REASON   AGE
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Bound    default/minminmsn-harbor-chartmuseum                rbd                     417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/minminmsn-harbor-jobservice                 rbd                     417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Bound    default/minminmsn-harbor-registry                   rbd                     417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/database-data-minminmsn-harbor-database-0   rbd                     417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/data-minminmsn-harbor-redis-0               rbd                     417d
[root@elasticsearch01 yaml]# kubectl get pvc
NAME                                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ceph-rbd-pv-claim                        Bound     ceph-rbd-pv                                20Gi       RWO                           540d
data-minminmsn-harbor-redis-0               Pending   pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
database-data-minminmsn-harbor-database-0   Pending   pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
minminmsn-harbor-chartmuseum                Pending   pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
minminmsn-harbor-jobservice                 Pending   pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
minminmsn-harbor-registry                   Bound     pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            rbd            12s
[root@elasticsearch01 yaml]# kubectl describe pvc minminmsn-harbor-registry
Name:          minminmsn-harbor-registry
Namespace:     default
StorageClass:  rbd
Status:        Bound
Volume:        pvc-e7985b55-7ded-11e9-a09d-52540089b2b6
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"minminmsn-harbor-registry","namespace":"default"},"spe...
               pv.kubernetes.io/bind-completed: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2000Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Events:        <none>
Mounted By:    <none>

使用Hlem從新部署Harbor鏡像庫

1.部署前先刪除版本

[root@elasticsearch01 harbor-helm]# helm delete --purge minminmsn
helm delete --purge minminmsn
release "minminmsn" deleted

2.修改Harbor-helm的values.yaml中PVC相關值
注意existingClaim: ""由空值改爲上面生成的PVC名字，注意對應關係，其餘不變，具體變動以下

persistence:
  enabled: true
  # Setting it to "keep" to avoid removing PVCs during a helm delete 
  # operation. Leaving it empty will delete PVCs after the chart deleted
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      # Use the existing PVC which must be created manually before bound
      existingClaim: "minminmsn-harbor-registry"
      # Specify the "storageClass" used to provision the volume. Or the default
      # StorageClass will be used(the default).
      # Set it to "-" to disable dynamic provisioning
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 2000Gi
    chartmuseum:
      existingClaim: "minminmsn-harbor-chartmuseum"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 50Gi
    jobservice:
      existingClaim: "minminmsn-harbor-jobservice"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi
    # If external database is used, the following settings for database will 
    # be ignored
    database:
      existingClaim: "database-data-minminmsn-harbor-database-0"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi
    # If external Redis is used, the following settings for Redis will 
    # be ignored
    redis:
      existingClaim: "data-minminmsn-harbor-redis-0"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi

3.從新部署

[root@elasticsearch01 harbor-helm]# helm  install . --name minminmsn
NAME:   minminmsn
LAST DEPLOYED: Wed Jul 15 11:18:13 2020
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                         TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)            AGE
minminmsn-harbor-adminserver    ClusterIP  10.254.58.23    <none>       80/TCP             1s
minminmsn-harbor-chartmuseum    ClusterIP  10.254.154.44   <none>       80/TCP             1s
minminmsn-harbor-clair          ClusterIP  10.254.25.107   <none>       6060/TCP           1s
minminmsn-harbor-core           ClusterIP  10.254.56.153   <none>       80/TCP             1s
minminmsn-harbor-database       ClusterIP  10.254.65.18    <none>       5432/TCP           1s
minminmsn-harbor-jobservice     ClusterIP  10.254.81.97    <none>       80/TCP             1s
minminmsn-harbor-notary-server  ClusterIP  10.254.99.90    <none>       4443/TCP           1s
minminmsn-harbor-notary-signer  ClusterIP  10.254.175.105  <none>       7899/TCP           1s
minminmsn-harbor-portal         ClusterIP  10.254.242.113  <none>       80/TCP             1s
minminmsn-harbor-redis          ClusterIP  10.254.127.40   <none>       6379/TCP           1s
minminmsn-harbor-registry       ClusterIP  10.254.158.222  <none>       5000/TCP,8080/TCP  1s

==> v1/Deployment
NAME                         DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
minminmsn-harbor-adminserver    1        1        1           0          1s
minminmsn-harbor-chartmuseum    1        1        1           0          1s
minminmsn-harbor-clair          1        0        0           0          1s
minminmsn-harbor-core           1        0        0           0          1s
minminmsn-harbor-jobservice     1        0        0           0          1s
minminmsn-harbor-notary-server  1        0        0           0          1s
minminmsn-harbor-notary-signer  1        0        0           0          1s
minminmsn-harbor-portal         1        0        0           0          1s
minminmsn-harbor-registry       1        0        0           0          1s

==> v1/StatefulSet
NAME                    DESIRED  CURRENT  AGE
minminmsn-harbor-database  1        1        1s
minminmsn-harbor-redis     1        1        1s

==> v1beta1/Ingress
NAME                   HOSTS                                                    ADDRESS  PORTS  AGE
minminmsn-harbor-ingress  core-harbor.minminmsn.com,notary-harbor.minminmsn.com  80, 443  1s

==> v1/Pod(related)
NAME                                          READY  STATUS             RESTARTS  AGE
minminmsn-harbor-adminserver-b5d58db8c-wmrbd     0/1    ContainerCreating  0         1s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb    0/1    Pending            0         1s
minminmsn-harbor-clair-54465ff7dd-d7bxx          0/1    Pending            0         1s
minminmsn-harbor-core-587cc5d9b5-2xxl9           0/1    Pending            0         1s
minminmsn-harbor-jobservice-764bb697d-wsxqx      0/1    Pending            0         1s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c  0/1    Pending            0         1s
minminmsn-harbor-notary-signer-8466d68f5b-klv76  0/1    Pending            0         1s
minminmsn-harbor-database-0                      0/1    Pending            0         1s
minminmsn-harbor-redis-0                         0/1    Pending            0         1s

==> v1/Secret
NAME                       TYPE    DATA  AGE
minminmsn-harbor-adminserver  Opaque  4     1s
minminmsn-harbor-chartmuseum  Opaque  1     1s
minminmsn-harbor-core         Opaque  4     1s
minminmsn-harbor-database     Opaque  1     1s
minminmsn-harbor-jobservice   Opaque  1     1s
minminmsn-harbor-registry     Opaque  1     1s

==> v1/ConfigMap
NAME                         DATA  AGE
minminmsn-harbor-adminserver    39    1s
minminmsn-harbor-chartmuseum    24    1s
minminmsn-harbor-clair          1     1s
minminmsn-harbor-core           1     1s
minminmsn-harbor-jobservice     1     1s
minminmsn-harbor-notary-server  5     1s
minminmsn-harbor-registry       2     1s

NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at https://core-harbor.minminmsn.com. 
For more details, please visit https://github.com/goharbor/harbor.

3.查看新生成Pods的信息
[root@elasticsearch01 harbor-helm]# kubectl get pods
NAME                                           READY   STATUS              RESTARTS   AGE
minminmsn-harbor-adminserver-b5d58db8c-wmrbd      0/1     ContainerCreating   0          9s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb     0/1     ContainerCreating   0          9s
minminmsn-harbor-clair-54465ff7dd-d7bxx           0/1     Running             0          9s
minminmsn-harbor-core-587cc5d9b5-2xxl9            0/1     Running             0          9s
minminmsn-harbor-database-0                       0/1     Init:0/1            0          9s
minminmsn-harbor-jobservice-764bb697d-wsxqx       0/1     ContainerCreating   0          9s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c   0/1     ContainerCreating   0          9s
minminmsn-harbor-notary-signer-8466d68f5b-klv76   0/1     ContainerCreating   0          9s
minminmsn-harbor-portal-64cf8b9b69-xm8nl          0/1     ContainerCreating   0          8s
minminmsn-harbor-redis-0                          0/1     ContainerCreating   0          9s
minminmsn-harbor-registry-755746c5bb-q8m55        0/2     ContainerCreating   0          8s

再等2分鐘查看就上恢復了

[root@elasticsearch01 harbor-helm]# kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
jenkins-0                                      1/1     Running   0          62d
rbd-provisioner-67b4857bcd-rjwlg               1/1     Running   0          61d
minminmsn-harbor-adminserver-b5d58db8c-wmrbd      1/1     Running   1          2m33s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb     1/1     Running   0          2m33s
minminmsn-harbor-clair-54465ff7dd-d7bxx           1/1     Running   1          2m33s
minminmsn-harbor-core-587cc5d9b5-2xxl9            1/1     Running   1          2m33s
minminmsn-harbor-database-0                       1/1     Running   0          2m33s
minminmsn-harbor-jobservice-764bb697d-wsxqx       1/1     Running   0          2m33s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c   1/1     Running   0          2m33s
minminmsn-harbor-notary-signer-8466d68f5b-klv76   1/1     Running   0          2m33s
minminmsn-harbor-portal-64cf8b9b69-xm8nl          1/1     Running   0          2m32s
minminmsn-harbor-redis-0                          1/1     Running   0          2m33s
minminmsn-harbor-registry-755746c5bb-q8m55        2/2     Running   0          2m32s

4.Harbor控制驗證
證書更新了項目也恢復了
https://core-harbor.minminmsn.com/harbor/projects