kubernetes中部署Heketi和GlusterFS(二)

時間 2021-01-19

標籤 node git github json bootstrap ubuntu api bash tcp 分佈式欄目 Git 简体版

原文原文鏈接

kubernetes中部署Heketi和GlusterFS(二)
在上一節中，Heketi的部署方式還不能用於生產環境，由於Heketi Pod的數據並無持久化，容易致使heketi的數據丟失，Heketi的數據保存在/var/lib/heketi/heketi.db文件中，所以須要把此目錄掛載到GlusterFS分佈式存儲中。node

按照上一節的步驟，執行heketi-cli topology load --json=topology-sample.jsongit

$ echo $HEKETI_CLI_SERVER
http://10.254.49.43:8080

$ heketi-cli topology load --json=topology-sample.json
Creating cluster ... ID: 5b930ef6081fd22e895c25a3dfb0c516
    Allowing file volumes on cluster.
    Allowing block volumes on cluster.
    Creating node 10.30.1.15 ... ID: b120572be40db6c1d979c3903876430b
        Adding device /dev/sdb ... OK
    Creating node 10.30.1.16 ... ID: 7ce13ffc5eabe64a3791e93233fd3c1a
        Adding device /dev/sdb ... OK
    Creating node 10.30.1.17 ... ID: f9abdc2e5d4cfa17c035a97f984a1a3b
        Adding device /dev/sdb ... OK

緊接着執行：將生成heketi-storage.json文件。github

$ heketi-cli setup-openshift-heketi-storage
Saving heketi-storage.json

若是在運行setup-openshift-heketi-storage子命令時heketi-cli報告「無空間」錯誤：
$ heketi-cli setup-openshift-heketi-storage
Error: Failed to allocate new volume: No space
則可能無心中運行topology load命令的時候，服務端和heketi-cli的版本不匹配形成的。json

中止正在運行的heketi pod：
kubectl scale deployment deploy-heketi --replicas=0

手動刪除存儲塊設備中的任何簽名：
加載拓撲的操做是在gluster 中添加了Peer，因此須要手動detach peer

而後繼續運行heketi pod：
kubectl scale deployment deploy-heketi --replicas=1。

用匹配版本的heketi-cli從新加載拓撲，而後重試該步驟。

執行完後，查看Pod deploy-heketi日誌信息，看看作了哪些事：bootstrap

#只截取了部分日誌，基本操做就是進入到各個glusterfs Pod建立brick目錄及建立一個副本爲3的Replicate volume, volume名爲heketidbstorage
[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir -p /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6
Result: 
[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir -p /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4
Result: 
[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir -p /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa

[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick

[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick

[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick

[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick
Result: 
[cmdexec] INFO 2018/07/09 07:07:26 Creating volume heketidbstorage replica 3
[kubeexec] DEBUG 2018/07/09 07:07:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: gluster --mode=script volume create heketidbstorage replica 3 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
Result: volume create: heketidbstorage: success: please start the volume to access data

進入任意GlusterFS Pod查看卷信息：ubuntu

$ kubectl exec glusterfs-25cm8 -it bash
[root@ubuntu16 /]# gluster volume info

Volume Name: heketidbstorage
Type: Replicate
Volume ID: c8da2a4a-3066-4708-a59d-201d22decd92
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick
Brick2: 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick
Brick3: 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@ubuntu16 /]#

分析下heketi-storage.json：api

#將建立以下資源信息：
Endpoints：
    name：heketi-storage-endpoints
         10.30.1.16:1 10.30.1.15:1 10.30.1.17:1
Service：    
    name: heketi-storage-endpoints  
job： 
   name: heketi-storage-copy-job
   images: heketi/heketi:dev
   聲明瞭 volume：heketi-storage 
         "volumes": [
                                {
                                    "name": "heketi-storage",
                                    "glusterfs": {
                                        "endpoints": "heketi-storage-endpoints",
                                        "path": "heketidbstorage"
                                    }
                                },
     掛載到 /heketi：
             volumeMounts": [
                                            {
                                                "name": "heketi-storage",
                                                "mountPath": "/heketi"
                                            },

   啓動時執行命令：cp /db/heketi.db /heketi 
   #由此可知，此job的做用就是複製heketi中的數據文件到 /heketi，而/heketi目錄掛載在了卷heketi-storage中，而heketi-storage volume是前面執行"heketi-cli setup-openshift-heketi-storage"時建立好了的

建立之：bash

$ kubectl create -f heketi-storage.json
secret "heketi-storage-secret" created
endpoints "heketi-storage-endpoints" created
service "heketi-storage-endpoints" created
job "heketi-storage-copy-job" created

當Job執行完後就能夠刪除它了：tcp

$ kubectl get job 
NAME                      DESIRED   SUCCESSFUL   AGE
heketi-storage-copy-job   1         1            1m

等到job完成後，刪除bootstrap Heketi實例相關的組件：分佈式

#把以前由heketi-bootstrap.json建立的資源刪除
$ kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"
deployment "deploy-heketi" deleted
job "heketi-storage-copy-job" deleted
pod "deploy-heketi-69bfbd4bbd-q8tsk" deleted
service "deploy-heketi" deleted
secret "heketi-storage-secret" deleted

以前建立的名爲deploy-heketi的pod,service已經刪除了：

$ kubectl get pod
NAME              READY     STATUS              RESTARTS   AGE
glusterfs-25cm8   1/1       Running             1          1h
glusterfs-8qrpt   1/1       Running             1          1h
glusterfs-c4859   1/1       Running             1          1h

$ kubectl get svc
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
heketi-storage-endpoints      ClusterIP   10.254.191.233   <none>        1/TCP      4m

最後，使用heketi-deployment.json文件從新部署heketi

$ cat heketi-deployment.json 
{
  "kind": "List",
  "apiVersion": "v1",
  "items": [
    {
      "kind": "Secret",
      "apiVersion": "v1",
      "metadata": {
        "name": "heketi-db-backup",
        "labels": {
          "glusterfs": "heketi-db",
          "heketi": "db"
        }
      },
      "data": {
      },
      "type": "Opaque"
    },
    {
      "kind": "Service",
      "apiVersion": "v1",
      "metadata": {
        "name": "heketi",
        "labels": {
          "glusterfs": "heketi-service",
          "deploy-heketi": "support"
        },
        "annotations": {
          "description": "Exposes Heketi Service"
        }
      },
      "spec": {
        "selector": {
          "name": "heketi"
        },
        "ports": [
          {
            "name": "heketi",
            "port": 8080,
            "targetPort": 8080
          }
        ]
      }
    },
    {
      "kind": "Deployment",
      "apiVersion": "extensions/v1beta1",
      "metadata": {
        "name": "heketi",
        "labels": {
          "glusterfs": "heketi-deployment"
        },
        "annotations": {
          "description": "Defines how to deploy Heketi"
        }
      },
      "spec": {
        "replicas": 1,
        "template": {
          "metadata": {
            "name": "heketi",
            "labels": {
              "name": "heketi",
              "glusterfs": "heketi-pod"
            }
          },
          "spec": {
            "serviceAccountName": "heketi-service-account",
            "containers": [
              {
                "image": "heketi/heketi:dev",
                "imagePullPolicy": "Always",
                "name": "heketi",
                "env": [
                  {
                    "name": "HEKETI_EXECUTOR",
                    "value": "kubernetes"
                  },
                  {
                    "name": "HEKETI_DB_PATH",
                    "value": "/var/lib/heketi/heketi.db"
                  },
                  {
                    "name": "HEKETI_FSTAB",
                    "value": "/var/lib/heketi/fstab"
                  },
                  {
                    "name": "HEKETI_SNAPSHOT_LIMIT",
                    "value": "14"
                  },
                  {
                    "name": "HEKETI_KUBE_GLUSTER_DAEMONSET",
                    "value": "y"
                  }
                ],
                "ports": [
                  {
                    "containerPort": 8080
                  }
                ],
                "volumeMounts": [
                  {
                    "mountPath": "/backupdb",
                    "name": "heketi-db-secret"
                  },
                  {
                    "name": "db",
                    "mountPath": "/var/lib/heketi"
                  },
                  {
                    "name": "config",
                    "mountPath": "/etc/heketi"
                  }
                ],
                "readinessProbe": {
                  "timeoutSeconds": 3,
                  "initialDelaySeconds": 3,
                  "httpGet": {
                    "path": "/hello",
                    "port": 8080
                  }
                },
                "livenessProbe": {
                  "timeoutSeconds": 3,
                  "initialDelaySeconds": 30,
                  "httpGet": {
                    "path": "/hello",
                    "port": 8080
                  }
                }
              }
            ],
            "volumes": [
              {
                "name": "db",
                "glusterfs": {
                  "endpoints": "heketi-storage-endpoints",
                  "path": "heketidbstorage"
                }
              },
              {
                "name": "heketi-db-secret",
                "secret": {
                  "secretName": "heketi-db-backup"
                }
              },
              {
                "name": "config",
                "secret": {
                  "secretName": "heketi-config-secret"
                }
              }
            ]
          }
        }
      }
    }
  ]
}

heketi-deployment.json 文件建立了以下資源：

Service:
     name: heketi
     port: 8080
Deployment：
    name: heketi
    replicas: 1
    image: heketi/heketi:dev
    volumeMounts: 
           name: db   mountPath: /var/lib/heketi 
   volumes: 
      endpoints: heketi-storage-endpoints   #由heketi-storage.json文件建立
       path：heketidbstorage  #這是gluster volume名，此volume是由"heketi-cli setup-openshift-heketi-storage"自動建立。
    # 將heketi容器內的/var/lib/heketi 目錄掛載到了GlusterFS volume 「heketidbstorage」中。

部署之：

$ kubectl create -f heketi-deployment.json 
secret "heketi-db-backup" created
service "heketi" created
deployment "heketi" created

$ kubectl get deployment 
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
heketi    1         1         1            1           45s
$ kubectl get svc 
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
heketi                        ClusterIP   10.254.239.189   <none>        8080/TCP   51s
heketi-storage-endpoints      ClusterIP   10.254.191.233   <none>        1/TCP      31m

驗證heketi是否在用在用gluster volume：

$ kubectl get pod
NAME                      READY     STATUS              RESTARTS   AGE
glusterfs-25cm8           1/1       Running             1          1h
glusterfs-8qrpt           1/1       Running             1          1h
glusterfs-c4859           1/1       Running             1          1h
heketi-7898db85dd-nb6kn   1/1       Running             0          1m

$ kubectl exec heketi-7898db85dd-nb6kn -it bash
[root@heketi-7898db85dd-nb6kn /]# mount |grep heketi
10.30.1.15:heketidbstorage on /var/lib/heketi type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

至此，heketi db已正確配置了GlusterFS卷。