獨立部署 GlusterFS+Heketi 實現 Kubernetes / OpenShift 共享存儲

時間  2019-11-24
標籤 獨立 部署 glusterfs+heketi glusterfs heketi 實現 kubernetes openshift 共享 存儲 简体版
原文   原文鏈接

1，準備工做

1.1 硬件信息

主機名 IP地址
gfs1 192.168.160.131
gfs2 192.168.160.132
gfs3/heketi 192.168.160.133
  • 20G 的裸盤 /dev/sdb
Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

1.2 環境準備

  • 默認狀況下，SELinux不容許從Pod寫入遠程Gluster服務器，要經過在每一個節點上執行SELinux來寫入GlusterFS卷
sudo setsebool -P virt_sandbox_use_fusefs on 
sudo setsebool -P virt_use_fusefs on

1.3 載入指定的個別模塊

modprobe dm_snapshot
modprobe dm_mirror
modprobe dm_thin_pool

2，安裝GlusterFS了

yum -y install glusterfs glusterfs-server glusterfs-fuse

2.1 須要爲GlusterFS peers打開幾個基本TCP端口，以便與OpenShift進行通訊並提供存儲：

firewall-cmd --add-port=24007-24008/tcp --add-port=49152-49664/tcp --add-port=2222/tcp
firewall-cmd --runtime-to-permanent

2.2 啓動GlusterFS的daemon進程了：

systemctl enable glusterd
systemctl start glusterd

3，在GlusterFS的一臺虛擬機上安裝heketi

yum -y install heketi heketi-client

3.1 啓動文件語法：

  • /usr/lib/systemd/system/heketi.service
[Unit]
Description=Heketi Server

[Service]
Type=simple
WorkingDirectory=/var/lib/heketi
EnvironmentFile=-/etc/heketi/heketi.json
User=heketi
ExecStart=/usr/bin/heketi --config=/etc/heketi/heketi.json
Restart=on-failure
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

3.2 重啓 heketi

systemctl daemon-reload
systemctl start heketi

3.3 建立密鑰並分發

ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
chown heketi:heketi /etc/heketi/heketi_key
for i in gfs1 gfs2 gfs3 ;do ssh-copy-id -i /etc/heketi/heketi_key.pub $i ;done

3.4 配置heketi來使用SSH。 編輯/etc/heketi/heketi.json文件

"executor":"ssh",
      "_sshexec_comment":"SSH username and private key file information",
      "sshexec":{
         "keyfile":"/etc/heketi/heketi_key",
         "user":"root",
         "port":"22",
         "fstab":"/etc/fstab"
      },

3.5 heketi將監聽8080端口，添加防火牆規則：

firewall-cmd --add-port=8080/tcp
firewall-cmd --runtime-to-permanent

3.6 重啓heketi：

systemctl enable heketi
systemctl restart heketi

3.7 測試 heketi 運行狀態：

Hello from Heketi

3.8 配置 GlusterFS 存儲池

  • vim /etc/heketi/topology.json
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "gfs1"
              ],
              "storage": [
                "192.168.160.131"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gfs2"
              ],
              "storage": [
                "192.168.160.132"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gfs3"
              ],
              "storage": [
                "192.168.160.133"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        }
      ]
    }
  ]
}

3.9 建立 GlusterFS 存儲池

export HEKETI_CLI_SERVER=http://gfs3:8080
heketi-cli --server=http://gfs3:8080 topology load --json=/etc/heketi/topology.json
  • 輸出信息
Creating cluster ... ID: d3a3f31dce28e06dbd1099268c4ebe84
    Allowing file volumes on cluster.
    Allowing block volumes on cluster.
    Creating node infra.test.com ... ID: ebfc1e8e2e7668311dc4304bfc1377cb
        Adding device /dev/sdb ... OK
    Creating node node1.test.com ... ID: 0ce162c3b8a65342be1aac96010251ef
        Adding device /dev/sdb ... OK
    Creating node node2.test.com ... ID: 62952de313e71eb5a4bfe5b76224e575
        Adding device /dev/sdb ...  OK

3.10 當前位於 gfs3， 查看集羣信息

  • heketi-cli topology info
Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84

    File:  true
    Block: true

    Volumes:

    Nodes:

    Node Id: 0ce162c3b8a65342be1aac96010251ef
    State: online
    Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84
    Zone: 1
    Management Hostnames: node1.test.com
    Storage Hostnames: 192.168.160.132
    Devices:
        Id:d6a5f0aba39a35d3d92f678dc9654eaa   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19      
            Bricks:

    Node Id: 62952de313e71eb5a4bfe5b76224e575
    State: online
    Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84
    Zone: 1
    Management Hostnames: node2.test.com
    Storage Hostnames: 192.168.160.133
    Devices:
        Id:dfd697f2215d2a304a44c5af44d352da   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19      
            Bricks:

    Node Id: ebfc1e8e2e7668311dc4304bfc1377cb
    State: online
    Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84
    Zone: 1
    Management Hostnames: infra.test.com
    Storage Hostnames: 192.168.160.131
    Devices:
        Id:e06b794b0b9f20608158081fbb5b5102   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19      
            Bricks:
  • heketi-cli node list
Id:0ce162c3b8a65342be1aac96010251ef Cluster:d3a3f31dce28e06dbd1099268c4ebe84
Id:62952de313e71eb5a4bfe5b76224e575 Cluster:d3a3f31dce28e06dbd1099268c4ebe84
Id:ebfc1e8e2e7668311dc4304bfc1377cb Cluster:d3a3f31dce28e06dbd1099268c4ebe84
  • gluster peer status
Number of Peers: 2

Hostname: gfs2
Uuid: ae6e998a-92c2-4c63-a7c6-c51a3b7e8fcb
State: Peer in Cluster (Connected)
Other names:
gfs2

Hostname: gfs1
Uuid: c8c46558-a8f2-46db-940d-4b19947cf075
State: Peer in Cluster (Connected)

4，測試

4.1 測試建立volume

  • heketi-cli --json volume create --size 3 --replica 3
{"size":3,"name":"vol_93060cd7698e9e48bd035f26bbfe57af","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{"data":4,"redundancy":2}},"glustervolumeoptions":["",""],"snapshot":{"enable":false,"factor":1},"id":"93060cd7698e9e48bd035f26bbfe57af","cluster":"d3a3f31dce28e06dbd1099268c4ebe84","mount":{"glusterfs":{"hosts":["192.168.160.132","192.168.160.133","192.168.160.131"],"device":"192.168.160.132:vol_93060cd7698e9e48bd035f26bbfe57af","options":{"backup-volfile-servers":"192.168.160.133,192.168.160.131"}}},"blockinfo":{},"bricks":[{"id":"16b8ddb1f2b2d3aa588d4d4a52bb7f6b","path":"/var/lib/heketi/mounts/vg_e06b794b0b9f20608158081fbb5b5102/brick_16b8ddb1f2b2d3aa588d4d4a52bb7f6b/brick","device":"e06b794b0b9f20608158081fbb5b5102","node":"ebfc1e8e2e7668311dc4304bfc1377cb","volume":"93060cd7698e9e48bd035f26bbfe57af","size":3145728},{"id":"9e60ac3b7259c4e8803d4e1f6a235021","path":"/var/lib/heketi/mounts/vg_d6a5f0aba39a35d3d92f678dc9654eaa/brick_9e60ac3b7259c4e8803d4e1f6a235021/brick","device":"d6a5f0aba39a35d3d92f678dc9654eaa","node":"0ce162c3b8a65342be1aac96010251ef","volume":"93060cd7698e9e48bd035f26bbfe57af","size":3145728},{"id":"e3f5ec732d5a8fe4b478af67c9caf85b","path":"/var/lib/heketi/mounts/vg_dfd697f2215d2a304a44c5af44d352da/brick_e3f5ec732d5a8fe4b478af67c9caf85b/brick","device":"dfd697f2215d2a304a44c5af44d352da","node":"62952de313e71eb5a4bfe5b76224e575","volume":"93060cd7698e9e48bd035f26bbfe57af","size":3145728}]}
  • heketi-cli volume list
Id:93060cd7698e9e48bd035f26bbfe57af    Cluster:d3a3f31dce28e06dbd1099268c4ebe84    Name:vol_93060cd7698e9e48bd035f26bbfe57af
  • heketi-cli volume info 93060cd7698e9e48bd035f26bbfe57af
Name: vol_93060cd7698e9e48bd035f26bbfe57af
Size: 3
Volume Id: 93060cd7698e9e48bd035f26bbfe57af
Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84
Mount: 192.168.160.132:vol_93060cd7698e9e48bd035f26bbfe57af
Mount Options: backup-volfile-servers=192.168.160.133,192.168.160.131
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3
  • gluster volume list
vol_93060cd7698e9e48bd035f26bbfe57af
  • gluster volume status
Status of volume: vol_93060cd7698e9e48bd035f26bbfe57af
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 192.168.160.132:/var/lib/heketi/mount
s/vg_d6a5f0aba39a35d3d92f678dc9654eaa/brick
_9e60ac3b7259c4e8803d4e1f6a235021/brick     49153     0          Y       30660
Brick 192.168.160.131:/var/lib/heketi/mount
s/vg_e06b794b0b9f20608158081fbb5b5102/brick
_16b8ddb1f2b2d3aa588d4d4a52bb7f6b/brick     49153     0          Y       21979
Brick 192.168.160.133:/var/lib/heketi/mount
s/vg_dfd697f2215d2a304a44c5af44d352da/brick
_e3f5ec732d5a8fe4b478af67c9caf85b/brick     49152     0          Y       61274
Self-heal Daemon on localhost               N/A       N/A        Y       61295
Self-heal Daemon on apps.test.com           N/A       N/A        Y       22000
Self-heal Daemon on 192.168.160.132         N/A       N/A        Y       30681
 
Task Status of Volume vol_93060cd7698e9e48bd035f26bbfe57af
------------------------------------------------------------------------------
There are no active volume tasks
  • gluster volume info vol_93060cd7698e9e48bd035f26bbfe57af
Volume Name: vol_93060cd7698e9e48bd035f26bbfe57af
Type: Replicate
Volume ID: ca4a9854-a33c-40ab-86c7-0d0d34004454
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.160.132:/var/lib/heketi/mounts/vg_d6a5f0aba39a35d3d92f678dc9654eaa/brick_9e60ac3b7259c4e8803d4e1f6a235021/brick
Brick2: 192.168.160.131:/var/lib/heketi/mounts/vg_e06b794b0b9f20608158081fbb5b5102/brick_16b8ddb1f2b2d3aa588d4d4a52bb7f6b/brick
Brick3: 192.168.160.133:/var/lib/heketi/mounts/vg_dfd697f2215d2a304a44c5af44d352da/brick_e3f5ec732d5a8fe4b478af67c9caf85b/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

5，在OpenShift中使用Gluster

5.1 OpenShift 建立 StorageClass YAML文件：

  • 編輯storage-class.yaml，其中resturl爲heketi的url，volumetype: replicate:3爲副本卷brick數量，建議爲3
  • cat storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: null
  name: gluster-heketi
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://gfs3:8080"
  restauthenabled: "true"
  volumetype: replicate:3
  • 建立 StorageClass
oc create -f storage-class.yaml
  • 查看 StorageClass
  • oc get sc
NAME             PROVISIONER               AGE
gluster-heketi   kubernetes.io/glusterfs   55m

5.2 OpenShift 建立 PVC YAML文件：

  • cat pvc.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: test-pvc
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
        storage: 1Gi
 storageClassName: gluster-heketi
  • 建立 PVC
oc create -f storage-class.yaml
  • 查看 PV，PVC
  • oc get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM           STORAGECLASS     REASON    AGE
persistentvolume/pvc-57362c7f-e6c2-11e9-8634-000c299365cc   1Gi        RWX            Delete           Bound     default/test1   gluster-heketi             57m

NAME                          STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
persistentvolumeclaim/test-pvc   Bound     pvc-57362c7f-e6c2-11e9-8634-000c299365cc  1Gi        RWX            gluster-heketi   57m
  • 掛載到虛機
mount -t glusterfs 192.168.160.132:vol_b96d0e18cef937dd56a161ae5fa5b9cb /mnt
  • df -h | grep vol_b96d0e18cef937dd56a161ae5fa5b9cb
192.168.160.132:vol_b96d0e18cef937dd56a161ae5fa5b9cb                                   1014M   43M  972M   5% /mnt

6，經常使用命令

查看集羣節點：gluster pool list
查看集羣狀態(默認不顯示當前主機)： gluster peer status
查看集羣volume ：gluster volume list
查看volume 信息：gluster volume info <VOLNAME>
查看volume狀態：gluster volume stats <VOLNAME>
強制啓動volume：gluster volume start <VOLNAME> force
查看volume須要修復的文件：gluster volume heal <VOLNAME> info
啓動徹底修復：gluster volume heal <VOLNAME> full
查看修復成功的文件：gluster volume heal <VOLNAME> info healed
查看修復失敗的文件：gluster volume heal <VOLNAME> info heal-failed
查看腦裂文件：gluster volume heal <VOLNAME> info split-brain

6.1 其它heketi客戶端經常使用命令

heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv cluster list
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv cluster info <cluster-id>
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv node info <node-id>
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume list
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume create --size=1 --replica=2
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume info <volune-id>
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume expand --volume=<volune-id> --expand-size=1
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume delete <volune-id>

6.2 初始化裸盤

  • 每一個glusterfs machine上掛一快一樣大小的未格式化過磁盤。 若是已經格式化過，使用下面命令重置磁盤(假設磁盤掛在/dev/sdb)：
pvcreate --metadatasize=128M --dataalignment=256K /dev/sdb

7， GlusterFS集羣故障處理

7.1 volume bricks掉線

  • 查看volume狀態：
gluster volume status <volume_name>
  • 當Online列有爲N的表示當前bricks不在線，登陸brick所在主機查看brick掛載
df -h |grep <BRICKNAME>
  • 若未查看到須要從新mount
cat /etc/fstab |grep <BRICKNAME> |xargs -i mount {}
  • 從新啓動掉線的bricks
gluster volume start <VOLNAME> force

7.2 bricks文件不一致修復

  • 查看bricks是否存在文件不一致
gluster volume heal <VOLNAME> info
  • 啓動自動修復
gluster volume heal <VOLNAME> full

7.3 bricks腦裂修復

gluster volume heal <VOLNAME> info
  • 若存在Is in split-brain內容則發生腦裂
1)  選擇較大的文件做爲源修復
gluster volume heal <VOLNAME> split-brain bigger-file <FILE>
2)  選擇以最新的mtime做爲源的文件
gluster volume heal <VOLNAME> split-brain latest-mtime <FILE>
3)  選擇副本中的磚塊之一做爲特定文件的源
gluster volume heal <VOLNAME> split-brain source-brick <HOSTNAME:BRICKNAME> <FILE>
4)  選擇副本的一個brick做爲全部文件的源
gluster volume heal <VOLNAME> split-brain source-brick <HOSTNAME:BRICKNAME>

7.4 更換brick

  • 將源bricks數據同步到新的bricks路徑
gluster volume replace-brick <VOLNAME> Server1:/home/gfs/r2_0 Server1:/home/gfs/r2_5 start
  • 在數據遷移的過程當中，能夠查看替換任務是否完成
gluster volume replace-brick <VOLNAME> Server1:/home/gfs/r2_0 Server1:/home/gfs/r2_5 status
  • 在數據遷移結束以後，執行commit命令結束任務，則進行Brick替換。使用volume info命令能夠查看到Brick已經被替換
gluster volume replace-brick <VOLNAME> Server1:/home/gfs/r2_0 Server1:/home/gfs/r2_5 commit

8， heketi服務故障處理

  • 報錯：
[heketi] ERROR 2018/07/02 09:08:19 /src/github.com/heketi/heketi/apps/glusterfs/app.go:172: Heketi was terminated while performing one or more operations. Server may refuse to start as long as pending operations are present in the db.
heketi服務沒法啓動
1) 導出heketi的heketi.db文件，文件的路徑在heketi.json文件裏面
heketi db export --dbfile=/var/lib/heketi/heketi.db --jsonfile=/tmp/heketidb1.json
2) 打開導出的db文件，好比上文的/tmp/heketidb1.json，查找```pendingoperations```選項，找到以後把與它相關的內容刪除
3) 將修改後的文件保存，切記要保存爲json後綴。而後將db文件再按照以下命令導入
heketi db import --jsonfile=/tmp/succ.json --dbfile=/var/lib/heketi/heketi.db
4) 重啓heketi 服務
systemctl start heketi

9，報錯找不到 集羣配置文件（只針對個別場景的報錯記錄）

Failed to provision volume with StorageClass "glusterfs-storage": failed to create volume: failed to create volume: No clusters configured
1, 根據pvc的YAML，找到 storageclass,
2, 根據storageclass的YAML,找到 heketi服務節點
3, 登陸 heketi 所在虛機，查看當前集羣節點，發現爲空，沒有集羣信息，heketi-cli --server=http://localhost:8080  cluster list
4, 查看本節點是否有掛載heketi盤，查看歷史記錄是否有掛載heketi存儲盤命令，
5, 經過/etc/heketi/topology.json存儲池，登陸 glusterfs 任意節點，gluster volume list | grep heketi 過濾heketi存儲盤與heketi服務對應歷史命令對應
6, 登陸 heketi 服務虛機，從新掛載heketi盤，而且重啓heketi服務
### 若是有keepalived服務，高可用
7, 查詢heketi節點的keepalive服務是否正常，查看配置文件 /etc/keepalived/keepalived.conf，是否有過漂移，
8, 查詢heketi備節點，掛載heketi存儲盤是否正常

8，參考文檔

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程