Ceph 與 volumes、PV/PVC 集成

1. Ceph與volumes結合

目標：實現Ceph RBD和kubernetes volumes集成。

rbd 卷容許將 Rados 塊設備 卷掛載到你的 Pod 中. 不像 emptyDir 那樣會在刪除 Pod 的同時也會被刪除，rbd 卷的內容在刪除 Pod 時 會被保存，卷只是被卸載。 這意味着 rbd 卷能夠被預先填充數據，而且這些數據能夠在 Pod 之間共享。

注意： 在使用 RBD 以前，你必須安裝運行 Ceph。

RBD 的一個特性是它能夠同時被多個用戶以只讀方式掛載。 這意味着你能夠用數據集預先填充卷，而後根據須要在儘量多的 Pod 中並行地使用卷。 不幸的是，RBD 卷只能由單個使用者以讀寫模式安裝。不容許同時寫入。

更多詳情請參考 RBD 示例

examples/volumes/rbd at master · kubernetes/examples · GitHub

Ceph與kubernetes完美集成-Happy雲實驗室-51CTO博客

1.1 準備工做

一、建立pool和用戶

[root@node-1 ~]# ceph osd pool create kubernetes 8 8

二、建立認證用戶

[root@node-1 ~]# ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes'
[client.kubernetes]
    key = AQDMup1emZMFOhAABlmnZFE2fF8puHeIhu+UPg==

三、建立secrets對象存儲將Ceph的認證key存儲在Secrets中

獲取步驟2生成的key，並將其加密爲base64格式

[root@node-1 ~]# echo AQDMup1emZMFOhAABlmnZFE2fF8puHeIhu+UPg== | base64 
QVFETXVwMWVtWk1GT2hBQUJsbW5aRkUyZkY4cHVIZUlodStVUGc9PQo=

建立定義secrets對象

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
type: "kubernetes.io/rbd"
data:
  key: QVFETXVwMWVtWk1GT2hBQUJsbW5aRkUyZkY4cHVIZUlodStVUGc9PQo=

生成secrets

[root@node-1 volumes]# kubectl apply -f secret.yaml 
secret/ceph-secret created

[root@node-1 volumes]# kubectl get secret
NAME                  TYPE                                  DATA   AGE
ceph-secret           kubernetes.io/rbd                     1      10s
default-token-hn65d   kubernetes.io/service-account-token   3      41d

1.2 容器中調用RBD volumes

一、建立rbd塊

[root@node-1 ~]# rbd create -p kubernetes --image-feature layering rbd.img --size 10G

[root@node-1 ~]# rbd info kubernetes/rbd.img
rbd image 'rbd.img':
    size 10 GiB in 2560 objects
    order 22 (4 MiB objects)
    id: 519576b8b4567
    block_name_prefix: rbd_data.519576b8b4567
    format: 2
    features: layering
    op_features: 
    flags: 
    create_timestamp: Mon Apr 20 23:27:02 2020

二、pod中引用RBD volumes

[root@node-1 volumes]# cat pods.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: volume-rbd-demo
spec:
  containers:
  - name: pod-with-rbd
    image: nginx:1.7.9
    imagePullPolicy: IfNotPresent
    ports:
    - name: www
      containerPort: 80
      protocol: TCP
    volumeMounts:
    - name: rbd-demo
      mountPath: /data
  volumes:
  - name: rbd-demo
    rbd:
      monitors:
      - 10.254.100.101:6789
      - 10.254.100.102:6789
      - 10.254.100.103:6789
      pool: kubernetes
      image: rbd.img 
      fsType: ext4 
      user: kubernetes
      secretRef:
        name: ceph-secret

1.3 測試驗證

一、生成pod

[root@node-1 volumes]# kubectl apply -f pods.yaml 
pod/volume-rbd-demo configured
[root@node-1 volumes]# kubectl get pods 
NAME                   READY   STATUS    RESTARTS   AGE
demo-8ffbcf7c5-r2wzf   1/1     Running   1          21h
volume-rbd-demo        1/1     Running   0          43m

二、查看掛載的狀況，能夠看到RBD塊存儲掛載至data目錄

[root@node-1 volumes]# kubectl exec -it volume-rbd-demo -- df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           50G  6.7G   41G  15% /
overlay          50G  6.7G   41G  15% /
tmpfs            64M     0   64M   0% /dev
tmpfs           920M     0  920M   0% /sys/fs/cgroup
/dev/rbd0       9.8G   37M  9.7G   1% /data

2. Ceph與PV/PVC集成

2.1 準備工做

參考步驟一，建立好pool，鏡像，用戶認證，secrets

2.2 定義PV和PVC

一、PV定義，定義一塊存儲，抽象化爲PV

[root@node-1 pv_and_pvc]# cat pv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: rbd-demo
spec:
  accessModes:
   - ReadWriteOnce
  capacity:
    storage: 10G
  rbd:
    monitors:
     - 10.254.100.101:6789
     - 10.254.100.102:6789
     - 10.254.100.103:6789
    pool: kubernetes
    image: demo-1.img 
    fsType: ext4
    user: kubernetes
    secretRef:
      name: ceph-secret
  persistentVolumeReclaimPolicy: Retain
  storageClassName: rbd

二、PVC定義，引用PV

[root@node-1 pv_and_pvc]# cat pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo
spec:
  accessModes:
   - ReadWriteOnce
  volumeName: rbd-demo
  resources:
    requests:
      storage: 10G
  storageClassName: rbd

三、生成PV和PVC

[root@node-1 pv_and_pvc]# kubectl apply -f pv.yaml 
persistentvolume/rbd-demo created

[root@node-1 pv_and_pvc]# kubectl get pv
NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
rbd-demo   10G        RWO            Retain           Available           rbd                     9s

[root@node-1 pv_and_pvc]# kubectl apply -f pvc.yaml 
persistentvolumeclaim/pvc-demo created
[root@node-1 pv_and_pvc]# kubectl get pvc
NAME       STATUS    VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-demo   Pending   rbd-demo   0                         rbd            2s

2.3 Pod中引用PVC

[root@node-1 pv_and_pvc]# cat pod-demo.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-demo
spec:
  containers:
  - name: demo
    image: nginx:1.7.9
    imagePullPolicy: IfNotPresent
    ports:
    - name: www
      protocol: TCP
      containerPort: 80
    volumeMounts:
    - name: rbd
      mountPath: /data
  volumes:
  - name: rbd
    persistentVolumeClaim:
      claimName: pvc-demo