k8s 1.19.2使用ceph做存儲後端

這份文檔各組件系統

• ceph：15.2.5
• k8s: 1.19.2
• os: centos7 7.8.2003

  在全部k8s node節點上安裝ceph-common

  不須要節點有/etc/ceph/目錄下的文件

  yum install -y ceph-common

  建立pool

  ceph osd pool create kube
  rbd pool init kube

建立用戶

ceph auth get-or-create \
  client.kube mon 'allow r' \
  osd 'allow class-read object_prefix rbd_children, allow rwx pool=kube' \
  -o /etc/ceph/ceph.client.kube.keyring

建立namespace rbd用於部署csi

kubectl create namespace rbd

建立csi configmap

ceph-csi當前僅支持舊版V1協議。

• 查看mon和clusterID

ceph mon dump

• 建立configmap

cat <<EOF > csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "01c1e960-fbce-11ea-a90e-52540079f755",
        "monitors": [
          "192.168.254.103:6789",
          "192.168.254.103:6789",
          "192.168.254.103:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config
  namespace: rbd
EOF

建立ksm configmap，新版本須要這個

參看https://github.com/ceph/ceph-csi/issues/834

cat <<EOF>>kms-config.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    {
    }
metadata:
  name: ceph-csi-encryption-kms-config
EOF

kubectl -n rbd create -f  kms-config.yaml

建立csi訪問ceph的secret

cat <<EOF > csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: rbd
stringData:
  userID: kube
  userKey: AQDFhmlfVphsGhAAh8f0Ck3gREjilODu4Sz/Zw==
EOF

配置csi插件

建立rbac

• 下載rbac文件

wget  https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml

• 將rbac中的namespace: default替換爲namespace: rbd

sed -i -E 's/(^\s+namespace:)(.*)/\1 rbd/g' csi-provisioner-rbac.yaml
sed -i  -E 's/(^\s+namespace:)(.*)/\1 rbd/g' csi-nodeplugin-rbac.yaml

• apply rbac

kubectl -n rbd apply -f csi-provisioner-rbac.yaml
kubectl -n rbd apply -f csi-nodeplugin-rbac.yaml

建立ceph-csi Provisioner和Node插件

• 下載插件文件

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml

• apply plugin

kubectl -n rbd apply -f  csi-rbdplugin-provisioner.yaml 
kubectl -n rbd apply -f   csi-rbdplugin.yaml

建立storageClass

注意增長fsType、imageFormat、imageFeatures
imageFeatures是必定要增長的，不然centos7上不能正常掛載rbd

cat <<EOF > csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 01c1e960-fbce-11ea-a90e-52540079f755
   pool: kube
   fsType: xfs
   imageFormat: "2"
   imageFeatures: "layering"
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: rbd
reclaimPolicy: Delete
mountOptions:
   - discard
EOF
kubectl apply -f storageClass.yaml

以block模式掛載

這種模式是直接將rbd設備以block模式掛載容器中，使用時須要進入容器格式，這種例子中將rbd設備掛載到/dev/xvda
這種模式下，進入容器後是沒有權限在/dev/xvda上建立文件系統、執行mount操做
注意：如下操做是在一個只有對ceph namespace有操做權限的用戶下執行的

建立pvc

cat <<EOF > raw-block-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
EOF
kubectl -n ceph apply -f raw-block-pvc.yaml

建立pod

cat <<EOF > raw-block-pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-raw-block-volume
spec:
  containers:
    - name: fc-container
      image: centos:7
      command: ["/bin/sh", "-c"]
      args: ["tail -f /dev/null"]
      volumeDevices:
        - name: data
          devicePath: /dev/xvda
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: raw-block-pvc
EOF
kubectl -n ceph apply -f raw-block-pod.yaml

以文件系統方式使用

這種模式會先將rbd設備格式化後再掛載給容器
注意：如下操做是在一個只有對ceph namespace有操做權限的用戶下執行的

建立pvc

cat <<EOF > pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
EOF
kubectl -n ceph apply -f pvc.yaml

建立Pod

cat <<EOF > pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx
      volumeMounts:
        - name: mypvc
          mountPath: /var/lib/www/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false
EOF
kubectl -n ceph apply -f pod.yaml

參考 文檔

https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/