在kubernetes中使用StorageClass綁定ceph rbd

使用StorageClass掛載ceph rbd須要用到2個cephx角色。一個用於建立和管理rbd，在這裏咱們使用admin,另外一個用於掛載rbd到Pod中，咱們能夠在ceph中手動建立一個。

準備：

假設已有一套ceph集羣，Monitor分別爲192.168.1.1，192.168.1.2和192.168.1.3

須要保證Monitor的6379端口和OSD的6800端口對kubelet節點開放

爲每一個kubelet節點都安裝ceph-common

1、建立一個k8s存儲池，用於存放數據卷

# ceph osd pool create k8s 64 64

2、建立cephx用戶，用於掛載由admin建立好的rbd

# ceph auth get-or-create client.k8s mon 'allow r' osd 'allow * pool=k8s'

3、獲取k8s用戶的key，以base64編碼

# ceph auth get-key client.k8s|base64

記錄輸出的內容，如：QVFEQjlmcFpjUE5lS0JBQU40NnZxZ2dIT0dRTEtaeUx1blNjR1E9PQ==

4、獲取admin用戶的key，以base64編碼

# ceph auth get-key client.admin|base64

記錄輸出的內容，如：QVFDUDNrQllRRVBJR0JBQXFuVXJQbHZQaC9xZEQ2ZGVZOXRoVXc9PQ==

5、建立storage-class-rbd.yaml

---
apiVersion: v1
data:
  key: QVFEQjlmcFpjUE5lS0JBQU40NnZxZ2dIT0dRTEtaeUx1blNjR1E9PQ==
kind: Secret
metadata:
  name: ceph-secret-user
  namespace: default
type: kubernetes.io/rbd
---
apiVersion: v1
data:
  key: QVFDUDNrQllRRVBJR0JBQXFuVXJQbHZQaC9xZEQ2ZGVZOXRoVXc9PQ==
kind: Secret
metadata:
  name: ceph-secret-admin
  namespace: kube-system
type: kubernetes.io/rbd
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rbd
provisioner: kubernetes.io/rbd
parameters:
  monitors: 192.168.1.1:6789,192.168.1.2:6789,192.168.1.3:6789
  adminId: admin
  adminSecretName: ceph-secret-admin
  adminSecretNamespace: kube-system
  pool: k8s
  userId: k8s
  userSecretName: ceph-secret-user
  fsType: xfs
  imageFormat: "2"
  imageFeatures: "layering"

6、建立StorageClass

# kubectl create -f ./storage-class-rbd.yaml

7、建立nginx-statefulset.yaml, 經過StatefulSet動態掛載卷

---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  volumeClaimTemplates:
  - metadata:
      annotations:
        volume.beta.kubernetes.io/storage-class: "rbd" # 這裏配置 上面建立的 storageclass 的名稱
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 2Gi 
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        volumeMounts:
        - mountPath: "/usr/share/nginx/html/"
          name: html

StatefulSet會爲每一個Pod建立一個rbd數據卷，並掛載到/usr/share/nginx/html/目錄中,數據卷會隨着StatefulSet的伸縮而動態建立和銷燬。