kubernetes存儲類與PV與PVC關係及實踐

StorageClass & PV & PVC關係圖

Volumes是最基礎的存儲抽象，其支持多種類型，包括本地存儲、NFS、FC以及衆多的雲存儲，咱們也能夠編寫本身的存儲插件來支持特定的存儲系統。Volume能夠被Pod直接使用，也能夠被PV使用。普通的Volume和Pod之間是一種靜態的綁定關係，在定義Pod的同時，經過volume屬性來定義存儲的類型，經過volumeMount來定義容器內的掛載點。html
PersistentVolume。與普通的Volume不一樣，PV是Kubernetes中的一個資源對象，建立一個PV至關於建立了一個存儲資源對象，這個資源的使用要經過PVC來請求。nginx
PersistentVolumeClaim。PVC是用戶對存儲資源PV的請求，根據PVC中指定的條件Kubernetes動態的尋找系統中的PV資源並進行綁定。目前PVC與PV匹配能夠經過StorageClassName、matchLabels或者matchExpressions三種方式。git
StorageClass。存儲類，目前kubernetes支持不少存儲，例如ceph,nfs,glusterfs等等。。。

接下來，本文使用前文《手把手教你使用rpm部署ceph集羣》建立好的ceph集羣來爲kubernetes提供存儲。github

建立存儲類

一、獲取admin keyapi

grep key /etc/ceph/ceph.client.admin.keyring |awk '{printf "%s", $NF }'|base64

QVFCZ2ZZOWJ1dGdBQ0JBQXN5dGdLZ1BFOGlsblIzWjJqNVVKMUE9PQ==

二、寫入 ceph-secret-admin.yamlbash

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret-admin
type: "kubernetes.io/rbd"
data:
  key: QVFCZ2ZZOWJ1dGdBQ0JBQXN5dGdLZ1BFOGlsblIzWjJqNVVKMUE9PQ==

三、建立secretapp

kubectl  apply -f ceph-secret-admin.yaml
kubectl  get secret
NAME                                 TYPE                                  DATA      AGE
ceph-secret-admin                    kubernetes.io/rbd                     1         6m

四、修改 rbd-storage-class.yamlide

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rbd
provisioner: kubernetes.io/rbd
parameters:
    monitors: 192.168.100.100:6789,192.168.100.101:6789,192.168.100.102:6789
    adminId: admin
    adminSecretName: ceph-secret-admin
    adminSecretNamespace: default
    pool: k8s
    userId: admin
    userSecretName: ceph-secret-admin
    userSecretNamespace: default
    fsType: ext4
    imageFormat: "2"
    imageFeatures: "layering"

五、建立storage class測試

kubectl apply -f rbd-storage-class.yaml

[root@qd01-stop-cloud001 rbd]# kubectl  get sc
NAME      PROVISIONER         AGE
rbd         kubernetes.io/rbd   4m

建立PVC & PV

六、建立pvc.yamlthis

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rbd
  resources:
    requests:
      storage: 1Gi

七、查看pvc,顯示狀態爲Bound表示成功

kubectl apply -f pvc.yaml
kubectl  get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
claim    Bound     pvc-130c2445-b4a5-11e8-9d27-782bcb3bb379   1Gi        RWO            slow           13m

測試驗證

八、建立pod.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: server
spec:
  replicas: 1
  selector:
    role: server
  template:
    metadata:
      labels:
        role: server
    spec:
      containers:
      - name: server
        image: nginx
        volumeMounts:
          - mountPath: /var/lib/www/html
            name: mypvc
      volumes:
        - name: mypvc
          persistentVolumeClaim:
            claimName: claim

九、查看掛載存儲

/dev/rbd0       1G  9.1M  0.98G   1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/k8s-image-kubernetes-dynamic-pvc-198f56b3-b4a5-11e8-97eb-782bcb3bb379

錯誤處理

若是出錯failed to create rbd image: executable file not found in $PATH
參考https://blog.csdn.net/aixiaoyang168/article/details/79120095
可使用下面的項目來建立存儲類

使用external-storage建立存儲類

$ git clone https://github.com/kubernetes-incubator/external-storage.git
$ tree external-storage/ceph/rbd/deploy/
├── README.md
├── non-rbac
│   └── deployment.yaml
└── rbac
    ├── clusterrole.yaml
    ├── clusterrolebinding.yaml
    ├── deployment.yaml
    └── serviceaccount.yaml

Install without RBAC roles:
cd $GOPATH/src/github.com/kubernetes-incubator/external-storage/ceph/rbd/deploy
kubectl apply -f ./non-rbac

Install with RBAC roles:
cd $GOPATH/src/github.com/kubernetes-incubator/external-storage/ceph/rbd/deploy
NAMESPACE=default     # change this if you want to deploy it in another namespace
sed -r -i "s/namespace: [^ ]+/namespace: $NAMESPACE/g" ./rbac/clusterrolebinding.yaml ./rbac/rolebinding.yaml
kubectl -n $NAMESPACE apply -f ./rbac

若是secret和provisioner不在同一個namespace中的話，獲取secret權限不夠。
解決方法：
如下文件添加secrets的權限
external-storage/ceph/rbd/deploy/rbac/clusterrole.yaml

- apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "create", "delete"]