在K8S集羣下爲應用配置本地卷（Local Volume）

本地卷描述

如Hadoop、ES等系統，其DataNode需大量存儲，且其自己提供了冗餘功能，那麼咱們就不必讓其從存儲系統中分配卷，而是像裸機部署同樣讓其使用本地節點上的存儲，local volumes出現以前，咱們可以使用HostPath掛載捲到容器中，但此方案有不少侷限性：

The prior mechanism of accessing local storage through hostPath volumes had many challenges. hostPath volumes were difficult to use in production at scale: operators needed to care for local disk management, topology, and scheduling of individual pods when using hostPath volumes, and could not use many Kubernetes features (like StatefulSets). Existing Helm charts that used remote volumes could not be easily ported to use hostPath volumes. The Local Persistent Volumes feature aims to address hostPath volumes’ portability, disk accounting, and scheduling challenges.

注意：本地卷僅適用於少許應用，如同HostPath同樣Pod被綁定到特定的主機上，若主機異常，則Pod無法調度到其餘節點，其適用場景：

• Caching of datasets that can leverage data gravity for fast processing
• Distributed storage systems that shard or replicate data across multiplenodes. Examples include distributed datastores like Cassandra, or distributedfile systems like Gluster or Ceph.

localvolume與hostpath相似，但其更靈活且統一，如應用使用hostpath，則只能在yaml文件中硬編碼，而localvolume如同普通pvc靈活可用，具體見下文。

手動配置本地卷

此節介紹在不使用local provisioner狀況下如何配置手動本地卷。

管理員必須爲本地卷建立一個存儲類，以下所示，建立了一個名爲local-storage的存儲類：

% kubectl create -f - <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF

注意：

• 此存儲類沒法動態提供存儲功能，全部PV需手動建立；
• volumeBindingMode: WaitForFirstConsumer：Pod調度前不先綁定PVC與PV，而是等待Pod被調度時，這樣可根據Pod資源等請求合理調度，如：selectors, affinity and anti-affinity policies；

宿主機準備目錄，即配置本地硬盤。如實驗環境okd-c01與okd-c02主機均配置了/mnt/test本地存儲，而對於OKD集羣需設置SeLinux權限：

chcon -R unconfined_u:object_r:svirt_sandbox_file_t:s0 /mnt/test

手動建立兩PV：example-local-pv-1、example-local-pv-2分別綁定兩主機存儲，如綁定okd-c01.zyl.io主機的本地卷以下：

% kubectl create -f - <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-local-pv-1
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/test
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - okd-c01.zyl.io
EOF

建立PVC：

% kubectl create -f - <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: example-local-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi 
  storageClassName: local-storage
EOF

注意：此時PVC不會立馬綁定到PV，其等待Pod被調度時纔會綁定到PV上，當前PVC狀態爲Pending：

% oc describe pvc example-local-claim
...
Events:
  Type    Reason                Age               From                         Message
  ----    ------                ----              ----                         -------
  Normal  WaitForFirstConsumer  5s (x2 over 10s)  persistentvolume-controller  waiting for first consumer to be created before binding

配置應用使用存儲：

% kubectl create -f - <<'EOF'
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: local-volume-test
  name: local-volume-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: local-volume-test
  template:
    metadata:
      labels:
        app: local-volume-test
    spec:
      containers:
      - image: busybox
        name: local-volume-test
        imagePullPolicy: IfNotPresent
        command: [ "/bin/sh", "-c", "while true; do sleep 2; echo $(date) $(hostname)>> /mnt/test/test.log; done" ]
        volumeMounts:
        - name: local-data
          mountPath: /mnt/test
      volumes:
      - name: local-data
        persistentVolumeClaim:
          claimName: example-local-claim
EOF

Pod調度後會發現PVC被綁定：

% oc get pvc example-local-claim 
NAME                  STATUS    VOLUME             CAPACITY   ACCESS MODES STORAGECLASS    AGE
example-local-claim   Bound     example-local-pv-2 1Gi        RWO          local-storage   7m

注意：

• 此時刪除Pod，可發現其仍然被調度到本地存儲example-local-pv-2所在的主機，即okd-c02.zyl.io；
• 刪除deployment後，PVC不會與PV解綁，即第一次Pod調度後，PVC就與PV關聯了；
• 刪除PVC後，發現PV一直處於Released狀態，致使PV沒法被重用，需管理員手動刪除PV並重建PV以便被重用：

When local volumes are manually created like this, the only supported persistentVolumeReclaimPolicy is 「Retain」. When the PersistentVolume is released from the PersistentVolumeClaim, an administrator must manually clean up and set up the local volume again for reuse.

自動配置本地卷

手動配置本地券中描述的PV需手動建立，而PVC刪除後PV無法重用，即需重建PV才行，當系統使用大量Local Volume時會加劇管理負擔，鑑於此，可經過external static provisioner協助簡化local volume配置。

當前版本（<=K8S 1.12及OKD 1.11）管理員需手動在主機上將volume掛載到特定目錄上，然後external static provisioner會掃描此目錄，其將自動建立PV，而當PVC被釋放後又會清理目錄內容並重建PV，其是半自動化的，但並不是動態提供，後續版本會提供動態提供支持，如管理員僅需提供LVM VG，此provisioner將自動完成格式化、掛載等步驟：

Dynamic provisioning of local volumes using LVM is under design and an alpha implementation will follow in a future release. This will eliminate the current need for an administrator to pre-partition, format and mount local volumes, as long as the workload’s performance requirements can tolerate sharing disks.

參考：

• Local Persistent Volumes for Kubernetes Goes Beta；
• [kubernetes-sigs/sig-storage-local-static-provisioner：K8S官方提供的external static provisioner；
• Configuring Local Volumes：OKD如何配置local provisioner；

如下描述如何使用OKD提供的local provisioner。

主機掛載本地卷

如本例在okd-c0[1-3]主機配置本地卷，需將目錄手動掛載到/mnt/local-storage/<storage-class-name>/<volume>目錄，如：

cat >> /etc/fstab <<EOF
/dev/datavg/d1 /mnt/local-storage/local-storage-1/d1 xfs defaults 0 0
/dev/datavg/d2 /mnt/local-storage/local-storage-1/d2 xfs defaults 0 0
/dev/datavg/d3 /mnt/local-storage/local-storage-2/d3 xfs defaults 0 0
/dev/datavg/d4 /mnt/local-storage/local-storage-2/d4 xfs defaults 0 0
EOF
mkdir -p /mnt/local-storage/local-storage-1/{d1,d2}
mkdir -p /mnt/local-storage/local-storage-2/{d3,d4}
mount -a

OKD/Openshift環境設置SeLinux權限：

chcon -R unconfined_u:object_r:svirt_sandbox_file_t:s0 /mnt/local-storage/

部署local provisioner

可選。在單獨的項目下部署local provisioner，建立項目：

oc new-project local-storage

建立一個ConfigMap，其被external provisioner使用，用於描述存儲類：

% kubectl create -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-volume-config
data:
    storageClassMap: |
        local-storage-1:
            hostDir: /mnt/local-storage/local-storage-1
            mountDir: /mnt/local-storage/local-storage-1
        local-storage-2:
            hostDir: /mnt/local-storage/local-storage-2
            mountDir: /mnt/local-storage/local-storage-2
EOF

注意：

• local-storage-1：爲StorageClass名稱，與/mnt/local-storage/<storage-class-name>對應；
• hostDir：本機目錄；
• moutDir：external provisioner將hostDir掛載到pod中的目錄，與hostDir保持一致便可。

provisioner以root權限運行，OKD集羣需賦權：

oc create serviceaccount local-storage-admin
oc adm policy add-scc-to-user privileged -z local-storage-admin

OKD集羣安裝模板：

oc create -f https://raw.githubusercontent.com/openshift/origin/release-3.11/examples/storage-examples/local-examples/local-storage-provisioner-template.yaml

從以上模板安裝應用（PS：其以DS模式運行在全部計算節點上）：

oc new-app -p CONFIGMAP=local-volume-config \
  -p SERVICE_ACCOUNT=local-storage-admin \
  -p NAMESPACE=local-storage \
  -p PROVISIONER_IMAGE=quay.io/external_storage/local-volume-provisioner:latest \
  local-storage-provisioner

建立所需的StorageClass：

% kubectl create -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: local-storage-1
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF

% kubectl create -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: local-storage-2
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF

注意：StorageClass建立完成後，PV纔會被自動建立。

接着，咱們手動建立PVC或者在Statefulset中使用此存儲卷：

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: example-local-claim
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage-1
  resources:
    requests:
      storage: 5Gi

---
kind: StatefulSet
...
 volumeClaimTemplates:
  - metadata:
      name: example-local-claim
    spec:
      accessModes:
      - ReadWriteOnce
      storageClassName: local-storage-1
      resources:
        requests:
          storage: 5Gi