容器編排系統K8s之crd資源

時間 2021-01-17

標籤 html node mysql linux nginx git 程序員 github sql mongodb 欄目 HTML 简体版

原文原文鏈接

　　前文咱們瞭解了k8s節點污點和pod的對節點污點容忍度相關話題，回顧請參考：http://www.javashuo.com/article/p-cymldyou-nz.html；今天咱們來聊一下擴展k8s相關話題；html

　　k8s上建立資源對象的過程node

　　咱們知道在k8s上，資源是有類型的，不一樣類型的資源，其定義方式和使用的字段各不相同；用戶建立一個資源，實際上就是把k8s抽象的資源作實例化，即把k8s抽象出來的資源，經過資源清單賦值，建立出來的對象就是咱們實例化對應類型資源的結果；用戶建立一個資源，首先會把請求發送給apiserver，經過apiserver的認證、受權、准入控制之後，對應建立資源的定義就存放在etcd中，控制器經過watch機制監視apiserver上的資源變更，經過對應資源變更事件觸發對應類型資源的控制器將對應資源建立出來，並經過控制器內部的和解循環監控着對應資源狀態是否和用戶定義的指望狀態同樣，若是發現不同，內部和解循環就會被觸發，對應控制器會向apiserver發起建立資源的請求，將對應資源重建，讓對應資源的狀態始終知足用戶指望的狀態；從上述的過程來看，用戶建立一個資源分兩個步驟，第一步是將對應請求發送給apiserver，經過apiserver把對應資源定義的信息存放在etcd中；第二個步驟是對應資源類型的控制器經過apiserver從etcd中讀取對應資源的定義，將其建立出來；對於etcd來講，它本是就一個kv數據庫，能夠存儲任意類型的kv數據，但在在k8s上，apiserver將不一樣類型的資源定義抽象成不一樣的資源，使得用戶建立對應資源，必須是知足對應類型資源定義的規範，而後將規範的定義存放在etcd中；簡單講apiserver就是把用戶存入etcd中的數據作了一層抽象，使得用戶不可以隨意將任意數據存儲到etcd中，存入etcd中的數據必須是知足對應apiserver接口定義的規範；這就比如咱們在使用mysql數據庫時，必須遵照對應庫中的對應表的定義；mysql

　　在k8s上建立自定義資源類型linux

　　在k8s上，資源的類型有不少，好比pod，service，PersistentVolume，PersistentVolumeClaim等等，這些都是一些基礎的資源類型；咱們要建立某種資源，直接使用對應的資源類型，實例化一個對象便可；假如咱們要在k8s上建立一個集羣，咱們是否是能夠直接使用某種類型的資源，實例化一個集羣對象呢？理論上是能夠的，可是前提是對應k8s上有對應類型的資源；有對應類型的資源，用戶就能夠把對應建立資源的定義存放在etcd中；除了有對應類型的資源，咱們還須要有對應的控制器將對應資源建立出來；這樣一來對於不一樣集羣或應用來講，其組織方式和邏輯都不一樣，使用的資源類型和控制器也有所不一樣；用戶要想實例化更高級的資源來，就必須本身手動實現定義其資源類型，將對應資源類型實例化爲對象；除此以外必要時還須要本身實現對應資源的控制器；簡單講用戶想要實現更高級的資源類型，就必須擴展示有k8s的資源類型和控制器；nginx

　　在k8s上擴展資源類型的方式有三種，第一種是crd，crd是k8s內建的資源類型，該類型資源主要用來建立用戶自定義資源類型的資源；即經過crd資源，能夠將用戶自定義資源類型轉換爲k8s上資源類型；第二種是自定義apiserver；這種方式要比第一種方式要複雜一點，須要用戶手動開發程序實現對應功能的apiserver，讓其用戶建立自定義類型資源可以經過自定義apiserver實現；第三種方式就是修改現有k8sapiserver，讓其支持對應用戶自定義資源類型；git

　　自定義控制器程序員

　　自定義資源類型咱們可使用crd資源實現，也可使用自定義apiserver或修改原有apiserver代碼實現，可是隻有資源類型是不可以讓對應自定義類型資源實例化爲一個自定義資源對象，只有自定義資源類型，用戶建立對應資源類型的資源對象時，只能把對應資源類型的定義信息寫入到etcd中，它不能真正的跑起來，要想真正的跑起來，咱們還須要一個自定義控制器，專門負責監聽對應的資源類型的資源變化，將對應資源實例化爲對應k8s上的資源對象；固然不是全部的自定義類型的資源都須要自定義控制器，若是對應自定義類型資源調用了底層基礎控制器來管控對應自定義資源，那麼對應自定義類型資源就不須要使用自定義控制器；咱們知道控制器是k8s上的一個重要組件，它的工做邏輯是註冊監聽在apiserver上對應類型的資源變更，若是對應資源狀態不知足用戶指望狀態，它就會根據內部的和解循環來請求apiserver將對應類型資源的定義發送給它，而後根據資源定義來重建對應的資源，讓其狀態始終和用戶指望的狀態保持一致；自定義控制器也是一樣的邏輯，使用自定義控制器的目的也是讓對應自定義類型資源可以被自定義控制器監聽，一旦對應資源發生變更，它可以將其在k8s上建立出來，並一直保持和用戶指望的狀態吻合；自定義控制器和自定義資源類型能夠分開實現，也能夠合併在一塊兒實現，即自定義控制器程序可以自動建立crd資源，讓其對應自定義類型資源可以被k8s識別並將其建立出來；具體是分開實現仍是合併在一塊兒實現，取決開發自定義控制器程序員；github

　　crd資源定義幫助sql

[root@master01 ~]# kubectl explain crd
KIND:     CustomResourceDefinition
VERSION:  apiextensions.k8s.io/v1

DESCRIPTION:
     CustomResourceDefinition represents a resource that should be exposed on
     the API server. Its name MUST be in the format <.spec.name>.<.spec.group>.

FIELDS:
   apiVersion   <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind <string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata     <Object>

   spec <Object> -required-
     spec describes how the user wants the resources to appear

   status       <Object>
     status indicates the actual state of the CustomResourceDefinition

[root@master01 ~]#

　　提示：crd資源是k8s上的標準資源之一，它的定義主要有apiVersion，kind，metadata，spec和status；其中kind類型爲CustomResourceDefinition，apiVersion是apiextensions.k8s.io/v1；這兩個是固定格式；spec字段是用來定義對應指定資源類型資源的相關屬性；mongodb

　　crd.spec字段說明

[root@master01 ~]# kubectl explain crd.spec
KIND:     CustomResourceDefinition
VERSION:  apiextensions.k8s.io/v1

RESOURCE: spec <Object>

DESCRIPTION:
     spec describes how the user wants the resources to appear

     CustomResourceDefinitionSpec describes how a user wants their resource to
     appear

FIELDS:
   conversion   <Object>
     conversion defines conversion settings for the CRD.

   group        <string> -required-
     group is the API group of the defined custom resource. The custom resources
     are served under `/apis/<group>/...`. Must match the name of the
     CustomResourceDefinition (in the form `<names.plural>.<group>`).

   names        <Object> -required-
     names specify the resource and kind names for the custom resource.

   preserveUnknownFields        <boolean>
     preserveUnknownFields indicates that object fields which are not specified
     in the OpenAPI schema should be preserved when persisting to storage.
     apiVersion, kind, metadata and known fields inside metadata are always
     preserved. This field is deprecated in favor of setting
     `x-preserve-unknown-fields` to true in
     `spec.versions[*].schema.openAPIV3Schema`. See
     https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields
     for details.

   scope        <string> -required-
     scope indicates whether the defined custom resource is cluster- or
     namespace-scoped. Allowed values are `Cluster` and `Namespaced`.

   versions     <[]Object> -required-
     versions is the list of all API versions of the defined custom resource.
     Version names are used to compute the order in which served versions are
     listed in API discovery. If the version string is "kube-like", it will sort
     above non "kube-like" version strings, which are ordered lexicographically.
     "Kube-like" versions start with a "v", then are followed by a number (the
     major version), then optionally the string "alpha" or "beta" and another
     number (the minor version). These are sorted first by GA > beta > alpha
     (where GA is a version with no suffix such as beta or alpha), and then by
     comparing major version, then minor version. An example sorted list of
     versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2,
     foo1, foo10.

[root@master01 ~]#

　　提示：crd.spec中group字段使用來描述對應自定義類型資源的羣組名稱，其值爲字符串；names字段是用來描述自定義類型資源對應類型，名稱等等，其值爲一個對象；scope字段用來定義對應自定義資源是那個級別的資源；該字段的值只能爲Cluster或Namespaced；versions字段是用來指定對應自定義資源的版本，以及對應類型資源的屬性字段等信息，該字段爲一個列表對象；

　　示例：定義一個自定義類型資源

[root@master01 ~]# cat crontab-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # 名字必需與下面的 spec 字段匹配，而且格式爲 '<名稱的複數形式>.<組名>'
  name: crontabs.stable.example.com
spec:
  # 組名稱，用於 REST API: /apis/<組>/<版本>
  group: stable.example.com
  # 列舉此 CustomResourceDefinition 所支持的版本
  versions:
    - name: v1
      # 每一個版本均可以經過 served 標誌來獨立啓用或禁止
      served: true
      # 其中一個且只有一個版本必需被標記爲存儲版本
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # 能夠是 Namespaced 或 Cluster
  scope: Namespaced
  names:
    # 名稱的複數形式，用於 URL：/apis/<組>/<版本>/<名稱的複數形式>
    plural: crontabs
    # 名稱的單數形式，做爲命令行使用時和顯示時的別名
    singular: crontab
    # kind 一般是單數形式的駝峯編碼（CamelCased）形式。你的資源清單會使用這一形式。
    kind: CronTab
    # shortNames 容許你在命令行使用較短的字符串來匹配資源
    shortNames:
    - ct
[root@master01 ~]#

　　應用資源清單前使用kubectl get crontab

[root@master01 ~]# kubectl get crontab
error: the server doesn't have a resource type "crontab"
[root@master01 ~]#

　　提示：在沒有應用資源清單前使用kubectl get crontab，它會提示咱們所沒有類型爲crontab的資源

　　應用資源清單

[root@master01 ~]# kubectl apply -f crontab-crd.yaml
customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created
[root@master01 ~]# kubectl get crontab
No resources found in default namespace.
[root@master01 ~]#

　　提示：應用資源清單後，再次使用kubectl get crontab就沒有報錯了，只是提示在default名稱空間沒有對應類型資源；

　　查看crd

[root@master01 ~]# kubectl get crd
NAME                                                  CREATED AT
bgpconfigurations.crd.projectcalico.org               2021-01-03T15:49:21Z
bgppeers.crd.projectcalico.org                        2021-01-03T15:49:21Z
blockaffinities.crd.projectcalico.org                 2021-01-03T15:49:21Z
clusterinformations.crd.projectcalico.org             2021-01-03T15:49:21Z
crontabs.stable.example.com                           2021-01-12T12:39:00Z
felixconfigurations.crd.projectcalico.org             2021-01-03T15:49:21Z
globalnetworkpolicies.crd.projectcalico.org           2021-01-03T15:49:21Z
globalnetworksets.crd.projectcalico.org               2021-01-03T15:49:21Z
hostendpoints.crd.projectcalico.org                   2021-01-03T15:49:21Z
ipamblocks.crd.projectcalico.org                      2021-01-03T15:49:21Z
ipamconfigs.crd.projectcalico.org                     2021-01-03T15:49:21Z
ipamhandles.crd.projectcalico.org                     2021-01-03T15:49:21Z
ippools.crd.projectcalico.org                         2021-01-03T15:49:21Z
kubecontrollersconfigurations.crd.projectcalico.org   2021-01-03T15:49:21Z
networkpolicies.crd.projectcalico.org                 2021-01-03T15:49:21Z
networksets.crd.projectcalico.org                     2021-01-03T15:49:22Z
[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME                          CREATED AT
crontabs.stable.example.com   2021-01-12T12:39:00Z
[root@master01 ~]#

　　查看詳情

[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME                          CREATED AT
crontabs.stable.example.com   2021-01-12T12:39:00Z
[root@master01 ~]# kubectl describe crd/crontabs.stable.example.com
Name:         crontabs.stable.example.com
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2021-01-12T12:39:00Z
  Generation:          1
  Managed Fields:
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:acceptedNames:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:conditions:
    Manager:      kube-apiserver
    Operation:    Update
    Time:         2021-01-12T12:39:00Z
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:conversion:
          .:
          f:strategy:
        f:group:
        f:names:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:scope:
        f:versions:
      f:status:
        f:storedVersions:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-01-12T12:39:00Z
  Resource Version:  805506
  UID:               b92a90f4-c953-4876-a496-030c9ba023fd
Spec:
  Conversion:
    Strategy:  None
  Group:       stable.example.com
  Names:
    Kind:       CronTab
    List Kind:  CronTabList
    Plural:     crontabs
    Short Names:
      ct
    Singular:  crontab
  Scope:       Namespaced
  Versions:
    Name:  v1
    Schema:
      openAPIV3Schema:
        Properties:
          Spec:
            Properties:
              Cron Spec:
                Type:  string
              Image:
                Type:  string
              Replicas:
                Type:  integer
            Type:      object
        Type:          object
    Served:            true
    Storage:           true
Status:
  Accepted Names:
    Kind:       CronTab
    List Kind:  CronTabList
    Plural:     crontabs
    Short Names:
      ct
    Singular:  crontab
  Conditions:
    Last Transition Time:  2021-01-12T12:39:00Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2021-01-12T12:39:00Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1
Events:  <none>
[root@master01 ~]#

　　使用自定義資源類型crontab建立資源

[root@master01 ~]# cat my-crontab.yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
[root@master01 ~]#

　　提示：上述資源清單表示建立一個類型爲Crontab的資源，該資源的羣組版本爲stable.example.com/v1;

　　應用資源清單

[root@master01 ~]# kubectl apply -f my-crontab.yaml
crontab.stable.example.com/my-new-cron-object created
[root@master01 ~]# kubectl get ct
NAME                 AGE
my-new-cron-object   5s
[root@master01 ~]# kubectl describe ct/my-new-cron-object
Name:         my-new-cron-object
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  stable.example.com/v1
Kind:         CronTab
Metadata:
  Creation Timestamp:  2021-01-12T12:45:29Z
  Generation:          1
  Managed Fields:
    API Version:  stable.example.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:cronSpec:
        f:image:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-01-12T12:45:29Z
  Resource Version:  806182
  UID:               31a88a3d-fa99-42b8-80f6-3e4559efdc40
Spec:
  Cron Spec:  * * * * */5
  Image:      my-awesome-cron-image
Events:       <none>
[root@master01 ~]#

　　提示：能夠看到對應類型資源已經建立成功；以上示例只是單純的crd的使用示例，沒有任何實質的做用；

　　部署自定義控制器

　　示例：部署mongodb-aperator

　　一、克隆項目

[root@master01 ~]# git clone https://github.com/mongodb/mongodb-kubernetes-operator.git
Cloning into 'mongodb-kubernetes-operator'...
remote: Enumerating objects: 95, done.
remote: Counting objects: 100% (95/95), done.
remote: Compressing objects: 100% (74/74), done.
remote: Total 4506 (delta 30), reused 60 (delta 15), pack-reused 4411
Receiving objects: 100% (4506/4506), 18.04 MiB | 183.00 KiB/s, done.
Resolving deltas: 100% (2621/2621), done.
[root@master01 ~]#

　　二、建立名稱空間mongodb，並進入到mongodb-kubernetes-operator目錄應用crd資源，建立自定義資源類型

[root@master01 mongodb-kubernetes-operator]# kubectl create ns mongodb
namespace/mongodb created
[root@master01 mongodb-kubernetes-operator]# kubectl get ns
NAME                   STATUS   AGE
default                Active   35d
ingress-nginx          Active   22d
kube-node-lease        Active   35d
kube-public            Active   35d
kube-system            Active   35d
kubernetes-dashboard   Active   11d
mongodb                Active   4s
[root@master01 mongodb-kubernetes-operator]# ls
agent    build  deploy     docs    go.sum      pkg        release.json      scripts  testdata  version
APACHE2  cmd    dev_notes  go.mod  LICENSE.md  README.md  requirements.txt  test     tools.go
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_mongodb_crd.yaml -n mongodb
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/mongodb.mongodb.com created
[root@master01 mongodb-kubernetes-operator]#

　　驗證：查看mongodb類型資源是否已經建立成功？

[root@master01 mongodb-kubernetes-operator]# kubectl get crd
NAME                                                  CREATED AT
bgpconfigurations.crd.projectcalico.org               2021-01-03T15:49:21Z
bgppeers.crd.projectcalico.org                        2021-01-03T15:49:21Z
blockaffinities.crd.projectcalico.org                 2021-01-03T15:49:21Z
clusterinformations.crd.projectcalico.org             2021-01-03T15:49:21Z
crontabs.stable.example.com                           2021-01-12T12:39:00Z
felixconfigurations.crd.projectcalico.org             2021-01-03T15:49:21Z
globalnetworkpolicies.crd.projectcalico.org           2021-01-03T15:49:21Z
globalnetworksets.crd.projectcalico.org               2021-01-03T15:49:21Z
hostendpoints.crd.projectcalico.org                   2021-01-03T15:49:21Z
ipamblocks.crd.projectcalico.org                      2021-01-03T15:49:21Z
ipamconfigs.crd.projectcalico.org                     2021-01-03T15:49:21Z
ipamhandles.crd.projectcalico.org                     2021-01-03T15:49:21Z
ippools.crd.projectcalico.org                         2021-01-03T15:49:21Z
kubecontrollersconfigurations.crd.projectcalico.org   2021-01-03T15:49:21Z
mongodb.mongodb.com                                   2021-01-13T06:38:22Z
networkpolicies.crd.projectcalico.org                 2021-01-03T15:49:21Z
networksets.crd.projectcalico.org                     2021-01-03T15:49:22Z
[root@master01 mongodb-kubernetes-operator]# kubectl get crd/mongodb.mongodb.com
NAME                  CREATED AT
mongodb.mongodb.com   2021-01-13T06:38:22Z
[root@master01 mongodb-kubernetes-operator]#

　　三、安裝operator

[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/operator/ -n mongodb
deployment.apps/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
serviceaccount/mongodb-kubernetes-operator created
[root@master01 mongodb-kubernetes-operator]#

　　提示：mongodb-kubernetes-operator這個項目是將自定義控制器和自定義資源類型分開實現的；其operator只負責建立和監聽對應資源類型的變化，在資源有變化時，實例化爲對應資源對象，並保持對應資源對象狀態吻合用戶指望狀態；上述四個清單中主要是建立了一個sa帳戶，並對對應的sa用戶受權；

　　operator.yaml內容

[root@master01 mongodb-kubernetes-operator]# cat deploy/operator/operator.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-kubernetes-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: mongodb-kubernetes-operator
  template:
    metadata:
      labels:
        name: mongodb-kubernetes-operator
    spec:
      serviceAccountName: mongodb-kubernetes-operator
      containers:
        - name: mongodb-kubernetes-operator
          image: quay.io/mongodb/mongodb-kubernetes-operator:0.3.0
          command:
            - mongodb-kubernetes-operator
          imagePullPolicy: Always
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "mongodb-kubernetes-operator"
            - name: AGENT_IMAGE # The MongoDB Agent the operator will deploy to manage MongoDB deployments
              value: quay.io/mongodb/mongodb-agent:10.19.0.6562-1
            - name: VERSION_UPGRADE_HOOK_IMAGE
              value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2
            - name: MONGODB_IMAGE
              value: "library/mongo"
            - name: MONGODB_REPO_URL
              value: "registry.hub.docker.com"
[root@master01 mongodb-kubernetes-operator]#

　　提示：上述資源主要用deploy控制器運行對應自定義控制器爲一個pod；

　　驗證：查看operator是否正常運行

[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb
NAME                                           READY   STATUS    RESTARTS   AGE
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          26s
[root@master01 mongodb-kubernetes-operator]#

　　提示：可以看到operator正常運行，就表示operator已經安裝成功；

　　驗證：使用自定義資源類型建立一個mongodb 副本集集羣

[root@master01 mongodb-kubernetes-operator]# cat deploy/crds/mongodb.com_v1_mongodb_cr.yaml 
---
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: example-mongodb
spec:
  members: 3
  type: ReplicaSet
  version: "4.2.6"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: my-user
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: my-user-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram

# the user credentials will be generated from this secret
# once the credentials are generated, this secret is no longer required
---
apiVersion: v1
kind: Secret
metadata:
  name: my-user-password
type: Opaque
stringData:
  password: 58LObjiMpxcjP1sMDW
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]#

　　應用清單

[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml -n mongodb
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              0/2     Pending   0          9s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          88s
[root@master01 mongodb-kubernetes-operator]#

　　提示：這裏能夠看到對應pod處於pending狀態；

　　查看pod詳細信息

[root@master01 mongodb-kubernetes-operator]# kubectl describe pod/example-mongodb-0 -n mongodb|grep -A 10 "Events"
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  66s (x2 over 66s)  default-scheduler  0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims.
[root@master01 mongodb-kubernetes-operator]#

　　提示：這裏提示沒有能夠用的pvc；

　　刪除mongodb名稱空間下pvc

[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb
NAME                            STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-volume-example-mongodb-0   Pending                                                     92s
[root@master01 mongodb-kubernetes-operator]# kubectl delete pvc --all -n mongodb
persistentvolumeclaim "data-volume-example-mongodb-0" deleted
[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb
No resources found in mongodb namespace.
[root@master01 mongodb-kubernetes-operator]#

　　建立pv和pvc

[root@master01 mongodb-kubernetes-operator]# cd
[root@master01 ~]# cat pv-demo.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v1
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v1
    server: 192.168.0.99
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v2
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v2
    server: 192.168.0.99
---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v3
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v3
    server: 192.168.0.99
[root@master01 ~]#

　　應用清單建立pv

[root@master01 ~]# kubectl apply -f pv-demo.yaml
persistentvolume/nfs-pv-v1 created
persistentvolume/nfs-pv-v2 created
persistentvolume/nfs-pv-v3 created
[root@master01 ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
nfs-pv-v1   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
nfs-pv-v2   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
nfs-pv-v3   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
[root@master01 ~]#

　　建立pvc清單

[root@master01 ~]# cat pvc-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-0 
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-1
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-2
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
[root@master01 ~]#

　　應用清單建立pvc

[root@master01 ~]# kubectl get pvc -n mongodb
No resources found in mongodb namespace.
[root@master01 ~]# kubectl apply -f pvc-demo.yaml
persistentvolumeclaim/data-volume-example-mongodb-0 created
persistentvolumeclaim/data-volume-example-mongodb-1 created
persistentvolumeclaim/data-volume-example-mongodb-2 created
[root@master01 ~]# kubectl get pvc -n mongodb    
NAME                            STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-volume-example-mongodb-0   Bound    nfs-pv-v1   1Gi        RWO,ROX,RWX                   6s
data-volume-example-mongodb-1   Bound    nfs-pv-v2   1Gi        RWO,ROX,RWX                   6s
data-volume-example-mongodb-2   Bound    nfs-pv-v3   1Gi        RWO,ROX,RWX                   6s
[root@master01 ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                   STORAGECLASS   REASON   AGE
nfs-pv-v1   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-0                           102s
nfs-pv-v2   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-1                           102s
nfs-pv-v3   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-2                           102s
[root@master01 ~]#

　　提示：能夠看到對應pvc和pv已經綁定好了；

　　驗證：查看mongodb副本集集羣是否跑起來了？

[root@master01 ~]# kubectl get pods -n mongodb
NAME                                           READY   STATUS            RESTARTS   AGE
example-mongodb-0                              2/2     Running           0          6m19s
example-mongodb-1                              0/2     PodInitializing   0          10s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running           0          7m38s
[root@master01 ~]# kubectl get pods -n mongodb -w
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              2/2     Running   0          6m35s
example-mongodb-1                              1/2     Running   0          26s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          7m54s
example-mongodb-1                              2/2     Running   0          43s
example-mongodb-2                              0/2     Pending   0          0s
example-mongodb-2                              0/2     Pending   0          0s
example-mongodb-2                              0/2     Init:0/1   0          0s
example-mongodb-2                              0/2     Init:0/1   0          1s
example-mongodb-2                              0/2     Terminating   0          4s
example-mongodb-2                              0/2     Terminating   0          6s
example-mongodb-2                              0/2     Terminating   0          20s
example-mongodb-2                              0/2     Terminating   0          20s
example-mongodb-2                              0/2     Pending       0          0s
example-mongodb-2                              0/2     Pending       0          0s
example-mongodb-2                              0/2     Init:0/1      0          0s
example-mongodb-2                              0/2     Init:0/1      0          1s
example-mongodb-2                              0/2     PodInitializing   0          7s
example-mongodb-2                              1/2     Running           0          14s
example-mongodb-2                              2/2     Running           0          36s
^C[root@master01 ~]# kubectl get pods -n mongodb 
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              2/2     Running   0          8m
example-mongodb-1                              2/2     Running   0          111s
example-mongodb-2                              2/2     Running   0          48s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          9m19s
[root@master01 ~]#

　　提示：能夠看到對應的pod已經正常跑起來了；

　　驗證：使用mongo 鏈接mongodbpod，看看對應副本集集羣是否工做正常？

[root@master01 ~]# kubectl get pods -n mongodb -o wide
NAME                                           READY   STATUS    RESTARTS   AGE     IP             NODE             NOMINATED NODE   READINESS GATES
example-mongodb-0                              2/2     Running   0          9m12s   10.244.4.101   node04.k8s.org   <none>           <none>
example-mongodb-1                              2/2     Running   0          3m3s    10.244.2.130   node02.k8s.org   <none>           <none>
example-mongodb-2                              2/2     Running   0          2m      10.244.1.130   node01.k8s.org   <none>           <none>
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          10m     10.244.3.116   node03.k8s.org   <none>           <none>
[root@master01 ~]# mongo 10.244.4.101
MongoDB shell version v4.4.3
connecting to: mongodb://10.244.4.101:27017/test?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b9d16fe9-6a74-4638-96e6-70aaf3c83bfa") }
MongoDB server version: 4.2.6
WARNING: shell and server versions do not match
example-mongodb:PRIMARY> show dbs
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
Error: Authentication failed.
0
example-mongodb:PRIMARY> use admin
switched to db admin
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
1
example-mongodb:PRIMARY> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
example-mongodb:PRIMARY> db.isMaster()
{
        "hosts" : [
                "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
                "example-mongodb-1.example-mongodb-svc.mongodb.svc.cluster.local:27017",
                "example-mongodb-2.example-mongodb-svc.mongodb.svc.cluster.local:27017"
        ],
        "setName" : "example-mongodb",
        "setVersion" : 1,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
        "me" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
        "electionId" : ObjectId("7fffffff0000000000000003"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1610520741, 1),
                        "t" : NumberLong(3)
                },
                "lastWriteDate" : ISODate("2021-01-13T06:52:21Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1610520741, 1),
                        "t" : NumberLong(3)
                },
                "majorityWriteDate" : ISODate("2021-01-13T06:52:21Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2021-01-13T06:52:27.873Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "connectionId" : 153,
        "minWireVersion" : 0,
        "maxWireVersion" : 8,
        "readOnly" : false,
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1610520741, 1),
                "signature" : {
                        "hash" : BinData(0,"EcWzL7O9Ue9kmm6cQ4FumkcIP6g="),
                        "keyId" : NumberLong("6917119940596072451")
                }
        },
        "operationTime" : Timestamp(1610520741, 1)
}
example-mongodb:PRIMARY>

　　提示：能夠看到三個mongodb pod是副本集關係；其中example-mongodb-0是主節點，其餘兩個是從節點；

　　最後仍是說一下，我在作上面的實驗中，雖然mongodb operator已經正常工做，可是用mongo這個客戶端工具鏈接到主節點無法寫數據，提示沒有權限；但對應的用戶在對應庫是有讀寫權限的；在admin庫下建立用戶能提示用戶添加成功，可是過幾秒查詢用戶信息，發現用戶不存在，不知道什麼緣由，有知道的朋友麻煩告知下（博主郵箱：linux-1874@qq.com），博主將感激涕零。。