kubernetes上安裝MongoDB-3.6.5集羣副本集方式

時間 2019-11-20

標籤 kubernetes 安裝 mongodb 3.6.5 集羣副本方式欄目 MongoDB 简体版

原文原文鏈接

1、安裝部署：

想直接一步建立集羣的小夥伴直接按如下步驟安裝(再日後是記錄本身出過的錯)：html

一、生成docker鏡像：

docker build -t 144.202.127.156/library/mongodb:3.4.10 . linux

docker push 144.202.127.156/library/mongodb:3.4.10 注：記得把地址換成本身的鏡像倉庫git

FROM  alpine:edge
MAINTAINER chengcuichao

RUN apk update && \
   echo http://dl-4.alpinelinux.org/alpine/edge/testing >> /etc/apk/repositories && \
   apk add --no-cache mongodb numactl numactl-tools

Dockerfile

二、在kubernete上建立：

kubectl create -f mongodb.ymlgithub

在此注意幾下幾點：mongodb

　　1）、將爲外置存儲的Secret和storageclass換成本身的。注：我用的是ceph，其餘的根據本身的更改，也能夠不用持久化存儲，就當測試用。docker

　　2）、鏡像換成本身的鏡像地址。api

　　3）、啓動參數--replSet換成本身的，不換也能夠。bash

apiVersion: v1
kind: Namespace
metadata:
  name: basic-app
---
apiVersion: v1
kind: Secret
metadata:
  namespace: basic-app
  name: ceph-secret
type: "kubernetes.io/rbd"
data:
  key: QVFEYmVRTmJZQ1B4TFJBQUg0QS9Tb01NZjF6NHB3L0p1Y3ZUQnc9PQ==
---
apiVersion: v1
kind: Secret
metadata:
  namespace: basic-app
  name: mongo-key
type: Opaque
data:
  key: UERVU0hWVU9KT1ZQVUVYT0JXWU8=
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb-config
  namespace: basic-app
data:
  mongodb: |
    systemLog:
      destination: file
      path: "/var/log/mongodb/mongodb.log"
      logAppend: true
    processManagement:
      fork: false
    net:
      port: 27017
      bindIp: 0.0.0.0
    security:
      keyFile: "/etc/conf.d/secret-key/key"
      authorization: enabled
    storage:
      dbPath: /var/lib/mongodb
    setParameter:
      enableLocalhostAuthBypass: true
      authenticationMechanisms: SCRAM-SHA-1
  mongos: |
    MONGOS_EXEC="/usr/bin/mongos"
    MONGOS_RUN="/var/run/mongodb"
    MONGOS_USER="mongodb"
    MONGOS_IP="127.0.0.1"
    MONGOS_PORT="27018"
    MONGOS_CONFIGDB=""
    MONGOS_OPTIONS=""
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: basic-app
  labels:
    name: mongo
spec:
  clusterIP: None
  ports:
  - port: 27017
    targetPort: 27017
  selector:
    app: mongo-cluster
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongodb
  namespace: basic-app
spec:
  serviceName: mongodb
  replicas: 3
  template:
    metadata:
      labels:
        app: mongo-cluster
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mongod
        image: 144.202.127.156/library/mongodb:3.6.5
        command: ["sh", "-c", "chmod 600 -R /etc/conf.d/secret-key;numactl --interleave=all mongod -f /etc/conf.d/mongodb --auth --replSet icsoc"]
        resources:
          limits:
            cpu: 2
            memory: 2G
          requests:
            cpu: 1
            memory: 1G
        volumeMounts:
        - name: mongodb-data
          mountPath: /var/lib/mongodb
        - name: mongo-config
          mountPath: /etc/conf.d
        - name: timezone-config
          mountPath: /etc/localtime
        - name: secret-key
          mountPath: /etc/conf.d/
        ports:
        - containerPort: 27017
        livenessProbe:
          tcpSocket:
            port: 27017
          initialDelaySeconds: 15
          periodSeconds: 20
      volumes:
      - name: mongo-config
        configMap:
          name: mongodb-config
      - name: timezone-config
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
      - name: secret-key
        secret:
          secretName: mongo-key
  volumeClaimTemplates:
  - metadata:
      name: mongodb-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "ceph-db"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 50Gi

mongodb.yml

三、初始化集羣：

1、連進容器內：
kubectl exec -it mongodb-0 /bin/sh

2、執行初始化副本集：
mongo
rs.initiate({_id: "icsoc", version: 1, members: [
       { _id: 0, host : "mongodb-0.mongodb.basic-app.svc.cluster.local:27017" },
       { _id: 1, host : "mongodb-1.mongodb.basic-app.svc.cluster.local:27017" },
       { _id: 2, host : "mongodb-2.mongodb.basic-app.svc.cluster.local:27017" }
 ]});

二、建立管理用戶：    具體可參考：https://docs.mongodb.com/manual/tutorial/enable-authentication/
use admin
db.createUser(
  {
    user: "myUserAdmin",
    pwd: "P@ssw0rd",
    roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
  }
)
db.auth("myUserAdmin","P@ssw0rd")

三、以後就能夠連進去建立用戶，賦予角色權限使用。

2、如下記錄建立詳細步驟：

一、在kubenetes安裝mongodb集羣大致思路爲：

　　1）、先在一個基礎的鏡像裏安裝mongodb，啓動正常後無報錯。app

　　2）、再理清楚mogodb副本集集羣是怎麼啓動的。tcp

　　3）、以後編寫在kubernetes建立資源的文件。

　　4）、先建立看看那進行不下去，再一個個解決。

二、在alpine:edge基礎上編寫dockerfile：

編寫好的Dockerfile以下：

FROM  alpine:edge
MAINTAINER chengcuichao

RUN apk update && \
   echo http://dl-4.alpinelinux.org/alpine/edge/testing >> /etc/apk/repositories && \
   apk add --no-cache mongodb numactl

COPY run.sh  /root/
RUN chmod +x /root/run.sh

CMD /root/run.sh

Dockerfile

先在docker上啓動，mongo連進去後報錯：

Server has startup warnings: 
2018-07-15T12:25:52.064+0800 W CONTROL  [main] --diaglog is deprecated and will be removed in a future release
2018-07-15T12:25:52.183+0800 I STORAGE  [initandlisten] 
2018-07-15T12:25:52.183+0800 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-07-15T12:25:52.183+0800 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem    第一個報錯 2018-07-15T12:26:02.364+0800 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.  第二個 2018-07-15T12:26:02.364+0800 I CONTROL  [initandlisten] 
2018-07-15T12:26:02.364+0800 I CONTROL  [initandlisten] 
2018-07-15T12:26:02.364+0800 I CONTROL  [initandlisten] ** WARNING: You are running on a NUMA machine.
2018-07-15T12:26:02.364+0800 I CONTROL  [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2018-07-15T12:26:02.364+0800 I CONTROL  [initandlisten] **              numactl --interleave=all mongod [other options]  第三個 2018-07-15T12:26:02.365+0800 I CONTROL  [initandlisten] 
2018-07-15T12:26:02.365+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-07-15T12:26:02.365+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'  第四個
2018-07-15T12:26:02.365+0800 I CONTROL  [initandlisten] 
2018-07-15T12:26:02.365+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-07-15T12:26:02.365+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'  第五個

　　1）、第一個報錯是文件系統的問題，還沒解決。

　　2）、第二個報錯是要開啓認證，在啓動參數上加入--auth就能夠。

　　3）、第三個報錯須要在mongod命令前面加上numactl --interleave=all，在docker容器裏執行numactl --interleave=all mongod -f /etc/conf.d/mongodb 在直接用docker起的容器裏執行會報錯：

set_mempolicy: Operation not permitted
setting interleave mask: Operation not permitted

但在statefulset的yml文件加上command: ["sh", "-c", "numactl --interleave=all mongod -f /etc/conf.d/mongodb --bind_ip 0.0.0.0"]，kubectl create -f mongodb.yml建立後不會報錯。

　　4）、第四個和第五個報錯須要執行：echo never > /sys/kernel/mm/transparent_hugepage/enabled，echo never > /sys/kernel/mm/transparent_hugepage/defrag，

可是在容器裏執行會報錯，就算用initContainers來爲mongodb的容器建立運行環境，可是還會報：

/bin/sh: can't create /sys/kernel/mm/transparent_hugepage/enabled: Read-only file system
/bin/sh: can't create /sys/kernel/mm/transparent_hugepage/defrag: Read-only file system

爲容器增長守護腳本，以daemonset方式運行：參考：http://pauldone.blogspot.com/2017/06/mongodb-kubernetes-production-settings.html https://github.com/kubernetes/contrib/tree/master/startup-script

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  namespace: basic-app
  name: hostvm-configurer
  labels:
    app: startup-script
spec:
  template:
    metadata:
      labels:
        app: startup-script
    spec:
      hostPID: true
      containers:
      - name: hostvm-configurer-container
#        image: gcr.io/google-containers/startup-script:v1
        image: 144.202.127.156/google_containers/startup-script:v1
        securityContext:
          privileged: true
        env:
        - name: STARTUP_SCRIPT
          value: |
            #! /bin/bash
            set -o errexit
            set -o pipefail
            set -o nounset

            # Disable hugepages
            echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled
            echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag