Kubernetes+Gitlab+Jenkins構建鏡像並建立Pod

1.從新構建 jnlp-slave

# ls
config  docker-18.06.1-ce.tgz  Dockerfile  kubectl

A.建立 config

# vim config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/ssl/ca.pem
    server: https://192.168.100.180:8443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: admin
  name: kubernetes
current-context: kubernetes
kind: Config
preferences: {}
users:
- name: admin
  user:
    client-certificate: /etc/kubernetes/ssl/admin.pem
    client-key: /etc/kubernetes/ssl/admin-key.pem

B.建立 Dockerfile

# vim Dockerfile
FROM jenkins/jnlp-slave:3.23-1-alpine
MAINTAINER zhi <wangzhijiansd@qq.com>

USER root
ARG DOCKER_GID=994
ENV DOCKER_VERSION=18.06.1-ce
ENV SSL_DIR /etc/kubernetes/ssl

# 提取 docker 二進制文件 
COPY docker-${DOCKER_VERSION}.tgz /var/tmp/
RUN  tar --strip-components=1 -xvzf /var/tmp/docker-${DOCKER_VERSION}.tgz -C /usr/local/bin \
    && rm -rf /var/tmp/docker-${DOCKER_VERSION}.tgz \
    && chmod -R 775 /usr/local/bin/docker
    
# 安裝 kubectl
COPY kubectl /usr/local/bin/

# 此處文件均爲空文件將在運行時由 ConfigMap 掛載爲 Volume 填充真實證書文件
COPY config ${SSL_DIR}/
RUN mkdir -p /root/.kube/ && mkdir -p ${SSL_DIR} \
    touch ${SSL_DIR}/ca.pem \
    touch ${SSL_DIR}/admin.pem \
    touch ${SSL_DIR}/admin-key.pem  
RUN export KUBE_CONFIG==${SSL_DIR}/config && kubectl config view 
RUN addgroup -g ${DOCKER_GID} docker && adduser jenkins docker 

# 暴露證書文件所在文件夾，在運行時由 ConfigMap 掛載爲 Volume 填充真實證書文件
VOLUME ${SSL_DIR}

USER jenkins:${DOCKER_GID}

注: docker 和 kubectl 跟 kubernetes 集羣的版本一致。同時說明一下，這裏GitLab 、Jenkins都部署在kube-ops這個namespace下。而docker使用了docker in docker 掛載宿主機/var/run/docker.dock進行操做。

C.構建 jnlp-slave 並推送至私有鏡像倉庫

# docker build -t 192.168.100.100/jenkinsci/jnlp-slave:latest .
# docker images|grep jnlp-slave
192.168.100.100/jenkinsci/jnlp-slave         latest            d3b287300a83        6 days ago          377MB
# docker push 192.168.100.100/jenkinsci/jnlp-slave:latest

2.建立並查看 configmap

# kubectl create configmap kubectl-cert-cm --from-file=/etc/kubernetes/ssl -n kube-ops
# kubectl describe configmap kubectl-cert-cm -n kube-ops

3.部署 Jenkins

注: 因爲沒法解決配置了專有 secret 而 Jenkins 依然去找 default 的 secret 的問題，這裏從新配置了 RBAC 以使用 default 的 secret 並綁定了 cluster-admin 角色。

A.建立 RBAC 

# vim jenkins-rbac.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: default
  namespace: kube-ops
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","get","update","list","watch","patch","delete"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","get","update","list","watch","patch","delete"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-ops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: default
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-ops

B.建立PVC

# vim jenkins-pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pv-claim
  namespace: kube-ops
  labels:
    app: jenkins
spec:
  storageClassName: kube-ops
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: "10Gi"

C.建立 deployment 和 service

# vim jenkins.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: kube-ops
spec:
  replicas: 1
  template:
    metadata:
      name: jenkins
      labels:
        name: jenkins
    spec:
      #serviceAccountName: jenkins
      serviceAccountName: default
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
      containers:
      - name: jenkins
        image: 192.168.100.100/jenkinsci/jenkins:lts
        resources:
          requests:
            memory: "1Gi"
            cpu: "1000m"
          limits:
            memory: "2Gi"
            cpu: "2000m"
        env:
        - name: JAVA_OPTS
          value: "-Duser.timezone=Asia/Shanghai"  
        ports:
        - name: web
          containerPort: 8080
        - name: agent
          containerPort: 50000
        volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins-data
        volumeMounts:
        - mountPath: /var/run/docker.sock
          readOnly: false
          name: docker-sock
      volumes:
      - name: jenkins-data
        persistentVolumeClaim:
          claimName: jenkins-pv-claim
      - name: docker-sock
        hostPath: 
          path: /var/run/docker.sock
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: kube-ops
  labels:
    name: jenkins
spec:
  type: LoadBalancer
  ports:
    - name: web
      port: 8080
      targetPort: web
    - name: agent
      port: 50000
      targetPort: agent
  selector:
    name: jenkins

注:這裏就不在贅述Jenkins的部署。也再也不贅述gitlab的部署，須要更改的地方大同小異。

4.配置Jenkins

5.新建項目

# git clone http://192.168.100.185:32030/wangzhijian/grafana.git
正克隆到 'grafana'...
Username for 'http://192.168.100.185:32030': wangzhijian
Password for 'http://wangzhijian@192.168.100.185:32030': 
remote: Counting objects: 37, done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 37 (delta 7), reused 37 (delta 7)
Unpacking objects: 100% (37/37), done.
# ls grafana/
build  grafana.yaml
# ls grafana/build/
build.sh  Dockerfile

注:該項非必須，只是想顯示一下文件存放目錄。原本我想使用本地添加hosts，而後使用域名來git，可是不知道是由於添加了traefik代理的緣由仍是traefik的SSL沒有徹底配置好的緣由，總之這裏沒有解決使用域名來git的問題。

使用IP進行git操做的格式:git clone http://nodeip:port/yourname/project.git

A.配置Dockerfile

# vim Dockerfile 
ARG GRAFANA_VERSION="latest"

FROM 192.168.100.100/grafana/grafana:${GRAFANA_VERSION}

USER grafana

ARG GF_INSTALL_PLUGINS=""

RUN if [ ! -z "${GF_INSTALL_PLUGINS}" ]; then \
    OLDIFS=$IFS; \
        IFS=','; \
    for plugin in ${GF_INSTALL_PLUGINS}; do \
        IFS=$OLDIFS; \
        grafana-cli --pluginsDir "$GF_PATHS_PLUGINS" plugins install ${plugin}; \
    done; \

B.配置構建腳本

# vim build.sh 
#!/bin/bash

IMAGE="192.168.100.100/grafana/grafana-plugins"

# 編譯鏡像
docker build -f build/Dockerfile -t $IMAGE \
  --build-arg "GRAFANA_VERSION=latest" \
  --build-arg "GF_INSTALL_PLUGINS=grafana-worldmap-panel,grafana-clock-panel,grafana-piechart-panel,alexanderzobnin-zabbix-app,grafana-kubernetes-app,grafana-simple-json-datasource,michaeldmoore-annunciator-panel" .
  
# 登陸Harbor鏡像倉庫
docker login -u admin -p Harbor12345 192.168.100.100

# 上傳鏡像 
docker push $IMAGE

# 清理鏡像
docker rmi $IMAGE

C.建立yaml文件

# vim grafana.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: grafana
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      name: grafana
      labels:
        k8s-app: grafana
    spec:
      containers:
      - name: grafana
        image: 192.168.100.100/grafana/grafana-plugins:latest
        ports:
        - containerPort: 3000
          protocol: TCP
        env:
        - name: GF_SERVER_HTTP_PORT
          value: "3000"
        - name: GF_AUTH_BASIC_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "false"
        - name: GF_SECURITY_ADMIN_USER
          value: "admin"
        - name: GF_SECURITY_ADMIN_PASSWORD
          value: "wangzhijian"
        - name: GF_SERVER_ROOT_URL
          value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
          value: /
        - name: GF_SMTP_ENABLED
          value: "true"
        - name: GF_SMTP_HOST
          value: smtp.qq.com:465
        - name: GF_SMTP_USER
          value: wangzhijiansd@qq.com
        - name: GF_SMTP_PASSWORD
          value: ********
        - name: GF_SMTP_FROM_ADDRESS
          value: wangzhijiansd@qq.com
---
apiVersion: v1
kind: Service
metadata:
  labels:
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Grafana
  name: grafana
  namespace: kube-system
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 3000
  selector:
    k8s-app: grafana

6.使用Jenkins新建任務

7.構建輸出

A.構建鏡像並推送到私有鏡像倉庫

B.部署grafana

C.查看部署狀況

# kubectl cluster-info|grep Grafana
Grafana is running at https://192.168.100.180:8443/api/v1/namespaces/kube-system/services/grafana/proxy
# kubectl -n kube-system get pod|grep grafana
grafana-576fd64977-2kj6c                1/1       Running   0          1d
# kubectl -n kube-system get service|grep grafana
grafana                   LoadBalancer   10.244.249.26    <pending>     80:36902/TCP      1d

這裏使用瀏覽器輸入http://nodeip:port,使用變量配置的用戶名(admin)和密碼(wangzhijian)進行登陸

構建鏡像時添加的一些插件顯示以下:

特別感謝:

爲了將kubectl運行起來，試過來不少方法，也搜了不少資料，可是依然不得其所，感謝做者和他的這篇文章:

基於 IBM Cloud Private 的 DevOps 實踐