linux運維、架構之路-K8s中部署Jenkins集羣高可用

1、在K8s中 部署Jenkins優勢

一、傳統Jenkins集羣架構一些問題

• Master發生故障時，整個流程都不可用
• Slave集羣的環境配置不同，來完成不一樣語言的編譯打包，可是這些差別化的配置致使管理起來不方便，維護麻煩
• 資源分配不均衡，有的slave要運行的job出現排隊等待，而有的salve處於空閒狀態
• 資源浪費，每臺slave多是物理機或者虛擬機，當slave處於空閒狀態時，也不能徹底釋放掉資源

二、K8s中Jenkins集羣架構優勢

• 當Jenkins Master接受到Build請求後，會根據配置的Label動態建立一個運行在Pod中的Jenkins Slave並註冊到Master上，當運行完Job後，這個Slave會被註銷而且這個Pod也會自動刪除，恢復到最初的狀態(這個策略能夠設置)
• 服務高可用，當Jenkins Master出現故障時，Kubernetes會自動建立一個新的Jenkins Master容器，而且將Volume分配給新建立的容器，保證數據不丟失，從而達到集羣服務高可用的做用
• 動態伸縮，合理使用資源，每次運行Job時，會自動建立一個Jenkins Slave，Job完成後，Slave自動註銷並刪除容器，資源自動釋放，而且Kubernetes會根據每一個資源的使用狀況，動態分配slave到空閒的節點上建立，下降出現因某節點資源利用率高，下降出現因某節點利用率高出現排隊的狀況
• 擴展性好，當Kubernetes集羣的資源嚴重不足致使Job排隊等待時，能夠很容器的添加一個Kubernetes Node到集羣，從而實現擴展

2、K8s 部署Jenkins

一、集羣環境

 二、k8s-node1部署NFS服務

①NFS服務端配置

mkdir -p /data/k8s chown -R nfsnobody.nfsnobody /data echo "/data/k8s 192.168.56.0/24(rw,no_root_squash,sync)" >/etc/exports systemctl enable rpcbind systemctl enable nfs systemctl start rpcbind systemctl start nf

②客戶端掛載

systemctl start rpcbind systemctl enable rpcbind mkdir /data/k8s -p mount -t nfs 192.168.56.11:/data/k8s /data/k8

三、建立Jenkins集羣所需的YAML文件

①建立命名空間和存放Jenkins的YAML目錄

kubectl create namespace demon mkdir -p /opt/jenkins

②爲Jenkins數據持久化存儲建立一個PV

cat >/opt/jenkins/jenkins_pv.yaml <<EOF apiVersion: v1 kind: PersistentVolume metadata: name: opspv spec: capacity: storage: 10Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Delete nfs: server: 192.168.56.11 path: /data/k8s --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: opspvc namespace: demon spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi EOF

③建立Jenkins集羣權限serviceAccount文件

cat >/opt/jenkins/jenkins_rbac.yaml <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: demon --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins rules: - apiGroups: ["extensions", "apps"] resources: ["deployments"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["services"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: jenkins namespace: demon roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins subjects: - kind: ServiceAccount name: jenkins namespace: demon EOF

④建立Jenkins Deployment

cat jenkins_deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: jenkins namespace: demon spec: template: metadata: labels: app: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccount: jenkins containers: - name: jenkins image: jenkins/jenkins:lts imagePullPolicy: IfNotPresent ports: - containerPort: 8080                #外部訪問端口 name: web protocol: TCP - containerPort: 50000              #jenkins save發現端口 name: agent protocol: TCP resources: limits: cpu: 1000m memory: 1Gi requests: cpu: 500m memory: 512Mi livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60          #容器初始化完成後，等待60秒進行探針檢查 timeoutSeconds: 5 failureThreshold: 12          #當Pod成功啓動且檢查失敗時，Kubernetes將在放棄以前嘗試failureThreshold次。放棄生存檢查意味着從新啓動Pod。而放棄就緒檢查，Pod將被標記爲未就緒。默認爲3.最小值爲1 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 volumeMounts: #須要將jenkins_home目錄掛載出來         - name: jenkinshome subPath: jenkins mountPath: /var/jenkins_home env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai securityContext: fsGroup: 1000 volumes: - name: jenkinshome persistentVolumeClaim: claimName: opspvc

⑤建立Jenkins SVC

cat >/opt/jenkins/jenkins_svc.yaml <<EOF apiVersion: v1 kind: Service metadata: name: jenkins namespace: demon labels: app: jenkins spec: selector: app: jenkins type: NodePort ports: - name: web port: 8080 targetPort: web nodePort: 30002
  - name: agent port: 50000 targetPort: agent EOF

⑥依次建立

[root@k8s-node1 jenkins]# ll 總用量 16
-rw-r--r-- 1 root root 2281 12月 27 14:59 jenkins_deployment.yaml -rw-r--r-- 1 root root  404 12月 27 14:57 jenkins_pv.yaml -rw-r--r-- 1 root root 1108 12月 27 14:58 jenkins_rbac.yaml -rw-r--r-- 1 root root  285 12月 27 14:59 jenkins_svc.yaml

[root@k8s-node1 jenkins]# kubectl apply -f jenkins_pv.yaml persistentvolume/opspv created persistentvolumeclaim/opspvc created [root@k8s-node1 jenkins]# kubectl apply -f jenkins_rbac.yaml serviceaccount/jenkins created clusterrole.rbac.authorization.k8s.io/jenkins created clusterrolebinding.rbac.authorization.k8s.io/jenkins created [root@k8s-node1 jenkins]# kubectl apply -f jenkins_deployment.yaml deployment.extensions/jenkins created [root@k8s-node1 jenkins]# kubectl apply -f jenkins_svc.yaml service/jenkins create

⑦查看結果

[root@k8s-node1 jenkins]# kubectl get pv,pvc,pod,svc -n demon NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/opspv   10Gi       RWX            Delete           Bound     demon/opspvc 1h NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/opspvc Bound opspv 10Gi RWX 1h NAME READY STATUS RESTARTS AGE pod/jenkins-6d7bc49b74-d9jxc   1/1       Running   0 1h NAME TYPE CLUSTER-IP     EXTERNAL-IP PORT(S) AGE service/jenkins   NodePort   10.1.148.201   <none>        8080:30002/TCP,50000:26723/TCP   1h

8080：端口爲咱們jenkins訪問端口  50000：端口爲jenkins save發現端口

⑧經過瀏覽器訪問集羣任意IP的svc端口

管理員密碼路徑：持久化在/data/k8s下，因此jenkins的全部配置都在這下面

cat /data/k8s/jenkins/secrets/initialAdminPassword

 

 直接推薦安裝便可

⑨安裝完成後咱們進入jenkins主頁面

 ⑩enkins–>插件–>安裝插件Kubernetes

 3、Jenkins中配置k8s

一、系統管理->系統配置

 

 二、配置拉到最下面找到Kubernetes插件

Name 配置的名稱 Kubernetes URL 這裏的URL是K8s內部的URL，實際上就是svcname https://kubernetes.default.svc.cluster.local Kubernetes Namespace k8s的命名空間 （實際上就是Jenkins所在的命名空間）

 三、Jenkins URL配置

 

 

 

Jenkins URL 這裏的URL是jenkins的svc名稱加上命名空間，實際上就是在k8s集羣內部訪問jenkins的一個方法，這裏也不須要修改 http://jenkins.demon.svc.cluster.local:8080

 四、配置添 Jenkins Slave Pod模板

Name = Pod 名稱 Namespave = Pod命名空間 Labels = Pod標籤

五、容器的模板配置

 

 

 六、建立volume的配置

 

 Jenkins Master收到Build請求時，會根據配置的Label動態建立一個運行在Pod中的Jenkins Slave並註冊到Master上，當Job運行完，這個Slave會被註銷而且這個Pod也會自動刪除，恢復到最初狀態

4、測試驗證

一、新建Job選擇流水線

 

 二、流水線Pipeline

def label = "jenkins-slave" podTemplate(label: label, cloud: 'kubernetes') { node(label) { stage('pull code') { echo "拉取代碼" } stage('build') { echo "代碼編譯" } stage('SonarQube') { echo "質量掃描" } } }

三、執行效果