Jenkis pipeline構建項目實踐

時間 2021-01-02

標籤 java node web docker shell api tomcat 安全 app webapp 欄目設計模式简体版

原文原文鏈接

在完成前文的jenkins server 在k8s環境部署以後，本文咱們來測試在k8s集羣環境中的jenkins pipeline構建項目和更新，具體環境要求以下：
一、jenkins pipeline插件安裝成功
二、要更新的應用已提早部署
三、Jenkins slave中須要有kubectl、svn、mvn客戶端且環境變量設置準確
四、Jenkis slave須要能和master的api-server進行正常通訊（這裏爲了簡便，前文構建jenkins server的時候直接對defalut這個service account作rabc受權。其餘可選方案是使用~/.kube/config文件實現客戶端的受權認證）
五、須要配置共享卷(slave編譯完成以後把war包copy到共享卷位置)java

1、建立構建任務

一、點擊「jenkins 」——「新建」——「pipeline」，輸入項目的名稱，點擊「OK」
node

二、下拉到「pipeline」配置，填寫腳本

具體腳本內容以下：web

podTemplate(name: 'jenkins-slave', cloud: 'kubernetes',
  namespace: 'default', label: 'jenkins-slave',
  serviceAccount: 'default', containers: [
  containerTemplate(
      name: 'jenkins-slave',
      image: 'harbor.59iedu.com/fjhb/jenkins-slave-toolkit:2018-08-10-v1',
      args: '${computer.jnlpmac} ${computer.name}',
      ttyEnabled: true,
      privileged: false,
      alwaysPullImage:true,
      )
  ],
  volumes: [
    persistentVolumeClaim(mountPath: '/tmp/', claimName: 'tomcat-jcsj-data')
  ]) {
  node('jenkins-slave') {
    stage('svn-checkout') {
      container('jnlp') {
          sh """
          svn checkout --username=yanglw --password=mypassword http://192.168.1.81/svn/fjhbjsb/k8s-pipeline-test --non-interactive 
          """
      }
    }

    stage('mvn-package') {
      container('jnlp') {
          sh """
          mkdir -p /var/jenkins_home && cd k8s-pipeline-test && mvn clean package && cp -rpf target/*.war /tmp/
          """
      }
    }

    stage('restart') {
      container('jnlp') {
          sh """
          pod_name=`kubectl  get pods -l name=jcsj-dev -o name | cut -d"/" -f2`
          kubectl delete pod \$pod_name
          """
      }
    }

    }
}

三、腳本釋義
podTemplate節指定建立pod的模板和環境docker

Name：爲pod的名稱前綴
Cloud：爲構建pod的雲環境，須要和前面新建的雲環境名稱同樣
Namespace：建立pod所在的namespace
Label: 建立pod對應的標籤
serviceAccount：pod使用sa，這裏使用default, 因此前文建立jenkins master deployment的時候多建立了一個default-role,這樣自動建立出來的的jenkins-slave具備對應的api權限shell

containerTemplate節api

指定建立容器的模板，當雲環境裏面配置pod模板後，容器模板以雲環境的配置爲準。tomcat

Volumes節安全

配置jenkins-slave pod掛載的卷，當雲環境裏面配置pod模板後，掛載卷以雲環境配置爲準app

stage節爲具體的pipeline步驟webapp

這裏第一步進行svn代碼遷出；
第二步進行編譯，並把包傳到共享捲上面；
第三步對pod進行刪除，由於應用採用deployment方式部署，因此能夠實現從新建立pod，達到更新的效果。

2、構建準備

在開始構建以前，有必要介紹一下jcsj-dev的環境，這個環境是須要提早部署好的。部署文件以下

# cat deploy.yml 
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jcsj-dev
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: jcsj-dev
    spec:
      containers:
      - name: jcsj-dev
        image: tomcat:latest
        imagePullPolicy: IfNotPresent
        ports:
        - name: web
          containerPort: 8080
        volumeMounts:
        - mountPath: /usr/local/tomcat/webapps
          name: tomcat-jcsj-data
      volumes:
      - name: tomcat-jcsj-data
        persistentVolumeClaim:
          claimName: tomcat-jcsj-data
---
apiVersion: v1
kind: Service
metadata:
  name: jcsj-dev
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 8453
  selector:
name: jcsj-dev

# cat pv.yaml 
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: tomcat-jcsj-data
spec:
  capacity:
    storage: 5Gi 
  accessModes:
  - ReadWriteMany 
  nfs: 
    path: /home/tomcat_jcsj
    server: 192.168.115.6
  persistentVolumeReclaimPolicy: Recycle 

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: tomcat-jcsj-data
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

3、更新項目代碼

更新index.jsp並提交

4、開始構建

點擊「當即構建」，構建過程當中能夠經過控制檯輸出看具體的日誌信息

構建過程當中觀察jenkins master的日誌輸出：

# kubectl logs -f jenkins-master-588b89c75f-ztvgm 
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Aug 13, 2018 10:14:22 AM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)
Aug 13, 2018 10:14:22 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: jenkins-slave-gjjbh in namespace default
Aug 13, 2018 10:14:22 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for Pod to be scheduled (0/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:28 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Container is waiting jenkins-slave-gjjbh [jnlp]: ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={})
Aug 13, 2018 10:14:28 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for Pod to be scheduled (1/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:34 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (1/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:35 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (2/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:36 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (3/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:37 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (4/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:38 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (5/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:39 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (6/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:40 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (7/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:41 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (8/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:42 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #1 from /172.30.66.4:47408
Aug 13, 2018 10:14:42 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (9/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:43 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (10/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:44 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (11/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:45 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (12/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:46 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (13/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:47 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (14/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:48 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (15/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:49 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (16/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:50 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (17/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:51 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (18/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:52 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (19/100): jenkins-slave-gjjbh
Aug 13, 2018 10:14:53 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (20/100): jenkins-slave-gjjbh
Aug 13, 2018 10:15:12 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1 doLaunch
INFO: Created process inside pod: [jenkins-slave-gjjbh], container: [jnlp] with pid:[-1]
Aug 13, 2018 10:15:21 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1 doLaunch
INFO: Created process inside pod: [jenkins-slave-gjjbh], container: [jnlp] with pid:[-1]
Aug 13, 2018 10:16:04 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1 doLaunch
INFO: Created process inside pod: [jenkins-slave-gjjbh], container: [jnlp] with pid:[-1]
Aug 13, 2018 10:16:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent jenkins-slave-gjjbh
Aug 13, 2018 10:16:06 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: Computer.threadPoolForRemoting [#8] for jenkins-slave-gjjbh terminated
java.nio.channels.ClosedChannelException
        at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
        at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
        at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
        at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
        at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
        at hudson.remoting.Channel.close(Channel.java:1295)
        at hudson.remoting.Channel.close(Channel.java:1263)
        at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
        at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
        at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Aug 13, 2018 10:16:06 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.PodTemplateStepExecution$PodTemplateCallback finished
INFO: Removing pod template and deleting pod jenkins-slave-7gnwx from cloud kubernetes
Aug 13, 2018 10:16:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminated Kubernetes instance for agent default/jenkins-slave-gjjbh
Aug 13, 2018 10:16:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer jenkins-slave-gjjbh
Terminated Kubernetes instance for agent default/jenkins-slave-gjjbh
Aug 13, 2018 10:16:06 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.PodTemplateStepExecution$PodTemplateCallback finished
WARNING: Failed to delete pod for agent default/jenkins-slave-7gnwx: not found
Aug 13, 2018 10:16:07 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: k8s-pipeline-test #1 completed: SUCCESS

5、驗證結果

6、後記

本文僅介紹的pipeline方式構建項目的過程，實際應用中還須要解決若干問題，例如：

一、svn checkout步驟須要明文配置用戶名和密碼，存在安全性問題
二、每次執行構建都須要全量遷出代碼，效率不高
三、要更新的應用須要提早部署好，不能適配首次編譯部署場景
四、對共享卷存在依賴，解耦問題須要解決

7、續

研究了pipeline的語法結合shell腳本，基本能夠解決上述問題，如下是pipeline代碼

podTemplate(name: 'jenkins-slave', cloud: 'kubernetes',
  namespace: 'default', label: 'jenkins-slave') 
  {
  node('jenkins-slave') {
    stage('svn-checkout') {
      container('jnlp') {
         checkout([$class: 'SubversionSCM', 
         additionalCredentials: [], 
         excludedCommitMessages: '', 
         excludedRegions: '', 
         excludedRevprop: '', 
         excludedUsers: '', 
         filterChangelog: false, 
         ignoreDirPropChanges: false, 
         includedRegions: '', 
         locations: [[cancelProcessOnExternalsFail: true, 
         credentialsId: 'ef21f2d3-cb39-468a-aa19-9175d96a365f', 
         depthOption: 'infinity', 
         ignoreExternalsOption: true, 
         local: '.', 
         remote: 'http://192.168.1.81/svn/fjhbjsb/k8s-pipeline-test']], 
         quietOperation: true, workspaceUpdater: [$class: 'UpdateUpdater']])
      }
    }

    stage('mvn-package') {
      container('jnlp') {
          sh """
          mvn clean package 
          """
      }
    }

    stage('build-docker-image') {
      container('jnlp') {
        sh '''
        IMAGE_NAME="harbor.59iedu.com/fjhb/jcsj:v$BUILD_NUMBER"   
        CONATINER_NAME="jcsj-dev" 
        echo "192.168.1.227 harbor.59iedu.com" >> /etc/hosts 
        sed -i  "s/BUILD_NUMBER/$BUILD_NUMBER/g" tomcat.yaml
        sed -i  "s/BUILD_NUMBER=1/BUILD_NUMBER=$BUILD_NUMBER/g" rolling-update.sh 
        docker login harbor.59iedu.com -u admin -p Harbor12345 
        docker build -t $IMAGE_NAME . 
        docker push $IMAGE_NAME
        '''

      }
    }

    stage('apply update') {
      container('jnlp') {
          sh '''
          NAMESPACE=default
          DEPLOYMENT=tomcat-jcsj
          REGISTRY=harbor.59iedu.com/fjhb/jcsj:v$BUILD_NUMBER
          FILE=tomcat.yaml       
          STATUS=$(kubectl get deployment -l k8s-app=$DEPLOYMENT -o name -n $NAMESPACE)
          if [ ! $STATUS ];then
          kubectl create -f $FILE
          else
          kubectl set image deployment/$DEPLOYMENT $DEPLOYMENT=$REGISTRY --namespace=$NAMESPACE
          fi    
          '''
      }
    }
    }
}

其中，svn-checkout節代碼經過「流水線語法」工具生成，同時須要給default sa用戶相應的權限,不然沒法實現apply update節配置

# kubectl create clusterrolebinding k8s-admin --clusterrole=cluster-admin  --user=system:serviceaccount:default:default

後續將介紹maven項目的構建和自動部署！

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。