15分鐘在阿里雲Kubernetes服務上快速創建Jenkins X Platform並運用GitOps管理應用發佈

本文主要介紹如何在阿里雲容器服務Kubernetes上快速安裝部署Jenkins X Platform並結合demo實踐演示GitOps的操做流程。

注意：
本文中使用的jx工具、cloud-environments等作過改造用以適配阿里雲Kubernetes容器服務，並未在自建Kubernetes集羣中作過驗證。

先決條件：
首先，須要在 阿里雲容器服務控制檯 建立一個Kubernetes集羣，本次實踐使用的環境信息以下：
master1 192.168.0.119
master2 192.168.0.120
master3 192.168.0.121
worker1 192.168.0.122
jx-node 192.168.0.123

1、 部署Jenkins X Platform並建立Staging Production Env

1. kubectl 、helm、git和jx的安裝以及初始化

$ scp 192.168.0.119:/usr/bin/kubectl /usr/bin/kubectl
$ scp 192.168.0.119:/usr/local/bin/helm /usr/local/bin/helm
$ mkdir -p $HOME/.kube
將容器服務控制檯上顯示的kubeconfig文件內容複製到計算機 $HOME/.kube/config

初始化Helm Client：

$ helm init --client-only --stable-repo-url https://aliacs-app-catalog.os...
$ helm repo update
安裝git

$ yum install git -y
安裝jx

$ wget http://acs-jenkins-x.oss-cn-b... && mv jx /usr/local/bin/ && chmod +x /usr/local/bin/jx
驗證jx：(注意不要選擇升級，jx作了定製化修改，目前版本1.3.699)

$ jx version
Error loading team settings. the server could not find the requested resource (get environments.jenkins.io)
Error loading team settings. the server could not find the requested resource (get environments.jenkins.io)
Failed to get git version: failed to run 'git version' command in directory '', output: '': exec: "git": executable file not found in $PATH
NAME VERSION
jx 1.3.699
Kubernetes cluster v1.11.5
kubectl v1.11.5
helm client v2.11.0+g2e55dbe
helm server v2.11.0+g2e55dbe
Operating System "CentOS Linux release 7.4.1708 (Core) "

A new jx version is available: 1.3.736
? Would you like to upgrade to the new jx version? No

1. 安裝jenkins-x-platform

$ mkdir -p ~/.jx
$ cd ~/.jx
$ jx install --provider=kubernetes \

--domain aliyunk8s-bj.com \
         --ingress-deployment=nginx-ingress-controller \
         --ingress-service=nginx-ingress-lb  \
         --cloud-environment-repo=https://github.com/AliyunContainerService/cloud-environments.git \
         --no-tiller=true \
         --no-default-environments=true

交互及日誌輸出以下：

Namespace jx created
Using helmBinary helm with feature flag: none
Context "kubernetes-admin-ccf3ebc497d9e4083aab8c89bb1b217bc" modified.
Storing the kubernetes provider kubernetes in the TeamSettings
Enabling helm template mode in the TeamSettings
? Please enter the name you wish to use with git: jenkins-x-bot
? Please enter the email address you wish to use with git: haoshuwei24@gmail.com
Git configured for user: jenkins-x-bot and email haoshuwei24@gmail.com
Trying to create ClusterRoleBinding kubernetes-admin-cluster-admin-binding for role: cluster-admin for user kubernetes-admin
clusterrolebindings.rbac.authorization.k8s.io "kubernetes-admin-cluster-admin-binding" not found
Created ClusterRoleBinding kubernetes-admin-cluster-admin-binding
Using helm2
Skipping tiller
helm installed and configured
existing ingress controller found, no need to install a new one
Waiting for external loadbalancer to be created and update the nginx-ingress-controller service in kube-system namespace
External loadbalancer created
Waiting to find the external host name of the ingress controller Service in namespace kube-system with name nginx-ingress-lb
If you are installing Jenkins X on premise you may want to use the '--on-premise' flag or specify the '--external-ip' flags. See: https://jenkins-x.io/getting-...
You can now configure your wildcard DNS aliyunk8s-bj.com to point to 39.97.21.40
nginx ingress controller installed and configured
Lets set up a Git user name and API token to be able to perform CI/CD

Creating a local Git user for GitHub server
? GitHub user name: jenkins-x-bot
To be able to create a repository on GitHub we need an API Token
Please click this URL https://github.com/settings/t...:user,read:org,user:email,write:repo_hook,delete_repo

Then COPY the token and enter in into the form below:

? API Token:
Select the CI/CD pipelines Git server and user
? Do you wish to use GitHub as the pipelines Git server: Yes
? Do you wish to use jenkins-x-bot as the pipelines Git user for GitHub server: Yes
Setting the pipelines Git server https://github.com and user name jenkins-x-bot.
Saving the Git authentication configurationCurrent configuration dir: /root/.jx
options.Flags.CloudEnvRepository: https://github.com/AliyunCont...
options.Flags.LocalCloudEnvironment: false
Cloning the Jenkins X cloud environments repo to /root/.jx/cloud-environments
Enumerating objects: 1317, done.
Counting objects: 100% (1317/1317), done.
Compressing objects: 100% (618/618), done.
Total 1317 (delta 673), reused 1317 (delta 673), pack-reused 0
No default password set, generating a random one
Generated helm values /root/.jx/extraValues.yaml
Creating Secret jx-install-config in namespace jx
Installing Jenkins X platform helm chart from: /root/.jx/cloud-environments/env-kubernetes
? Select Jenkins installation type: Static Master Jenkins
? Pick workload build pack: Kubernetes Workloads: Automated CI+CD with GitOps Promotion
Setting the team build pack to kubernetes-workloads repo: https://github.com/AliyunCont... ref: master
Installing jx into namespace jx
Adding values file /root/.jx/cloud-environments/env-kubernetes/myvalues.yaml
Adding values file /root/.jx/adminSecrets.yaml
Adding values file /root/.jx/extraValues.yaml
Adding values file /root/.jx/cloud-environments/env-kubernetes/secrets.yaml
Fetched chart jenkins-x/jenkins-x-platform to dir /tmp/helm-template-workdir-860490114/jenkins-x/chartFiles/jenkins-x-platform
Generating Chart Template 'template --name jenkins-x --namespace jx /tmp/helm-template-workdir-860490114/jenkins-x/chartFiles/jenkins-x-platform --output-dir /tmp/helm-template-workdir-860490114/jenkins-x/output --debug --values /root/.jx/cloud-environments/env-kubernetes/myvalues.yaml --values /root/.jx/adminSecrets.yaml --values /root/.jx/extraValues.yaml --values /root/.jx/cloud-environments/env-kubernetes/secrets.yaml'
Applying generated chart jenkins-x/jenkins-x-platform YAML via kubectl in dir: /tmp/helm-template-workdir-860490114/jenkins-x/output
deployment.extensions/jenkins-x-chartmuseum created
persistentvolumeclaim/jenkins-x-chartmuseum created
secret/jenkins-x-chartmuseum created
service/jenkins-x-chartmuseum created
role.rbac.authorization.k8s.io/cleanup created
rolebinding.rbac.authorization.k8s.io/cleanup created
serviceaccount/cleanup created
clusterrole.rbac.authorization.k8s.io/controllercommitstatus-jx created
clusterrolebinding.rbac.authorization.k8s.io/controllercommitstatus-jx created
deployment.apps/jenkins-x-controllercommitstatus created
role.rbac.authorization.k8s.io/controllercommitstatus created
rolebinding.rbac.authorization.k8s.io/controllercommitstatus created
serviceaccount/jenkins-x-controllercommitstatus created
clusterrole.rbac.authorization.k8s.io/controllerrole-jx created
clusterrolebinding.rbac.authorization.k8s.io/controllerrole-jx created
deployment.apps/jenkins-x-controllerrole created
role.rbac.authorization.k8s.io/controllerrole created
rolebinding.rbac.authorization.k8s.io/controllerrole created
serviceaccount/jenkins-x-controllerrole created
clusterrole.rbac.authorization.k8s.io/controllerteam-jx created
clusterrolebinding.rbac.authorization.k8s.io/controllerteam-jx created
deployment.apps/jenkins-x-controllerteam created
role.rbac.authorization.k8s.io/controllerteam created
rolebinding.rbac.authorization.k8s.io/controllerteam created
serviceaccount/jenkins-x-controllerteam created
clusterrole.rbac.authorization.k8s.io/controllerworkflow-jx created
clusterrolebinding.rbac.authorization.k8s.io/controllerworkflow-jx created
deployment.apps/jenkins-x-controllerworkflow created
role.rbac.authorization.k8s.io/controllerworkflow created
rolebinding.rbac.authorization.k8s.io/controllerworkflow created
serviceaccount/jenkins-x-controllerworkflow created
configmap/exposecontroller created
role.rbac.authorization.k8s.io/expose created
rolebinding.rbac.authorization.k8s.io/expose created
serviceaccount/expose created
clusterrole.rbac.authorization.k8s.io/gcactivities-jx created
clusterrolebinding.rbac.authorization.k8s.io/gcactivities-jx created
cronjob.batch/jenkins-x-gcactivities created
role.rbac.authorization.k8s.io/gcactivities created
rolebinding.rbac.authorization.k8s.io/gcactivities created
serviceaccount/jenkins-x-gcactivities created
cronjob.batch/jenkins-x-gcpods created
role.rbac.authorization.k8s.io/gcpods created
rolebinding.rbac.authorization.k8s.io/gcpods created
serviceaccount/jenkins-x-gcpods created
clusterrole.rbac.authorization.k8s.io/gcpreviews-jx created
clusterrolebinding.rbac.authorization.k8s.io/gcpreviews-jx created
cronjob.batch/jenkins-x-gcpreviews created
role.rbac.authorization.k8s.io/gcpreviews created
rolebinding.rbac.authorization.k8s.io/gcpreviews created
serviceaccount/jenkins-x-gcpreviews created
deployment.extensions/jenkins-x-heapster created
service/heapster created
configmap/jenkins created
persistentvolumeclaim/jenkins created
service/jenkins-agent created
deployment.extensions/jenkins created
service/jenkins created
configmap/jenkins-x-git-kinds created
clusterrolebinding.rbac.authorization.k8s.io/jenkins-jx-role-binding created
secret/jenkins created
serviceaccount/jenkins created
configmap/jenkins-tests created
configmap/nexus created
deployment.extensions/jenkins-x-nexus created
persistentvolumeclaim/jenkins-x-nexus created
secret/nexus created
service/nexus created
role.rbac.authorization.k8s.io/committer created
clusterrolebinding.rbac.authorization.k8s.io/jenkins-x-team-controller created
configmap/jenkins-x-team-controller created
secret/jenkins-docker-cfg created
configmap/jenkins-x-devpod-config created
configmap/jenkins-x-docker-registry created
configmap/jenkins-x-extensions created
configmap/jenkins-x-pod-templates created
secret/jx-basic-auth created
role.rbac.authorization.k8s.io/jx-view created
secret/jenkins-maven-settings created
secret/jenkins-npm-token created
role.rbac.authorization.k8s.io/owner created
secret/jenkins-release-gpg created
secret/jenkins-ssh-config created
role.rbac.authorization.k8s.io/viewer created
Applying Helm hook post-upgrade YAML via kubectl in file: /tmp/helm-template-workdir-860490114/jenkins-x/helmHooks/jenkins-x-platform/charts/expose/templates/job.yaml
job.batch/expose created
Waiting for helm post-upgrade hook Job expose to complete before removing it
若是是首次部署jenkins x platfrom的話，此處需等待幾分鐘進行鏡像的拉取和容器的啓動：

$ kubectl -n jx get po
NAME READY STATUS RESTARTS AGE
expose-2bhmv 0/1 ContainerCreating 0 1m
jenkins-568884c766-sxc7d 0/1 Init:0/1 0 1m
jenkins-x-chartmuseum-6cf566bfb-jm7cv 0/1 ContainerCreating 0 1m
jenkins-x-controllercommitstatus-6f57d857d8-kzllt 1/1 Running 0 1m
jenkins-x-controllerrole-57d864c96f-4lrrk 1/1 Running 0 1m
jenkins-x-controllerteam-c48fc44f-tsrkh 0/1 ContainerCreating 0 1m
jenkins-x-controllerworkflow-c758649d6-2kpfj 0/1 ContainerCreating 0 1m
jenkins-x-heapster-7fbdb867d9-wt6sl 0/2 ContainerCreating 0 1m
jenkins-x-nexus-5d5455cfd7-4xsck 0/1 ContainerCreating 0 1m
接下來的日誌交互及輸出以下：

^@^@^@Deleting helm hook sources from file: /tmp/helm-template-workdir-860490114/jenkins-x/helmHooks/jenkins-x-platform/charts/expose/templates/job.yaml
job.batch "expose" deleted
Removing Kubernetes resources from older releases using selector: jenkins.io/chart-release=jenkins-x,jenkins.io/version!=0.0.3193
waiting for install to be ready, if this is the first time then it will take a while to download images
^@Jenkins X deployments ready in namespace jx

********************************************************

     NOTE: Your admin password is: qub6n#mKkh0oN!S59nPp

********************************************************

Getting Jenkins API Token
Using url http://jenkins.jx.aliyunk8s-b...
Unable to automatically find API token with chromedp using URL http://jenkins.jx.aliyunk8s-b...
Error: creating the chrome client: fork/exec /usr/bin/google-chrome: no such file or directory
Please go to http://jenkins.jx.aliyunk8s-b... and click Show API Token to get your API Token
Then COPY the token and enter in into the form below:

? API Token:
此處若是沒有DNS service解析域名jenkins.jx.aliyunk8s-bj.com的話， 須要先在jx-node上手動綁定hosts：

$ kubectl -n jx get ing
NAME HOSTS ADDRESS PORTS AGE
chartmuseum chartmuseum.jx.aliyunk8s-bj.com xx.xx.xx.xx 80 5m
jenkins jenkins.jx.aliyunk8s-bj.com xx.xx.xx.xx 80 5m
nexus nexus.jx.aliyunk8s-bj.com xx.xx.xx.xx 80 5m
$ echo "xx.xx.xx.xx jenkins.jx.aliyunk8s-bj.com" >> /etc/hosts
根據日誌提示生成jenkins api token：
image
完成安裝：

Created user admin API Token for Jenkins server jenkins.jx.aliyunk8s-bj.com at http://jenkins.jx.aliyunk8s-b...
Updating Jenkins with new external URL details http://jenkins.jx.aliyunk8s-b...

Jenkins X installation completed successfully

********************************************************

     NOTE: Your admin password is: qub6n#mKkh0oN!S59nPp

********************************************************

Your Kubernetes context is now set to the namespace: jx
To switch back to your original namespace use: jx namespace default
For help on switching contexts see: https://jenkins-x.io/developi...

To import existing projects into Jenkins: jx import
To create a new Spring Boot microservice: jx create spring -d web -d actuator
To create a new microservice from a quickstart: jx create quickstart

1. 建立staging env

$ jx create env -n staging -l Staging --namespace staging --fork-git-repo='https://github.com/haoshuwei/...' --domain=aliyunk8s-bj.com --promotion=Auto --prefix='bj'

....
Creating GitHub webhook for haoshuwei/environment-bj-staging for url http://jenkins.jx.aliyunk8s-b...
因爲咱們默認使用Ingress暴露服務， 因此須要修改environment-bj-staging的webhook：

$ kubectl -n jx get svc |grep LoadBalancer
kubectl -n jx get svc |grep LoadBalancer
jenkins LoadBalancer 172.19.11.179 xx.xx.xx.xx 8080:30456/TCP 13h
image

jenkins會自動建立environment-bj-staging job並進行掃描和構建：
image

1. 建立production env:

$ jx create env -n production -l Production --namespace production --fork-git-repo='https://github.com/haoshuwei/...' --domain=aliyunk8s-bj.com --promotion=Manual --prefix='bj'

...
Creating GitHub webhook for haoshuwei/environment-bj-production for url http://jenkins.jx.aliyunk8s-b...
修改environment-hz-production的webhook。

jenkins會自動建立environment-bj-production job並進行掃描和構建：
image

1. 配置訪問阿里雲容器鏡像服務的docker registry secret：

$ kubectl -n jx delete secrets jenkins-docker-cfg
secret "jenkins-docker-cfg" deleted
$ docker login -u xxx -p xxx registry.cn-beijing.aliyuncs.com
$ kubectl create secret generic jenkins-docker-cfg -n jx --from-file=/root/.docker/config.json
secret/jenkins-docker-cfg created
2、 運用GitOps管理應用發佈

1. 安裝openjdk

$ yum install -y java-1.8.0-openjdk

1. 建立一個Spring示例應用

$ jx create spring -d web -d actuator
? Language: java
? Group: com.example
? Artifact: jenkins-x-demo
Created Spring Boot project at /root/.jx/jenkins-x-demo
No username defined for the current Git server!
? Do you wish to use jenkins-x-bot as the Git user name: Yes
The directory /root/.jx/jenkins-x-demo is not yet using git
? Would you like to initialise git now? Yes
? Commit message: Initial import

Git repository created
selected pack: /root/.jx/draft/packs/github.com/AliyunContainerService/jenkins-x-kubernetes/packs/maven
? Which organisation do you want to use? jenkins-x-bot
replacing placeholders in directory /root/.jx/jenkins-x-demo
app name: jenkins-x-demo, git server: github.com, org: jenkins-x-bot, Docker registry org: jenkins-x-bot
skipping directory "/root/.jx/jenkins-x-demo/.git"
Using Git provider GitHub at https://github.com

About to create repository jenkins-x-demo on server https://github.com with user jenkins-x-bot
? Enter the new repository name: jenkins-x-demo

Creating repository jenkins-x-bot/jenkins-x-demo
Pushed Git repository to https://github.com/haoshuwei/...

Created Jenkins Project: http://jenkins.jx.aliyunk8s-b...

Watch pipeline activity via: jx get activity -f jenkins-x-demo -w
Browse the pipeline log via: jx get build logs haoshuwei/jenkins-x-demo/master
Open the Jenkins console via jx console
You can list the pipelines via: jx get pipelines
When the pipeline is complete: jx get applications

For more help on available commands see: https://jenkins-x.io/developi...

Note that your first pipeline may take a few minutes to start while the necessary images get downloaded!

Creating GitHub webhook for haoshuwei/jenkins-x-demo for url http://jenkins.jx.aliyunk8s-b...
修改 jenkins-x-demo的webhook。

jenkins-x-demo項目成功建立後，jenkins job jenkins-x-demo會自動掃描並構建jenkins-x-demo-0.0.1.tgz發佈到chartmuseum並自動promote到staging環境：
image
查看environment-bj-staging項目的PR：
image
PR的Check經過後會自動merge：
image
主要修改內容爲：
image
而後environment-bj-staging項目自動構建並部署jenkins-x-demo應用到Staging Env。

1. 訪問Staging環境中的jenkins-x-demo應用服務

$ jx get apps
APPLICATION PRODUCTION PODS URL STAGING PODS URL
jx-jenkins-x-demo 0.0.1 1/1 http://jenkins-x-demo.staging...
訪問http://jenkins-x-demo.staging...：
image

1. 開發jenkins-x-demo應用併發布新版本到Staging環境

步驟3中， 咱們訪問了一個並無準備好進入生產環境的jenkins-x-demo應用， 下面咱們添加一個新的feature：

$ cd jenkins-x-demo
$ git checkout -b feature/add-index
新建src/main/resources/static/index.html並寫入以下內容：

<html>
<head>

<title> Jenkins X Spring Demo </title>

</head>
<body bgcolor=white>

<table border="0" cellpadding="10">
  <tr>
    <td>
      <img height="300" width="300" src="https://github.com/jenkins-x/jenkins-x-website/raw/e5aae999166a67d6220aa469eed1f23e0996c1f7/images/logo.png">
    </td>
    <td>
      <h1>Jenkins X Spring Demo</h1>
    </td>
  </tr>
</table>

</body>

</html>
提交和推送後咱們在GitHub上建立一個feature/add-index到master分支的PR：
image
jenkins會自動對PR進行構建和部署，這個過程當中jenkins-x-demo的部署並非部署到Staging環境，而是會部署爲Preview環境提供預覽：
image
image
預覽jenkins-x-demo應用http://jenkins-x-demo.jx-haos...：
image

1. 發佈新版本jenkins-x-demo應用到Staging環境

確認提交的PR沒問題後，便可merge到master分支：
image

jenkins會從新構建jenkins-x-demo v0.0.2併發布到Staging環境:

$ jx get apps
APPLICATION PRODUCTION PODS URL STAGING PODS URL
jx-jenkins-x-demo 0.0.2 1/1 http://jenkins-x-demo.staging...

1. jenkins-x-demo應用推送到Production環境

一旦應用經過測試驗證並準備好進入生產後， 咱們就能夠手動發佈應用到Production環境：

$ jx get apps
APPLICATION PRODUCTION PODS URL STAGING PODS URL
jx-jenkins-x-demo 0.0.2 1/1 http://jenkins-x-demo.product... 0.0.2 1/1 http://jenkins-x-demo.staging...
訪問Production環境中的jenkins-x-demo：
image

3、 Production環境中應用的回滾
回滾到版本v0.x.x

$ jx promote --version 0.x.x --env production --timeout 20m
4、 刪除或重裝Jenkins X Platform
$ kubectl config current-context
$ jx uninstall
$ cd ~/.jx
$ rm -rf *
5、 結束語
目前Jenkins X Platform還在不斷完善和改進中， 好比支持多集羣等問題https://github.com/jenkins-x/...， 咱們也會持續優化Jenkins X 在阿里雲Kubernetes容器服務上的最佳實踐並更新博客。