jenkins X實踐系列（3） —— jenkins X 安裝拾遺

jx是雲原生CICD，devops的一個最佳實踐之一，目前在快速的發展成熟中。最近調研了JX，這裏爲第3篇，介紹下如何安裝jenkins x。html

前置條件

安裝K8S
安裝ceph集羣（jx須要storage class建立pv）
申請一個域名（可選），能夠修改hosts實現
helm
git私服

安裝ceph集羣

這裏的服務器爲centos 7。mysql

使用官方的ceph-deploy安裝便可，先安裝ceph-deploy，而後每臺機器安裝運行環境linux

pip install ceph-deploy
 export CEPH_DEPLOY_REPO_URL=http://mirrors.ustc.edu.cn/ceph/rpm-jewel/el7
 export CEPH_DEPLOY_GPG_URL=http://mirrors.ustc.edu.cn/ceph/keys/release.asc
 ceph-deploy install docker86-156 docker86-155 docker86-154

而後安裝集羣nginx

ceph-deploy new docker86-156 docker86-155 docker86-154

修改配置文件git

cat <<EOF >>ceph.conf
#osd_journal_size = 10000
public network = 192.168.86.0/24
osd_pool_default_size = 2
osd_pool_default_min_size = 1
osd_crush_chooseleaf_type = 1
osd_crush_update_on_start = true
max_open_files = 131072
osd pool default pg num = 128
osd pool default pgp num = 128
mon_pg_warn_max_per_osd = 0
mon clock drift allowed = 2
mon clock drift warn backoff = 30
mon_pg_warn_max_per_osd = 300
EOF

分發配置文件：github

ceph-deploy --overwrite-conf config push docker86-155 docker86-154 docker86-156

安裝服務web

ceph-deploy mon create-initial
ceph-deploy admin docker86-156 docker86-155 docker86-154

安裝osdsql

ceph-deploy disk zap  docker86-156:sdb docker86-155:sdb docker86-154:sdb
ceph-deploy osd prepare docker86-156:sdb docker86-155:sdb docker86-154:sdb
ceph-deploy osd activate docker86-156:sdb1 docker86-154:sdb1

添加pool

ceph osd pool create k8smeta 128
ceph osd pool create k8sdata 128
ceph fs new k8s k8smeta k8sdata
ceph osd pool ls detail

K8S使用ceph

生成 Ceph secretdocker

grep key /etc/ceph/ceph.client.admin.keyring |awk '{printf "%s", $NF}'|base64

假如獲得: $SECRET==shell

在k8s建立Secret

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
  namespace: default
type: "kubernetes.io/rbd"  
data:
  key: $SECRET==
EOF

建立StorageClass

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: ceph-web
provisioner: kubernetes.io/rbd
parameters:
  monitors: 192.168.86.156,192.168.86.155,192.168.86.154
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: default
  pool: rbd
  userId: admin
  userSecretName: ceph-secret
EOF

能夠將ceph設爲默認的storage class：

kubectl patch storageclass ceph-web -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

安裝git私服gitea(可選）

若是已經有git的，或者直接使用GitHub的跳過。

建立PV：

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: cephfs-github-pv
  namespace: gitea
  labels:
    name: cephfs-github-pv
spec:
  capacity:
    storage: 200Gi
  accessModes:
    - ReadWriteMany
  cephfs:
    monitors: 
    - 192.168.86.156:6789
    path: /github
    user: admin
    secretRef:
      name: ceph-secret
    readOnly: false
  persistentVolumeReclaimPolicy: Retain
EOF

PVC

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-github-pvc
  namespace: gitea
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 200Gi
  selector:
    matchLabels:
      name: cephfs-github-pv
EOF

gitea部署：

cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: giteamysql
  namespace: gitea
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: giteamysql
    spec:
      containers:
      - image: gitea/gitea:latest
        imagePullPolicy: IfNotPresent
        name: gitea
        resources: {}
        volumeMounts:
        - name: ceph
          mountPath: /data 
      volumes:
        - name: ceph
          persistentVolumeClaim:
            claimName: cephfs-github-pvc
EOF

建立服務：

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: giteamysql-service
  namespace: gitea
  labels:
    app: charts
spec:
  ports:
    - port: 80
      targetPort: 3000 
  selector:
    app: giteamysql
  type: NodePort
EOF

建立ingress

cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    kubernetes.io/tls-acme: 'true'
  name: giteamysql-ingress
  namespace: gitea
spec:
  rules:
  - host: github.youdomain.com
    http:
      paths:
      - backend:
          serviceName: giteamysql-service
          servicePort: 80
        path: /
EOF

一切正常的話，打開github.youdomain.com，按提示進行安裝，設置管理員密碼便可。

安裝好後建立一個token， $git_access_token

域名與tls

將域名的通配符，a記錄到k8s集羣。

申請TLS證書，使用certbot

$ yum -y install yum-utils
$ yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
 $ sudo yum install certbot

而後申請

certbot certonly --manual -d *.domain.com --email youmail@domain.com

這個會要求建立一個A記錄，按提示建立便可。

一些OK的話，服務器/etc/letsencrypt/live/domain.com/ 會生成tls證書。

$ ll /etc/letsencrypt/live/iflyresearch.com/
total 4
lrwxrwxrwx. 1 root root  40 Oct 17 15:11 cert.pem -> ../../archive/iflyresearch.com/cert1.pem
drwxr-xr-x  2 root root  78 Nov 14 09:33 certs
lrwxrwxrwx. 1 root root  41 Oct 17 15:11 chain.pem -> ../../archive/iflyresearch.com/chain1.pem
lrwxrwxrwx. 1 root root  45 Oct 17 15:11 fullchain.pem -> ../../archive/iflyresearch.com/fullchain1.pem
lrwxrwxrwx. 1 root root  43 Oct 17 15:11 privkey.pem -> ../../archive/iflyresearch.com/privkey1.pem

要在k8s使用，須要建立secret：

kubectl create secret tls research-tls-secret --cert=cert.pem --key=./privkey.pem -n=kube-system

安裝helm

jx依賴helm，須要先安裝，能夠參見本系列第一篇

安裝jenkins X

先建立一個namespace：incubation

寫入 ceph-secret：

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
  namespace: incubation
type: "kubernetes.io/rbd"  
data:
  key: $SECRET==
EOF

下載jx執行文件：

wget https://github.com/jenkins-x/jx/releases/download/v1.3.380/jx-linux-amd64.tar.gz 
  tar xzv jx-linux-amd64.tar.gz  -C ~/.jx/bin
  export PATH=$PATH:~/.jx/bin
  echo 'export PATH=$PATH:~/.jx/bin' >> ~/.bashrc

而後執行安裝命令：

jx install --external-ip=192.168.86.214 --namespace='incubation'  --git-provider-url='http://github.iflyresearch.com' --git-username='jqpeng' --git-api-token=' $git_access_token' --domain='iflyresearch.com'  --provider=kubernetes

$git_access_token 替換爲你的token
external-ip 填寫k8s的虛擬ip

而後按提示，輸出jenkins等access_token等參數。

注意:

安裝過程，會依賴一些gcr.io的鏡像，請參見《google gcr.io、k8s.gcr.io 國內鏡像》解決
若是使用gitea，安裝修改下vim gitAuth.yaml，修改kind爲gitea

做者：Jadepeng 出處：jqpeng的技術記事本--http://www.cnblogs.com/xiaoqi 您的支持是對博主最大的鼓勵，感謝您的認真閱讀。本文版權歸做者全部，歡迎轉載，但未經做者贊成必須保留此段聲明，且在文章頁面明顯位置給出原文鏈接，不然保留追究法律責任的權利。