Jenkins kubernetes-plugin 使用 Gradle 構建時 Permission denied 問題

問題描述

在使用 Jenkins 的 kubernetes-plugin 插件時發現使用 Gradle 鏡像 構建項目時出現 Permission denied 錯誤，具體錯誤以下：

[test] Running shell script
sh: 1: cannot create /home/jenkins/workspace/test@tmp/durable-b7fece0a/jenkins-log.txt: Permission denied
sh: 1: cannot create /home/jenkins/workspace/test@tmp/durable-b7fece0a/jenkins-result.txt.tmp: Permission denied
sh: 1: ps: not found
mv: cannot stat '/home/jenkins/workspace/test@tmp/durable-b7fece0a/jenkins-result.txt.tmp': No such file or directory

問題排查

既然出現 Permission denied 確定要從權限入手了，看錯誤信息是在工做目錄發生的錯誤，由於 kubernetes-plugin 這個插件會將工做目錄掛載出去，以保證全部容器都能訪問，因此可能就是就是各個容器的權限不統一形成的，下邊驗證下這個猜測。

驗證

個人 Jenkinsfile 以下：

#!/usr/bin/env groovy
pipeline {
  agent {
    kubernetes {
        label "mypod-${UUID.randomUUID().toString()}"
        yaml """
apiVersion: v1
kind: Pod
metadata:
  name: jenkins-agents
spec:
  containers:
  - name: gradle
    image: gradle:4.9-jdk8-alpine
    command:
    - cat
    tty: true
    volumeMounts:
    - mountPath: /home/gradle/.gradle/caches
      name: gradle-caches
      readOnly: false
  volumes:
  - name: gradle-caches
    persistentVolumeClaim:
      claimName: gradle-caches
"""
    }
  }
    stages {
        stage('編譯測試') {
            stages {
                stage('拉取代碼') {
                    steps {
                        git credentialsId: 'gitlab-jenkins', url: 'xxxxxxxx'
                    }
                }
                stage('Gradle 編譯打包') {
                    steps {
                        container('gradle') {
                            sh 'gradle build'
                        }
                    }
                }
            }
        }
    }
}

一共只用到兩個容器：除了默認的 jnlp 就只有 gradle 了，所用鏡像分別是 jenkins/jnlp-slave:3.10-1 和 gradle:4.9-jdk8-alpine。

首先查看下兩個鏡像默認用戶的信息：

$ docker run --rm jenkins/jnlp-slave:3.10-1 id
uid=10000(jenkins) gid=10000(jenkins) groups=10000(jenkins)

$ docker run --rm gradle:4.9-jdk8-alpine id
uid=1000(gradle) gid=1000(gradle) groups=1000(gradle)

能夠看到 jnpl 的 jenkins 用的 uid 和 gid 竟然是 10000，而 gradle 的 gradle 用戶的 uid 和 gid 爲 1000。

爲了弄清楚到底怎麼回事，找到 Dockerfile 一探究竟，首先找到 jenkins/jnlp-slave

[jenkins/jnlp-slave]

FROM jenkins/slave:3.23-1
MAINTAINER Oleg Nenashev <o.v.nenashev@gmail.com>
LABEL Description="This is a base image, which allows connecting Jenkins agents via JNLP protocols" Vendor="Jenkins project" Version="3.23"

COPY jenkins-slave /usr/local/bin/jenkins-slave

ENTRYPOINT ["jenkins-slave"]

只是個空殼，找到 jenkins/slave

FROM openjdk:8-jdk
MAINTAINER Oleg Nenashev <o.v.nenashev@gmail.com>

ARG user=jenkins
ARG group=jenkins
ARG uid=10000
ARG gid=10000

ENV HOME /home/${user}
RUN groupadd -g ${gid} ${group}
RUN useradd -c "Jenkins user" -d $HOME -u ${uid} -g ${gid} -m ${user}
LABEL Description="This is a base image, which provides the Jenkins agent executable (slave.jar)" Vendor="Jenkins project" Version="3.23"

ARG VERSION=3.23
ARG AGENT_WORKDIR=/home/${user}/agent

RUN curl --create-dirs -sSLo /usr/share/jenkins/slave.jar https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/${VERSION}/remoting-${VERSION}.jar \
  && chmod 755 /usr/share/jenkins \
  && chmod 644 /usr/share/jenkins/slave.jar

USER ${user}
ENV AGENT_WORKDIR=${AGENT_WORKDIR}
RUN mkdir /home/${user}/.jenkins && mkdir -p ${AGENT_WORKDIR}

VOLUME /home/${user}/.jenkins
VOLUME ${AGENT_WORKDIR}
WORKDIR /home/${user}

能夠看到 Dockerfile 裏指定的 uid 和 gid 確實是 10000，至於爲何是 10000 沒有提到。

解決方案

既然已經找到了問題所在，那麼只要把 gradle 鏡像的 uid 和 gid 也改成 10000 應該就能夠了，下面就試一下。

首先找到 gradle 的原始 Dockerfile 並修改以下：

FROM openjdk:8-jdk-alpine

CMD ["gradle"]

ENV GRADLE_HOME /opt/gradle
ENV GRADLE_VERSION 4.9

ARG GRADLE_DOWNLOAD_SHA256=e66e69dce8173dd2004b39ba93586a184628bc6c28461bc771d6835f7f9b0d28
RUN set -o errexit -o nounset \
    && echo "Installing build dependencies" \
    && apk add --no-cache --virtual .build-deps \
        ca-certificates \
        openssl \
        unzip \
    \
    && echo "Downloading Gradle" \
    && wget -O gradle.zip "https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip" \
    \
    && echo "Checking download hash" \
    && echo "${GRADLE_DOWNLOAD_SHA256} *gradle.zip" | sha256sum -c - \
    \
    && echo "Installing Gradle" \
    && unzip gradle.zip \
    && rm gradle.zip \
    && mkdir /opt \
    && mv "gradle-${GRADLE_VERSION}" "${GRADLE_HOME}/" \
    && ln -s "${GRADLE_HOME}/bin/gradle" /usr/bin/gradle \
    \
    && apk del .build-deps \
    \
    && echo "Adding gradle user and group" \
    && addgroup -S -g 10000 gradle \
    && adduser -D -S -G gradle -u 10000 -s /bin/ash gradle \
    && mkdir /home/gradle/.gradle \
    && chown -R gradle:gradle /home/gradle \
    \
    && echo "Symlinking root Gradle cache to gradle Gradle cache" \
    && ln -s /home/gradle/.gradle /root/.gradle

# Create Gradle volume
USER gradle
VOLUME "/home/gradle/.gradle"
WORKDIR /home/gradle

RUN set -o errexit -o nounset \
    && echo "Testing Gradle installation" \
    && gradle --version

只是把 uid 和 gid 由 1000 改成 10000

結果

鏡像構建完成後使用新鏡像從新 build 項目，問題解決！