ARM64上Docker編譯問題與解決

編譯過程

在96Boards 的410C上編譯Docker（這裏使用Dokcer 1.14 Master版本，方法參考），遇到些問題，解決方法以下：

修改Docker源碼目錄下的Dockerfile.aarch64文件

• 修改 FROM aarch64/ubuntu:wily 去掉wily，由於wily版不少源都無法下載。
• Docker編譯須要GOROOT_BOOTSTRAP，在安裝包中加入： golang
• 在docker-py安裝前運行：pip install setuptools
• 須要下載go的源代碼安裝，訪問不了的設置代理，示意：curl -x 192.168.1.11:9999，把這裏的地址改爲本身的代理服務器。
• 內存不足，啓用虛存支持，參考：http://www.javashuo.com/article/p-ourgqptd-u.html
• 主存儲空間不足，把Docker放到U盤或SD卡上，參考：http://www.javashuo.com/article/p-xerainwt-hk.html

** 基於ARM64的不少容器資源：https://github.com/multiarch**

運行問題解決

由於不明緣由，最後make deb編譯失敗，但到目錄中看，bundles中dockerD和docker都已經生成好了。

• 拷貝到/usr/bin目錄中，修改/lib/systemd/system/docker.service文件。
• 但systemctl docker start運行時又報container找不到錯誤。
• 到 **hack/dockerfile/install-binaries.sh **中找到containerd和runc的相應安裝代碼，手工運行、安裝到/usr/local/bin，創建軟鏈接，docker就能夠正常啓動了。

注意：

• 因爲最近github.com強制使用了https，致使go get和git clone 原來的http網站失敗。
• 出錯顯示，網站找不到，參考這裏的方法解決。

SSL CA錯誤的解決：
運行：git config --global http.sslverify false.
或者：env GIT_SSL_NO_VERIFY=true
添加CA：http://askubuntu.com/questions/645818/how-to-install-certificates-for-command-line

安裝containerd和runc

export GOPATH="/home/linaro" 

#複製DockerD服務器到系統目錄
cp docker1.14.0-dev /usr/bin/
#修改/lib/systemd/system/docker.service中的ExecStart=/usr/bin/dockerd爲ExecStart=/usr/bin/docker1.14.0-dev

#之後使用dox來運行新的Docker客戶端。
cp  docker1.14.0 /usr/local/bin/dox
 
# Install runc
export RUNC_COMMIT=f59ba3cdd76fdc08c004f42aa915996f6f420899
git clone https://github.com/docker/runc.git "$GOPATH/src/github.com/opencontainers/runc"
cd "$GOPATH/src/github.com/opencontainers/runc"
git checkout -q "$RUNC_COMMIT"
make static BUILDTAGS="seccomp apparmor selinux" 
cp runc /usr/local/bin/docker-runc
ln -s /usr/local/bin/docker-runc /usr/local/bin/runc

# Install containerd
export CONTAINERD_COMMIT=2a5e70cbf65457815ee76b7e5dd2a01292d9eca8
git clone https://github.com/docker/containerd.git "$GOPATH/src/github.com/docker/containerd" 
cd "$GOPATH/src/github.com/docker/containerd"
git checkout -q "$CONTAINERD_COMMIT"
make static
cp bin/containerd /usr/local/bin/docker-containerd
ln -s /usr/local/bin/docker-containerd  /usr/local/bin/containerd

cp bin/containerd-shim /usr/local/bin/docker-containerd-shim
ln -s /usr/local/bin/docker-containerd-shim /usr/local/bin/containerd-shim

cp bin/ctr /usr/local/bin/docker-containerd-ctr
ln -s /usr/local/bin/docker-containerd-ctr /usr/local/bin/containerd-ctr

修改的docker.service文件

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

修改的Dockerfile.aarch64文件

下面是修改過的Dockerfile.aarch64，查看原始文件：

# This file describes the standard way to build Docker on aarch64, using docker
#
# Usage:
#
# # Assemble the full dev environment. This is slow the first time.
# docker build -t docker -f Dockerfile.aarch64 .
#
# # Mount your source in an interactive container for quick testing:
# docker run -v `pwd`:/go/src/github.com/docker/docker --privileged -i -t docker bash
#
# # Run the test suite:
# docker run --privileged docker hack/make.sh test-unit test-integration-cli test-docker-py
#
# Note: AppArmor used to mess with privileged mode, but this is no longer
# the case. Therefore, you don't have to disable it anymore.
#

FROM aarch64/ubuntu:wily

# Packaged dependencies
RUN apt-get update && apt-get install -y \
	apparmor \
	aufs-tools \
	automake \
	bash-completion \
	btrfs-tools \
	build-essential \
	cmake \
	createrepo \
	curl \
	dpkg-sig \
	g++ \
	gcc \
	git \
	iptables \
	jq \
	libapparmor-dev \
	libc6-dev \
	libcap-dev \
	libltdl-dev \
	libsqlite3-dev \
	libsystemd-dev \
	mercurial \
	net-tools \
	parallel \
	pkg-config \
	python-dev \
	python-mock \
	python-pip \
	python-websocket \
	gccgo \
        golang \
	iproute2 \
	iputils-ping \
	vim-common \
	--no-install-recommends

# Get lvm2 source for compiling statically
ENV LVM2_VERSION 2.02.103
RUN mkdir -p /usr/local/lvm2 \
	&& curl -fsSL "https://mirrors.kernel.org/sourceware/lvm2/LVM2.${LVM2_VERSION}.tgz" \
		| tar -xzC /usr/local/lvm2 --strip-components=1
# See https://git.fedorahosted.org/cgit/lvm2.git/refs/tags for release tags

# Fix platform enablement in lvm2 to support aarch64 properly
RUN set -e \
	&& for f in config.guess config.sub; do \
		curl -fsSL -o "/usr/local/lvm2/autoconf/$f" "http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=$f;hb=HEAD"; \
	done
# "arch.c:78:2: error: #error the arch code needs to know about your machine type"

# Compile and install lvm2
RUN cd /usr/local/lvm2 \
	&& ./configure \
		--build="$(gcc -print-multiarch)" \
		--enable-static_link \
	&& make device-mapper \
	&& make install_device-mapper
# See https://git.fedorahosted.org/cgit/lvm2.git/tree/INSTALL

# Install seccomp: the version shipped in trusty is too old
ENV SECCOMP_VERSION 2.3.1
RUN set -x \
	&& export SECCOMP_PATH="$(mktemp -d)" \
	&& curl -fsSL "https://github.com/seccomp/libseccomp/releases/download/v${SECCOMP_VERSION}/libseccomp-${SECCOMP_VERSION}.tar.gz" \
		| tar -xzC "$SECCOMP_PATH" --strip-components=1 \
	&& ( \
		cd "$SECCOMP_PATH" \
		&& ./configure --prefix=/usr/local \
		&& make \
		&& make install \
		&& ldconfig \
	) \
	&& rm -rf "$SECCOMP_PATH"

# Install Go
# We don't have official binary tarballs for ARM64, eigher for Go or bootstrap,
# so we use gccgo as bootstrap to build Go from source code.
# We don't use the official ARMv6 released binaries as a GOROOT_BOOTSTRAP, because
# not all ARM64 platforms support 32-bit mode. 32-bit mode is optional for ARMv8.
ENV GO_VERSION 1.7.4
RUN mkdir /usr/src/go && curl -x 192.168.1.11:9999 -fsSL https://golang.org/dl/go${GO_VERSION}.src.tar.gz | tar -v -C /usr/src/go -xz --strip-components=1 \
	&& cd /usr/src/go/src \
	&& GOOS=linux GOARCH=arm64 GOROOT_BOOTSTRAP="$(go env GOROOT)" ./make.bash

ENV PATH /usr/src/go/bin:$PATH
ENV GOPATH /go

# Only install one version of the registry, because old version which support
# schema1 manifests is not working on ARM64, we should skip integration-cli
# tests for schema1 manifests on ARM64.
ENV REGISTRY_COMMIT 47a064d4195a9b56133891bbb13620c3ac83a827
RUN set -x \
	&& export GOPATH="$(mktemp -d)" \
	&& git clone https://github.com/docker/distribution.git "$GOPATH/src/github.com/docker/distribution" \
	&& (cd "$GOPATH/src/github.com/docker/distribution" && git checkout -q "$REGISTRY_COMMIT") \
	&& GOPATH="$GOPATH/src/github.com/docker/distribution/Godeps/_workspace:$GOPATH" \
		go build -o /usr/local/bin/registry-v2 github.com/docker/distribution/cmd/registry \
	&& rm -rf "$GOPATH"

# Install notary and notary-server
ENV NOTARY_VERSION v0.4.2
RUN set -x \
	&& export GOPATH="$(mktemp -d)" \
	&& git clone https://github.com/docker/notary.git "$GOPATH/src/github.com/docker/notary" \
	&& (cd "$GOPATH/src/github.com/docker/notary" && git checkout -q "$NOTARY_VERSION") \
	&& GOPATH="$GOPATH/src/github.com/docker/notary/vendor:$GOPATH" \
		go build -o /usr/local/bin/notary-server github.com/docker/notary/cmd/notary-server \
	&& GOPATH="$GOPATH/src/github.com/docker/notary/vendor:$GOPATH" \
		go build -o /usr/local/bin/notary github.com/docker/notary/cmd/notary \
	&& rm -rf "$GOPATH"

# Get the "docker-py" source so we can run their integration tests
ENV DOCKER_PY_COMMIT e2655f658408f9ad1f62abdef3eb6ed43c0cf324
RUN git clone https://github.com/docker/docker-py.git /docker-py \
	&& cd /docker-py \
	&& git checkout -q $DOCKER_PY_COMMIT \
	&& pip install -r test-requirements.txt

# Set user.email so crosbymichael's in-container merge commits go smoothly
RUN git config --global user.email 'docker-dummy@example.com'

# Add an unprivileged user to be used for tests which need it
RUN groupadd -r docker
RUN useradd --create-home --gid docker unprivilegeduser

VOLUME /var/lib/docker
WORKDIR /go/src/github.com/docker/docker
ENV DOCKER_BUILDTAGS apparmor pkcs11 seccomp selinux

# Let us use a .bashrc file
RUN ln -sfv $PWD/.bashrc ~/.bashrc

# Register Docker's bash completion.
RUN ln -sv $PWD/contrib/completion/bash/docker /etc/bash_completion.d/docker

# Get useful and necessary Hub images so we can "docker load" locally instead of pulling
COPY contrib/download-frozen-image-v2.sh /go/src/github.com/docker/docker/contrib/
RUN ./contrib/download-frozen-image-v2.sh /docker-frozen-images \
	aarch64/buildpack-deps:jessie@sha256:6aa1d6910791b7ac78265fd0798e5abd6cb3f27ae992f6f960f6c303ec9535f2 \
	aarch64/busybox:latest@sha256:b23a6a37cf269dff6e46d2473b6e227afa42b037e6d23435f1d2bc40fc8c2828 \
	aarch64/debian:jessie@sha256:4be74a41a7c70ebe887b634b11ffe516cf4fcd56864a54941e56bb49883c3170 \
	aarch64/hello-world:latest@sha256:65a4a158587b307bb02db4de41b836addb0c35175bdc801367b1ac1ddeb9afda
# See also "hack/make/.ensure-frozen-images" (which needs to be updated any time this list is)

# Install tomlv, vndr, runc, containerd, tini, docker-proxy
# Please edit hack/dockerfile/install-binaries.sh to update them.
COPY hack/dockerfile/binaries-commits /tmp/binaries-commits
COPY hack/dockerfile/install-binaries.sh /tmp/install-binaries.sh
RUN /tmp/install-binaries.sh tomlv vndr runc containerd tini proxy

# Wrap all commands in the "docker-in-docker" script to allow nested containers
ENTRYPOINT ["hack/dind"]

# Upload docker source
COPY . /go/src/github.com/docker/docker