openshift上使用devicemapper

 

環境：openshift v3.6.173.0.5

openshift上devicemapper與官方文檔中的描述略有不一樣，在官方文檔的描述中，容器使用的lvm文件系統掛載在/var/lib/devicemapper下，所以能夠在/var/lib/devicemapper/metadata中找到容器對應的devicemapper的配置信息，在/var/lib/devicemapper/mnt中能夠看到容器的文件系統，描述以下：

The /var/lib/docker/devicemapper/metadata/ directory contains metadata about the Devicemapper configuration itself and about each image and container layer that exist. The devicemapper storage driver uses snapshots, and this metadata include information about those snapshots. These files are in JSON format.

The /var/lib/devicemapper/mnt/ directory contains a mount point for each image and container layer that exists. Image layer mount points are empty, but a container’s mount point shows the container’s filesystem as it appears from within the container.

在openshfit中使用docker inspect查看一個容器的devicemapper信息以下，使用的塊設備爲 docker-253:0-101504694-13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f

"GraphDriver": {
    "Name": "devicemapper",
    "Data": {
        "DeviceId": "29",
        "DeviceName": "docker-253:0-101504694-13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f",
        "DeviceSize": "10737418240"
    }
},

但在/var/lib/docker/devicemapper/mnt中該容器對應的文件掛載爲空，系統mount命令也查找不到該容器對應的塊設備掛載信息

# pwd
/var/lib/docker/devicemapper
# du -d 2|grep 13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f 0       ./mnt/13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f-init
0       ./mnt/13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f

查看該容器對應的進程，pid爲19422

# docker inspect 17ba06eb4578|grep Pid
  "Pid": 19422,
  "PidMode": "",
  "PidsLimit": 0,

在/proc/19422/ns中能夠看到其命名空間相關的信息，對比pid 爲1的mnt ns(mnt -> mnt:[4026531840])，能夠看到該容器與根進程不屬於同一個mount 命名空間，所以在根進程所在的mount命名空間中沒法查看到該容器的掛載信息

# ll
total 0
lrwxrwxrwx. 1 1000010000 root 0 Apr 12 09:43 ipc -> ipc:[4026532228]
lrwxrwxrwx. 1 1000010000 root 0 Apr 11 19:47 mnt -> mnt:[4026532521]
lrwxrwxrwx. 1 1000010000 root 0 Apr 12 09:43 net -> net:[4026532231]
lrwxrwxrwx. 1 1000010000 root 0 Apr 11 19:47 pid -> pid:[4026532523]
lrwxrwxrwx. 1 1000010000 root 0 Apr 12 09:43 user -> user:[4026531837]
lrwxrwxrwx. 1 1000010000 root 0 Apr 11 19:47 uts -> uts:[4026532522]

使用nsenter -t 19422 -m mnt -u命令進入到進程所在的mount和uts命名空間，使用mount命令能夠看到容器的塊設備掛載到了該mount命名空間的根目錄

/dev/mapper/docker-253:0-101504694-13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f on / type xfs (rw,relatime,context="system_u:object_r:container_file_t:s0:c2,c3",nouuid,attr2,inode64,sunit=1024,swidth=1024,noquota)

固然也能夠在/proc/19422/mounts和/proc/19422/mountinfo中看到與該進程相關的mount信息，其中也包括容器的塊設備掛載信息

使用lsns能夠看系統中的命名空間與進程的對應關係，下面列出了pid爲19422的相關命名空間，NPROCS表示該命名空間下面的進程數目，PID表示該命名空間下的最小PID值。能夠看出19422與根進程屬於同一個user命名空間。其中mnt命名空間中有2個進程，一個是19422，另外一個是19422的父進程(此處爲java)。所以在容器異常退出後，能夠經過進入未退出進程的命名空間定位問題。

[root@lab-node1 proc]# lsns -p 19422
        NS TYPE  NPROCS   PID USER       COMMAND
4026531837 user     338     1 root       /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026532228 ipc        3 18413 1001       /usr/bin/pod
4026532231 net        3 18413 1001       /usr/bin/pod
4026532521 mnt        2 19422 1000010000 /bin/sh /opt/eap/bin/standalone.sh -Djavax.net.ssl.keyStore=/opt/hawkular/auth/hawkular-metrics.keystore -Djavax.net.ssl.tru
4026532522 uts        2 19422 1000010000 /bin/sh /opt/eap/bin/standalone.sh -Djavax.net.ssl.keyStore=/opt/hawkular/auth/hawkular-metrics.keystore -Djavax.net.ssl.tru
4026532523 pid        2 19422 1000010000 /bin/sh /opt/eap/bin/standalone.sh -Djavax.net.ssl.keyStore=/opt/hawkular/auth/hawkular-metrics.keystore -Djavax.net.ssl.tru

 

參考：

https://xuxinkun.github.io/2019/04/02/deviemapper-docker/

https://docs.docker.com/engine/reference/commandline/dockerd/