記一次曲折的運維經歷-2020-04-23

jenkins build jobs error：

 

ERROR: Build step failed with exception com.github.dockerjava.api.exception.DockerClientException: Could not build image: devmapper: Thin Pool has 38659 free data blocks which is less than minimum required 38911 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior at com.github.dockerjava.core.command.BuildImageResultCallback.getImageId(BuildImageResultCallback.java:79) at com.github.dockerjava.core.command.BuildImageResultCallback.awaitImageId(BuildImageResultCallback.java:51) at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher$Run.buildImage(DockerBuilderPublisher.java:387) at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher$Run.run(DockerBuilderPublisher.java:313) at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher.perform(DockerBuilderPublisher.java:463) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.build(MavenModuleSetBuild.java:946) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:896) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1816) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Build step 'Build / Publish Docker Image' marked build as failure

Finished: FAILURE

 

度孃的答案：device-mapper在刪除鏡像時沒有回收，這是一個內核bug。

解決方法：

標準的解決辦法是打Linux的補丁，使得在內核層面解決問題。

若是不想打補丁，不知道怎麼打補丁，也不想去倒騰了，能夠手工臨時釋放空間的辦法。

咱們先來手工清理空間：

1.清理exited進程：

docker rm $(docker ps -q -f status=exited)

2.清理dangling volumes：

docker volume rm $(docker volume ls -qf dangling=true)

3.清理dangling image：

docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

本覺得清理完成後，jenkins構建不會出現相似問題，以上操做不會對running的容器產生影響。

 

誰承想jenkins掛了，先把jenkins搞起來吧，ok，jenkins起來了，卻發現原有的jobs不見了，這時在系統管理，看到一堆的錯誤提示須要升級jenkins相關插件。

 

升級完相關插件後，jenkins構建時又出現其餘錯誤，

CMD /data/tomcat/bin/startup.sh start ERROR: Build step failed with exception java.lang.NullPointerException: uri was not specified at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at com.github.dockerjava.core.DefaultDockerClientConfig$Builder.withDockerHost(DefaultDockerClientConfig.java:337) at io.jenkins.docker.client.DockerAPI.makeClient(DockerAPI.java:244) at io.jenkins.docker.client.DockerAPI.getOrMakeClient(DockerAPI.java:200) at io.jenkins.docker.client.DockerAPI.getClient(DockerAPI.java:169) at io.jenkins.docker.client.DockerAPI.getClient(DockerAPI.java:152) at com.nirima.jenkins.plugins.docker.DockerCloud.isTriton(DockerCloud.java:743) at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher.getDockerAPI(DockerBuilderPublisher.java:254) at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher.perform(DockerBuilderPublisher.java:463) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.build(MavenModuleSetBuild.java:946) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:896) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1816) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Build step 'Build / Publish Docker Image' marked build as failure Finished: FAILURE

在系統管理-cloud-add a new cloud

Docker Host URI ：tcp://內網ip:2375

2375 docker 的遠程鏈接端口。

 

jenkins 再次構建沒有以上錯誤，但回到了原始的問題。

嘗試構建其餘服務會不會有相似問題，在構建的過程當中沒有發現相似問題，啓動容器時出現相似錯誤。

 

綜上出現異常，docker build 和 docker run 都會報一樣的錯誤。 那就用docker info 查看docker 的詳細信息吧，這裏咱們發現

Data Space Available: 20.27 GB

Metadata Space Available: 2.076 GB

原有200G,可用20G，

df -h free -hm 都夠用吶

空間不夠用了？ 不該該啊，先前磁盤不夠用，報的異常是設備上沒有可用空間，不是這個錯誤，

 

docker使用的是thinpool，thin-pool由一個metadata設備和data設備組成，按需分配數據塊，刪除時也回收數據塊，提升存儲空間利用率。

 

使用 lsblk 以及lvs 命令觀察當前的分配狀況，與此同時收集device-mapper的日誌，持續跟蹤direct-lvm工做狀況，能夠直接使用:

# journalctl -fu dm-event.service

-- Logs begin at Mon 2020-02-10 22:41:43 CST. --

Apr 23 16:48:40 VM_4_11_centos lvm[1529]: Insufficient free space: 9728 extents needed, but only 1538 available

Apr 23 16:48:40 VM_4_11_centos lvm[1529]: Failed command for vgdocker-thinpool.

Apr 23 17:31:30 VM_4_11_centos lvm[1529]: Insufficient free space: 9728 extents needed, but only 1538 available

Apr 23 17:31:30 VM_4_11_centos lvm[1529]: Failed command for vgdocker-thinpool.

Apr 23 18:14:20 VM_4_11_centos lvm[1529]: Insufficient free space: 9728 extents needed, but only 1538 available

Apr 23 18:14:20 VM_4_11_centos lvm[1529]: Failed command for vgdocker-thinpool.

Apr 23 18:57:10 VM_4_11_centos lvm[1529]: Insufficient free space: 9728 extents needed, but only 1538 available

Apr 23 18:57:10 VM_4_11_centos lvm[1529]: Failed command for vgdocker-thinpool.

Apr 23 19:40:00 VM_4_11_centos lvm[1529]: Insufficient free space: 9728 extents needed, but only 1538 available

Apr 23 19:40:00 VM_4_11_centos lvm[1529]: Failed command for vgdocker-thinpool.

 

可用空間不足：須要9728個數據塊，但只有1538個可用

 

***刪除垃圾鏡像和容器後從新構建服務此問題獲得修復。