從新配置語義分割實驗環境遇到的坑

新的框架是semseg, by hs-z https://github.com/hszhao/semsegnode

1. 安裝apex報錯：fatal error: gnu-crypt.h: No such file or directory

本質上是cryptacular的pip源有問題，使用conda install cryptacular便可python

2. pip install老是安裝到別的虛擬環境裏

這是由於當前正在使用的pip並不是當前虛擬環境裏的。這裏conda install會默認安裝到當前虛擬環境，可是pip並不會。linux

因此使用 whereis pip查看想要的當前虛擬環境的pip程序的位置，而後使用絕對路徑來執行pip install便可git

3. 關於pip和conda的源

今天是2019年6月10日，目前conda的清華源由於版權問題已經關閉，而pip的清華源仍能夠正常使用。github

4. ModuleNotFoundError: No module named 'yaml'

應該是服務器

conda install pyaml

5. TypeError: Class advice impossible in Python3. Use the @implementer class decorator instead

首先切換當前的CUDA版本與pytorch的CUDA版本一致多線程

而後卸載已經安裝過的apex。app

而後：框架

git clone https://github.com/NVIDIA/apex.git && cd apex && pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" .

對於多虛擬環境，可能會有些錯亂。此時使用 less

whereis pip

來找到你當前虛擬環境的pip執行程序的位置。而後使用pip的絕對路徑進行操做。

包括apex上面的最後一步的python也可使用其絕對路徑來安裝，保證必定安裝到了正確的位置

不該該使用下面的這行命令來安裝apex：

git clone https://github.com/NVIDIA/apex.git && cd apex && python setup.py install --cuda_ext --cpp_ext

緣由多是我在conda虛擬環境中

參考：https://github.com/NVIDIA/apex/issues/214#issuecomment-476399539

6. 什麼報錯都沒有，用PDB也沒有。在 x = self.layer0(x) 處消失

batch size和輸入圖片尺寸小一些就行了

7. Warning: apex was installed without --cpp_ext. Falling back to Python flatten and unflatten.

在已經正確安裝的前提下，還堅持報這個錯誤，說明是個有深度的錯誤。

根據：https://discuss.pytorch.org/t/undefined-symbol-when-import-lltm-cpp-extension/32627/2

兩個解決方法：

build cpp extensions with -D_GLIBCXX_USE_CXX11_ABI=1.
build pytorch with -D_GLIBCXX_USE_CXX11_ABI=0

可是apex如何設置額外的編譯參數我也不會。根據他們提供的export方法，即：

export CFLAGS="-D_GLIBCXX_USE_CXX11_ABI=1 $CFLAGS"

而後再編譯apex，發如今編譯過程當中這個參數仍是等於0，沒有效果。

最終根據下面這段話：

The best way to solve this problem in any case is to compile Pytorch from source and use that same compiler for the extension. Then all problems go away.

決定仍是把pytorch和apex都在本機上從源代碼編譯一遍得了。

而後發現pytorch從源代碼編譯頗有困難……遇到了一堆找不到解決辦法的BUG，最後想了想把pytorch安裝回去吧。

以前安裝pytorch和此次的途徑不一樣：

以前的方式是：

conda install pytorch torchvision cudatoolkit=10.0

爲了加快下載速度，就不想從pytorch官方源下載，而是選擇了從conda源下載。

然而在python中，使用：

torch._C._GLIBCXX_USE_CXX11_ABI

發現是True，也就是說

-D_GLIBCXX_USE_CXX11_ABI=1

不知足要求

此次使用了pytorch官方的源：

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

神奇的事情發生了，此次

-D_GLIBCXX_USE_CXX11_ABI=0了

仍是pytorch官方靠譜……conda上是收錄的官方的編譯包，更新的不夠及時。

而後就OK了……………………

總結一下，

1. 靠譜的仍是官方，不要圖省事，也不要總想着本身去編譯，那樣子問題更多。

2. 遇到問題要到github上相應倉庫的issue去查詢，這也很重要。特別是，要用英文查詢。中文查詢都是二手信息。

3. Google的搜索能力的確很厲害，儘可能用Google！

8. ValueError: batch_size should be a positive integer value, but got batch_size=0

在config文件裏的

batch_size_val: 8  # batch size for validation during training, memory and speed tradeoff

須要設置爲和GPU同樣的數量，雖然不知道爲何。

9. cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

data_root不正確，沒有讀取到數據

10. Exception: process 0 terminated with signal SIGSEGV

內存不足

目前還沒法解決

只能把分佈式訓練給關了，暫時能夠運行，可是很慢

11. pip下載速度慢

linux下，修改 ~/.pip/pip.conf (沒有就建立一個)，修改 index-url至tuna，內容以下：

 [global] index-url = https://pypi.tuna.tsinghua.edu.cn/simple

12. 查看pytorch對應的cuda版本

print(torch.version.cuda)

13. libSM.so.6: cannot open shared object file: No such file or directory

# https://stackoverflow.com/questions/47113029/importerror-libsm-so-6-cannot-open-shared-object-file-no-such-file-or-directo

pip install opencv-python-headless # also contrib, if needed pip install opencv-contrib-python-headless

14. fatal error: gnu-crypt.h: No such file or directory

在安裝apex過程當中出現的。應該使用

conda install cryptacular

而後再安裝apex

15. OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).

OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and ope
rating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://
www.intel.com/software/products/support/.

Traceback (most recent call last): │··································
File "tool/train.py", line 456, in <module> │··································
main() │··································
File "tool/train.py", line 106, in main │··································
mp.spawn(main_worker, nprocs=args.ngpus_per_node, args=(args.ngpus_per_node, args)) │··································
File "/home/xxx/.conda/envs/pytorchseg10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn │··································
while not spawn_context.join(): │··································
File "/home/xxx/.conda/envs/pytorchseg10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 107, in join │··································
(error_index, name) │··································
Exception: process 2 terminated with signal SIGABRT

在新服務器上試圖從新配置環境，然而遇到這個問題

解決方法：

在train.sh裏添加這一行

export KMP_INIT_AT_FORK=FALSE

此時的train.sh爲

#!/bin/sh PARTITION=gpu PYTHON=python dataset=$1 exp_name=$2 exp_dir=exp/${dataset}/${exp_name} model_dir=${exp_dir}/model result_dir=${exp_dir}/result config=config/${dataset}/${dataset}_${exp_name}.yaml now=$(date +"%Y%m%d_%H%M%S") mkdir -p ${model_dir} ${result_dir} cp tool/train.sh tool/train.py ${config} ${exp_dir} export PYTHONPATH=./ export KMP_INIT_AT_FORK=FALSE #sbatch -p $PARTITION --gres=gpu:8 -c16 --job-name=train \ $PYTHON -u tool/train.py \ --config=${config} \ 2>&1 | tee ${model_dir}/train-$now.log

View Code

參考：https://github.com/ContinuumIO/anaconda-issues/issues/11294

16. RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1556653114079/work/torch/lib/c10d/ProcessGroupNCCL.cpp:272, unhandled cuda error

不少地方（https://github.com/pytorch/pytorch/issues/23534）

說是NCCL的版本問題，因而打印版本：（在命令行裏，運行python以前）

export NCCL_DEBUG=VERSION

而後執行程序，看到個人版本是2.4.8

此時，標題中的報錯再也不出現……

因此這個錯誤是一種表象，掩蓋了實際的錯誤

可是運行了一段時間後又自動斷開了，再次運行仍是這個錯誤…… 很奇怪

在打印了debug信息後：加入

export NCCL_DEBUG=info

發現有一個錯誤：

Cuda failure 'an illegal memory access was encountered'

這個問題沒有通行的解決方案，每一個人的問題都不太同樣。

我忽然發現，每次到GPU3的進程開始初始化時就會報錯，而後取消使用GPU3 發現錯誤解決了…… 難道是硬件壞了

通過排查，已經肯定只有在GPU3上有問題。運行屢次，報錯不一樣，這裏記錄一下

報錯記錄1：

Traceback (most recent call last): File "tool/train.py", line 480, in <module> main() File "tool/train.py", line 115, in main main_worker(args.train_gpu, args.ngpus_per_node, args) File "tool/train.py", line 290, in main_worker loss_train, mIoU_train, mAcc_train, allAcc_train = train(train_loader, model, optimizer, epoch) File "tool/train.py", line 335, in train output, main_loss, aux_loss = model(input, target) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward return self.module(*inputs[0], **kwargs[0]) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/segmentation_exp_LINUX/L012_cell/models/pspnet.py", line 93, in forward x = self.layer4(x_tmp) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/segmentation_exp_LINUX/L012_cell/models/resnet.py", line 82, in forward out = self.conv2(out) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward return self.conv2d_forward(input, self.weight) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward self.padding, self.dilation, self.groups) RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

View Code

報錯記錄2：

Traceback (most recent call last): File "tool/train.py", line 480, in <module> main() File "tool/train.py", line 115, in main main_worker(args.train_gpu, args.ngpus_per_node, args) File "tool/train.py", line 290, in main_worker loss_train, mIoU_train, mAcc_train, allAcc_train = train(train_loader, model, optimizer, epoch) File "tool/train.py", line 335, in train output, main_loss, aux_loss = model(input, target) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward return self.module(*inputs[0], **kwargs[0]) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/segmentation_exp_LINUX/L012_cell/models/pspnet.py", line 92, in forward x_tmp = self.layer3(x) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/segmentation_exp_LINUX/L012_cell/models/resnet.py", line 82, in forward out = self.conv2(out) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward return self.conv2d_forward(input, self.weight) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward self.padding, self.dilation, self.groups) RuntimeError: cublas runtime error : resource allocation failed at /opt/conda/conda-bld/pytorch_1568696969690/work/aten/src/THC/THCGeneral.cpp:216

View Code

對於報錯1，

根據 https://github.com/qqwweee/keras-yolo3/issues/332#issuecomment-517989338

應該安裝補丁： https://developer.nvidia.com/cuda-10.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal

安裝後仍是沒用

對於報錯記錄2：根據https://github.com/huggingface/transfer-learning-conv-ai/issues/10#issuecomment-496111466

增長了export CUDA_LAUNCH_BLOCKING=1

下面是報錯記錄3：

Traceback (most recent call last): File "tool/train.py", line 480, in <module> main() File "tool/train.py", line 115, in main main_worker(args.train_gpu, args.ngpus_per_node, args) File "tool/train.py", line 290, in main_worker loss_train, mIoU_train, mAcc_train, allAcc_train = train(train_loader, model, optimizer, epoch) File "tool/train.py", line 335, in train output, main_loss, aux_loss = model(input, target) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward return self.module(*inputs[0], **kwargs[0]) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/segmentation_exp_LINUX/L012_cell/models/pspnet.py", line 90, in forward x = self.layer1(x) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/segmentation_exp_LINUX/L012_cell/models/resnet.py", line 90, in forward residual = self.downsample(x) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 545, in __call__ result = self.forward(*input, **kwargs) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward return self.conv2d_forward(input, self.weight) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward self.padding, self.dilation, self.groups) RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

View Code

添加了下面一行後暫時能夠了，

torch.backends.cudnn.benchmark = True

好了十分鐘，又壞了，報錯和記錄3同樣……

==================================================================================================================

不肯定是否是仍是第三塊卡的問題，因此此次使用其餘的全部卡，訓練多一些時間看看

使用了別的全部的卡一塊兒訓練了好久都沒有問題。

總結兩點：

1. 只使用第三塊卡會有問題

2. 單卡時，至關於很是普通的訓練方式，並不會出發多線程、多進程以及分佈式的代碼

再嘗試一下是否是能夠經過軟件層面解決，不行的話就只能歸因於顯卡壞掉了。或者服務器有問題

有人說是CUDNN的版本問題，先將CUDNN關閉：

torch.backends.cudnn.enabled = False

關閉後，在單獨使用第三塊卡時候的確能夠運行了。

而後測試使用全部卡+關閉CUDNN。在運行了20個ITERS後報錯：

[2019-09-29 20:03:08,046 INFO train.py line 404 98898] Epoch: [44/200][20/186] Data 0.001 (0.106) Batch 1.274 (1.504) Remain 12:11:15 MainLoss 0.1086 AuxLoss 0.1171 Loss 0.1555 Accuracy 0.9622. terminate called after throwing an instance of 'c10::Error' what(): CUDA error: an illegal memory access was encountered (insert_events at /opt/conda/conda-bld/pytorch_1568696969690/work/c10/cuda/CUDACachingAllocator.cpp:569) frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7ffa46ff5477 in /home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: <unknown function> + 0x17044 (0x7ffa47231044 in /home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) frame #2: <unknown function> + 0x1cccb (0x7ffa47236ccb in /home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) frame #3: c10::TensorImpl::release_resources() + 0x4d (0x7ffa46fe2e8d in /home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/lib/libc10.so) frame #4: <unknown function> + 0x1c2789 (0x7ffa7892f789 in /home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #5: <unknown function> + 0x445d2b (0x7ffa78bb2d2b in /home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #6: <unknown function> + 0x445d61 (0x7ffa78bb2d61 in /home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #7: <unknown function> + 0x1a184f (0x558f4667b84f in /home/lzx/.conda/envs/seg/bin/python) frame #8: <unknown function> + 0xfd1a8 (0x558f465d71a8 in /home/lzx/.conda/envs/seg/bin/python) frame #9: <unknown function> + 0x10e3c7 (0x558f465e83c7 in /home/lzx/.conda/envs/seg/bin/python) frame #10: <unknown function> + 0x10e3dd (0x558f465e83dd in /home/lzx/.conda/envs/seg/bin/python) frame #11: <unknown function> + 0x10e3dd (0x558f465e83dd in /home/lzx/.conda/envs/seg/bin/python) frame #12: <unknown function> + 0xf5777 (0x558f465cf777 in /home/lzx/.conda/envs/seg/bin/python) frame #13: <unknown function> + 0xf57e3 (0x558f465cf7e3 in /home/lzx/.conda/envs/seg/bin/python) frame #14: <unknown function> + 0xf5766 (0x558f465cf766 in /home/lzx/.conda/envs/seg/bin/python) frame #15: <unknown function> + 0x1db5e3 (0x558f466b55e3 in /home/lzx/.conda/envs/seg/bin/python) frame #16: _PyEval_EvalFrameDefault + 0x2a5a (0x558f466a7e4a in /home/lzx/.conda/envs/seg/bin/python) frame #17: _PyFunction_FastCallKeywords + 0xfb (0x558f4663dccb in /home/lzx/.conda/envs/seg/bin/python) frame #18: _PyEval_EvalFrameDefault + 0x6a3 (0x558f466a5a93 in /home/lzx/.conda/envs/seg/bin/python) frame #19: _PyFunction_FastCallKeywords + 0xfb (0x558f4663dccb in /home/lzx/.conda/envs/seg/bin/python) frame #20: _PyEval_EvalFrameDefault + 0x416 (0x558f466a5806 in /home/lzx/.conda/envs/seg/bin/python) frame #21: _PyEval_EvalCodeWithName + 0x2f9 (0x558f465ee539 in /home/lzx/.conda/envs/seg/bin/python) frame #22: _PyFunction_FastCallKeywords + 0x387 (0x558f4663df57 in /home/lzx/.conda/envs/seg/bin/python) frame #23: _PyEval_EvalFrameDefault + 0x14dc (0x558f466a68cc in /home/lzx/.conda/envs/seg/bin/python) frame #24: _PyEval_EvalCodeWithName + 0x2f9 (0x558f465ee539 in /home/lzx/.conda/envs/seg/bin/python) frame #25: PyEval_EvalCodeEx + 0x44 (0x558f465ef424 in /home/lzx/.conda/envs/seg/bin/python) frame #26: PyEval_EvalCode + 0x1c (0x558f465ef44c in /home/lzx/.conda/envs/seg/bin/python) frame #27: <unknown function> + 0x22ab74 (0x558f46704b74 in /home/lzx/.conda/envs/seg/bin/python) frame #28: PyRun_StringFlags + 0x7d (0x558f4670fddd in /home/lzx/.conda/envs/seg/bin/python) frame #29: PyRun_SimpleStringFlags + 0x3f (0x558f4670fe3f in /home/lzx/.conda/envs/seg/bin/python) frame #30: <unknown function> + 0x235f3d (0x558f4670ff3d in /home/lzx/.conda/envs/seg/bin/python) frame #31: _Py_UnixMain + 0x3c (0x558f467102bc in /home/lzx/.conda/envs/seg/bin/python) frame #32: __libc_start_main + 0xf0 (0x7ffa91705830 in /lib/x86_64-linux-gnu/libc.so.6) frame #33: <unknown function> + 0x1db062 (0x558f466b5062 in /home/lzx/.conda/envs/seg/bin/python) /home/lzx/.conda/envs/seg/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 14 leaked semaphores to clean up at shutdown len(cache)) /home/lzx/.conda/envs/seg/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 14 leaked semaphores to clean up at shutdown len(cache)) /home/lzx/.conda/envs/seg/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 14 leaked semaphores to clean up at shutdown len(cache)) /home/lzx/.conda/envs/seg/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 14 leaked semaphores to clean up at shutdown len(cache)) /home/lzx/.conda/envs/seg/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 14 leaked semaphores to clean up at shutdown len(cache)) /home/lzx/.conda/envs/seg/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 14 leaked semaphores to clean up at shutdown len(cache)) Traceback (most recent call last): File "tool/train.py", line 488, in <module> main() File "tool/train.py", line 114, in main mp.spawn(main_worker, nprocs=args.ngpus_per_node, args=(args.ngpus_per_node, args)) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception: -- Process 3 terminated with the following error: Traceback (most recent call last): File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/home/lzx/segmentation_exp_LINUX/L012_cell/tool/train.py", line 298, in main_worker loss_train, mIoU_train, mAcc_train, allAcc_train = train(train_loader, model, optimizer, epoch) File "/home/lzx/segmentation_exp_LINUX/L012_cell/tool/train.py", line 351, in train scaled_loss.backward() File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/tensor.py", line 120, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/home/lzx/.conda/envs/seg/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

View Code

但是爲何最後仍是說CUDNN有錯，不是都關閉了嗎？

根據：https://blog.csdn.net/qq_39938666/article/details/86611474

多是python版本有問題。因而選擇和他同樣的3.6.6進行嘗試

從新建立了Python=3.6.6的環境，仍是不正確……

在別的卡都正確，只有使用GPU03不正確，說明代碼沒寫錯，就是卡的問題。或者就是不兼容

我嘗試將這些卡調換順序，可是報錯一直都是將GPU02有問題。（之前是GPU03，後來不知道爲何一直是GPU03了，會不會是電源插口有問題？）

目前的報錯（單卡GPU02）是cuda runtime error (77) : an illegal memory access was encountered

根據有的回答https://ethereum.stackexchange.com/questions/65652/error-cuda-mining-an-illegal-memory-access-was-encountered

我正在嘗試將CUDA升級到10.1，目前是10.0

然而並無用…… 先就用7塊卡吧，

10月03日更新

今天算是解決了這個問題，誤打誤撞的

主要是參考了：https://github.com/pytorch/pytorch/issues/22050#issuecomment-521030783

這我的的頭像我很熟悉，是pytorch論壇裏常常回復別人消息的NVIDIA員工

他說除了使用conda。也要嘗試使用pip安裝。

因而我嘗試了使用pip，將pytorch安裝在別的環境裏。而後別的環境和之前的環境，都再也不有問題了。

這本質上應該是CUDNN被pip安裝的東西覆蓋了。你們都說這個是CUDNN有問題。可能pip的這個版本正好是OK的

大神給的命令行是

pip3 install torch torchvision

可是，你們都知道pip可能會更新源的包。因此這裏貼一下個人實際的下載到的包：

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting torch Downloading https://pypi.tuna.tsinghua.edu.cn/packages/30/57/d5cceb0799c06733eefce80c395459f28970ebb9e896846ce96ab579a3f1/torch-1.2.0-cp36-cp36m-manylinux1_x86_64.whl (748.8MB)
     |████████████████████████████████| 748.9MB 68kB/s Collecting torchvision Downloading https://pypi.tuna.tsinghua.edu.cn/packages/06/e6/a564eba563f7ff53aa7318ff6aaa5bd8385cbda39ed55ba471e95af27d19/torchvision-0.4.0-cp36-cp36m-manylinux1_x86_64.whl (8.8MB)
     |████████████████████████████████| 8.8MB 1.0MB/s Requirement already satisfied: numpy in /home/lzx/.conda/envs/pytorchseg10/lib/python3.6/site-packages (from torch) (1.17.2) Requirement already satisfied: pillow>=4.1.1 in /home/lzx/.conda/envs/pytorchseg10/lib/python3.6/site-packages (from torchvision) (6.1.0) Requirement already satisfied: six in /home/lzx/.conda/envs/pytorchseg10/lib/python3.6/site-packages (from torchvision) (1.12.0) Installing collected packages: torch, torchvision Successfully installed torch-1.2.0 torchvision-0.4.0

僅供參考