Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED

[ RUN      ] PowerLayerTest/3.TestPowerOneGradient F0319 15:50:19.414253 22426 math_functions.cu:92] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED *** Check failure stack trace: *** @ 0x7f574fe7e78d google::LogMessage::Fail() @ 0x7f574fe80d43 google::LogMessage::SendToLog() @ 0x7f574fe7e31b google::LogMessage::Flush() @ 0x7f574fe7fc8e  google::LogMessageFatal::~LogMessageFatal() @ 0x7f574e348e4e  caffe::caffe_gpu_scal<>() @ 0x7f574e336906  caffe::PowerLayer<>::Forward_gpu() @ 0x455772  caffe::Layer<>::Forward() @ 0x496300  caffe::GradientChecker<>::CheckGradientSingle() @ 0x50ea94  caffe::GradientChecker<>::CheckGradientEltwise() @ 0x6b69b5  caffe::PowerLayerTest<>::TestBackward() @ 0x7ac6b3  testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x7a5cca testing::Test::Run() @ 0x7a5e18 testing::TestInfo::Run() @ 0x7a5ef5 testing::TestCase::Run() @ 0x7a71cf testing::internal::UnitTestImpl::RunAllTests() @ 0x7a74f3 testing::UnitTest::Run() @ 0x44bfa9 main @ 0x7f574d5a8830  __libc_start_main @ 0x451c39 _start Makefile:478: recipe for target 'runtest' failed make: *** [runtest] 已放棄 (core dumped)

 首先明確，這是make runtest的錯誤，因此必定不能是代碼問題。必定是個人配置問題。雖然是segnet做者改得caffe，可是應該沒有問題。不過我仍是打算用官方的caffe跑一下試試。

我以前那篇教程，安裝教程http://www.cnblogs.com/SweetBeens/p/8525131.html，提到若是ubuntu16.04裝9,可能會有問題，那麼問題來了。官網給出的doc提到了這個問題：

The GPU program failed to execute. This is often caused by a launch failure of the kernel on the GPU, which can be caused by multiple reasons.

To correct: check that the hardware, an appropriate version of the driver, and the cuBLAS library are correctly installed.

github上有個比較火的討論貼：

https://github.com/BVLC/caffe/issues/2417

你們的解決方法是裝cuda8.0,還有說不用cudnn。

但因爲我不到黃河心不死，不想用cuda8

那麼有以下思路：

1,跑官網caffe，看看是否是由於segnet_caffe版本過低或者什麼的

2.改下gcc版本

3.更換driver和cuda版本

等我試試。並且爲何別人的有一些可用16.04+9.1，奇怪了。

擬解決過程

1.我跑了官網的caffe，出現以下錯誤：

[  FAILED  ] EmbedLayerTest/3.TestForward, where TypeParam = caffe::GPUDevice<double> (1 ms) [ RUN ] EmbedLayerTest/3.TestGradient [ OK ] EmbedLayerTest/3.TestGradient (101 ms) [ RUN ] EmbedLayerTest/3.TestSetUp [ OK ] EmbedLayerTest/3.TestSetUp (0 ms) [ RUN ] EmbedLayerTest/3.TestForwardWithBias F0319 17:26:25.959848 30839 math_functions.cu:42] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED *** Check failure stack trace: *** @ 0x7f787028d78d google::LogMessage::Fail() @ 0x7f787028fd43 google::LogMessage::SendToLog() @ 0x7f787028d31b google::LogMessage::Flush() @ 0x7f787028ec8e  google::LogMessageFatal::~LogMessageFatal() @ 0x7f786e105672  caffe::caffe_gpu_gemm<>() @ 0x7f786e13d75a  caffe::EmbedLayer<>::Forward_gpu() @ 0x476522  caffe::Layer<>::Forward() @ 0x4f5ef6  caffe::EmbedLayerTest_TestForwardWithBias_Test<>::TestBody() @ 0x90b393  testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x9049aa testing::Test::Run() @ 0x904af8 testing::TestInfo::Run() @ 0x904bd5 testing::TestCase::Run() @ 0x905eaf testing::internal::UnitTestImpl::RunAllTests() @ 0x9061d3 testing::UnitTest::Run() @ 0x469fed main @ 0x7f786d2ec830  __libc_start_main @ 0x471a69 _start Makefile:532: recipe for target 'runtest' failed make: *** [runtest] 已放棄 (core dumped)

 問題解決

cuda版本問題。在個人電腦配置環境下，必須是ubuntu 16.04+cuda8。不能夠用9.1.

那麼借鑑這個思路，請各位看下本身的版本有沒有不按要求整的。基本限制就是cuda，python。

我簡化的安裝步驟，更方便查漏補缺：

http://www.cnblogs.com/SweetBeens/p/8652083.html

詳細的安裝版本制約:

http://www.cnblogs.com/SweetBeens/p/8525131.html

以及利用anaconda2隨意配置多個版本：

http://www.cnblogs.com/SweetBeens/p/8650460.html

卸載cuda9.1安裝cuda8

http://www.cnblogs.com/SweetBeens/p/8616797.html