[mcj]Could not create cudnn handle: CUDNN_STATUS_INTERNAL

使用tensorflow的时候出错：

2019-08-02 14:10:57.531973: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-08-02 14:10:57.534123: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
08/02/2019 14:10:57 CRITICAL Error caught! Exiting...
08/02/2019 14:10:57 ERROR    Caught exception in thread: 'training_0'
08/02/2019 14:10:59 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/home/ubuntu/bigdisk/mcj/faceswap/lib/cli.py", line 122, in execute_script
    process.process()
  File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 98, in process
    self.end_thread(thread, err)
  File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 124, in end_thread
    thread.join()
  File "/home/ubuntu/bigdisk/mcj/faceswap/lib/multithreading.py", line 460, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/home/ubuntu/bigdisk/mcj/faceswap/lib/multithreading.py", line 391, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 150, in training
    raise err
  File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 140, in training
    self.run_training_cycle(model, trainer)
  File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 222, in run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/home/ubuntu/bigdisk/mcj/faceswap/plugins/train/trainer/_base.py", line 211, in train_one_step
    raise err
  File "/home/ubuntu/bigdisk/mcj/faceswap/plugins/train/trainer/_base.py", line 176, in train_one_step
    loss[side] = batcher.train_one_batch(do_preview)
  File "/home/ubuntu/bigdisk/mcj/faceswap/plugins/train/trainer/_base.py", line 276, in train_one_batch
    loss = self.model.predictors[self.side].train_on_batch(*batch)
  File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/keras/engine/training.py", line 1217, in train_on_batch
    outputs = self.train_function(ins)
  File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in __call__
    run_metadata_ptr)
  File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node encoder/conv_0_conv2d/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@train...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/encoder/conv_0_conv2d/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv_0_conv2d/kernel/read)]]
	 [[{{node training/Adam/gradients/AddN_1/_325}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_840_training/Adam/gradients/AddN_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
08/02/2019 14:10:59 CRITICAL An unexpected crash has occurred. Crash report written to '/home/ubuntu/bigdisk/mcj/faceswap/crash_report.2019.08.02.141059741177.log'. Please verify you are running the latest version of faceswap before reporting

2019-08-02 14:10:57.531973: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-08-02 14:10:57.534123: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

08/02/2019 14:10:57 CRITICAL Error caught! Exiting...

08/02/2019 14:10:57 ERROR Caught exception in thread: 'training_0'

08/02/2019 14:10:59 ERROR Got Exception on main handler:

Traceback (most recent call last):

File "/home/ubuntu/bigdisk/mcj/faceswap/lib/cli.py", line 122, in execute_script

process.process()

File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 98, in process

self.end_thread(thread, err)

File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 124, in end_thread

thread.join()

File "/home/ubuntu/bigdisk/mcj/faceswap/lib/multithreading.py", line 460, in join

raise thread.err[1].with_traceback(thread.err[2])

File "/home/ubuntu/bigdisk/mcj/faceswap/lib/multithreading.py", line 391, in run

self._target(*self._args, **self._kwargs)

File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 150, in training

raise err

File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 140, in training

self.run_training_cycle(model, trainer)

File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 222, in run_training_cycle

trainer.train_one_step(viewer, timelapse)

File "/home/ubuntu/bigdisk/mcj/faceswap/plugins/train/trainer/_base.py", line 211, in train_one_step

raise err

File "/home/ubuntu/bigdisk/mcj/faceswap/plugins/train/trainer/_base.py", line 176, in train_one_step

loss[side] = batcher.train_one_batch(do_preview)

File "/home/ubuntu/bigdisk/mcj/faceswap/plugins/train/trainer/_base.py", line 276, in train_one_batch

loss = self.model.predictors[self.side].train_on_batch(*batch)

File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/keras/engine/training.py", line 1217, in train_on_batch

outputs = self.train_function(ins)

File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__

return self._call(inputs)

File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call

fetched = self._callable_fn(*array_vals)

File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in __call__

run_metadata_ptr)

File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__

c_api.TF_GetCode(self.status.status))

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

[[{{node encoder/conv_0_conv2d/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@train...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/encoder/conv_0_conv2d/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv_0_conv2d/kernel/read)]]

[[{{node training/Adam/gradients/AddN_1/_325}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_840_training/Adam/gradients/AddN_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

08/02/2019 14:10:59 CRITICAL An unexpected crash has occurred. Crash report written to '/home/ubuntu/bigdisk/mcj/faceswap/crash_report.2019.08.02.141059741177.log'. Please verify you are running the latest version of faceswap before reporting

解决方案如下：


隐藏内容，解锁需要先评论本文
评论后刷新解锁