使用tensorflow的时候出错：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
2019-08-02 14:10:57.531973: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2019-08-02 14:10:57.534123: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 08/02/2019 14:10:57 CRITICAL Error caught! Exiting... 08/02/2019 14:10:57 ERROR Caught exception in thread: 'training_0' 08/02/2019 14:10:59 ERROR Got Exception on main handler: Traceback (most recent call last): File "/home/ubuntu/bigdisk/mcj/faceswap/lib/cli.py", line 122, in execute_script process.process() File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 98, in process self.end_thread(thread, err) File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 124, in end_thread thread.join() File "/home/ubuntu/bigdisk/mcj/faceswap/lib/multithreading.py", line 460, in join raise thread.err[1].with_traceback(thread.err[2]) File "/home/ubuntu/bigdisk/mcj/faceswap/lib/multithreading.py", line 391, in run self._target(*self._args, **self._kwargs) File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 150, in training raise err File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 140, in training self.run_training_cycle(model, trainer) File "/home/ubuntu/bigdisk/mcj/faceswap/scripts/train.py", line 222, in run_training_cycle trainer.train_one_step(viewer, timelapse) File "/home/ubuntu/bigdisk/mcj/faceswap/plugins/train/trainer/_base.py", line 211, in train_one_step raise err File "/home/ubuntu/bigdisk/mcj/faceswap/plugins/train/trainer/_base.py", line 176, in train_one_step loss[side] = batcher.train_one_batch(do_preview) File "/home/ubuntu/bigdisk/mcj/faceswap/plugins/train/trainer/_base.py", line 276, in train_one_batch loss = self.model.predictors[self.side].train_on_batch(*batch) File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/keras/engine/training.py", line 1217, in train_on_batch outputs = self.train_function(ins) File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__ return self._call(inputs) File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in __call__ run_metadata_ptr) File "/home/ubuntu/.conda/envs/facewap/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__ c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node encoder/conv_0_conv2d/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@train...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/encoder/conv_0_conv2d/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv_0_conv2d/kernel/read)]] [[{{node training/Adam/gradients/AddN_1/_325}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_840_training/Adam/gradients/AddN_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] 08/02/2019 14:10:59 CRITICAL An unexpected crash has occurred. Crash report written to '/home/ubuntu/bigdisk/mcj/faceswap/crash_report.2019.08.02.141059741177.log'. Please verify you are running the latest version of faceswap before reporting |
解决方案如下：
隐藏内容，解锁需要先评论本文
评论后刷新解锁