Page 1 of 1

Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Posted: Thu Feb 06, 2020 12:50 pm
by congo
Hi everybody,

I installed Ubuntu 19.10 on my new PC (which has a RTX 2080 Super) plus nvdia-driver-435.
I'm getting the error "Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR" whenever I try to extract, sort or train anything. This seems to be a known bug and enabling "allow growth" makes extraction and training possible, however there is no option like this for sorting.
Is there a way to fix this problem so all tools can be used?

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Posted: Thu Feb 06, 2020 2:04 pm
by torzdf
At the moment, not in faceswap....

However, you can force allow growth by setting an environment variable prior to launching faceswap:
https://www.tensorflow.org/guide/gpu wrote: Another way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Posted: Fri Feb 07, 2020 9:52 am
by congo
Thanks, that did the trick. Both key an value are case sensitive, just in case somebody else wants to try this out.

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Posted: Mon Feb 24, 2020 7:49 pm
by graza
Hi

I'm having difficulty setting this environment variable (embarrassingly).

So, I have the same problem, probably because I'm using an RTX2060. After starting training:

704953: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

The solution appears to be to set the environment variable to allow growth.

I've spent a long time looking at the conda documentation, but this command (from docs.conda.io)
conda env config vars set my_var=value is failing to work

So I activated the faceswap environment, then entered in terminal:
conda env config vars set TF_FORCE_GPU_ALLOW_GROWTH=true

but this error is returned:
usage: conda-env [-h] {create,export,list,remove,update} ...
conda-env: error: invalid choice: 'config' (choose from 'create', 'export', 'list', 'remove', 'update')


but its the command on the conda website!!!!

I've also tried setting the variable in the _config.yml file, but still get the error when starting training

Can anyone help? :roll:

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Posted: Tue Feb 25, 2020 12:24 am
by torzdf
Yeah, this is an inconsistent bug with no apparent pattern between GPU/Driver/OS

We have you covered though (outside of Sort).

For Extract go Settings > Configure Extract Settings and check Allow Growth.
For Train and Convert enable the Allow Growth option in the main options panel.

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Posted: Sat Mar 07, 2020 2:01 pm
by graza
Thanks for the reply. I can't believe I didn't spot that option on the setings panel! Am now currently training my first model!!!!! :D