Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Installing and setting up FaceSwap


Forum rules

Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
congo
Posts: 16
Joined: Mon Dec 16, 2019 3:09 pm
Has thanked: 7 times
Been thanked: 1 time

Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by congo »

Hi everybody,

I installed Ubuntu 19.10 on my new PC (which has a RTX 2080 Super) plus nvdia-driver-435.
I'm getting the error "Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR" whenever I try to extract, sort or train anything. This seems to be a known bug and enabling "allow growth" makes extraction and training possible, however there is no option like this for sorting.
Is there a way to fix this problem so all tools can be used?

by torzdf » Thu Feb 06, 2020 2:04 pm

At the moment, not in faceswap....

However, you can force allow growth by setting an environment variable prior to launching faceswap:

https://www.tensorflow.org/guide/gpu wrote:

Another way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.

Go to full post
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by torzdf »

At the moment, not in faceswap....

However, you can force allow growth by setting an environment variable prior to launching faceswap:

https://www.tensorflow.org/guide/gpu wrote:

Another way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.

My word is final

User avatar
congo
Posts: 16
Joined: Mon Dec 16, 2019 3:09 pm
Has thanked: 7 times
Been thanked: 1 time

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by congo »

Thanks, that did the trick. Both key an value are case sensitive, just in case somebody else wants to try this out.

User avatar
graza
Posts: 2
Joined: Mon Feb 24, 2020 7:41 pm

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by graza »

Hi

I'm having difficulty setting this environment variable (embarrassingly).

So, I have the same problem, probably because I'm using an RTX2060. After starting training:

704953: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

The solution appears to be to set the environment variable to allow growth.

I've spent a long time looking at the conda documentation, but this command (from docs.conda.io)
conda env config vars set my_var=value is failing to work

So I activated the faceswap environment, then entered in terminal:
conda env config vars set TF_FORCE_GPU_ALLOW_GROWTH=true

but this error is returned:
usage: conda-env [-h] {create,export,list,remove,update} ...
conda-env: error: invalid choice: 'config' (choose from 'create', 'export', 'list', 'remove', 'update')

but its the command on the conda website!!!!

I've also tried setting the variable in the _config.yml file, but still get the error when starting training

Can anyone help? :roll:

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by torzdf »

Yeah, this is an inconsistent bug with no apparent pattern between GPU/Driver/OS

We have you covered though (outside of Sort).

For Extract go Settings > Configure Extract Settings and check Allow Growth.
For Train and Convert enable the Allow Growth option in the main options panel.

My word is final

User avatar
graza
Posts: 2
Joined: Mon Feb 24, 2020 7:41 pm

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by graza »

Thanks for the reply. I can't believe I didn't spot that option on the setings panel! Am now currently training my first model!!!!! :D

Locked