Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Installing and setting up FaceSwap
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Post Reply
User avatar
congo
Posts: 14
Joined: Mon Dec 16, 2019 3:09 pm
Has thanked: 6 times

Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by congo » Thu Feb 06, 2020 12:50 pm

Hi everybody,

I installed Ubuntu 19.10 on my new PC (which has a RTX 2080 Super) plus nvdia-driver-435.
I'm getting the error "Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR" whenever I try to extract, sort or train anything. This seems to be a known bug and enabling "allow growth" makes extraction and training possible, however there is no option like this for sorting.
Is there a way to fix this problem so all tools can be used?
by torzdf » Thu Feb 06, 2020 2:04 pm
At the moment, not in faceswap....

However, you can force allow growth by setting an environment variable prior to launching faceswap:
https://www.tensorflow.org/guide/gpu wrote: Another way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.
Go to full post

User avatar
torzdf
Posts: 559
Joined: Fri Jul 12, 2019 12:53 am
Answers: 88
Has thanked: 17 times
Been thanked: 121 times

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by torzdf » Thu Feb 06, 2020 2:04 pm

At the moment, not in faceswap....

However, you can force allow growth by setting an environment variable prior to launching faceswap:
https://www.tensorflow.org/guide/gpu wrote: Another way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.
My word is final

User avatar
congo
Posts: 14
Joined: Mon Dec 16, 2019 3:09 pm
Has thanked: 6 times

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by congo » Fri Feb 07, 2020 9:52 am

Thanks, that did the trick. Both key an value are case sensitive, just in case somebody else wants to try this out.

User avatar
graza
Posts: 2
Joined: Mon Feb 24, 2020 7:41 pm

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by graza » Mon Feb 24, 2020 7:49 pm

Hi

I'm having difficulty setting this environment variable (embarrassingly).

So, I have the same problem, probably because I'm using an RTX2060. After starting training:

704953: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

The solution appears to be to set the environment variable to allow growth.

I've spent a long time looking at the conda documentation, but this command (from docs.conda.io)
conda env config vars set my_var=value is failing to work

So I activated the faceswap environment, then entered in terminal:
conda env config vars set TF_FORCE_GPU_ALLOW_GROWTH=true

but this error is returned:
usage: conda-env [-h] {create,export,list,remove,update} ...
conda-env: error: invalid choice: 'config' (choose from 'create', 'export', 'list', 'remove', 'update')


but its the command on the conda website!!!!

I've also tried setting the variable in the _config.yml file, but still get the error when starting training

Can anyone help? :roll:

User avatar
torzdf
Posts: 559
Joined: Fri Jul 12, 2019 12:53 am
Answers: 88
Has thanked: 17 times
Been thanked: 121 times

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by torzdf » Tue Feb 25, 2020 12:24 am

Yeah, this is an inconsistent bug with no apparent pattern between GPU/Driver/OS

We have you covered though (outside of Sort).

For Extract go Settings > Configure Extract Settings and check Allow Growth.
For Train and Convert enable the Allow Growth option in the main options panel.
My word is final

User avatar
graza
Posts: 2
Joined: Mon Feb 24, 2020 7:41 pm

Re: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Post by graza » Sat Mar 07, 2020 2:01 pm

Thanks for the reply. I can't believe I didn't spot that option on the setings panel! Am now currently training my first model!!!!! :D

Post Reply