First run... cannot create cudnn handle

Installing and setting up FaceSwap


Forum rules

Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Grassone
Posts: 45
Joined: Sun Apr 19, 2020 7:32 pm
Has thanked: 6 times
Been thanked: 2 times

First run... cannot create cudnn handle

Post by Grassone »

First run on Linux after installation.

No luck.
This is the console output...

Code: Select all

2020-09-26 00:39:05.333060: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-09-26 00:39:05.337996: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-09-26 00:39:05.342392: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-09-26 00:39:05.346375: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
09/26/2020 00:39:06 CRITICAL Error caught! Exiting...
09/26/2020 00:39:06 ERROR    Caught exception in thread: '_training_0'
09/26/2020 00:39:07 ERROR    Got Exception on main handler:

And this is a suspicious diagnostic in the crash log...

tensorflow.python.framework.errors_impl.UnknownError:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node original/encoder_1/conv_128_0_conv2d/Conv2D (defined at /faceswap/plugins/train/trainer/_base.py:212) ]] [Op:__inference_train_function_8730]

Function call stack:
train_function


============ System Information ============
encoding:            UTF-8
git_branch:          Not Found
git_commits:         Not Found
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce RTX 2070
gpu_devices_active:  GPU_0
gpu_driver:          450.66
gpu_vram:            GPU_0: 7981MB
os_machine:          x86_64
os_platform:         Linux-5.4.0-48-generic-x86_64-with-glibc2.10
os_release:          5.4.0-48-generic
py_command:          /home/fulvio-amd/faceswap/faceswap.py train -A /home/fulvio-amd/Documenti/Ritorno a Casablanca/_HB_emasked -ala /home/fulvio-amd/Documenti/Ritorno a Casablanca/_HB_emasked/alignments_merged_20200604_190124.fsa -B /home/fulvio-amd/Documenti/Ritorno a Casablanca/_FG_emasked -alb /home/fulvio-amd/Documenti/Ritorno a Casablanca/_FG_emasked/alignments_merged_20200612_190817.fsa -m /home/fulvio-amd/Documenti/Ritorno a Casablanca/__Modelli -t original -bs 16 -it 1000000 -s 250 -ss 25000 -ps 50 -L INFO -gui
py_conda_version:    conda 4.8.5
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           16
sys_processor:       x86_64
sys_ram:             Total: 15938MB, Available: 12061MB, Used: 3461MB, Free: 718MB

=============== Pip Packages ===============
absl-py @ file:///tmp/build/80754af9/absl-py_1600297518631/work
astunparse==1.6.3
blinker==1.4
brotlipy==0.7.0
cachetools @ file:///tmp/build/80754af9/cachetools_1596822027882/work
certifi==2020.6.20
cffi @ file:///tmp/build/80754af9/cffi_1600699146221/work
chardet==3.0.4
click==7.1.2
cryptography @ file:///tmp/build/80754af9/cryptography_1601046815590/work
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.3.3
google-auth @ file:///tmp/build/80754af9/google-auth_1600960338579/work
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio @ file:///tmp/build/80754af9/grpcio_1597424474635/work
h5py @ file:///tmp/build/80754af9/h5py_1593454122442/work
idna @ file:///tmp/build/80754af9/idna_1593446292537/work
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg==0.4.2
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1593446406207/work
joblib==0.16.0
Keras-Preprocessing==1.1.0
kiwisolver==1.2.0
Markdown @ file:///tmp/build/80754af9/markdown_1597433240441/work
matplotlib @ file:///tmp/build/80754af9/matplotlib-base_1592846008246/work
numpy==1.19.2
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.4.0.44
opt-einsum==3.1.0
pathlib==1.0.1
Pillow @ file:///tmp/build/80754af9/pillow_1594307295532/work
protobuf==3.12.4
psutil @ file:///tmp/build/80754af9/psutil_1598370257551/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
PyJWT==1.7.1
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1594392929924/work
pyparsing==2.4.7
PySocks==1.7.1
python-dateutil==2.8.1
requests @ file:///tmp/build/80754af9/requests_1592841827918/work
requests-oauthlib==1.3.0
rsa @ file:///tmp/build/80754af9/rsa_1596998415516/work
scikit-learn==0.23.2
scipy==1.5.2
sip==4.19.13
six==1.15.0
tensorboard==2.2.1
tensorboard-plugin-wit==1.6.0
tensorflow==2.2.0
tensorflow-estimator==2.2.0
termcolor==1.1.0
threadpoolctl==2.1.0
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1600709023549/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1597086586889/work
Werkzeug==1.0.1
wrapt==1.12.1
zipp==3.1.0

============== Conda Packages ==============
# packages in environment at /home/fulvio-amd/miniconda3/envs/faceswap:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_tflow_select 2.1.0 gpu
absl-py 0.10.0 py38_0
astunparse 1.6.3 py_0
blas 1.0 openblas
blinker 1.4 py38_0
brotlipy 0.7.0 py38h7b6447c_1000
c-ares 1.16.1 h7b6447c_0
ca-certificates 2020.7.22 0
cachetools 4.1.1 py_0
certifi 2020.6.20 py38_0
cffi 1.14.3 py38he30daa8_0
chardet 3.0.4 py38_1003
click 7.1.2 py_0
cryptography 3.1.1 py38h1ba5d50_0
cudatoolkit 10.1.243 h6bb024c_0
cudnn 7.6.5 cuda10.1_0
cupti 10.1.168 0
cycler 0.10.0 py38_0
dbus 1.13.16 hb2f20db_0
expat 2.2.9 he6710b0_2
fastcluster 1.1.26 py38hcb8c335_1 conda-forge fontconfig 2.13.0 h9420a91_0
freetype 2.10.2 h5ab3b9f_0
gast 0.3.3 py_0
git 2.23.0 pl526hacde149_0
glib 2.65.0 h3eb4bd4_0
google-auth 1.21.3 py_0
google-auth-oauthlib 0.4.1 py_2
google-pasta 0.2.0 py_0
grpcio 1.31.0 py38hf8bcb03_0
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb31296c_0
h5py 2.10.0 py38hd6299e0_1
hdf5 1.10.6 hb1b8bf9_0
icu 58.2 he6710b0_3
idna 2.10 py_0
imageio 2.9.0 py_0
importlib-metadata 1.7.0 py38_0
jpeg 9b h024ee3a_2
keras-preprocessing 1.1.0 py_1
kiwisolver 1.2.0 py38hfd86e86_0
krb5 1.18.2 h173b8e3_0
lcms2 2.11 h396b838_0
ld_impl_linux-64 2.33.1 h53a641e_7
libblas 3.8.0 17_openblas conda-forge libcblas 3.8.0 17_openblas conda-forge libcurl 7.71.1 h20c2e04_1
libedit 3.1.20191231 h14c3975_1
libffi 3.3 he6710b0_2
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.5.0 hdf63c60_16 conda-forge liblapack 3.8.0 17_openblas conda-forge libopenblas 0.3.10 pthreads_hb3c22a3_4 conda-forge libpng 1.6.37 hbc83047_0
libprotobuf 3.12.4 hd408876_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.1.0 h2733197_1
libuuid 1.0.3 h1bed415_2
libxcb 1.14 h7b6447c_0
libxml2 2.9.10 he19cac6_1
lz4-c 1.9.2 he6710b0_1
markdown 3.2.2 py38_0
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38hef1b27d_0
ncurses 6.2 he6710b0_1
numpy 1.19.1 py38hbc27379_2 conda-forge nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 py_0
olefile 0.46 py_0
openssl 1.1.1h h7b6447c_0
opt_einsum 3.1.0 py_0
pathlib 1.0.1 py_1
pcre 8.44 he6710b0_0
perl 5.26.2 h14c3975_0
pillow 7.2.0 py38hb39fc2d_0
pip 20.2.2 py38_0
protobuf 3.12.4 py38he6710b0_0
psutil 5.7.2 py38h7b6447c_0
pyasn1 0.4.8 py_0
pyasn1-modules 0.2.8 py_0
pycparser 2.20 py_2
pyjwt 1.7.1 py38_0
pyopenssl 19.1.0 py_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py38h05f1152_4
pysocks 1.7.1 py38_0
python 3.8.5 h7579374_1
python-dateutil 2.8.1 py_0
python_abi 3.8 1_cp38 conda-forge qt 5.9.7 h5867ecd_1
readline 8.0 h7b6447c_0
requests 2.24.0 py_0
requests-oauthlib 1.3.0 py_0
rsa 4.6 py_0
scipy 1.5.2 py38habc2bb6_0
setuptools 49.6.0 py38_0
sip 4.19.13 py38he6710b0_0
six 1.15.0 py_0
sqlite 3.33.0 h62c20be_0
tensorboard 2.2.1 pyh532a8cf_0
tensorboard-plugin-wit 1.6.0 py_0
tensorflow 2.2.0 gpu_py38hb782248_0
tensorflow-base 2.2.0 gpu_py38h83e3d50_0
tensorflow-estimator 2.2.0 pyh208ff02_0
tensorflow-gpu 2.2.0 h0d30ee6_0
termcolor 1.1.0 py38_1
tk 8.6.10 hbc83047_0
tornado 6.0.4 py38h7b6447c_1
tqdm 4.49.0 py_0
urllib3 1.25.10 py_0
werkzeug 1.0.1 py_0
wheel 0.35.1 py_0
wrapt 1.12.1 py38h7b6447c_1
xz 5.2.5 h7b6447c_0
zipp 3.1.0 py_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.5 h9ceee32_0 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- extract.ini --------- [global] allow_growth: False [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 70 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 [align.fan] batch-size: 12 --------- train.ini --------- [global] coverage: 68.75 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 reflect_padding: False allow_growth: False mixed_precision: False convert_batchsize: 16 [global.loss] loss_function: ssim mask_loss_function: mse l2_reg_term: 100 eye_multiplier: 3 mouth_multiplier: 2 penalized_mask_loss: True mask_type: extended mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False [model.villain] lowmem: False [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.dlight] features: best details: good output_size: 256 [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.dfl_h128] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 disable_warp: False color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 10 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [scaling.sharpen] method: unsharp_mask amount: 150 radius: 0.3 threshold: 5.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate
User avatar
abigflea
Posts: 182
Joined: Sat Feb 22, 2020 10:59 pm
Answers: 2
Has thanked: 20 times
Been thanked: 62 times

Re: First run... cannot create cudnn handle

Post by abigflea »

#1 enable Allow growth.
Let me read through the log if there is something else

:o I dunno what I'm doing :shock:
2X RTX 3090 : RTX 3080 : RTX: 2060 : 2x RTX 2080 Super : Ghetto 1060

User avatar
abigflea
Posts: 182
Joined: Sat Feb 22, 2020 10:59 pm
Answers: 2
Has thanked: 20 times
Been thanked: 62 times

Re: First run... cannot create cudnn handle

Post by abigflea »

Allow growth and I see you have a 2070 (me too!)

You are starting with the original Model?

:o I dunno what I'm doing :shock:
2X RTX 3090 : RTX 3080 : RTX: 2060 : 2x RTX 2080 Super : Ghetto 1060

User avatar
Grassone
Posts: 45
Joined: Sun Apr 19, 2020 7:32 pm
Has thanked: 6 times
Been thanked: 2 times

Re: First run... cannot create cudnn handle

Post by Grassone »

Thanks.
1) I enabled the "Allow growth option" and now it works.
2) I started with the basic-vanilla-plain original model to have a comparison with my old GTX card (2 gig.) The 2070 is probably 10 times faster...

Locked