Cannot get NVIDIA GPU to be used on training (v

Installing and setting up FaceSwap


Forum rules

Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
jamesthegreat
Posts: 2
Joined: Sun Jan 10, 2021 8:44 pm

Cannot get NVIDIA GPU to be used on training (v

Post by jamesthegreat »

I cannot get the GPU to be used at all, it is 100% on CPU.

What I've done:
Used DDU to remove my (current) nvidia driver and installed v457.51
Installed (and re-installed) with NVIDIA GPU selected

Running detects the GPU which it didn't before the driver rollback and looks like it's upposed to be using the GPU, but I get the exact same EG/s that I did on the CPU and looking at GPU usage through various tools I don't see it kicking in.

Any suggestions?

Thanks

Code: Select all

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         15bbc95 Update Dockerfile (#1112)
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce RTX 2070 SUPER
gpu_devices_active:  GPU_0
gpu_driver:          457.51
gpu_vram:            GPU_0: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.19041-SP0
os_release:          10
py_command:          C:\Users\jball\faceswap/faceswap.py gui
py_conda_version:    conda 4.9.2
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           20
sys_processor:       Intel64 Family 6 Model 165 Stepping 5, GenuineIntel
sys_ram:             Total: 65310MB, Available: 54138MB, Used: 11171MB, Free: 54138MB

=============== Pip Packages ===============
absl-py==0.11.0
astunparse==1.6.3
cachetools==4.2.0
certifi==2020.12.5
chardet==4.0.0
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
flatbuffers==1.12
gast==0.3.3
google-auth==1.24.0
google-auth-oauthlib==0.4.2
google-pasta==0.2.0
grpcio==1.32.0
h5py==2.10.0
idna==2.10
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1609799311556/work
joblib @ file:///tmp/build/80754af9/joblib_1607970656719/work
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/ci/kiwisolver_1604014703538/work
Markdown==3.3.3
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1603466732592/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.5.1.48
opt-einsum==3.3.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1609786840597/work
protobuf==3.14.0
psutil @ file:///C:/ci/psutil_1598370330503/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
pywin32==227
requests==2.25.1
requests-oauthlib==1.3.0
rsa==4.7
scikit-learn @ file:///C:/ci/scikit-learn_1598377018496/work
scipy @ file:///C:/ci/scipy_1604596260408/work
sip==4.19.13
six @ file:///C:/ci/six_1605187374963/work
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow-estimator==2.4.0
tensorflow-gpu==2.4.0
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado @ file:///C:/ci/tornado_1606942392901/work
tqdm @ file:///tmp/build/80754af9/tqdm_1609788246169/work
typing-extensions==3.7.4.3
urllib3==1.26.2
Werkzeug==1.0.1
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\jball\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   0.11.0                   pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
ca-certificates 2020.12.8 haa95532_0
cachetools 4.2.0 pypi_0 pypi certifi 2020.12.5 py38haa95532_0
chardet 4.0.0 pypi_0 pypi cudatoolkit 10.1.243 h74a9793_0
cudnn 7.6.5 cuda10.1_0
cycler 0.10.0 py38_0
fastcluster 1.1.26 py38h251f6bf_2 conda-forge ffmpeg 4.3.1 ha925a31_0 conda-forge ffmpy 0.2.3 pypi_0 pypi flatbuffers 1.12 pypi_0 pypi freetype 2.10.4 hd328e21_0
gast 0.3.3 pypi_0 pypi git 2.23.0 h6bb4b03_0
google-auth 1.24.0 pypi_0 pypi google-auth-oauthlib 0.4.2 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.32.0 pypi_0 pypi h5py 2.10.0 pypi_0 pypi icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 pypi_0 pypi imageio 2.9.0 py_0
imageio-ffmpeg 0.4.3 pyhd8ed1ab_0 conda-forge intel-openmp 2020.2 254
joblib 1.0.0 pyhd3eb1b0_0
jpeg 9b hb83a4c4_2
keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.3.0 py38hd77b12b_0
libpng 1.6.37 h2a8f88b_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 hf4a77e7_3
markdown 3.3.3 pypi_0 pypi matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38h64f37c6_0
mkl 2020.2 256
mkl-service 2.3.0 py38h196d8e1_0
mkl_fft 1.2.0 py38h45dec08_0
mkl_random 1.1.1 py38h47e9c7a_0
numpy 1.19.2 py38hadc3359_0
numpy-base 1.19.2 py38ha3acd2a_0
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 pypi_0 pypi olefile 0.46 py_0
opencv-python 4.5.1.48 pypi_0 pypi openssl 1.1.1i h2bbff1b_0
opt-einsum 3.3.0 pypi_0 pypi pathlib 1.0.1 py_1
pillow 8.1.0 py38h4fa10fc_0
pip 20.3.3 py38haa95532_0
protobuf 3.14.0 pypi_0 pypi psutil 5.7.2 py38he774522_0
pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pyparsing 2.4.7 py_0
pyqt 5.9.2 py38ha925a31_4
python 3.8.5 h5fd99cc_1
python-dateutil 2.8.1 py_0
python_abi 3.8 1_cp38 conda-forge pywin32 227 py38he774522_1
qt 5.9.7 vc14h73c81de_0
requests 2.25.1 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rsa 4.7 pypi_0 pypi scikit-learn 0.23.2 py38h47e9c7a_0
scipy 1.5.2 py38h14eb087_0
setuptools 51.0.0 py38haa95532_2
sip 4.19.13 py38ha925a31_0
six 1.15.0 py38haa95532_0
sqlite 3.33.0 h2a8f88b_0
tensorboard 2.4.0 pypi_0 pypi tensorboard-plugin-wit 1.7.0 pypi_0 pypi tensorflow-estimator 2.4.0 pypi_0 pypi tensorflow-gpu 2.4.0 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
tornado 6.1 py38h2bbff1b_0
tqdm 4.55.1 pyhd3eb1b0_0
typing-extensions 3.7.4.3 pypi_0 pypi urllib3 1.26.2 pypi_0 pypi vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 1.0.1 pypi_0 pypi wheel 0.36.2 pyhd3eb1b0_0
wincertstore 0.2 py38_0
wrapt 1.12.1 pypi_0 pypi xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.5 h04227a9_0 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: none amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 70 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] centering: face coverage: 68.75 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 reflect_padding: False allow_growth: False mixed_precision: False convert_batchsize: 16 [global.loss] loss_function: ssim mask_loss_function: mse l2_reg_term: 100 eye_multiplier: 3 mouth_multiplier: 2 penalized_mask_loss: True mask_type: extended mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False [model.dfaker] output_size: 128 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 disable_warp: False color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Cannot get NVIDIA GPU to be used on training (v

Post by torzdf »

For some reason you have Tensorflow 2.4 installed.

Whilst Faceswap will run on this, it is not supported by the installer, so you would need to manually install Cuda/cuDNN at a global level (which I don't really recommend, and we won't provide direct support for).

My suggestion is to completely remove Faceswap + re-install from the installer which should bring the correct version (2.3) in along with the Cuda toolkit.

My word is final

User avatar
jamesthegreat
Posts: 2
Joined: Sun Jan 10, 2021 8:44 pm

Re: Cannot get NVIDIA GPU to be used on training (v

Post by jamesthegreat »

Wow what a difference - from 14 EG/S to over 300. Looks like it's running on the GPU now!

For reference if anyone else has the same problem I deleted the faceswap folder and re-installed and that's it, works perfectly now.

Thanks

Locked