application wont run on GPU, low EGs

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
andi_zein
Posts: 4
Joined: Fri Jul 03, 2020 6:00 pm
Has thanked: 1 time

application wont run on GPU, low EGs

Post by andi_zein »

hello, at first i thought it was okay to extract on CPU (because i cant make it run on GPU), but when i start to train, i realized it takes long time to compute. i've tried many solutions like:

  • reinstall and tick on GPU (Nvidia)
  • conda install tensorflow-gpu==1.15.0
  • install cuda cudnn from nvidia
  • install cuda cudnn from conda packages
  • update faceswap
  • check nvml.dll in system32 or nvidia folder
    is there any solution i could try? thank you,

Image

Image

Code: Select all

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         6595cdf Add icons
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce 820M
gpu_devices_active:  GPU_0
gpu_driver:          385.54
gpu_vram:            GPU_0: 2048MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.19041-SP0
os_release:          10
py_command:          C:\Users\hp\faceswap/faceswap.py gui
py_conda_version:    conda 4.8.3
py_implementation:   CPython
py_version:          3.7.7
py_virtual_env:      True
sys_cores:           4
sys_processor:       Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
sys_ram:             Total: 10176MB, Available: 5175MB, Used: 5000MB, Free: 5175MB

=============== Pip Packages ===============
absl-py==0.9.0
astor==0.8.0
astroid==2.2.5
astunparse==1.6.3
blinker==1.4
Brotli==1.0.7
brotlipy==0.7.0
cachetools==4.1.0
certifi==2020.6.20
cffi==1.14.0
chardet==3.0.4
click==7.1.2
cloudpickle==1.4.1
cryptography==2.9.2
cycler==0.10.0
cytoolz==0.10.1
dask @ file:///tmp/build/80754af9/dask-core_1592842333140/work
decorator==4.4.2
enum34==1.1.10
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.2.2
google-auth==1.14.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.27.2
h5py==2.10.0
idna @ file:///tmp/build/80754af9/idna_1593446292537/work
imageio==2.8.0
imageio-ffmpeg==0.4.2
isort==4.3.21
joblib==0.15.1
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.2.0
lazy-object-proxy==1.4.1
Markdown==3.1.1
matplotlib @ file:///C:/ci/matplotlib-base_1592846084747/work
mccabe==0.6.1
mkl-fft==1.1.0
mkl-random==1.1.1
mkl-service==2.3.0
networkx==2.4
numpy==1.18.5
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.2.0.34
opt-einsum==3.1.0
pathlib==1.0.1
Pillow==7.1.2
plaidml==0.7.0
plaidml-keras==0.7.0
protobuf==3.12.3
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycparser==2.20
PyJWT==1.7.1
pylint==2.3.1
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyreadline==2.1
PySocks==1.7.1
python-dateutil==2.8.1
PyWavelets==1.1.1
pywin32==227
PyYAML==5.3.1
requests @ file:///tmp/build/80754af9/requests_1592841827918/work
requests-oauthlib==1.3.0
rsa==4.0
scikit-image==0.16.2
scikit-learn @ file:///C:/ci/scikit-learn_1592847564598/work
scipy @ file:///C:/ci/scipy_1592916958183/work
six==1.15.0
tensorboard==2.2.1
tensorboard-plugin-wit==1.6.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1
tensorflow-gpu==1.15.0
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
toolz==0.10.0
toposort==1.5
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1593446365756/work
typed-ast==1.4.0
urllib3==1.25.9
Werkzeug==0.16.1
win-inet-pton==1.1.0
wincertstore==0.2
wrapt==1.11.2

============== Conda Packages ==============
# packages in environment at C:\Users\hp\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu  
absl-py 0.9.0 py37_0
astor 0.8.0 py37_0
blas 1.0 mkl
blinker 1.4 py37_0
brotli 1.0.7 pypi_0 pypi brotlipy 0.7.0 py37he774522_1000
ca-certificates 2020.6.24 0
cachetools 4.1.0 py_1
certifi 2020.6.20 py37_0
cffi 1.14.0 py37h7a1dbc1_0
chardet 3.0.4 py37_1003
click 7.1.2 py_0
cloudpickle 1.4.1 py_0
cryptography 2.9.2 py37h7a1dbc1_0
cudatoolkit 10.0.130 0
cudnn 7.6.4 cuda10.0_0 anaconda cycler 0.10.0 py37_0
cytoolz 0.10.1 py37he774522_0
dask-core 2.19.0 py_0
decorator 4.4.2 py_0
enum34 1.1.10 pypi_0 pypi fastcluster 1.1.26 pypi_0 pypi ffmpy 0.2.3 pypi_0 pypi freetype 2.10.2 hd328e21_0
gast 0.2.2 py37_0
git 2.23.0 h6bb4b03_0
google-auth 1.14.1 py_0
google-auth-oauthlib 0.4.1 py_2
google-pasta 0.2.0 py_0
grpcio 1.27.2 py37h351948d_0
h5py 2.10.0 py37h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 py_0
imageio 2.8.0 py_0
imageio-ffmpeg 0.4.2 pypi_0 pypi intel-openmp 2020.1 216
joblib 0.15.1 py_0
jpeg 9b hb83a4c4_2
keras 2.2.4 0
keras-applications 1.0.8 py_0
keras-base 2.2.4 py37_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.2.0 py37h74a9793_0
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.12.3 h7bd577a_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 h62dcd97_0
markdown 3.1.1 py37_0
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py37h64f37c6_0
mkl 2020.1 216
mkl-service 2.3.0 py37hb782905_0
mkl_fft 1.1.0 py37h45dec08_0
mkl_random 1.1.1 py37h47e9c7a_0
networkx 2.4 py_0
numpy 1.18.5 py37h6530119_0
numpy-base 1.18.5 py37hc3f5095_0
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 py_0
olefile 0.46 py37_0
opencv-python 4.2.0.34 pypi_0 pypi openssl 1.1.1g he774522_0
opt_einsum 3.1.0 py_0
pathlib 1.0.1 py37_1
pillow 7.1.2 py37hcc1f983_0
pip 20.1.1 py37_1
plaidml 0.7.0 pypi_0 pypi plaidml-keras 0.7.0 pypi_0 pypi protobuf 3.12.3 py37h33f27b4_0
psutil 5.7.0 py37he774522_0
pyasn1 0.4.8 py_0
pyasn1-modules 0.2.7 py_0
pycparser 2.20 py_0
pyjwt 1.7.1 py37_0
pyopenssl 19.1.0 py37_0
pyparsing 2.4.7 py_0
pyqt 5.9.2 py37h6538335_2
pyreadline 2.1 py37_1
pysocks 1.7.1 py37_0
python 3.7.7 h81c818b_4
python-dateutil 2.8.1 py_0
pywavelets 1.1.1 py37he774522_0
pywin32 227 py37he774522_1
pyyaml 5.3.1 py37he774522_1
qt 5.9.7 vc14h73c81de_0
requests 2.24.0 py_0
requests-oauthlib 1.3.0 py_0
rsa 4.0 py_0
scikit-image 0.16.2 py37h47e9c7a_0
scikit-learn 0.23.1 py37h25d0782_0
scipy 1.5.0 py37h9439919_0
setuptools 47.3.1 py37_0
sip 4.19.8 py37h6538335_0
six 1.15.0 py_0
sqlite 3.32.3 h2a8f88b_0
tensorboard 2.2.1 pyh532a8cf_0
tensorboard-plugin-wit 1.6.0 py_0
tensorflow 1.15.0 gpu_py37hc3743a6_0
tensorflow-base 1.15.0 gpu_py37h1afeea4_0
tensorflow-estimator 1.15.1 pyh2649769_0
tensorflow-gpu 1.15.0 h0d30ee6_0
termcolor 1.1.0 py37_1
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
toolz 0.10.0 py_0
toposort 1.5 pypi_0 pypi tornado 6.0.4 py37he774522_1
tqdm 4.47.0 py_0
urllib3 1.25.9 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_3
werkzeug 0.16.1 py_0
wheel 0.34.2 py37_0
win_inet_pton 1.1.0 py37_0
wincertstore 0.2 py37_0
wrapt 1.12.1 py37he774522_1
xz 5.2.5 h62dcd97_0
yaml 0.2.5 he774522_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.4 ha9fde0e_3 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: unsharp_mask amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 70 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] coverage: 68.75 mask_type: vgg-obstructed mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False icnr_init: False conv_aware_init: True reflect_padding: False penalized_mask_loss: True loss_function: ssim learning_rate: 5e-05 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
by bryanlyon » Fri Jul 03, 2020 9:39 pm

Unfortunately an 820m with 2gb of vram is unlikely to be viable. I'm afraid that you're going to have to rely on CPU or cloud training.

Go to full post
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: application wont run on GPU, low EGs

Post by bryanlyon »

Unfortunately an 820m with 2gb of vram is unlikely to be viable. I'm afraid that you're going to have to rely on CPU or cloud training.

User avatar
police.bike
Posts: 22
Joined: Tue Jun 30, 2020 3:37 pm
Has thanked: 7 times
Been thanked: 5 times

Re: application wont run on GPU, low EGs

Post by police.bike »

My 2 cents... I had a similar lower end graphics and I stayed with Original Model. By adjusting coverage % to > 82% and opting ssim in loss function for training, i was able to get very good result.

User avatar
andi_zein
Posts: 4
Joined: Fri Jul 03, 2020 6:00 pm
Has thanked: 1 time

Re: application wont run on GPU, low EGs

Post by andi_zein »

bryanlyon wrote: Fri Jul 03, 2020 9:39 pm

Unfortunately an 820m with 2gb of vram is unlikely to be viable. I'm afraid that you're going to have to rely on CPU or cloud training.

ah, i see. thank you for the reply.
are there any characteristics between viable gpu and non viable GPU? maybe you have a kind of list?
oh yeah, about cloud training, oot, i've got usage limited and i already wait for few days, yet i still cant connect to GPU. :?

Last edited by andi_zein on Sat Jul 04, 2020 5:58 am, edited 1 time in total.
User avatar
andi_zein
Posts: 4
Joined: Fri Jul 03, 2020 6:00 pm
Has thanked: 1 time

Re: application wont run on GPU, low EGs

Post by andi_zein »

police.bike wrote: Sat Jul 04, 2020 2:52 am

My 2 cents... I had a similar lower end graphics and I stayed with Original Model. By adjusting coverage % to > 82% and opting ssim in loss function for training, i was able to get very good result.

good for you. how many iterations and how long does it takes?
i think i will look for a configuration which is faster to calculated. increasing the batch size looks promising. i dont know if adjusting coverage could make any different in time. cheers :D

User avatar
torzdf
Posts: 2671
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: application wont run on GPU, low EGs

Post by torzdf »

You may be able to train the Lightweight model on that card. It is specifically written for 2GB GPU cards, but it will be dependent on what else is using the GPU.

Try it with a very low batchsize (2) and if it trains, you can try upping the batchsize in increments.

If it doesn't train, try memory saving gradients and/or optimizer savings.

My word is final

User avatar
andi_zein
Posts: 4
Joined: Fri Jul 03, 2020 6:00 pm
Has thanked: 1 time

Re: application wont run on GPU, low EGs

Post by andi_zein »

torzdf wrote: Sat Jul 04, 2020 8:44 am

You may be able to train the Lightweight model on that card. It is specifically written for 2GB GPU cards, but it will be dependent on what else is using the GPU.

Try it with a very low batchsize (2) and if it trains, you can try upping the batchsize in increments.

If it doesn't train, try memory saving gradients and/or optimizer savings.

sounds good, doesn't work.
change model to Lightweight with batch size 2, no changes.
then tick memory saving gradients and/or optimizer savings, still no changes.
i think, i will just train it on google colab. thanks anyway. ;)

Locked