Can't engage GPU

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
anvar
Posts: 5
Joined: Sun Jun 14, 2020 12:24 pm

Can't engage GPU

Post by anvar »

Hi, there.

I'm running faceswap gui on windows 10.
It fails to engage my GPU for both extraction and training (GeForce GTX 1650).
In case of extraction procedure I checked "Allow Growth" feature and used CPU (It's not a big deal).

But as you know, training without GPU will take days. What am I doing wrong?

This is error I get:

Code: Select all

06/14/2020 20:30:32 INFO     Loading Trainer from Original plugin...
06/14/2020 20:30:34 INFO     Enabled TensorBoard Logging
06/14/2020 20:31:04 CRITICAL Error caught! Exiting...
06/14/2020 20:31:04 ERROR    Caught exception in thread: '_training_0'
06/14/2020 20:31:04 ERROR    You do not have enough GPU memory available to train the selected model at the selected settings. You can try a number of things:
06/14/2020 20:31:04 ERROR    1) Close any other application that is using your GPU (web browsers are particularly bad for this).
06/14/2020 20:31:04 ERROR    2) Lower the batchsize (the amount of images fed into the model each iteration).
06/14/2020 20:31:04 ERROR    3) Try 'Memory Saving Gradients' and/or 'Optimizer Savings' and/or 'Ping Pong Training'.
06/14/2020 20:31:04 ERROR    4) Use a more lightweight model, or select the model's 'LowMem' option (in config) if it has one.
Process exited.

And this is the Output system information:

Code: Select all

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         127d3db Dependencies update (#1028)
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1650
gpu_devices_active:  GPU_0
gpu_driver:          446.14
gpu_vram:            GPU_0: 4096MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.18362-SP0
os_release:          10
py_command:          C:\Users\admin\faceswap/faceswap.py gui
py_conda_version:    conda 4.8.3
py_implementation:   CPython
py_version:          3.7.7
py_virtual_env:      True
sys_cores:           12
sys_processor:       Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram:             Total: 32495MB, Available: 20090MB, Used: 12404MB, Free: 20090MB

=============== Pip Packages ===============
absl-py==0.9.0
astor==0.8.0
blinker==1.4
cachetools==4.1.0
certifi==2020.4.5.1
cffi==1.14.0
chardet==3.0.4
click==7.1.2
cloudpickle==1.4.1
cryptography==2.9.2
cycler==0.10.0
cytoolz==0.10.1
dask==2.17.2
decorator==4.4.2
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.2.2
google-auth==1.14.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.27.2
h5py==2.10.0
idna==2.9
imageio==2.8.0
imageio-ffmpeg==0.4.2
joblib==0.15.1
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.2.0
Markdown==3.1.1
matplotlib==3.1.3
mkl-fft==1.0.15
mkl-random==1.1.1
mkl-service==2.3.0
networkx==2.4
numpy==1.18.1
nvidia-ml-py3==7.352.1
oauthlib==3.1.0
olefile==0.46
opencv-python==4.2.0.34
opt-einsum==3.1.0
pathlib==1.0.1
Pillow==7.1.2
protobuf==3.12.3
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycparser==2.20
PyJWT==1.7.1
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyreadline==2.1
PySocks==1.7.1
python-dateutil==2.8.1
PyWavelets==1.1.1
pywin32==227
PyYAML==5.3.1
requests==2.23.0
requests-oauthlib==1.3.0
rsa==4.0
scikit-image==0.16.2
scikit-learn==0.22.1
scipy==1.4.1
six==1.15.0
tensorboard==2.2.1
tensorboard-plugin-wit==1.6.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1
termcolor==1.1.0
toolz==0.10.0
toposort==1.5
tornado==6.0.4
tqdm==4.46.0
urllib3==1.25.8
Werkzeug==0.16.1
win-inet-pton==1.1.0
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\admin\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu  
absl-py 0.9.0 py37_0
astor 0.8.0 py37_0
blas 1.0 mkl
blinker 1.4 py37_0
ca-certificates 2020.1.1 0
cachetools 4.1.0 py_1
certifi 2020.4.5.1 py37_0
cffi 1.14.0 py37h7a1dbc1_0
chardet 3.0.4 py37_1003
click 7.1.2 py_0
cloudpickle 1.4.1 py_0
cryptography 2.9.2 py37h7a1dbc1_0
cudatoolkit 10.0.130 0
cudnn 7.6.5 cuda10.0_0
cycler 0.10.0 py37_0
cytoolz 0.10.1 py37he774522_0
dask-core 2.17.2 py_0
decorator 4.4.2 py_0
fastcluster 1.1.26 pypi_0 pypi ffmpy 0.2.3 pypi_0 pypi freetype 2.9.1 ha9979f8_1
gast 0.2.2 py37_0
git 2.23.0 h6bb4b03_0
google-auth 1.14.1 py_0
google-auth-oauthlib 0.4.1 py_2
google-pasta 0.2.0 py_0
grpcio 1.27.2 py37h351948d_0
h5py 2.10.0 py37h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.9 py_1
imageio 2.8.0 py_0
imageio-ffmpeg 0.4.2 pypi_0 pypi intel-openmp 2020.1 216
joblib 0.15.1 py_0
jpeg 9b hb83a4c4_2
keras 2.2.4 0
keras-applications 1.0.8 py_0
keras-base 2.2.4 py37_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.2.0 py37h74a9793_0
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.12.3 h7bd577a_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 h62dcd97_0
markdown 3.1.1 py37_0
matplotlib 3.1.3 py37_0
matplotlib-base 3.1.3 py37h64f37c6_0
mkl 2020.1 216
mkl-service 2.3.0 py37hb782905_0
mkl_fft 1.0.15 py37h14836fe_0
mkl_random 1.1.1 py37h47e9c7a_0
networkx 2.4 py_0
numpy 1.18.1 py37h93ca92e_0
numpy-base 1.18.1 py37hc3f5095_1
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 py_0
olefile 0.46 py37_0
opencv-python 4.2.0.34 pypi_0 pypi openssl 1.1.1g he774522_0
opt_einsum 3.1.0 py_0
pathlib 1.0.1 py37_1
pillow 7.1.2 py37hcc1f983_0
pip 20.0.2 py37_3
protobuf 3.12.3 py37h33f27b4_0
psutil 5.7.0 py37he774522_0
pyasn1 0.4.8 py_0
pyasn1-modules 0.2.7 py_0
pycparser 2.20 py_0
pyjwt 1.7.1 py37_0
pyopenssl 19.1.0 py37_0
pyparsing 2.4.7 py_0
pyqt 5.9.2 py37h6538335_2
pyreadline 2.1 py37_1
pysocks 1.7.1 py37_0
python 3.7.7 h81c818b_4
python-dateutil 2.8.1 py_0
pywavelets 1.1.1 py37he774522_0
pywin32 227 py37he774522_1
pyyaml 5.3.1 py37he774522_0
qt 5.9.7 vc14h73c81de_0
requests 2.23.0 py37_0
requests-oauthlib 1.3.0 py_0
rsa 4.0 py_0
scikit-image 0.16.2 py37h47e9c7a_0
scikit-learn 0.22.1 py37h6288b17_0
scipy 1.4.1 py37h9439919_0
setuptools 47.1.1 py37_0
sip 4.19.8 py37h6538335_0
six 1.15.0 py_0
sqlite 3.31.1 h2a8f88b_1
tensorboard 2.2.1 pyh532a8cf_0
tensorboard-plugin-wit 1.6.0 py_0
tensorflow 1.15.0 gpu_py37hc3743a6_0
tensorflow-base 1.15.0 gpu_py37h1afeea4_0
tensorflow-estimator 1.15.1 pyh2649769_0
tensorflow-gpu 1.15.0 h0d30ee6_0
termcolor 1.1.0 py37_1
tk 8.6.8 hfa6e2cd_0
toolz 0.10.0 py_0
toposort 1.5 pypi_0 pypi tornado 6.0.4 py37he774522_1
tqdm 4.46.0 py_0
urllib3 1.25.8 py37_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_2
werkzeug 0.16.1 py_0
wheel 0.34.2 py37_0
win_inet_pton 1.1.0 py37_0
wincertstore 0.2 py37_0
wrapt 1.12.1 py37he774522_1
xz 5.2.5 h62dcd97_0
yaml 0.1.7 hc54c509_2
zlib 1.2.11 h62dcd97_4
zstd 1.4.4 ha9fde0e_3 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: unsharp_mask amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 70 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] coverage: 68.75 mask_type: none mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False icnr_init: False conv_aware_init: False reflect_padding: False penalized_mask_loss: True loss_function: mae learning_rate: 5e-05 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Can't engage GPU

Post by torzdf »

That looks like it is engaging your GPU but you don't have enough VRAM available.

Please provide the full crash report.

My word is final

User avatar
anvar
Posts: 5
Joined: Sun Jun 14, 2020 12:24 pm

Re: Can't engage GPU

Post by anvar »

I'm not running anything heavy. All browsers are off.

This is the full report:

Code: Select all

06/16/2020 21:57:00 MainProcess     MainThread      logger          log_setup                 INFO     Log level set to: INFO
06/16/2020 21:57:02 MainProcess     MainThread      train           _get_images               INFO     Model A Directory: C:\faceswap\dataset1\inputA
06/16/2020 21:57:02 MainProcess     MainThread      train           _get_images               INFO     Model B Directory: C:\faceswap\dataset1\inputB
06/16/2020 21:57:02 MainProcess     MainThread      train           process                   INFO     Training data directory: C:\faceswap\dataset1\modelOriginal
06/16/2020 21:57:02 MainProcess     MainThread      train           _monitor                  INFO     ===================================================
06/16/2020 21:57:02 MainProcess     MainThread      train           _monitor                  INFO       Starting
06/16/2020 21:57:02 MainProcess     MainThread      train           _monitor                  INFO       Press 'Stop' to save and quit
06/16/2020 21:57:02 MainProcess     MainThread      train           _monitor                  INFO     ===================================================
06/16/2020 21:57:03 MainProcess     _training_0     train           _training                 INFO     Loading data, this may take a while...
06/16/2020 21:57:03 MainProcess     _training_0     plugin_loader   _import                   INFO     Loading Model from Original plugin...
06/16/2020 21:57:03 MainProcess     _training_0     _base           load                      INFO     No existing state file found. Generating.
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     _base           load_models               INFO     Creating new 'original' model in folder: 'C:\faceswap\dataset1\modelOriginal'
06/16/2020 21:57:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     plugin_loader   _import                   INFO     Loading Trainer from Original plugin...
06/16/2020 21:57:06 MainProcess     _training_0     _base           _set_tensorboard          INFO     Enabled TensorBoard Logging
06/16/2020 21:57:07 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n
06/16/2020 21:57:07 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n
06/16/2020 21:57:30 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
06/16/2020 21:57:30 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    You do not have enough GPU memory available to train the selected model at the selected settings. You can try a number of things:
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    1) Close any other application that is using your GPU (web browsers are particularly bad for this).
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    2) Lower the batchsize (the amount of images fed into the model each iteration).
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    3) Try 'Memory Saving Gradients' and/or 'Optimizer Savings' and/or 'Ping Pong Training'.
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    4) Use a more lightweight model, or select the model's 'LowMem' option (in config) if it has one.
User avatar
anvar
Posts: 5
Joined: Sun Jun 14, 2020 12:24 pm

Re: Can't engage GPU

Post by anvar »

torzdf wrote: Mon Jun 15, 2020 9:55 am

That looks like it is engaging your GPU but you don't have enough VRAM available.

Please provide the full crash report.

Code: Select all

06/16/2020 21:57:00 MainProcess     MainThread      logger          log_setup                 INFO     Log level set to: INFO
06/16/2020 21:57:02 MainProcess     MainThread      train           _get_images               INFO     Model A Directory: C:\faceswap\dataset1\inputA
06/16/2020 21:57:02 MainProcess     MainThread      train           _get_images               INFO     Model B Directory: C:\faceswap\dataset1\inputB
06/16/2020 21:57:02 MainProcess     MainThread      train           process                   INFO     Training data directory: C:\faceswap\dataset1\modelOriginal
06/16/2020 21:57:02 MainProcess     MainThread      train           _monitor                  INFO     ===================================================
06/16/2020 21:57:02 MainProcess     MainThread      train           _monitor                  INFO       Starting
06/16/2020 21:57:02 MainProcess     MainThread      train           _monitor                  INFO       Press 'Stop' to save and quit
06/16/2020 21:57:02 MainProcess     MainThread      train           _monitor                  INFO     ===================================================
06/16/2020 21:57:03 MainProcess     _training_0     train           _training                 INFO     Loading data, this may take a while...
06/16/2020 21:57:03 MainProcess     _training_0     plugin_loader   _import                   INFO     Loading Model from Original plugin...
06/16/2020 21:57:03 MainProcess     _training_0     _base           load                      INFO     No existing state file found. Generating.
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n
06/16/2020 21:57:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     _base           load_models               INFO     Creating new 'original' model in folder: 'C:\faceswap\dataset1\modelOriginal'
06/16/2020 21:57:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n
06/16/2020 21:57:04 MainProcess     _training_0     plugin_loader   _import                   INFO     Loading Trainer from Original plugin...
06/16/2020 21:57:06 MainProcess     _training_0     _base           _set_tensorboard          INFO     Enabled TensorBoard Logging
06/16/2020 21:57:07 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n
06/16/2020 21:57:07 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n
06/16/2020 21:57:30 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
06/16/2020 21:57:30 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    You do not have enough GPU memory available to train the selected model at the selected settings. You can try a number of things:
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    1) Close any other application that is using your GPU (web browsers are particularly bad for this).
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    2) Lower the batchsize (the amount of images fed into the model each iteration).
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    3) Try 'Memory Saving Gradients' and/or 'Optimizer Savings' and/or 'Ping Pong Training'.
06/16/2020 21:57:30 MainProcess     MainThread      launcher        execute_script            ERROR    4) Use a more lightweight model, or select the model's 'LowMem' option (in config) if it has one.
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Can't engage GPU

Post by bryanlyon »

This is not the full crash log. The crash log is saved to a file in the faceswap folder labelled as crash_report.[YEAR].[MONTH].[DAY].[TIMESTAMP].log . Attach that file to a post here for us to examine.

User avatar
anvar
Posts: 5
Joined: Sun Jun 14, 2020 12:24 pm

Re: Can't engage GPU

Post by anvar »

There is no such file in any folder.

User avatar
djandg
Posts: 43
Joined: Mon Dec 09, 2019 7:00 pm
Has thanked: 4 times
Been thanked: 2 times

Re: Can't engage GPU

Post by djandg »

On your machine you should find the crash logs in

C:\Users\admin\faceswap\

User avatar
anvar
Posts: 5
Joined: Sun Jun 14, 2020 12:24 pm

Re: Can't engage GPU

Post by anvar »

I have only one log file in that folder (looked in subfolders as well):
faceswap.log
and the contents is same as above:

Code: Select all

06/16/2020 21:52:00 MainProcess     MainThread      logger          log_setup                 INFO     Log level set to: INFO
06/16/2020 21:52:02 MainProcess     MainThread      train           _get_images               INFO     Model A Directory: C:\faceswap\dataset1\inputA
06/16/2020 21:52:02 MainProcess     MainThread      train           _get_images               INFO     Model B Directory: C:\faceswap\dataset1\inputB
06/16/2020 21:52:02 MainProcess     MainThread      train           process                   INFO     Training data directory: C:\faceswap\dataset1\modelOriginal
06/16/2020 21:52:02 MainProcess     MainThread      train           _monitor                  INFO     ===================================================
06/16/2020 21:52:02 MainProcess     MainThread      train           _monitor                  INFO       Starting
06/16/2020 21:52:02 MainProcess     MainThread      train           _monitor                  INFO       Press 'Stop' to save and quit
06/16/2020 21:52:02 MainProcess     MainThread      train           _monitor                  INFO     ===================================================
06/16/2020 21:52:03 MainProcess     _training_0     train           _training                 INFO     Loading data, this may take a while...
06/16/2020 21:52:03 MainProcess     _training_0     plugin_loader   _import                   INFO     Loading Model from Original plugin...
06/16/2020 21:52:03 MainProcess     _training_0     _base           load                      INFO     No existing state file found. Generating.
06/16/2020 21:52:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n
06/16/2020 21:52:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n
06/16/2020 21:52:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n
06/16/2020 21:52:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n
06/16/2020 21:52:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n
06/16/2020 21:52:03 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n
06/16/2020 21:52:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n
06/16/2020 21:52:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n
06/16/2020 21:52:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n
06/16/2020 21:52:04 MainProcess     _training_0     _base           load_models               INFO     Creating new 'original' model in folder: 'C:\faceswap\dataset1\modelOriginal'
06/16/2020 21:52:04 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n
06/16/2020 21:52:04 MainProcess     _training_0     plugin_loader   _import                   INFO     Loading Trainer from Original plugin...
06/16/2020 21:52:06 MainProcess     _training_0     _base           _set_tensorboard          INFO     Enabled TensorBoard Logging
06/16/2020 21:52:06 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n
06/16/2020 21:52:07 MainProcess     _training_0     module_wrapper  _tfmw_add_deprecation_warning DEBUG    From C:\Users\admin\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n
06/16/2020 21:52:30 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
06/16/2020 21:52:30 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
06/16/2020 21:52:30 MainProcess     MainThread      launcher        execute_script            ERROR    You do not have enough GPU memory available to train the selected model at the selected settings. You can try a number of things:
06/16/2020 21:52:30 MainProcess     MainThread      launcher        execute_script            ERROR    1) Close any other application that is using your GPU (web browsers are particularly bad for this).
06/16/2020 21:52:30 MainProcess     MainThread      launcher        execute_script            ERROR    2) Lower the batchsize (the amount of images fed into the model each iteration).
06/16/2020 21:52:30 MainProcess     MainThread      launcher        execute_script            ERROR    3) Try 'Memory Saving Gradients' and/or 'Optimizer Savings' and/or 'Ping Pong Training'.
06/16/2020 21:52:30 MainProcess     MainThread      launcher        execute_script            ERROR    4) Use a more lightweight model, or select the model's 'LowMem' option (in config) if it has one.

There is nor crash report anywhere. I did systemwide search, nothing.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Can't engage GPU

Post by torzdf »

Extraction should definitely not fail, and should generate a crash report (the error message you posted is from training). If you are still having issues with Extract, then please provide the crash report....

For training, you should try to follow the steps listed in the original output message.

My word is final

Locked