immediate training error

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
SirDeepsALot
Posts: 35
Joined: Sat May 28, 2022 7:10 am
Has thanked: 7 times
Been thanked: 2 times

immediate training error

Post by SirDeepsALot »

I was having some problems with faceswap training crashing, so I uninstalled it (and miniconda) and reinstalled it. On reinstall it said there was no NVIDA card, when there definitely is. I uninstalled again and tried to start a training session based on the extraction I had finalised before the reinstall when I got this:

Code: Select all

Loading...
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\dkroker\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 1892, in __call__
    return self.func(*args)
  File "C:\Users\dkroker\faceswap\lib\gui\display.py", line 169, in _update_displaybook
    self._command_display(command)
  File "C:\Users\dkroker\faceswap\lib\gui\display.py", line 99, in _command_display
    build_tabs()
  File "C:\Users\dkroker\faceswap\lib\gui\display.py", line 123, in _train_tabs
    GraphDisplay(self, "graph", helptext, 5000)
  File "C:\Users\dkroker\faceswap\lib\gui\display_command.py", line 207, in __init__
    super().__init__(parent, tab_name, helptext, wait_time, command)
  File "C:\Users\dkroker\faceswap\lib\gui\display_page.py", line 173, in __init__
    DisplayPage.__init__(self, parent, tab_name, helptext)
  File "C:\Users\dkroker\faceswap\lib\gui\display_page.py", line 25, in __init__
    ttk.Frame.__init__(self, parent)
  File "C:\Users\dkroker\MiniConda3\envs\faceswap\lib\tkinter\ttk.py", line 735, in __init__
    Widget.__init__(self, master, "ttk::frame", kw)
  File "C:\Users\dkroker\MiniConda3\envs\faceswap\lib\tkinter\ttk.py", line 552, in __init__
    tkinter.Widget.__init__(self, master, widgetname, kw=kw)
  File "C:\Users\dkroker\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 2572, in __init__
    self.tk.call(
_tkinter.TclError: window name "!graphdisplay2" already exists in parent
Setting Faceswap backend to NVIDIA
08/26/2022 02:35:13 INFO     Log level set to: INFO
08/26/2022 02:35:19 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "C:\Users\dkroker\faceswap\lib\cli\launcher.py", line 199, in execute_script
    script = self._import_script()
  File "C:\Users\dkroker\faceswap\lib\cli\launcher.py", line 51, in _import_script
    module = import_module(mod)
  File "C:\Users\dkroker\MiniConda3\envs\faceswap\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "C:\Users\dkroker\faceswap\scripts\train.py", line 12, in <module>
    import cv2
ModuleNotFoundError: No module named 'cv2'
08/26/2022 02:35:19 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\dkroker\faceswap\crash_report.2022.08.26.023516392566.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Process exited.

crash file attached.

Attachments
crash_report.2022.08.26.023516392566.log
(23.4 KiB) Downloaded 96 times
User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: immediate training error

Post by torzdf »

Can you also post the contents of faceswap_setup.log

I really want to get to the bottom of why opencv sometimes doesn't install

Thanks.

My word is final

User avatar
SirDeepsALot
Posts: 35
Joined: Sat May 28, 2022 7:10 am
Has thanked: 7 times
Been thanked: 2 times

Re: immediate training error

Post by SirDeepsALot »

When I try and post the faceswap setup log it says the file is too big (it's a little over 2mb) so I"ve compressed it into a rar.

Attachments
faceswap_setup.rar
(71.18 KiB) Downloaded 100 times
User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: immediate training error

Post by torzdf »

Ok, several packages failed to install because of a network timeout.

It looks like this was an intermittent issue at your end rather than a permanent one, so you should hopefully be able to fix it by doing:

Start > Anaconda Prompt (cmd)

Then in the window that pops up:

Code: Select all

conda activate faceswap
cd faceswap
python setup.py --installer --nvidia

This will try to pull in any missing packages.

Keep an eye on the output to make sure the remaining packages show that they successfully install.

My word is final

User avatar
SirDeepsALot
Posts: 35
Joined: Sat May 28, 2022 7:10 am
Has thanked: 7 times
Been thanked: 2 times

Re: immediate training error

Post by SirDeepsALot »

I've done this, uninstalled and reinstalled and done this again - and each time it says it's "good to go". I'm not getting error messages specifically, but the prompt still says that I do not have a GPU:

08/27/2022 03:44:16 INFO Training data directory: F:\(DEEP)\Model\nadine to mik
08/27/2022 03:44:16 INFO ===================================================
08/27/2022 03:44:16 INFO Starting
08/27/2022 03:44:16 INFO ===================================================
08/27/2022 03:44:16 INFO Loading data, this may take a while...
08/27/2022 03:44:16 INFO Loading Model from Realface plugin...
08/27/2022 03:44:17 INFO No existing state file found. Generating.
08/27/2022 03:44:17 INFO Storing Mixed Precision compatible layers. Please ignore any following warnings about using mixed precision.
08/27/2022 03:44:17 WARNING Mixed precision compatibility check (mixed_float16): WARNING
The dtype policy mixed_float16 may run slowly because this machine does not have a GPU. Only Nvidia GPUs with compute capability of at least 7.0 run quickly with mixed_float16.
If you will use compatible GPU(s) not attached to this host, e.g. by running a multi-worker model, you can ignore this warning. This message will only be logged once
08/27/2022 03:44:22 INFO Loading Trainer from Original plugin...

User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: immediate training error

Post by torzdf »

I will need the latest output from help > output system info and the latest setup log, as this is a different error than before.

My word is final

User avatar
SirDeepsALot
Posts: 35
Joined: Sat May 28, 2022 7:10 am
Has thanked: 7 times
Been thanked: 2 times

Re: immediate training error

Post by SirDeepsALot »

here you go mate, my cpu is clocking out to 100% again so it's definitely using CPU isn't of GPU.

Code: Select all

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         326110f bugfix - timelapse image loader multithreading.py - typing + docs
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce GTX 1080
gpu_devices_active:  GPU_0
gpu_driver:          516.01
gpu_vram:            GPU_0: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.19044-SP0
os_release:          10
py_command:          C:\Users\dkroker\faceswap/faceswap.py gui
py_conda_version:    conda 4.14.0
py_implementation:   CPython
py_version:          3.9.12
py_virtual_env:      True
sys_cores:           12
sys_processor:       Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram:             Total: 32710MB, Available: 21911MB, Used: 10799MB, Free: 21911MB

=============== Pip Packages ===============
absl-py==1.2.0
astunparse==1.6.3
cachetools==5.2.0
certifi @ file:///C:/Windows/TEMP/abs_e9b7158a-aa56-4a5b-87b6-c00d295b01fanefpc8_o/croots/recipe/certifi_1655968940823/work/certifi
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1632508026186/work
colorama @ file:///C:/Windows/TEMP/abs_9439aeb1-0254-449a-96f7-33ab5eb17fc8apleb4yn/croots/recipe/colorama_1657009099097/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
dm-tree==0.1.5
fastcluster @ file:///D:/bld/fastcluster_1649783471014/work
ffmpy==0.3.0
flatbuffers==1.12
fonttools==4.25.0
gast==0.4.0
google-auth==2.11.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.47.0
h5py==3.7.0
imageio @ file:///C:/Windows/TEMP/abs_24c1b783-7540-4ca9-a1b1-0e8aa8e6ae64hb79ssux/croots/recipe/imageio_1658785038775/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1649960641006/work
importlib-metadata==4.12.0
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1633637554808/work
keras==2.9.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/ci/kiwisolver_1653292407425/work
libclang==14.0.6
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib @ file:///C:/ci/matplotlib-suite_1660169687702/work
mkl-fft==1.3.1
mkl-random @ file:///C:/ci/mkl_random_1626186184308/work
mkl-service==2.4.0
munkres==1.1.4
numexpr @ file:///C:/Windows/Temp/abs_e2036a32-9fe9-47f3-a04c-dbb1c232ba4b334exiur/croots/recipe/numexpr_1656940304835/work
numpy @ file:///C:/Windows/Temp/abs_2a1e1vbeag/croots/recipe/numpy_and_numpy_base_1659432712056/work
nvidia-ml-py==11.515.48
oauthlib==3.2.0
opencv-python==4.6.0.66
opt-einsum==3.3.0
packaging @ file:///tmp/build/80754af9/packaging_1637314298585/work
Pillow==9.2.0
ply==3.11
protobuf==3.19.4
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work
PyQt5==5.15.7
PyQt5-sip @ file:///C:/Windows/Temp/abs_d7gmd2jg8i/croots/recipe/pyqt-split_1659273064801/work/pyqt_sip
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==302
pywinpty @ file:///C:/ci_310/pywinpty_1644230983541/work/target/wheels/pywinpty-2.0.2-cp39-none-win_amd64.whl
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn @ file:///D:/bld/scikit-learn_1659726281030/work
scipy @ file:///C:/bld/scipy_1658811088396/work
sip @ file:///C:/Windows/Temp/abs_b8fxd17m2u/croots/recipe/sip_1659012372737/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.9.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.9.0
tensorflow-gpu==2.9.1
tensorflow-io-gcs-filesystem==0.26.0
tensorflow-probability @ file:///tmp/build/80754af9/tensorflow-probability_1633017132682/work
termcolor==1.1.0
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
tornado @ file:///C:/ci/tornado_1606924294691/work
tqdm @ file:///C:/ci/tqdm_1650636210717/work
typing_extensions @ file:///C:/Windows/TEMP/abs_dd2d0moa85/croots/recipe/typing_extensions_1659638831135/work
Werkzeug==2.2.2
wincertstore==0.2
wrapt==1.14.1
zipp==3.8.1

============== Conda Packages ==============
# packages in environment at C:\Users\dkroker\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   0.15.0             pyhd3eb1b0_0  
aom                       3.4.0                h0e60522_1    conda-forge
blas                      1.0                         mkl  
brotli                    1.0.9                h2bbff1b_7  
brotli-bin                1.0.9                h2bbff1b_7  
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2022.07.19           haa95532_0  
certifi                   2022.6.15        py39haa95532_0  
cloudpickle               2.0.0              pyhd3eb1b0_0  
colorama                  0.4.5            py39haa95532_0  
cycler                    0.11.0             pyhd3eb1b0_0  
decorator                 5.1.1              pyhd3eb1b0_0  
dm-tree                   0.1.5            py39hf11a4ad_0  
expat                     2.4.8                h39d44d4_0    conda-forge
fastcluster               1.2.6            py39h2e25243_1    conda-forge
ffmpeg                    5.1.0           gpl_h36dc936_101    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.0               hce3cb01_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.10.4               hd328e21_0  
gast                      0.5.3              pyhd3eb1b0_0  
git                       2.34.1               haa95532_0  
glib                      2.69.1               h5dc1a3c_1  
gst-plugins-base          1.18.5               h9e645db_0  
gstreamer                 1.18.5               hd78058f_0  
icu                       58.2                 ha925a31_3  
imageio                   2.19.3           py39haa95532_0  
imageio-ffmpeg            0.4.7              pyhd8ed1ab_0    conda-forge
intel-openmp              2021.4.0          haa95532_3556  
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h2bbff1b_0  
kiwisolver                1.4.2            py39hd77b12b_0  
lerc                      3.0                  hd77b12b_0  
libblas                   3.9.0           1_h8933c1f_netlib    conda-forge
libbrotlicommon           1.0.9                h2bbff1b_7  
libbrotlidec              1.0.9                h2bbff1b_7  
libbrotlienc              1.0.9                h2bbff1b_7  
libcblas                  3.9.0           5_hd5c7e75_netlib    conda-forge
libclang                  12.0.0          default_h627e005_2  
libdeflate                1.8                  h2bbff1b_5  
libffi                    3.4.2                hd77b12b_4  
libiconv                  1.16                 h2bbff1b_2  
liblapack                 3.9.0           5_hd5c7e75_netlib    conda-forge
libogg                    1.3.5                h2bbff1b_1  
libpng                    1.6.37               h2a8f88b_0  
libtiff                   4.4.0                h8a3f274_0  
libvorbis                 1.3.7                he774522_0  
libwebp                   1.2.2                h2bbff1b_0  
libxml2                   2.9.14               h0ad7f3c_0  
libxslt                   1.1.35               h2bbff1b_0  
libzlib                   1.2.12               h8ffe710_2    conda-forge
lz4-c                     1.9.3                h2bbff1b_1  
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
matplotlib                3.5.2            py39haa95532_0  
matplotlib-base           3.5.2            py39hd77b12b_0  
mkl                       2021.4.0           haa95532_640  
mkl-service               2.4.0            py39h2bbff1b_0  
mkl_fft                   1.3.1            py39h277e83a_0  
mkl_random                1.2.2            py39hf11a4ad_0  
msys2-conda-epoch         20160418                      1    conda-forge
munkres                   1.1.4                      py_0  
numexpr                   2.8.3            py39hb80d3ca_0  
numpy                     1.23.1           py39h7a0a035_0  
numpy-base                1.23.1           py39hca35cd5_0  
openh264                  2.3.0                h0e60522_0    conda-forge
openssl                   1.1.1q               h2bbff1b_0  
packaging                 21.3               pyhd3eb1b0_0  
pcre                      8.45                 hd77b12b_0  
pillow                    9.2.0            py39hdc2b20a_1  
pip                       22.1.2           py39haa95532_0  
ply                       3.11             py39haa95532_0  
psutil                    5.9.0            py39h2bbff1b_0  
pyparsing                 3.0.9            py39haa95532_0  
pyqt                      5.15.7           py39hd77b12b_0  
pyqt5-sip                 12.11.0          py39hd77b12b_0  
python                    3.9.12               h6244533_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python_abi                3.9                      2_cp39    conda-forge
pywin32                   302              py39h2bbff1b_2  
pywinpty                  2.0.2            py39h5da7b33_0  
qt-main                   5.15.2               he8e5bd7_7  
qt-webengine              5.15.9               hb9a9bb5_4  
qtwebkit                  5.212                h3ad3cdb_4  
scikit-learn              1.1.2            py39hfd4428b_0    conda-forge
scipy                     1.8.1            py39h5567194_2    conda-forge
setuptools                63.4.1           py39haa95532_0  
sip                       6.6.2            py39hd77b12b_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.39.2               h2bbff1b_0  
svt-av1                   1.2.0                h0e60522_0    conda-forge
tensorflow-probability    0.14.0             pyhd3eb1b0_0  
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
tornado                   6.1              py39h2bbff1b_0  
tqdm                      4.64.0           py39haa95532_0  
typing-extensions         4.3.0            py39haa95532_0  
typing_extensions         4.3.0            py39haa95532_0  
tzdata                    2022a                hda174b7_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py39haa95532_2  
winpty                    0.4.3                         4  
x264                      1!164.3095           h8ffe710_2    conda-forge
x265                      3.5                  h2d74725_3    conda-forge
xz                        5.2.5                h8cc25b3_1  
zlib                      1.2.12               h8cc25b3_2  
zstd                      1.5.2                h19a0ad4_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0
erosion_top:              0.0
erosion_bottom:           0.0
erosion_left:             0.0
erosion_right:            0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
separate_mask:            False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
separate_mask:            False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
cpu:                      True
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.bisenet_fp]
batch-size:               8
cpu:                      False
weights:                  faceswap
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.custom]
batch-size:               8
centering:                face
fill:                     False

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                face
coverage:                 62.5
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
autoclip:                 False
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
loss_function_2:          mse
loss_weight_2:            100
loss_function_3:          None
loss_weight_3:            0
loss_function_4:          None
loss_weight_4:            0
mask_loss_function:       mse
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.phaze_a]
output_size:              128
shared_fc:                None
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              7
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          None
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_upscales_in_fc:       0
dec_norm:                 None
dec_min_filters:          64
dec_max_filters:          512
dec_slope_mode:           full
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
fs_original_use_alt:      False
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.realface]
input_size:               128
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4
User avatar
SirDeepsALot
Posts: 35
Joined: Sat May 28, 2022 7:10 am
Has thanked: 7 times
Been thanked: 2 times

Re: immediate training error

Post by SirDeepsALot »

and it's definitely installed from the command prompt:

User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: immediate training error

Post by torzdf »

Ok, this time you do have OpenCV, but you don't have cuda toolkit....

I would need to see the faceswap_setup_log again, but if I were to guess (I am) I would say you had intermittent connection issues during install again.

You can try to pull it in with:

Start > Anaconda Prompt (cmd)

Code: Select all

conda activate faceswap
conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1

I cannot guarantee that this is the only package which failed to install.

My word is final

User avatar
SirDeepsALot
Posts: 35
Joined: Sat May 28, 2022 7:10 am
Has thanked: 7 times
Been thanked: 2 times

Re: immediate training error

Post by SirDeepsALot »

Hmm same problem. Still says this machine has no GPU, although I now have faceswap_setup.log and a faceswap_setup.log 1

Attachments
faceswap_setup.log
(9.81 KiB) Downloaded 100 times
User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: immediate training error

Post by torzdf »

Ok, I suspect just running setup.py/update_deps does not work because of the way we have to handle cuda dependencies in Conda at the moment (it sees TF-GPU is installed, so assumed cuda is installed as it is a loose dependency).

Doing this, as per my earlier post should pull in those packages for you though. And if it doesn't, it will give you verbose output as to why it doesn't:

torzdf wrote: Sat Aug 27, 2022 6:12 am

Ok, this time you do have OpenCV, but you don't have cuda toolkit....

You can try to pull it in with:

Start > Anaconda Prompt (cmd)

Code: Select all

conda activate faceswap
conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1

I cannot guarantee that this is the only package which failed to install.

My word is final

User avatar
SirDeepsALot
Posts: 35
Joined: Sat May 28, 2022 7:10 am
Has thanked: 7 times
Been thanked: 2 times

Re: immediate training error

Post by SirDeepsALot »

ok it worked, however, I have realface issues now.

As I know what it's like to be in your shoes (I run a tiny software company - unfortunately, PHP-focused so I can't help you with this) I'll give you the full details of everything in case it helps you avoid future product issues. I actually had followed your instructions in the last post already, the command prompt said it downloaded the repo, but still failed to find the GPU on startup. All I've down now is I've just re-executed the command and now it works, but here's what it came up with when I tried to run it:

Not sure if that helps. It does now say that it is using NVIDA at startup:

though I can't just do an update of Faceswap anymore either:

And apparently have mulitple code branches now? Not sure if that is normal:

I am wondering if the uninstall is not uninstalling cleanly. I go to PC Settings > apps > uninstall miniconda then go to C:/Users/<username>/ then delete the faceswap and .conda folders. Reboot. Then reinstall. But even after I reinstall some of the settings seeeeem to remain, like I ticked "include glasses" once in Mask settings, and I don't think that's a default setting yet it's ticked after every reinstall? If it is ticked on default, please ignore.

Anyway, now it finds the NVIDIA driver but Realface is crapping out, I do have a browser open and that is all, but I've previously been able to play low-graphics games and have a browser open without this being a problem:

Loading...
Setting Faceswap backend to NVIDIA
08/28/2022 17:02:32 INFO Log level set to: INFO
08/28/2022 17:02:36 INFO Model A Directory: 'F:\(DEEP)\output a\Nadine sage - amateur allure\missionary edit 2' (415 images)
08/28/2022 17:02:36 INFO Model B Directory: 'F:\(DEEP)\output b\mik' (644 images)
08/28/2022 17:02:36 INFO Training data directory: F:\(DEEP)\Model\nadine to mik\realface_logs
08/28/2022 17:02:36 INFO ===================================================
08/28/2022 17:02:36 INFO Starting
08/28/2022 17:02:36 INFO ===================================================
08/28/2022 17:02:36 INFO Loading data, this may take a while...
08/28/2022 17:02:36 INFO Loading Model from Realface plugin...
08/28/2022 17:02:37 INFO No existing state file found. Generating.
08/28/2022 17:02:37 INFO Storing Mixed Precision compatible layers. Please ignore any following warnings about using mixed precision.
08/28/2022 17:02:37 WARNING Mixed precision compatibility check (mixed_float16): WARNING
Your GPU may run slowly with dtype policy mixed_float16 because it does not have compute capability of at least 7.0. Your GPU:
NVIDIA GeForce GTX 1080, compute capability 6.1
See https://developer.nvidia.com/cuda-gpus for a list of GPUs and their compute capabilities.
If you will use compatible GPU(s) not attached to this host, e.g. by running a multi-worker model, you can ignore this warning. This message will only be logged once
08/28/2022 17:02:42 INFO Loading Trainer from Original plugin...
08/28/2022 17:03:12 CRITICAL Error caught! Exiting...
08/28/2022 17:03:12 ERROR Caught exception in thread: '_training'
08/28/2022 17:03:12 ERROR You do not have enough GPU memory available to train the selected model at the selected settings. You can try a number of things:
08/28/2022 17:03:12 ERROR 1) Close any other application that is using your GPU (web browsers are particularly bad for this).
08/28/2022 17:03:12 ERROR 2) Lower the batchsize (the amount of images fed into the model each iteration).
08/28/2022 17:03:12 ERROR 3) Try enabling 'Mixed Precision' training.
08/28/2022 17:03:12 ERROR 4) Use a more lightweight model, or select the model's 'LowMem' option (in config) if it has one.
Process exited.

Currently the GPU is hovering between 5-8% usage with the browsers open, though it can spike up to around 20%. Is RealFace really that resource-intensive? I'm using an NVIDIA GeForce GTX 1080,

Thanks in advance for all your help.

User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: immediate training error

Post by torzdf »

Ok. if its finding the GPU now, I am going to ignore the earlier messages (I would expect to see it downloading cudatoolkit/cudnn, but there is no evidence of that :/)

  • setting backend to NVIDIA is just an internal Faceswap thing, it does not mean that your GPU will actually be used (the only thing that dictates this is that Cuda/cuDNN is installed)
  • Not being able to update means that tracked files have changed in your environment somehow. Sometimes this just happens. Troubleshooting this is too much work, so a delete and re-install is the easiest method.
  • Multiple branches have always existed (it just runs standard git commands to update/get branches etc)
  • Include glasses is defaulted true, that is fine. (fwiw, the .ini config files are not stored in the repo. They are auto-generated on Faceswap's first run). You can keep your settings after a full install by backing up the contents of the /faceswap/config folder prior to deletion, and copying them back in after re-installing Faceswap.
  • Just uninstalling is not enough to clean up badly installed dependencies/conflicts. This is a Windows implementation of Python issue. It is why following every one of these steps solves 99% of issues, and why we have it as an FAQ item: app.php/faqpage#f1r1
  • I don't know what to add to the output message at the end there. You have run out of VRAM, you need to follow the instructions in the message

My word is final

Locked