Python crashes when trying to train a model.

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
SilverFox
Posts: 6
Joined: Sun Nov 06, 2022 3:06 pm

Python crashes when trying to train a model.

Post by SilverFox »

I actually don't really know where to put my question as it seem a little confusing.
I installed FS long time ago using the installer, everything was just fine, it installed miniconda and well, it worked.
Than some day i updated FS and than it didn't worked, delelted FS and re installed the old version, still didn't really worked until i've decided to manually install it, delelted miniconda and installed anaconda3 and than i got it to work.
Everything was just fine but than i didn't used FS for more than a month, a few days ago i've started and it worked but i was wondering why it got so slow, checking the task manager showed that it used a lot of the CPU so i was thinking about that FS don't used my card anymore and switched to CPU for some reason (i didn't changed anythin on conda or FS, i also didn't made any updates on my system).

I've tried to re install FS again but didn't got it to work again, got lots of DLL errors of missing files which i haven't got before.
After some time i managed to install FS again and re installed missing stuff via pip and it also finds my graphics card, so it should work again but every time i start training i got the message that python have crashed and the training process ends, extracting works fine. From the error message itself it looks like it have something to do with python itself as FS should work, i don't know if i am still missing any package i should install on FS ENV.

Btw. my sys is the following.

Code: Select all

Win7 64 ultimate
i7 11700K
RTX 2060 12GB
32GB DDR4 3200

installed packages on FS ENV in conda are as followed (numbers are displayed versions)

Code: Select all

absl-py 1.3.0
astunparse 1.6.3
blas 1.0
brotli 1.0.9
brotli-bin 1.0.9
ca-certificates 2022.10.11
cachetools 5.2.0
certifi 2022.9.24
charset-normalizer 2.1.1
cloudpickle 2.0.0
colorama 0.4.5
cudatoolkit 11.2.2
cudnn 8.1.0.77
cycler 0.11.0
decorator 5.1.1
dm-tree 0.1.5
fastcluster 1.2.6
ffmpeg 4.3.1
ffmpy 0.3.0
flatbuffers 1.12
fonttools 4.25.0
freetype 2.12.1
gast 0.5.3
git 2.34.1
glib 2.69.1
google-auth 2.14.0
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
grpico 1.50.0
gst-plugin-base 1.18.5
gstreamer 1.18.5
h5py 3.7.0
icu 58.2
idna 3.4
imageio 2.19.3
imageio-ffmpeg 0.4.7
importlib-metadata 5.0.0
intel-openmp 2021.4.0
joblib 1.2.0
jpeg 9e
keras 2.9.0
keras-preprocessing 1.1.2
kiwisolver 1.4.2
lerc 3.0
libblas 3.9.0
libbrotlicommon 1.0.9
libbrotlidec 1.0.9
libbrotlienc 1.0.9
libcblas 3.9.0
libclang 12.0.0
libdeflate 1.8
libffi 3.4.2
libiconv 1.16
liblapack 3.9.0
libogg 1.3.5
libpng 1.6.37
libtiff 4.4.0
libvorbis 1.3.7
libwebp 1.2.4
libwebp-base 1.2.4
libxml2 2.9.14
libxslt 1.1.35
lz4-c 1.9.3
m2w64-gcc-libgfortran 5.3.0
m2w64-gcc-libs 5.3.0
m2w64-gcc-libs-core 5.3.0
m2w64-gmp 6.1.0
m2w64-libwinpthread-git 5.0.0.46...
markdown 3.4.1
markupsafe 2.1.1
matplotlib 3.5.3
matplotlib-base 3.5.3
mkl 2021.4.0
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
mkl-fft 1.3.1
mkl_random 1.2.2
msys2-conda-epoch 20160418
munkres 1.1.4
numexpr 2.8.3
numpy 1.23.3
nvidia-ml-py 11.515.75
oauthlib 3.2.2
opencv-python 4.6.0.66
openssl 1.1.1s
opt-einsum 3.3.0
packagin 21.3
pcre 8.45
pillow 9.2.0
pip 22.2.2
ply 3.11
protobuf 3.19.6
psutil 5.9.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pynvx 1.0.0
pyparsing 3.0.9
pyqt 5.15.7
pyqt5 5.15.7
pyt5-sip 12.11.0
python 3.8.13
python-dateutil 2.8.2
python_abi 3.8
pywin32 302
pywinpty 2.0.2
qt-main 5.12.2
qt-webengine 5.15.9
qtwebkit 5.212
requests 2.28.1
requests-oauthlib 1.3.1
rsa 4.9
scikit-lean 1.0.2
scipy 1.8.1
setuptools 65.5.0
sip 6.6.2
six 1.16.0
sqlite 3.39.3
tensorboard 2.9.1
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorflow-estimator 2.9.0
tensorflow-gpu 2.9.2
tensorflow-io-gcs-filesystem 0.27.0
tensorflow-probability 0.14.0
termcolor 2.1.0
threadpdoolctl 3.1.0
tk 8.6.12
toml 0.10.2
tornado 6.2
tqdm 4.64.1
typing-extensions 4.3.0
typing_extensions 4.3.0
urllib3 1.26.12
vc 14.2
vs2015_runtime 14.27.29...
werkzeug 2.2.2
wheel 0.37.1
wincertstore 0.2
winpty 0.4.3
wrapt 1.14.1
xz 5.2.6
zipp 3.10.0
zlib 1.2.13
zstd 1.5.2

no error log was created, just the faceswap log

Code: Select all

11/06/2022 16:59:54 MainProcess     MainThread                     logger          log_setup                      INFO     Log level set to: INFO
11/06/2022 16:59:56 MainProcess     MainThread                     train           _get_images                    INFO     Model A Directory: 'C:\Users\User\Documents\FS\16.0\2\F' (3087 images)
11/06/2022 16:59:56 MainProcess     MainThread                     train           _get_images                    INFO     Model B Directory: 'C:\Users\User\Documents\FS\16.0\1\F' (14861 images)
11/06/2022 16:59:56 MainProcess     MainThread                     train           process                        INFO     Training data directory: C:\Users\User\Documents\FS\16.0\2\M
11/06/2022 16:59:56 MainProcess     MainThread                     train           _output_startup_info           INFO     ===================================================
11/06/2022 16:59:56 MainProcess     MainThread                     train           _output_startup_info           INFO       Starting
11/06/2022 16:59:56 MainProcess     MainThread                     train           _output_startup_info           INFO     ===================================================
11/06/2022 16:59:56 MainProcess     _training                      train           _training                      INFO     Loading data, this may take a while...
11/06/2022 16:59:56 MainProcess     _training                      plugin_loader   _import                        INFO     Loading Model from Realface plugin...
11/06/2022 16:59:56 MainProcess     _training                      deprecation     new_func                       DEBUG    From C:\Users\User\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\util\deprecation.py:561: calling function (from tensorflow.python.eager.def_function) with experimental_relax_shapes is deprecated and will be removed in a future version.\nInstructions for updating:\nexperimental_relax_shapes is deprecated, use reduce_retracing instead
11/06/2022 16:59:57 MainProcess     _training                      model           _load                          INFO     No existing state file found. Generating.
11/06/2022 16:59:57 MainProcess     _training                      settings        get_mixed_precision_layers     INFO     Storing Mixed Precision compatible layers. Please ignore any following warnings about using mixed precision.
11/06/2022 16:59:57 MainProcess     _training                      device_compatibility_check _log_device_compatibility_check INFO     Mixed precision compatibility check (mixed_float16): OK\nYour GPU will likely run quickly with dtype policy mixed_float16 as it has compute capability of at least 7.0. Your GPU: NVIDIA GeForce RTX 2060, compute capability 7.5
11/06/2022 16:59:59 MainProcess     _training                      plugin_loader   _import                        INFO     Loading Trainer from Original plugin...

Error message is as followed: Python stopped working.

Code: Select all

Appcrash 
Version: 3.8.3150.1013
Error modul name: cublasLt64_11.dll
Error modul version: 6.14.11.1141
code: c000001d

i am not sure if i am still missing some packages or if a version mismatch of something cause this issue. Didn't checked if other functions works but extracting is just fine.

User avatar
SilverFox
Posts: 6
Joined: Sun Nov 06, 2022 3:06 pm

Re: Python crashes when trying to train a model.

Post by SilverFox »

Just for little addition.
I've just tested if FC works in CPU mode with the option to exclude CPU's and there is no problem or crash.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Python crashes when trying to train a model.

Post by torzdf »

This looks like a Cuda version conflict. In the first instance do this (do not skip any steps):
app.php/faqpage#f1r1

If that does not solve your issue, post back following these instructions:
app.php/rules#rule-4b

My word is final

User avatar
SilverFox
Posts: 6
Joined: Sun Nov 06, 2022 3:06 pm

Re: Python crashes when trying to train a model.

Post by SilverFox »

thanks a lot for the fast reply.
I deinstalled everything and rebooted my system, than started the installer but got an error after downloading Miniconda.
The error is in creating a virtual environment and the installation get aborted.
That was the reason why i installed anaconda3 manually when i got first issues with FS.
(it was anaconda 3 2020.7 with Python 3.8)

I've tried installing FS via installer twice now and everytime it couldn't create the environment for FS and installation got aborted.
I've than installed anaconda3 2020.07 again and created the FS environment, opened the terminal and followed the steps here.

https://github.com/deepfakes/faceswap/b ... nvironment

while the installation from terminal i've got these messages:

Code: Select all

"opencv-python>= 4.6.0.0" not available in Conda. Installing with pip
"ffmpy>= 0.3.0" not available in Conda. Installing with pip
"nvidia-ml-py 11.515,<300" not available in Conda. Installing with pip

checking Anaconda navigator, all these three show up in FS environment (but not in base root, do they have to be there too?)
But while installing pywin32 i've got this error message twice

Code: Select all

"?init@PyWinBufferView@@QEAA_NPEAU_object@@_N1@Z wasnt found in pywintypes38.dll"

Installing Cuda Toolkit also wasn't possible as it said this: couldnt install Cuda Toolkit with Conda. Please install this package manually.

So i guess what i should do now is just install the Cuda Toolkit.
Versions i've still got on my HDD is 10.0.13._411.31 and 7.0.28

couldn't got the gui to start but using update_deps.py via terminal helped here. Gui did start and first attempt of training seemed to work but slow and also looked like it's still using CPU for training.
Loaded another model, started training and got an error (at least no python crash)

Code: Select all

11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'model.unbalanced', title: 'lowmem', datatype: '<class 'bool'>', default: 'False', info: 'Lower memory mode. Set to 'True' if having issues with VRAM useage.\nNB: Models with a changed lowmem mode are not compatible with each other.\nNB: lowmem will override cutom nodes and complexity settings.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'model.unbalanced', title: 'nodes', datatype: '<class 'int'>', default: '1024', info: 'Number of nodes for decoder. Don't change this unless you know what you are doing!', rounding: '64', min_max: (512, 4096), choices: [], gui_radio: False, fixed: True, group: network)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'model.unbalanced', title: 'complexity_encoder', datatype: '<class 'int'>', default: '128', info: 'Encoder Convolution Layer Complexity. sensible ranges: 128 to 160.', rounding: '16', min_max: (64, 1024), choices: [], gui_radio: False, fixed: True, group: network)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'model.unbalanced', title: 'complexity_decoder_a', datatype: '<class 'int'>', default: '384', info: 'Decoder A Complexity.', rounding: '16', min_max: (64, 1024), choices: [], gui_radio: False, fixed: True, group: network)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'model.unbalanced', title: 'complexity_decoder_b', datatype: '<class 'int'>', default: '512', info: 'Decoder B Complexity.', rounding: '16', min_max: (64, 1024), choices: [], gui_radio: False, fixed: True, group: network)
11/07/2022 11:31:30 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Added defaults: model.unbalanced
11/07/2022 11:31:30 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Adding defaults: (filename: villain_defaults.py, module_path: plugins.train.model, plugin_type: model
11/07/2022 11:31:30 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Importing defaults module: plugins.train.model.villain_defaults
11/07/2022 11:31:30 MainProcess     _training                      config          add_section                    DEBUG    Add section: (title: 'model.villain', info: 'A Higher resolution version of the Original Model by VillainGuy.\nExtremely VRAM heavy. Don't try to run this if you have a small GPU.\n')
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'model.villain', title: 'lowmem', datatype: '<class 'bool'>', default: 'False', info: 'Lower memory mode. Set to 'True' if having issues with VRAM useage.\nNB: Models with a changed lowmem mode are not compatible with each other.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
11/07/2022 11:31:30 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Added defaults: model.villain
11/07/2022 11:31:30 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Adding defaults: (filename: original_defaults.py, module_path: plugins.train.trainer, plugin_type: trainer
11/07/2022 11:31:30 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Importing defaults module: plugins.train.trainer.original_defaults
11/07/2022 11:31:30 MainProcess     _training                      config          add_section                    DEBUG    Add section: (title: 'trainer.original', info: 'Original Trainer Options.\nWARNING: The defaults for augmentation will be fine for 99.9% of use cases. Only change them if you absolutely know what you are doing!')
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'preview_images', datatype: '<class 'int'>', default: '14', info: 'Number of sample faces to display for each side in the preview when training.', rounding: '2', min_max: (2, 16), choices: None, gui_radio: False, fixed: True, group: evaluation)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'zoom_amount', datatype: '<class 'int'>', default: '5', info: 'Percentage amount to randomly zoom each training image in and out.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'rotation_range', datatype: '<class 'int'>', default: '10', info: 'Percentage amount to randomly rotate each training image.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'shift_range', datatype: '<class 'int'>', default: '5', info: 'Percentage amount to randomly shift each training image horizontally and vertically.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'flip_chance', datatype: '<class 'int'>', default: '50', info: 'Percentage chance to randomly flip each training image horizontally.\nNB: This is ignored if the 'no-flip' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: True, group: image augmentation)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_lightness', datatype: '<class 'int'>', default: '30', info: 'Percentage amount to randomly alter the lightness of each training image.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: True, group: color augmentation)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_ab', datatype: '<class 'int'>', default: '8', info: 'Percentage amount to randomly alter the 'a' and 'b' colors of the L*a*b* color space of each training image.\nNB: This is ignored if the 'no-augment-color' optionis enabled', rounding: '1', min_max: (0, 50), choices: None, gui_radio: False, fixed: True, group: color augmentation)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_clahe_chance', datatype: '<class 'int'>', default: '50', info: 'Percentage chance to perform Contrast Limited Adaptive Histogram Equalization on each training image.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: False, group: color augmentation)
11/07/2022 11:31:30 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_clahe_max_size', datatype: '<class 'int'>', default: '4', info: 'The grid size dictates how much Contrast Limited Adaptive Histogram Equalization is performed on any training image selected for clahe. Contrast will be applied randomly with a gridsize of 0 up to the maximum. This value is a multiplier calculated from the training image size.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (1, 8), choices: None, gui_radio: False, fixed: True, group: color augmentation)
11/07/2022 11:31:30 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Added defaults: trainer.original
11/07/2022 11:31:30 MainProcess     _training                      config          handle_config                  DEBUG    Handling config: (section: model.realface, configfile: 'C:\Users\User\faceswap\config\train.ini')
11/07/2022 11:31:30 MainProcess     _training                      config          check_exists                   DEBUG    Config file exists: 'C:\Users\User\faceswap\config\train.ini'
11/07/2022 11:31:30 MainProcess     _training                      config          load_config                    VERBOSE  Loading config: 'C:\Users\User\faceswap\config\train.ini'
11/07/2022 11:31:30 MainProcess     _training                      config          validate_config                DEBUG    Validating config
11/07/2022 11:31:30 MainProcess     _training                      config          check_config_change            DEBUG    Default config has not changed
11/07/2022 11:31:30 MainProcess     _training                      config          check_config_choices           DEBUG    Checking config choices
11/07/2022 11:31:30 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'keras_encoder' to list ['keras_encoder'] for section 'model.phaze_a', option 'freeze_layers'
11/07/2022 11:31:30 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'encoder' to list ['encoder'] for section 'model.phaze_a', option 'load_layers'
11/07/2022 11:31:30 MainProcess     _training                      config          check_config_choices           DEBUG    Checked config choices
11/07/2022 11:31:30 MainProcess     _training                      config          validate_config                DEBUG    Validated config
11/07/2022 11:31:30 MainProcess     _training                      config          handle_config                  DEBUG    Handled config
11/07/2022 11:31:30 MainProcess     _training                      config          __init__                       DEBUG    Initialized: Config
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'learning_rate')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 5e-05)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'epsilon_exponent')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: -7)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'autoclip')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'allow_growth')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'mixed_precision')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'nan_protection')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: True)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'convert_batchsize')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 16)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: ssim)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_2')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mse)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_2')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 100)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_3')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: None)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_3')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 0)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_4')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: None)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_4')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 0)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mask_loss_function')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mse)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'eye_multiplier')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 3)
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mouth_multiplier')
11/07/2022 11:31:30 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 2)
11/07/2022 11:31:30 MainProcess     _training                      config          changeable_items               DEBUG    Alterable for existing models: {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}
11/07/2022 11:31:30 MainProcess     _training                      model           __init__                       DEBUG    Initializing State: (model_dir: 'C:\Users\User\Documents\FS\16.0\1\M', model_name: 'realface', config_changeable_items: '{'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}', no_logs: False
11/07/2022 11:31:30 MainProcess     _training                      serializer      get_serializer                 DEBUG    <lib.serializer._JSONSerializer object at 0x000000001093C820>
11/07/2022 11:31:30 MainProcess     _training                      model           _load                          DEBUG    Loading State
11/07/2022 11:31:30 MainProcess     _training                      model           _load                          INFO     No existing state file found. Generating.
11/07/2022 11:31:30 MainProcess     _training                      model           _new_session_id                DEBUG    1
11/07/2022 11:31:30 MainProcess     _training                      model           _create_new_session            DEBUG    Creating new session. id: 1
11/07/2022 11:31:30 MainProcess     _training                      model           __init__                       DEBUG    Initialized State:
11/07/2022 11:31:30 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Settings: (arguments: Namespace(batch_size=16, colab=False, configfile=None, distributed=False, distribution_strategy='default', exclude_gpus=None, freeze_weights=False, func=<bound method ScriptExecutor.execute_script of <lib.cli.launcher.ScriptExecutor object at 0x000000000385EEE0>>, input_a='C:\\Users\\User\\Documents\\FS\\16.0\\1\\F', input_b='C:\\Users\\User\\Documents\\FS\\16.0\\2\\F', iterations=1000000, load_weights=None, logfile=None, loglevel='INFO', model_dir='C:\\Users\\User\\Documents\\FS\\16.0\\1\\M', no_augment_color=False, no_flip=False, no_logs=False, no_warp=False, preview=False, redirect_gui=True, save_interval=250, snapshot_interval=25000, summary=False, timelapse_input_a=None, timelapse_input_b=None, timelapse_output=None, trainer='realface', warp_to_landmarks=False, write_image=False), mixed_precision: False, allow_growth: False, is_predict: False)
11/07/2022 11:31:30 MainProcess     _training                      settings        _set_tf_settings               DEBUG    Not setting any specific Tensorflow settings
11/07/2022 11:31:30 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    use_mixed_precision: False
11/07/2022 11:31:30 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    Disabling mixed precision. (Compute dtype: float32, variable_dtype: float32)
11/07/2022 11:31:30 MainProcess     _training                      settings        _get_strategy                  DEBUG    Using strategy: <tensorflow.python.distribute.distribute_lib._DefaultDistributionStrategy object at 0x0000000012D93B80>
11/07/2022 11:31:30 MainProcess     _training                      settings        __init__                       DEBUG    Initialized Settings
11/07/2022 11:31:30 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Loss: (color_order: bgr)
11/07/2022 11:31:30 MainProcess     _training                      settings        _get_mask_channels             DEBUG    uses_masks: (True, True, True), mask_channels: [3, 4, 5]
11/07/2022 11:31:30 MainProcess     _training                      settings        __init__                       DEBUG    Initialized: Loss
11/07/2022 11:31:30 MainProcess     _training                      model           __init__                       DEBUG    Initialized ModelBase (Model)
11/07/2022 11:31:30 MainProcess     _training                      realface        check_input_output             DEBUG    Input and output sizes are valid
11/07/2022 11:31:30 MainProcess     _training                      realface        get_dense_width_upscalers_numbers DEBUG    dense_width: 4, upscalers_no: 5
11/07/2022 11:31:30 MainProcess     _training                      settings        strategy_scope                 DEBUG    Using strategy scope: <tensorflow.python.distribute.distribute_lib._DefaultDistributionContext object at 0x000000001067A780>
11/07/2022 11:31:30 MainProcess     _training                      io              _load                          DEBUG    Loading model: C:\Users\User\Documents\FS\16.0\1\M\realface.h5
11/07/2022 11:31:30 MainProcess     _training                      multithreading  run                            DEBUG    Error in thread (_training): Unable to open file (bad object header version number)
11/07/2022 11:31:30 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
11/07/2022 11:31:30 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
11/07/2022 11:31:30 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
11/07/2022 11:31:30 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
11/07/2022 11:31:30 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
11/07/2022 11:31:30 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training'
11/07/2022 11:31:30 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training'
Traceback (most recent call last):
  File "C:\Users\User\faceswap\lib\cli\launcher.py", line 217, in execute_script
    process.process()
  File "C:\Users\User\faceswap\scripts\train.py", line 218, in process
    self._end_thread(thread, err)
  File "C:\Users\User\faceswap\scripts\train.py", line 258, in _end_thread
    thread.join()
  File "C:\Users\User\faceswap\lib\multithreading.py", line 217, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\User\faceswap\lib\multithreading.py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\User\faceswap\scripts\train.py", line 280, in _training
    raise err
  File "C:\Users\User\faceswap\scripts\train.py", line 268, in _training
    model = self._load_model()
  File "C:\Users\User\faceswap\scripts\train.py", line 296, in _load_model
    model.build()
  File "C:\Users\User\faceswap\plugins\train\model\_base\model.py", line 302, in build
    model = self._io._load()  # pylint:disable=protected-access
  File "C:\Users\User\faceswap\plugins\train\model\_base\io.py", line 152, in _load
    model = load_model(self._filename, compile=False)
  File "C:\Users\User\anaconda3\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\User\anaconda3\lib\site-packages\h5py\_hl\files.py", line 406, in __init__
    fid = make_fid(name, mode, userblock_size,
  File "C:\Users\User\anaconda3\lib\site-packages\h5py\_hl\files.py", line 173, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5f.pyx", line 88, in h5py.h5f.open
OSError: Unable to open file (bad object header version number)

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         128a4d0 Merge branch 'staging'
gpu_cuda:            10.0
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce RTX 2060
gpu_devices_active:  GPU_0
gpu_driver:          473.04
gpu_vram:            GPU_0: 12079MB
os_machine:          AMD64
os_platform:         Windows-7-6.1.7601-SP1
os_release:          7
py_command:          C:\Users\User\faceswap\faceswap.py train -A C:/Users/User/Documents/FS/16.0/1/F -B C:/Users/User/Documents/FS/16.0/2/F -m C:/Users/User/Documents/FS/16.0/1/M -t realface -bs 16 -it 1000000 -D default -s 250 -ss 25000 -L INFO -gui
py_conda_version:    conda 22.9.0
py_implementation:   CPython
py_version:          3.8.3
py_virtual_env:      False
sys_cores:           16
sys_processor:       Intel64 Family 6 Model 167 Stepping 1, GenuineIntel
sys_ram:             Total: 32655MB, Available: 27359MB, Used: 5295MB, Free: 27359MB

=============== Pip Packages ===============
absl-py==1.3.0
alabaster==0.7.12
anaconda-client==1.7.2
anaconda-navigator==1.9.12
anaconda-project==0.8.3
argh==0.26.2
asn1crypto==1.3.0
astroid @ file:///C:/ci/astroid_1592487315634/work
astropy==4.0.1.post1
astunparse==1.6.3
atomicwrites==1.4.0
attrs==19.3.0
autopep8 @ file:///tmp/build/80754af9/autopep8_1592412889138/work
Babel==2.8.0
backcall==0.2.0
backports.functools-lru-cache @ file:///tmp/build/80754af9/backports.functools_lru_cache_1618170165463/work
backports.shutil-get-terminal-size==1.0.0
backports.tempfile @ file:///home/linux1/recipes/ci/backports.tempfile_1610991236607/work
backports.weakref==1.0.post1
bcrypt==3.1.7
beautifulsoup4==4.9.1
bitarray @ file:///C:/ci/bitarray_1594751093906/work
bkcharts==0.2
bleach==3.1.5
bokeh @ file:///C:/ci/bokeh_1593178781838/work
boto==2.49.0
Bottleneck==1.3.2
brotlipy==0.7.0
cachetools==5.2.0
certifi @ file:///C:/b/abs_ac29jvt43w/croot/certifi_1665076682579/work/certifi
cffi==1.14.0
chardet==3.0.4
click==7.1.2
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1594141588948/work
clyent==1.2.2
colorama==0.4.3
comtypes==1.1.7
conda==22.9.0
conda-build==3.22.0
conda-package-handling @ file:///C:/b/abs_44a2d5ajjt/croot/conda-package-handling_1666940385350/work
conda-verify==3.4.2
contextlib2==0.6.0.post1
cryptography==2.9.2
cycler==0.10.0
Cython @ file:///C:/ci/cython_1594829190914/work
cytoolz==0.10.1
dask @ file:///tmp/build/80754af9/dask-core_1594156306305/work
decorator==4.4.2
defusedxml==0.6.0
diff-match-patch @ file:///tmp/build/80754af9/diff-match-patch_1594828741838/work
distributed @ file:///C:/ci/distributed_1594742844291/work
dm-tree==0.1.7
docutils==0.16
entrypoints==0.3
et-xmlfile==1.0.1
fastcache==1.1.0
fastcluster @ file:///D:/bld/fastcluster_1649783471515/work
ffmpy==0.3.0
filelock==3.0.12
flake8==3.8.3
Flask==1.1.2
flatbuffers==1.12
fonttools==4.25.0
fsspec==0.7.4
future==0.18.2
gast==0.4.0
gevent @ file:///C:/ci/gevent_1593005471151/work
glob2==0.7
gmpy2==2.0.8
google-auth==2.14.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
greenlet==0.4.16
grpcio==1.50.0
h5py==2.10.0
HeapDict==1.0.1
html5lib @ file:///tmp/build/80754af9/html5lib_1593446221756/work
idna @ file:///tmp/build/80754af9/idna_1593446292537/work
imageio @ file:///C:/Windows/TEMP/abs_24c1b783-7540-4ca9-a1b1-0e8aa8e6ae64hb79ssux/croots/recipe/imageio_1658785038775/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1649960641006/work
imagesize==1.2.0
importlib-metadata==5.0.0
intervaltree @ file:///tmp/build/80754af9/intervaltree_1594361675072/work
ipykernel @ file:///C:/ci/ipykernel_1594745408489/work/dist/ipykernel-5.3.2-py3-none-any.whl
ipython @ file:///C:/ci/ipython_1593447482397/work
ipython-genutils==0.2.0
ipywidgets==7.5.1
isort==4.3.21
itsdangerous==1.1.0
jdcal==1.4.1
jedi @ file:///C:/ci/jedi_1592833825077/work
Jinja2==2.11.2
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
json5==0.9.5
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client @ file:///tmp/build/80754af9/jupyter_client_1594826976318/work
jupyter-console==6.1.0
jupyter-core==4.6.3
jupyterlab==2.1.5
jupyterlab-server @ file:///tmp/build/80754af9/jupyterlab_server_1594164409481/work
keras==2.9.0
Keras-Preprocessing==1.1.2
keyring @ file:///C:/ci/keyring_1593109799227/work
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
libarchive-c==2.9
libclang==14.0.6
llvmlite==0.33.0+1.g022ab0f
locket==0.2.0
lxml @ file:///C:/ci/lxml_1594822774489/work
Markdown==3.4.1
MarkupSafe==1.1.1
matplotlib @ file:///C:/ci/matplotlib-suite_1660169687702/work
mccabe==0.6.1
menuinst==1.4.16
mistune==0.8.4
mkl-fft==1.3.1
mkl-random @ file:///C:/ci/mkl_random_1626186184278/work
mkl-service==2.4.0
mock==4.0.2
more-itertools==8.4.0
mpmath==1.1.0
msgpack==1.0.0
multipledispatch==0.6.0
munkres==1.1.4
navigator-updater==0.2.1
nbconvert==5.6.1
nbformat==5.0.7
networkx @ file:///tmp/build/80754af9/networkx_1594377231366/work
nltk @ file:///tmp/build/80754af9/nltk_1592496090529/work
nose==1.3.7
notebook==6.0.3
numba==0.50.1
numexpr @ file:///C:/Windows/Temp/abs_e2036a32-9fe9-47f3-a04c-dbb1c232ba4b334exiur/croots/recipe/numexpr_1656940304835/work
numpy==1.23.4
numpydoc @ file:///tmp/build/80754af9/numpydoc_1594166760263/work
nvidia-ml-py==11.515.75
oauthlib==3.2.2
olefile==0.46
opencv-python==4.6.0.66
openpyxl @ file:///tmp/build/80754af9/openpyxl_1594167385094/work
opt-einsum==3.3.0
packaging==20.4
pandas @ file:///C:/ci/pandas_1592833613419/work
pandocfilters==1.4.2
paramiko==2.7.1
parso==0.7.0
partd==1.1.0
path==13.1.0
pathlib2==2.3.5
pathtools==0.1.2
patsy==0.5.1
pep8==1.7.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.2.0
pkginfo==1.5.0.1
pluggy==0.13.1
ply==3.11
prometheus-client==0.8.0
prompt-toolkit==3.0.5
protobuf==3.19.6
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
py @ file:///tmp/build/80754af9/py_1593446248552/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.6.0
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pycurl==7.43.0.5
pydocstyle @ file:///tmp/build/80754af9/pydocstyle_1592848020240/work
pyflakes==2.2.0
Pygments==2.6.1
pylint @ file:///C:/ci/pylint_1592482039483/work
PyNaCl @ file:///C:/ci/pynacl_1595000047588/work
pyodbc===4.0.0-unsupported
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1594392929924/work
pyparsing==2.4.7
pyreadline==2.1
pyrsistent==0.16.0
PySocks==1.7.1
pytest==5.4.3
python-dateutil==2.8.1
python-jsonrpc-server @ file:///tmp/build/80754af9/python-jsonrpc-server_1594397536060/work
python-language-server @ file:///C:/ci/python-language-server_1594162130238/work
pytz==2020.1
PyWavelets==1.1.1
pywin32==302
pywin32-ctypes==0.2.0
pywinpty @ file:///C:/ci_310/pywinpty_1644230983541/work/target/wheels/pywinpty-2.0.2-cp38-none-win_amd64.whl
PyYAML==5.3.1
pyzmq==19.0.1
QDarkStyle==2.8.1
QtAwesome==0.7.2
qtconsole @ file:///tmp/build/80754af9/qtconsole_1592848611704/work
QtPy==1.9.0
regex @ file:///C:/ci/regex_1593419644658/work
requests @ file:///tmp/build/80754af9/requests_1592841827918/work
requests-oauthlib==1.3.1
rope==0.17.0
rsa==4.9
Rtree==0.9.4
ruamel-yaml==0.15.87
scikit-image==0.16.2
scikit-learn @ file:///D:/bld/scikit-learn_1640464350231/work
scipy @ file:///C:/ci/scipy_1661330047857/work
seaborn==0.10.1
Send2Trash==1.5.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
sip==4.19.13
six==1.15.0
snowballstemmer==2.0.0
sortedcollections==1.2.1
sortedcontainers==2.2.2
soupsieve==2.0.1
Sphinx @ file:///tmp/build/80754af9/sphinx_1594223420021/work
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
sphinxcontrib-websupport @ file:///tmp/build/80754af9/sphinxcontrib-websupport_1593446360927/work
spyder @ file:///C:/ci/spyder_1594830825244/work
spyder-kernels @ file:///C:/ci/spyder-kernels_1594751670175/work
SQLAlchemy @ file:///C:/ci/sqlalchemy_1593445271541/work
statsmodels==0.11.1
sympy @ file:///C:/ci/sympy_1594234545115/work
tables==3.6.1
tblib==1.6.0
tensorboard==2.9.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.9.2
tensorflow-estimator==2.9.0
tensorflow-gpu==2.9.2
tensorflow-io-gcs-filesystem==0.27.0
tensorflow-probability==0.14.0
termcolor==2.1.0
terminado==0.8.3
testpath==0.4.4
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
toml @ file:///tmp/build/80754af9/toml_1592853716807/work
toolz==0.10.0
tornado==6.0.4
tqdm @ file:///C:/b/abs_0axbz66qik/croots/recipe/tqdm_1664392691071/work
traitlets==4.3.3
typing-extensions @ file:///C:/Windows/TEMP/abs_dd2d0moa85/croots/recipe/typing_extensions_1659638831135/work
ujson==1.35
unicodecsv==0.14.1
urllib3==1.25.9
watchdog @ file:///C:/ci/watchdog_1593447437088/work
wcwidth @ file:///tmp/build/80754af9/wcwidth_1593447189090/work
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
win-inet-pton==1.1.0
win-unicode-console==0.5
wincertstore==0.2
wrapt==1.11.2
xlrd==1.2.0
XlsxWriter==1.2.9
xlwings==0.19.5
xlwt==1.3.0
xmltodict @ file:///Users/ktietz/demo/mc3/conda-bld/xmltodict_1629301980723/work
yapf @ file:///tmp/build/80754af9/yapf_1593528177422/work
zict==2.0.0
zipp==3.1.0
zope.event==4.4
zope.interface==4.7.1

============== Conda Packages ==============
# packages in environment at C:\Users\User\anaconda3:
#
# Name                    Version                   Build  Channel
_anaconda_depends         2020.07                  py38_0  
_ipyw_jlab_nb_ext_conf    0.1.0                    py38_0  
absl-py                   1.3.0                    pypi_0    pypi
alabaster                 0.7.12                     py_0  
anaconda                  custom                   py38_1  
anaconda-client           1.7.2                    py38_0  
anaconda-navigator        1.9.12                   py38_0  
anaconda-project          0.8.4                      py_0  
argh                      0.26.2                   py38_0  
asn1crypto                1.3.0                    py38_0  
astroid                   2.4.2                    py38_0  
astropy                   4.0.1.post1      py38he774522_1  
astunparse                1.6.3                    pypi_0    pypi
atomicwrites              1.4.0                      py_0  
attrs                     19.3.0                     py_0  
autopep8                  1.5.3                      py_0  
babel                     2.8.0                      py_0  
backcall                  0.2.0                      py_0  
backports                 1.0                        py_2  
backports.functools_lru_cache 1.6.4              pyhd3eb1b0_0  
backports.shutil_get_terminal_size 1.0.0                    py38_2  
backports.tempfile        1.0                pyhd3eb1b0_1  
backports.weakref         1.0.post1                  py_1  
bcrypt                    3.1.7            py38he774522_1  
beautifulsoup4            4.9.1                    py38_0  
bitarray                  1.4.0            py38he774522_0  
bkcharts                  0.2                      py38_0  
blas                      1.0                         mkl  
bleach                    3.1.5                      py_0  
blosc                     1.19.0               h7bd577a_0  
bokeh                     2.1.1                    py38_0  
boto                      2.49.0                   py38_0  
bottleneck                1.3.2            py38h2a96729_1  
brotli                    1.0.9                h2bbff1b_7  
brotli-bin                1.0.9                h2bbff1b_7  
brotlipy                  0.7.0           py38he774522_1000  
bzip2                     1.0.8                he774522_0  
ca-certificates           2022.10.11           haa95532_0  
cachetools                5.2.0                    pypi_0    pypi
certifi                   2022.9.24        py38haa95532_0  
cffi                      1.14.0           py38h7a1dbc1_0  
chardet                   3.0.4                 py38_1003  
click                     7.1.2                      py_0  
cloudpickle               1.5.0                      py_0  
clyent                    1.2.2                    py38_1  
colorama                  0.4.3                      py_0  
comtypes                  1.1.7                 py38_1001  
conda                     22.9.0           py38haa95532_0  
conda-build               3.22.0           py38haa95532_0  
conda-env                 2.6.0                         1  
conda-package-handling    1.9.0            py38h8cc25b3_1  
conda-verify              3.4.2                      py_1  
console_shortcut          0.1.1                         4  
contextlib2               0.6.0.post1                py_0  
cryptography              2.9.2            py38h7a1dbc1_0  
curl                      7.71.1               h2a8f88b_1  
cycler                    0.10.0                   py38_0  
cython                    0.29.21          py38ha925a31_0  
cytoolz                   0.10.1           py38he774522_0  
dask                      2.20.0                     py_0  
dask-core                 2.20.0                     py_0  
decorator                 4.4.2                      py_0  
defusedxml                0.6.0                      py_0  
diff-match-patch          20200713                   py_0  
distributed               2.20.0                   py38_0  
dm-tree                   0.1.7                    pypi_0    pypi
docutils                  0.16                     py38_1  
entrypoints               0.3                      py38_0  
et_xmlfile                1.0.1                   py_1001  
fastcache                 1.1.0            py38he774522_0  
fastcluster               1.2.6            py38hcc40339_1    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.3.0                    pypi_0    pypi
fftw                      3.3.9                h2bbff1b_1  
filelock                  3.0.12                     py_0  
flake8                    3.8.3                      py_0  
flask                     1.1.2                      py_0  
flatbuffers               1.12                     pypi_0    pypi
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.10.4               hd328e21_0  
fsspec                    0.7.4                      py_0  
future                    0.18.2                   py38_1  
gast                      0.4.0                    pypi_0    pypi
get_terminal_size         1.0.0                h38e98db_0  
gevent                    20.6.2           py38he774522_0  
glob2                     0.7                        py_0  
gmpy2                     2.0.8            py38h7edee0f_3  
google-auth               2.14.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
greenlet                  0.4.16           py38he774522_0  
grpcio                    1.50.0                   pypi_0    pypi
h5py                      2.10.0           py38h5e291fa_0  
hdf5                      1.10.4               h7ebc959_0  
heapdict                  1.0.1                      py_0  
html5lib                  1.1                        py_0  
icc_rt                    2019.0.0             h0cc432a_1  
icu                       58.2                 ha925a31_3  
idna                      2.10                       py_0  
imageio                   2.19.3           py38haa95532_0  
imageio-ffmpeg            0.4.7              pyhd8ed1ab_0    conda-forge
imagesize                 1.2.0                      py_0  
importlib-metadata        5.0.0                    pypi_0    pypi
importlib_metadata        1.7.0                         0  
intel-openmp              2021.4.0          haa95532_3556  
intervaltree              3.0.2                      py_1  
ipykernel                 5.3.2            py38h5ca1d4c_0  
ipython                   7.16.1           py38h5ca1d4c_0  
ipython_genutils          0.2.0                    py38_0  
ipywidgets                7.5.1                      py_0  
isort                     4.3.21                   py38_0  
itsdangerous              1.1.0                      py_0  
jdcal                     1.4.1                      py_0  
jedi                      0.17.1                   py38_0  
jinja2                    2.11.2                     py_0  
joblib                    0.16.0                     py_0  
jpeg                      9b                   hb83a4c4_2  
json5                     0.9.5                      py_0  
jsonschema                3.2.0                    py38_0  
jupyter                   1.0.0                    py38_7  
jupyter_client            6.1.6                      py_0  
jupyter_console           6.1.0                      py_0  
jupyter_core              4.6.3                    py38_0  
jupyterlab                2.1.5                      py_0  
jupyterlab_server         1.2.0                      py_0  
keras                     2.9.0                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
keyring                   21.2.1                   py38_0  
kiwisolver                1.2.0            py38h74a9793_0  
krb5                      1.18.2               hc04afaa_0  
lazy-object-proxy         1.4.3            py38he774522_0  
libarchive                3.4.2                h5e25573_0  
libblas                   3.9.0           1_h8933c1f_netlib    conda-forge
libbrotlicommon           1.0.9                h2bbff1b_7  
libbrotlidec              1.0.9                h2bbff1b_7  
libbrotlienc              1.0.9                h2bbff1b_7  
libcblas                  3.9.0           5_hd5c7e75_netlib    conda-forge
libclang                  14.0.6                   pypi_0    pypi
libcurl                   7.71.1               h2a8f88b_1  
libiconv                  1.15                 h1df5818_7  
liblief                   0.10.1               ha925a31_0  
libllvm9                  9.0.1                h21ff451_0  
libpng                    1.6.37               h2a8f88b_0  
libsodium                 1.0.18               h62dcd97_0  
libspatialindex           1.9.3                h33f27b4_0  
libssh2                   1.9.0                h7a1dbc1_1  
libtiff                   4.1.0                h56a325e_1  
libwebp                   1.2.4                h2bbff1b_0  
libwebp-base              1.2.4                h2bbff1b_0  
libxml2                   2.9.10               h464c3ec_1  
libxslt                   1.1.34               he774522_0  
llvmlite                  0.33.0           py38ha925a31_0  
locket                    0.2.0                    py38_1  
lxml                      4.5.2            py38h1350720_0  
lz4-c                     1.9.2                h62dcd97_0  
lzo                       2.10                 he774522_2  
m2-msys2-runtime          2.5.0.17080.65c939c               3  
m2-patch                  2.7.5                         2  
m2w64-gcc-libgfortran     5.3.0                         6  
m2w64-gcc-libs            5.3.0                         7  
m2w64-gcc-libs-core       5.3.0                         7  
m2w64-gmp                 6.1.0                         2  
m2w64-libwinpthread-git   5.0.0.4634.697f757               2  
markdown                  3.4.1                    pypi_0    pypi
markupsafe                1.1.1            py38he774522_0  
matplotlib                3.5.2            py38haa95532_0  
matplotlib-base           3.5.2            py38hd77b12b_0  
mccabe                    0.6.1                    py38_1  
menuinst                  1.4.16           py38he774522_1  
mistune                   0.8.4           py38he774522_1000  
mkl                       2021.4.0           haa95532_640  
mkl-service               2.4.0            py38h2bbff1b_0  
mkl_fft                   1.3.1            py38h277e83a_0  
mkl_random                1.2.2            py38hf11a4ad_0  
mock                      4.0.2                      py_0  
more-itertools            8.4.0                      py_0  
mpc                       1.1.0                h7edee0f_1  
mpfr                      4.0.2                h62dcd97_1  
mpir                      3.0.0                hec2e145_1  
mpmath                    1.1.0                    py38_0  
msgpack-python            1.0.0            py38h74a9793_1  
msys2-conda-epoch         20160418                      1  
multipledispatch          0.6.0                    py38_0  
munkres                   1.1.4                      py_0  
navigator-updater         0.2.1                    py38_0  
nbconvert                 5.6.1                    py38_0  
nbformat                  5.0.7                      py_0  
networkx                  2.4                        py_1  
nltk                      3.5                        py_0  
nose                      1.3.7                    py38_2  
notebook                  6.0.3                    py38_0  
numba                     0.50.1           py38h47e9c7a_0  
numexpr                   2.8.3            py38hb80d3ca_0  
numpy                     1.23.4                   pypi_0    pypi
numpy-base                1.21.5           py38hca35cd5_3  
numpydoc                  1.1.0                      py_0  
nvidia-ml-py              11.515.75                pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
olefile                   0.46                       py_0  
opencv-python             4.6.0.66                 pypi_0    pypi
openpyxl                  3.0.4                      py_0  
openssl                   1.1.1s               h2bbff1b_0  
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 20.4                       py_0  
pandas                    1.0.5            py38h47e9c7a_0  
pandoc                    2.10                          0  
pandocfilters             1.4.2                    py38_1  
paramiko                  2.7.1                      py_0  
parso                     0.7.0                      py_0  
partd                     1.1.0                      py_0  
path                      13.1.0                   py38_0  
path.py                   12.4.0                        0  
pathlib2                  2.3.5                    py38_0  
pathtools                 0.1.2                      py_1  
patsy                     0.5.1                    py38_0  
pep8                      1.7.1                    py38_0  
pexpect                   4.8.0                    py38_0  
pickleshare               0.7.5                 py38_1000  
pillow                    9.2.0            py38hdc2b20a_1  
pip                       20.1.1                   py38_1  
pkginfo                   1.5.0.1                  py38_0  
pluggy                    0.13.1                   py38_0  
ply                       3.11                     py38_0  
powershell_shortcut       0.0.1                         3  
prometheus_client         0.8.0                      py_0  
prompt-toolkit            3.0.5                      py_0  
prompt_toolkit            3.0.5                         0  
protobuf                  3.19.6                   pypi_0    pypi
psutil                    5.9.0            py38h2bbff1b_0  
py                        1.9.0                      py_0  
py-lief                   0.10.1           py38ha925a31_0  
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycodestyle               2.6.0                      py_0  
pycosat                   0.6.3            py38he774522_0  
pycparser                 2.20                       py_2  
pycurl                    7.43.0.5         py38h7a1dbc1_0  
pydocstyle                5.0.2                      py_0  
pyflakes                  2.2.0                      py_0  
pygments                  2.6.1                      py_0  
pylint                    2.5.3                    py38_0  
pynacl                    1.4.0            py38h62dcd97_1  
pyodbc                    4.0.30           py38ha925a31_0  
pyopenssl                 19.1.0                     py_1  
pyparsing                 2.4.7                      py_0  
pyqt                      5.9.2            py38ha925a31_4  
pyreadline                2.1                      py38_1  
pyrsistent                0.16.0           py38he774522_0  
pysocks                   1.7.1                    py38_0  
pytables                  3.6.1            py38ha5be198_0  
pytest                    5.4.3                    py38_0  
python                    3.8.3                he1778fa_2  
python-dateutil           2.8.1                      py_0  
python-jsonrpc-server     0.3.4                      py_1  
python-language-server    0.34.1                   py38_0  
python-libarchive-c       2.9                        py_0  
python_abi                3.8                      2_cp38    conda-forge
pytz                      2020.1                     py_0  
pywavelets                1.1.1            py38he774522_0  
pywin32                   302              py38h2bbff1b_2  
pywin32-ctypes            0.2.0                 py38_1000  
pywinpty                  2.0.2            py38h5da7b33_0  
pyyaml                    5.3.1            py38he774522_1  
pyzmq                     19.0.1           py38ha925a31_1  
qdarkstyle                2.8.1                      py_0  
qt                        5.9.7            vc14h73c81de_0  
qtawesome                 0.7.2                      py_0  
qtconsole                 4.7.5                      py_0  
qtpy                      1.9.0                      py_0  
regex                     2020.6.8         py38he774522_0  
requests                  2.24.0                     py_0  
requests-oauthlib         1.3.1                    pypi_0    pypi
rope                      0.17.0                     py_0  
rsa                       4.9                      pypi_0    pypi
rtree                     0.9.4            py38h21ff451_1  
ruamel_yaml               0.15.87          py38he774522_1  
scikit-image              0.16.2           py38h47e9c7a_0  
scikit-learn              1.0.2            py38hb60ee80_0    conda-forge
scipy                     1.7.3            py38h7a0a035_2  
seaborn                   0.10.1                     py_0  
send2trash                1.5.0                    py38_0  
setuptools                49.2.0                   py38_0  
simplegeneric             0.8.1                    py38_2  
singledispatch            3.4.0.3                  py38_0  
sip                       4.19.13          py38ha925a31_0  
six                       1.15.0                     py_0  
snappy                    1.1.8                h33f27b4_0  
snowballstemmer           2.0.0                      py_0  
sortedcollections         1.2.1                      py_0  
sortedcontainers          2.2.2                      py_0  
soupsieve                 2.0.1                      py_0  
sphinx                    3.1.2                      py_0  
sphinxcontrib             1.0                      py38_1  
sphinxcontrib-applehelp   1.0.2                      py_0  
sphinxcontrib-devhelp     1.0.2                      py_0  
sphinxcontrib-htmlhelp    1.0.3                      py_0  
sphinxcontrib-jsmath      1.0.1                      py_0  
sphinxcontrib-qthelp      1.0.3                      py_0  
sphinxcontrib-serializinghtml 1.1.4                      py_0  
sphinxcontrib-websupport  1.2.3                      py_0  
spyder                    4.1.4                    py38_0  
spyder-kernels            1.9.2                    py38_0  
sqlalchemy                1.3.18           py38he774522_0  
sqlite                    3.32.3               h2a8f88b_0  
statsmodels               0.11.1           py38he774522_0  
sympy                     1.6.1                    py38_0  
tbb                       2020.0               h74a9793_0  
tblib                     1.6.0                      py_0  
tensorboard               2.9.1                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.9.2                    pypi_0    pypi
tensorflow-estimator      2.9.0                    pypi_0    pypi
tensorflow-gpu            2.9.2                    pypi_0    pypi
tensorflow-io-gcs-filesystem 0.27.0                   pypi_0    pypi
tensorflow-probability    0.14.0                   pypi_0    pypi
termcolor                 2.1.0                    pypi_0    pypi
terminado                 0.8.3                    py38_0  
testpath                  0.4.4                      py_0  
threadpoolctl             2.1.0              pyh5ca1d4c_0  
tk                        8.6.10               he774522_0  
toml                      0.10.1                     py_0  
toolz                     0.10.0                     py_0  
tornado                   6.0.4            py38he774522_1  
tqdm                      4.64.1           py38haa95532_0  
traitlets                 4.3.3                    py38_0  
typing-extensions         4.3.0            py38haa95532_0  
typing_extensions         4.3.0            py38haa95532_0  
ujson                     1.35             py38he774522_0  
unicodecsv                0.14.1                   py38_0  
urllib3                   1.25.9                     py_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
watchdog                  0.10.3                   py38_0  
wcwidth                   0.2.5                      py_0  
webencodings              0.5.1                    py38_1  
werkzeug                  1.0.1                      py_0  
wheel                     0.34.2                   py38_0  
widgetsnbextension        3.5.1                    py38_0  
win_inet_pton             1.1.0                    py38_0  
win_unicode_console       0.5                      py38_0  
wincertstore              0.2                      py38_0  
winpty                    0.4.3                         4  
wrapt                     1.11.2           py38he774522_0  
xlrd                      1.2.0                      py_0  
xlsxwriter                1.2.9                      py_0  
xlwings                   0.19.5                   py38_0  
xlwt                      1.3.0                    py38_0  
xmltodict                 0.12.0             pyhd3eb1b0_0  
xz                        5.2.5                h62dcd97_0  
yaml                      0.2.5                he774522_0  
yapf                      0.30.0                     py_0  
zeromq                    4.3.2                ha925a31_2  
zict                      2.0.0                      py_0  
zipp                      3.1.0                      py_0  
zlib                      1.2.11               h62dcd97_4  
zope                      1.0                      py38_1  
zope.event                4.4                      py38_0  
zope.interface            4.7.1            py38he774522_0  
zstd                      1.4.5                ha9fde0e_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0
erosion_top:              0.0
erosion_bottom:           0.0
erosion_left:             0.0
erosion_right:            0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
separate_mask:            False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
separate_mask:            False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False
aligner_min_scale:        0.07
aligner_max_scale:        2.0
aligner_distance:         22.5
aligner_roll:             45.0
save_filtered:            False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
cpu:                      True
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.bisenet_fp]
batch-size:               8
cpu:                      False
weights:                  faceswap
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.custom]
batch-size:               8
centering:                face
fill:                     False

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

[recognition.vgg_face2]
batch-size:               16
cpu:                      False

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                face
coverage:                 87.5
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
autoclip:                 False
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
loss_function_2:          mse
loss_weight_2:            100
loss_function_3:          none
loss_weight_3:            0
loss_function_4:          none
loss_weight_4:            0
mask_loss_function:       mse
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.phaze_a]
output_size:              128
shared_fc:                none
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              7
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          none
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_upscales_in_fc:       0
dec_norm:                 none
dec_min_filters:          64
dec_max_filters:          512
dec_slope_mode:           full
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
fs_original_use_alt:      False
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

is the error because cudnn is missing?

User avatar
SilverFox
Posts: 6
Joined: Sun Nov 06, 2022 3:06 pm

Re: Python crashes when trying to train a model.

Post by SilverFox »

I just did some research 'coz that output showed that there is no cudnn installed and i was wondering why.
Installation with FS said i need tensorflow 2.7.0 - 2.10.0 to work but from what i've found out, Tensorflow 2.7.0 is for Cuda 11.2 but RTX2060 is Turing Architecture, so it's Cuda 10.2
looking at this website https://spltech.co.uk/installing-tensor ... u-support/
i should need tensorflow 2.3.0 and cudnn 7.6 to get it to work but installing FS packages said i need at least TF2.7, so does that mean that Turing architecture isn't longer supported?

https://docs.nvidia.com/deeplearning/cu ... index.html

So what's the workaround here now?

Code: Select all

11/08/2022 01:10:14 MainProcess     MainThread                     logger          log_setup                      INFO     Log level set to: INFO
11/08/2022 01:10:16 MainProcess     MainThread                     launcher        _handle_import_error           ERROR    The minimum supported Tensorflow is version 2.7 but you have version 2.3 installed. Please upgrade Tensorflow.
11/08/2022 01:10:16 MainProcess     MainThread                     launcher        _handle_import_error           INFO     Press "ENTER" to dismiss the message and close FaceSwap

is there anything else i have to downgrade to get FS to work? Updating to TF 2.7 won't work or i would only have CPU mode as it isn't compatible with RTX2000 Gen.
What package have a dependency for TF2.7? Keras? or anything else?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Python crashes when trying to train a model.

Post by torzdf »

You are overcomplicating it.

You should remove any and all globally installed versions of Cuda and cuDNN. These will conflict with the version that Faceswap installs locally.

The lookup for cuDNN in the system output does not currently work, but both of these should say "no global version found".

Latest Faceswap works fine with any GPU on or above the 9xx series.

These messages:

Code: Select all

"opencv-python>= 4.6.0.0" not available in Conda. Installing with pip
"ffmpy>= 0.3.0" not available in Conda. Installing with pip
"nvidia-ml-py 11.515,<300" not available in Conda. Installing with pip

Are normal. Those packages are not available in Conda, but the setup script will always check Conda first.

To resolve the other install issues I would need a copy if the generated faceswap_setup.log from within your faceswap folder.

My word is final

Locked