I"m new here, so please excuse my cluelessness.
just for context, i made one video with moderate success. I followed some random "faceswap for noobs" video online. it wasnt great, but relatively okay for a first attempt. for my second attempt, I've been using the guides and reading some forums posted here, and following the recommended settings generally speaking. everything seemed to be going okay, and after about 10 hours or so, i had 50K+ iterations and the target faces (B) were looking pretty good in the preview.
the training was stopped and restarted and everything seemed okay.... but about 10 mins later it crashed. once i came back and found it had exited, I tried to convert what i already had, but that crashed too. then i tried restarting the training and that crashed as well. i no longer have the console output, but attached is the log file from that first crash
any help will be greatly appreciated, i dont want to throw out 12 hours of training if i dont have to.
Code: Select all
01/17/2023 09:35:28 MainProcess _training serializer save DEBUG filename: C:\Users\doggy\Desktop\fsdir\modelAB\original_state.json, data type: <class 'dict'>
01/17/2023 09:35:28 MainProcess _training serializer _check_extension DEBUG Original filename: 'C:\Users\doggy\Desktop\fsdir\modelAB\original_state.json', final filename: 'C:\Users\doggy\Desktop\fsdir\modelAB\original_state.json'
01/17/2023 09:35:28 MainProcess _training serializer marshal DEBUG data type: <class 'dict'>
01/17/2023 09:35:28 MainProcess _training serializer marshal DEBUG returned data type: <class 'bytes'>
01/17/2023 09:35:28 MainProcess _training model save DEBUG Saved State
01/17/2023 09:35:28 MainProcess _training io save INFO [Saved models] - Average loss since last save: face_a: 0.03209, face_b: 0.03462
01/17/2023 09:35:32 MainProcess _training _base generate_preview DEBUG Generating preview (is_timelapse: False)
01/17/2023 09:35:32 MainProcess _training _base generate_preview DEBUG Generated samples: is_timelapse: False, images: {'feed': {'a': (14, 64, 64, 3), 'b': (14, 64, 64, 3)}, 'samples': {'a': (14, 94, 94, 3), 'b': (14, 94, 94, 3)}, 'sides': {'a': (14, 64, 64, 1), 'b': (14, 64, 64, 1)}}
01/17/2023 09:35:32 MainProcess _training _base compile_sample DEBUG Compiling samples: (side: 'a', samples: 14)
01/17/2023 09:35:32 MainProcess _training _base compile_sample DEBUG Compiling samples: (side: 'b', samples: 14)
01/17/2023 09:35:32 MainProcess _training _base compile_sample DEBUG Compiled Samples: {'a': [(14, 64, 64, 3), (14, 94, 94, 3), (14, 64, 64, 1)], 'b': [(14, 64, 64, 3), (14, 94, 94, 3), (14, 64, 64, 1)]}
01/17/2023 09:35:32 MainProcess _training _base show_sample DEBUG Showing sample
01/17/2023 09:35:32 MainProcess _training _base _get_predictions DEBUG Getting Predictions
01/17/2023 09:35:36 MainProcess _training _base _get_predictions DEBUG Returning predictions: {'a_a': (14, 64, 64, 3), 'b_b': (14, 64, 64, 3), 'a_b': (14, 64, 64, 3), 'b_a': (14, 64, 64, 3)}
01/17/2023 09:35:36 MainProcess _training _base _to_full_frame DEBUG side: 'a', number of sample arrays: 3, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
01/17/2023 09:35:36 MainProcess _training _base _process_full DEBUG full_size: 94, prediction_size: 64, color: (0.0, 0.0, 1.0)
01/17/2023 09:35:36 MainProcess _training _base _process_full DEBUG Overlayed background. Shape: (14, 94, 94, 3)
01/17/2023 09:35:36 MainProcess _training _base _compile_masked DEBUG masked shapes: [(14, 64, 64, 3), (14, 64, 64, 3), (14, 64, 64, 3)]
01/17/2023 09:35:36 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:35:36 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:35:36 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:35:36 MainProcess _training _base _get_headers DEBUG side: 'a', width: 94
01/17/2023 09:35:36 MainProcess _training _base _get_headers DEBUG height: 20, total_width: 282
01/17/2023 09:35:36 MainProcess _training _base _get_headers DEBUG texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(53, 7), (86, 7), (75, 7)], text_x: [20, 98, 197], text_y: 13
01/17/2023 09:35:36 MainProcess _training _base _get_headers DEBUG header_box.shape: (20, 282, 3)
01/17/2023 09:35:36 MainProcess _training _base _to_full_frame DEBUG side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
01/17/2023 09:35:36 MainProcess _training _base _process_full DEBUG full_size: 94, prediction_size: 64, color: (0.0, 0.0, 1.0)
01/17/2023 09:35:36 MainProcess _training _base _process_full DEBUG Overlayed background. Shape: (14, 94, 94, 3)
01/17/2023 09:35:36 MainProcess _training _base _compile_masked DEBUG masked shapes: [(14, 64, 64, 3), (14, 64, 64, 3), (14, 64, 64, 3)]
01/17/2023 09:35:36 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:35:36 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:35:36 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:35:36 MainProcess _training _base _get_headers DEBUG side: 'b', width: 94
01/17/2023 09:35:36 MainProcess _training _base _get_headers DEBUG height: 20, total_width: 282
01/17/2023 09:35:36 MainProcess _training _base _get_headers DEBUG texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(44, 7), (64, 7), (75, 7)], text_x: [25, 109, 197], text_y: 13
01/17/2023 09:35:36 MainProcess _training _base _get_headers DEBUG header_box.shape: (20, 282, 3)
01/17/2023 09:35:36 MainProcess _training _base _duplicate_headers DEBUG side: a header.shape: (20, 282, 3)
01/17/2023 09:35:36 MainProcess _training _base _duplicate_headers DEBUG side: b header.shape: (20, 282, 3)
01/17/2023 09:35:36 MainProcess _training _base _stack_images DEBUG Stack images
01/17/2023 09:35:36 MainProcess _training _base get_transpose_axes DEBUG Even number of images to stack
01/17/2023 09:35:36 MainProcess _training _base _stack_images DEBUG Stacked images
01/17/2023 09:35:36 MainProcess _training _base _compile_preview DEBUG Compiled sample
01/17/2023 09:35:37 MainProcess _training train _show DEBUG Updating preview: (name: Training - 'S': Save Now. 'R': Refresh Preview. 'M': Toggle Mask. 'F': Toggle Screen Fit-Actual Size. 'ENTER': Save and Quit)
01/17/2023 09:35:37 MainProcess _training train _show DEBUG Generating preview for GUI
01/17/2023 09:35:37 MainProcess _training train _show DEBUG Generated preview for GUI: 'C:\Users\doggy\faceswap\lib\gui\.cache\preview\.gui_training_preview.png'
01/17/2023 09:35:37 MainProcess _training train _show DEBUG Updated preview: (name: Training - 'S': Save Now. 'R': Refresh Preview. 'M': Toggle Mask. 'F': Toggle Screen Fit-Actual Size. 'ENTER': Save and Quit)
01/17/2023 09:35:37 MainProcess _training train _run_training_cycle INFO [Preview Updated]
01/17/2023 09:44:47 MainProcess _training _base output_timelapse DEBUG Ouputting time-lapse
01/17/2023 09:44:47 MainProcess _training _base output_timelapse DEBUG Getting time-lapse samples
01/17/2023 09:44:47 MainProcess _training _base generate_preview DEBUG Generating preview (is_timelapse: True)
01/17/2023 09:44:47 MainProcess _training _base generate_preview DEBUG Generated samples: is_timelapse: True, images: {'feed': {'a': (14, 64, 64, 3), 'b': (14, 64, 64, 3)}, 'samples': {'a': (14, 94, 94, 3), 'b': (14, 94, 94, 3)}, 'sides': {'a': (14, 64, 64, 1), 'b': (14, 64, 64, 1)}}
01/17/2023 09:44:47 MainProcess _training _base compile_sample DEBUG Compiling samples: (side: 'a', samples: 14)
01/17/2023 09:44:47 MainProcess _training _base compile_sample DEBUG Compiling samples: (side: 'b', samples: 14)
01/17/2023 09:44:47 MainProcess _training _base compile_sample DEBUG Compiled Samples: {'a': [(14, 64, 64, 3), (14, 94, 94, 3), (14, 64, 64, 1)], 'b': [(14, 64, 64, 3), (14, 94, 94, 3), (14, 64, 64, 1)]}
01/17/2023 09:44:47 MainProcess _training _base output_timelapse DEBUG Got time-lapse samples: {'a': 3, 'b': 3}
01/17/2023 09:44:47 MainProcess _training _base show_sample DEBUG Showing sample
01/17/2023 09:44:47 MainProcess _training _base _get_predictions DEBUG Getting Predictions
01/17/2023 09:44:51 MainProcess _training _base _get_predictions DEBUG Returning predictions: {'a_a': (14, 64, 64, 3), 'b_b': (14, 64, 64, 3), 'a_b': (14, 64, 64, 3), 'b_a': (14, 64, 64, 3)}
01/17/2023 09:44:51 MainProcess _training _base _to_full_frame DEBUG side: 'a', number of sample arrays: 3, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
01/17/2023 09:44:51 MainProcess _training _base _process_full DEBUG full_size: 94, prediction_size: 64, color: (0.0, 0.0, 1.0)
01/17/2023 09:44:51 MainProcess _training _base _process_full DEBUG Overlayed background. Shape: (14, 94, 94, 3)
01/17/2023 09:44:51 MainProcess _training _base _compile_masked DEBUG masked shapes: [(14, 64, 64, 3), (14, 64, 64, 3), (14, 64, 64, 3)]
01/17/2023 09:44:51 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:44:51 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:44:51 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:44:51 MainProcess _training _base _get_headers DEBUG side: 'a', width: 94
01/17/2023 09:44:51 MainProcess _training _base _get_headers DEBUG height: 20, total_width: 282
01/17/2023 09:44:51 MainProcess _training _base _get_headers DEBUG texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(53, 7), (86, 7), (75, 7)], text_x: [20, 98, 197], text_y: 13
01/17/2023 09:44:51 MainProcess _training _base _get_headers DEBUG header_box.shape: (20, 282, 3)
01/17/2023 09:44:51 MainProcess _training _base _to_full_frame DEBUG side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
01/17/2023 09:44:51 MainProcess _training _base _process_full DEBUG full_size: 94, prediction_size: 64, color: (0.0, 0.0, 1.0)
01/17/2023 09:44:51 MainProcess _training _base _process_full DEBUG Overlayed background. Shape: (14, 94, 94, 3)
01/17/2023 09:44:52 MainProcess _training _base _compile_masked DEBUG masked shapes: [(14, 64, 64, 3), (14, 64, 64, 3), (14, 64, 64, 3)]
01/17/2023 09:44:52 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:44:52 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:44:52 MainProcess _training _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 94, 94, 3)
01/17/2023 09:44:52 MainProcess _training _base _get_headers DEBUG side: 'b', width: 94
01/17/2023 09:44:52 MainProcess _training _base _get_headers DEBUG height: 20, total_width: 282
01/17/2023 09:44:52 MainProcess _training _base _get_headers DEBUG texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(44, 7), (64, 7), (75, 7)], text_x: [25, 109, 197], text_y: 13
01/17/2023 09:44:52 MainProcess _training _base _get_headers DEBUG header_box.shape: (20, 282, 3)
01/17/2023 09:44:52 MainProcess _training _base _duplicate_headers DEBUG side: a header.shape: (20, 282, 3)
01/17/2023 09:44:52 MainProcess _training _base _duplicate_headers DEBUG side: b header.shape: (20, 282, 3)
01/17/2023 09:44:52 MainProcess _training _base _stack_images DEBUG Stack images
01/17/2023 09:44:52 MainProcess _training _base get_transpose_axes DEBUG Even number of images to stack
01/17/2023 09:44:52 MainProcess _training _base _stack_images DEBUG Stacked images
01/17/2023 09:44:52 MainProcess _training _base _compile_preview DEBUG Compiled sample
01/17/2023 09:44:52 MainProcess _training _base output_timelapse DEBUG Created time-lapse: 'C:\Users\doggy\Desktop\fsdir\timelapse_output\1673977492.jpg'
01/17/2023 09:44:52 MainProcess _training train _run_training_cycle DEBUG Saving (save_iterations: True, save_now: False) Iteration: (iteration: 32750)
01/17/2023 09:44:52 MainProcess _training io save DEBUG Backing up and saving models
01/17/2023 09:44:52 MainProcess _training io _get_save_averages DEBUG Getting save averages
01/17/2023 09:44:52 MainProcess _training io _get_save_averages DEBUG Average losses since last save: [0.03278469024598599, 0.034648136049509046]
01/17/2023 09:44:52 MainProcess _training io _should_backup DEBUG Should backup: False
01/17/2023 09:44:53 MainProcess _training multithreading run DEBUG Error in thread (_training): Can't decrement id ref count (unable to extend file properly)
01/17/2023 09:44:54 MainProcess MainThread train _monitor DEBUG Thread error detected
01/17/2023 09:44:54 MainProcess MainThread train _monitor DEBUG Closed Monitor
01/17/2023 09:44:54 MainProcess MainThread train _end_thread DEBUG Ending Training thread
01/17/2023 09:44:54 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...
01/17/2023 09:44:54 MainProcess MainThread multithreading join DEBUG Joining Threads: '_training'
01/17/2023 09:44:54 MainProcess MainThread multithreading join DEBUG Joining Thread: '_training'
01/17/2023 09:44:54 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training'
Traceback (most recent call last):
File "C:\Users\doggy\faceswap\lib\cli\launcher.py", line 217, in execute_script
process.process()
File "C:\Users\doggy\faceswap\scripts\train.py", line 218, in process
self._end_thread(thread, err)
File "C:\Users\doggy\faceswap\scripts\train.py", line 258, in _end_thread
thread.join()
File "C:\Users\doggy\faceswap\lib\multithreading.py", line 217, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\doggy\faceswap\lib\multithreading.py", line 96, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\doggy\faceswap\scripts\train.py", line 280, in _training
raise err
File "C:\Users\doggy\faceswap\scripts\train.py", line 270, in _training
self._run_training_cycle(model, trainer)
File "C:\Users\doggy\faceswap\scripts\train.py", line 372, in _run_training_cycle
model.save(is_exit=False)
File "C:\Users\doggy\faceswap\plugins\train\model\_base\model.py", line 436, in save
self._io.save(is_exit=is_exit)
File "C:\Users\doggy\faceswap\plugins\train\model\_base\io.py", line 207, in save
self._plugin.model.save(self._filename, include_optimizer=include_optimizer)
File "C:\Users\doggy\MiniConda3\envs\faceswap\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\doggy\MiniConda3\envs\faceswap\lib\site-packages\h5py\_hl\files.py", line 552, in close
self.id._close_open_objects(h5f.OBJ_LOCAL | h5f.OBJ_FILE)
File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5f.pyx", line 360, in h5py.h5f.FileID._close_open_objects
RuntimeError: Can't decrement id ref count (unable to extend file properly)
============ System Information ============
encoding: cp1252
git_branch: master
git_commits: bcef3b4 Merge branch 'staging'
gpu_cuda: No global version found. Check Conda packages for Conda Cuda
gpu_cudnn: No global version found. Check Conda packages for Conda cuDNN
gpu_devices: GPU_0: NVIDIA GeForce RTX 2060
gpu_devices_active: GPU_0
gpu_driver: 516.94
gpu_vram: GPU_0: 6144MB
os_machine: AMD64
os_platform: Windows-10-10.0.19045-SP0
os_release: 10
py_command: C:\Users\doggy\faceswap\faceswap.py train -A C:/Users/doggy/Desktop/fsdir/extract_A -B C:/Users/doggy/Desktop/fsdir/extract_B -m C:/Users/doggy/Desktop/fsdir/modelAB -t original -bs 8 -it 1000000 -D central-storage -s 250 -ss 25000 -tia C:/Users/doggy/Desktop/fsdir/extract_A -tib C:/Users/doggy/Desktop/fsdir/extract_B -to C:/Users/doggy/Desktop/fsdir/timelapse_output -L INFO -gui
py_conda_version: conda 22.11.1
py_implementation: CPython
py_version: 3.9.15
py_virtual_env: True
sys_cores: 12
sys_processor: AMD64 Family 23 Model 1 Stepping 1, AuthenticAMD
sys_ram: Total: 32718MB, Available: 6286MB, Used: 26432MB, Free: 6286MB
=============== Pip Packages ===============
absl-py @ file:///C:/b/abs_5babsu7y5x/croot/absl-py_1666362945682/work
astunparse==1.6.3
cachetools==5.2.0
certifi==2022.12.7
charset-normalizer==2.1.1
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1632508026186/work
colorama @ file:///C:/Windows/TEMP/abs_9439aeb1-0254-449a-96f7-33ab5eb17fc8apleb4yn/croots/recipe/colorama_1657009099097/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
dm-tree @ file:///C:/b/abs_10z0iy5knj/croot/dm-tree_1671027465819/work
fastcluster @ file:///D:/bld/fastcluster_1649783471014/work
ffmpy==0.3.0
flatbuffers==22.12.6
flit_core @ file:///opt/conda/conda-bld/flit-core_1644941570762/work/source/flit_core
fonttools==4.25.0
gast==0.4.0
google-auth==2.15.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.1
h5py==3.7.0
idna==3.4
imageio @ file:///C:/Windows/TEMP/abs_24c1b783-7540-4ca9-a1b1-0e8aa8e6ae64hb79ssux/croots/recipe/imageio_1658785038775/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1649960641006/work
importlib-metadata==5.2.0
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1663332044897/work
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/ci/kiwisolver_1653292407425/work
libclang==14.0.6
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib @ file:///C:/ci/matplotlib-suite_1660169687702/work
mkl-fft==1.3.1
mkl-random @ file:///C:/ci/mkl_random_1626186184308/work
mkl-service==2.4.0
munkres==1.1.4
numexpr @ file:///C:/b/abs_a7kbak88hk/croot/numexpr_1668713882979/work
numpy @ file:///C:/b/abs_5ct9ex77k9/croot/numpy_and_numpy_base_1668593740598/work
nvidia-ml-py==11.515.75
oauthlib==3.2.2
opencv-python==4.6.0.66
opt-einsum==3.3.0
packaging @ file:///tmp/build/80754af9/packaging_1637314298585/work
Pillow==9.3.0
ply==3.11
protobuf==3.19.6
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work
PyQt5==5.15.7
PyQt5-sip @ file:///C:/Windows/Temp/abs_d7gmd2jg8i/croots/recipe/pyqt-split_1659273064801/work/pyqt_sip
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==305.1
pywinpty @ file:///C:/ci_310/pywinpty_1644230983541/work/target/wheels/pywinpty-2.0.2-cp39-none-win_amd64.whl
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn @ file:///D:/bld/scikit-learn_1659726281030/work
scipy @ file:///C:/bld/scipy_1658811088396/work
sip @ file:///C:/Windows/Temp/abs_b8fxd17m2u/croots/recipe/sip_1659012372737/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.10.0
tensorflow-gpu==2.10.1
tensorflow-io-gcs-filesystem==0.29.0
tensorflow-probability @ file:///tmp/build/80754af9/tensorflow-probability_1633017132682/work
termcolor==2.1.1
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
tornado @ file:///C:/ci/tornado_1662458743919/work
tqdm @ file:///C:/b/abs_0axbz66qik/croots/recipe/tqdm_1664392691071/work
typing_extensions @ file:///C:/b/abs_89eui86zuq/croot/typing_extensions_1669923792806/work
urllib3==1.26.13
Werkzeug==2.2.2
wincertstore==0.2
wrapt==1.14.1
zipp==3.11.0
============== Conda Packages ==============
# packages in environment at C:\Users\doggy\MiniConda3\envs\faceswap:
#
# Name Version Build Channel
absl-py 1.3.0 py39haa95532_0
astunparse 1.6.3 pypi_0 pypi
blas 1.0 mkl
brotli 1.0.9 h2bbff1b_7
brotli-bin 1.0.9 h2bbff1b_7
ca-certificates 2022.12.7 h5b45459_0 conda-forge
cachetools 5.2.0 pypi_0 pypi
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
charset-normalizer 2.1.1 pypi_0 pypi
cloudpickle 2.0.0 pyhd3eb1b0_0
colorama 0.4.5 py39haa95532_0
cudatoolkit 11.2.2 h933977f_10 conda-forge
cudnn 8.1.0.77 h3e0f4f4_0 conda-forge
cycler 0.11.0 pyhd3eb1b0_0
decorator 5.1.1 pyhd3eb1b0_0
dm-tree 0.1.7 py39hd77b12b_1
fastcluster 1.2.6 py39h2e25243_1 conda-forge
ffmpeg 4.3.1 ha925a31_0 conda-forge
ffmpy 0.3.0 pypi_0 pypi
flatbuffers 22.12.6 pypi_0 pypi
flit-core 3.6.0 pyhd3eb1b0_0
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.12.1 ha860e81_0
gast 0.4.0 pypi_0 pypi
git 2.34.1 haa95532_0
glib 2.69.1 h5dc1a3c_2
google-auth 2.15.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.51.1 pypi_0 pypi
gst-plugins-base 1.18.5 h9e645db_0
gstreamer 1.18.5 hd78058f_0
h5py 3.7.0 pypi_0 pypi
icu 58.2 ha925a31_3
idna 3.4 pypi_0 pypi
imageio 2.19.3 py39haa95532_0
imageio-ffmpeg 0.4.7 pyhd8ed1ab_0 conda-forge
importlib-metadata 5.2.0 pypi_0 pypi
intel-openmp 2021.4.0 haa95532_3556
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h2bbff1b_0
keras 2.10.0 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.4.2 py39hd77b12b_0
lerc 3.0 hd77b12b_0
libblas 3.9.0 1_h8933c1f_netlib conda-forge
libbrotlicommon 1.0.9 h2bbff1b_7
libbrotlidec 1.0.9 h2bbff1b_7
libbrotlienc 1.0.9 h2bbff1b_7
libcblas 3.9.0 5_hd5c7e75_netlib conda-forge
libclang 14.0.6 pypi_0 pypi
libdeflate 1.8 h2bbff1b_5
libffi 3.4.2 hd77b12b_6
libiconv 1.16 h2bbff1b_2
liblapack 3.9.0 5_hd5c7e75_netlib conda-forge
libogg 1.3.5 h2bbff1b_1
libpng 1.6.37 h2a8f88b_0
libtiff 4.4.0 h8a3f274_2
libvorbis 1.3.7 he774522_0
libwebp 1.2.4 h2bbff1b_0
libwebp-base 1.2.4 h2bbff1b_0
libxml2 2.9.14 h0ad7f3c_0
libxslt 1.1.35 h2bbff1b_0
lz4-c 1.9.4 h2bbff1b_0
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
markdown 3.4.1 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib 3.5.2 py39haa95532_0
matplotlib-base 3.5.2 py39hd77b12b_0
mkl 2021.4.0 haa95532_640
mkl-service 2.4.0 py39h2bbff1b_0
mkl_fft 1.3.1 py39h277e83a_0
mkl_random 1.2.2 py39hf11a4ad_0
msys2-conda-epoch 20160418 1 conda-forge
munkres 1.1.4 py_0
numexpr 2.8.4 py39h5b0cc5e_0
numpy 1.23.4 py39h3b20f71_0
numpy-base 1.23.4 py39h4da318b_0
nvidia-ml-py 11.515.75 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
opencv-python 4.6.0.66 pypi_0 pypi
openssl 1.1.1s h2bbff1b_0
opt-einsum 3.3.0 pypi_0 pypi
packaging 21.3 pyhd3eb1b0_0
pcre 8.45 hd77b12b_0
pillow 9.3.0 py39hdc2b20a_0
pip 22.3.1 py39haa95532_0
ply 3.11 py39haa95532_0
protobuf 3.19.6 pypi_0 pypi
psutil 5.9.0 py39h2bbff1b_0
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pyparsing 3.0.9 py39haa95532_0
pyqt 5.15.7 py39hd77b12b_0
pyqt5-sip 12.11.0 py39hd77b12b_0
python 3.9.15 h6244533_2
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.9 2_cp39 conda-forge
pywin32 305 py39h2bbff1b_0
pywinpty 2.0.2 py39h5da7b33_0
qt-main 5.15.2 he8e5bd7_7
qt-webengine 5.15.9 hb9a9bb5_4
qtwebkit 5.212 h3ad3cdb_4
requests 2.28.1 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.9 pypi_0 pypi
scikit-learn 1.1.2 py39hfd4428b_0 conda-forge
scipy 1.8.1 py39h5567194_2 conda-forge
setuptools 65.5.0 py39haa95532_0
sip 6.6.2 py39hd77b12b_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.40.0 h2bbff1b_0
tensorboard 2.10.1 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tensorflow-estimator 2.10.0 pypi_0 pypi
tensorflow-gpu 2.10.1 pypi_0 pypi
tensorflow-io-gcs-filesystem 0.29.0 pypi_0 pypi
tensorflow-probability 0.14.0 pyhd3eb1b0_0
termcolor 2.1.1 pypi_0 pypi
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tk 8.6.12 h2bbff1b_0
toml 0.10.2 pyhd3eb1b0_0
tornado 6.2 py39h2bbff1b_0
tqdm 4.64.1 py39haa95532_0
typing-extensions 4.4.0 py39haa95532_0
typing_extensions 4.4.0 py39haa95532_0
tzdata 2022g h04d1e81_0
urllib3 1.26.13 pypi_0 pypi
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 2.2.2 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
wincertstore 0.2 py39haa95532_2
winpty 0.4.3 4
wrapt 1.14.1 pypi_0 pypi
xz 5.2.8 h8cc25b3_0
zipp 3.11.0 pypi_0 pypi
zlib 1.2.13 h8cc25b3_0
zstd 1.5.2 h19a0ad4_0
=============== State File =================
{
"name": "original",
"sessions": {
"1": {
"timestamp": 1673918226.3963852,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1750,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"autoclip": false,
"allow_growth": false,
"mixed_precision": true,
"nan_protection": true,
"convert_batchsize": 16,
"loss_function": "ssim",
"loss_function_2": "mse",
"loss_weight_2": 100,
"loss_function_3": null,
"loss_weight_3": 0,
"loss_function_4": null,
"loss_weight_4": 0,
"mask_loss_function": "mse",
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"2": {
"timestamp": 1673919681.2474973,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 12500,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"autoclip": false,
"allow_growth": false,
"mixed_precision": true,
"nan_protection": true,
"convert_batchsize": 16,
"loss_function": "ssim",
"loss_function_2": "mse",
"loss_weight_2": 100,
"loss_function_3": null,
"loss_weight_3": 0,
"loss_function_4": null,
"loss_weight_4": 0,
"mask_loss_function": "mse",
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"3": {
"timestamp": 1673930399.201518,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 8,
"iterations": 750,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"autoclip": false,
"allow_growth": false,
"mixed_precision": true,
"nan_protection": true,
"convert_batchsize": 16,
"loss_function": "ssim",
"loss_function_2": "mse",
"loss_weight_2": 100,
"loss_function_3": null,
"loss_weight_3": 0,
"loss_function_4": null,
"loss_weight_4": 0,
"mask_loss_function": "mse",
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"4": {
"timestamp": 1673931037.1198366,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 8,
"iterations": 6000,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"autoclip": false,
"allow_growth": false,
"mixed_precision": true,
"nan_protection": true,
"convert_batchsize": 16,
"loss_function": "ssim",
"loss_function_2": "mse",
"loss_weight_2": 100,
"loss_function_3": null,
"loss_weight_3": 0,
"loss_function_4": null,
"loss_weight_4": 0,
"mask_loss_function": "mse",
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"5": {
"timestamp": 1673935120.988235,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 8,
"iterations": 32500,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"autoclip": false,
"allow_growth": false,
"mixed_precision": true,
"nan_protection": true,
"convert_batchsize": 16,
"loss_function": "ssim",
"loss_function_2": "mse",
"loss_weight_2": 100,
"loss_function_3": null,
"loss_weight_3": 0,
"loss_function_4": null,
"loss_weight_4": 0,
"mask_loss_function": "mse",
"eye_multiplier": 3,
"mouth_multiplier": 2
}
}
},
"lowest_avg_loss": {
"a": 0.03208780471980572,
"b": 0.03462204637378454
},
"iterations": 53500,
"mixed_precision_layers": [],
"config": {
"centering": "legacy",
"coverage": 67.5,
"optimizer": "adam",
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"autoclip": false,
"allow_growth": false,
"mixed_precision": true,
"nan_protection": true,
"convert_batchsize": 16,
"loss_function": "ssim",
"loss_function_2": "mse",
"loss_weight_2": 100,
"loss_function_3": null,
"loss_weight_3": 0,
"loss_function_4": null,
"loss_weight_4": 0,
"mask_loss_function": "mse",
"eye_multiplier": 3,
"mouth_multiplier": 2,
"penalized_mask_loss": true,
"mask_type": "extended",
"mask_blur_kernel": 3,
"mask_threshold": 4,
"learn_mask": false,
"lowmem": false
}
}
================= Configs ==================
--------- .faceswap ---------
backend: nvidia
--------- convert.ini ---------
[color.color_transfer]
clip: True
preserve_paper: True
[color.manual_balance]
colorspace: HSV
balance_1: 0.0
balance_2: 0.0
balance_3: 0.0
contrast: 0.0
brightness: 0.0
[color.match_hist]
threshold: 99.0
[mask.mask_blend]
type: normalized
kernel_size: 3
passes: 4
threshold: 4
erosion: 0.0
erosion_top: 0.0
erosion_bottom: 0.0
erosion_left: 0.0
erosion_right: 0.0
[scaling.sharpen]
method: none
amount: 150
radius: 0.3
threshold: 5.0
[writer.ffmpeg]
container: mp4
codec: libx264
crf: 23
preset: medium
tune: none
profile: auto
level: auto
skip_mux: False
[writer.gif]
fps: 25
loop: 0
palettesize: 256
subrectangles: False
[writer.opencv]
format: png
draw_transparent: False
separate_mask: False
jpg_quality: 75
png_compress_level: 3
[writer.pillow]
format: png
draw_transparent: False
separate_mask: False
optimize: False
gif_interlace: True
jpg_quality: 75
png_compress_level: 3
tif_compression: tiff_deflate
--------- extract.ini ---------
[global]
allow_growth: False
aligner_min_scale: 0.07
aligner_max_scale: 2.0
aligner_distance: 22.5
aligner_roll: 45.0
aligner_features: True
filter_refeed: True
save_filtered: False
realign_refeeds: True
filter_realign: True
[align.fan]
batch-size: 12
[detect.cv2_dnn]
confidence: 50
[detect.mtcnn]
minsize: 20
scalefactor: 0.709
batch-size: 8
cpu: True
threshold_1: 0.6
threshold_2: 0.7
threshold_3: 0.7
[detect.s3fd]
confidence: 70
batch-size: 4
[mask.bisenet_fp]
batch-size: 8
cpu: False
weights: faceswap
include_ears: False
include_hair: False
include_glasses: True
[mask.custom]
batch-size: 8
centering: face
fill: False
[mask.unet_dfl]
batch-size: 8
[mask.vgg_clear]
batch-size: 6
[mask.vgg_obstructed]
batch-size: 2
[recognition.vgg_face2]
batch-size: 16
cpu: False
--------- gui.ini ---------
[global]
fullscreen: False
tab: extract
options_panel_width: 30
console_panel_height: 20
icon_size: 14
font: default
font_size: 9
autosave_last_session: prompt
timeout: 120
auto_load_model_stats: True
--------- train.ini ---------
[global]
centering: legacy
coverage: 67.5
icnr_init: False
conv_aware_init: False
optimizer: adam
learning_rate: 5e-05
epsilon_exponent: -7
autoclip: False
reflect_padding: False
allow_growth: False
mixed_precision: True
nan_protection: True
convert_batchsize: 16
[global.loss]
loss_function: ssim
loss_function_2: mse
loss_weight_2: 100
loss_function_3: None
loss_weight_3: 0
loss_function_4: None
loss_weight_4: 0
mask_loss_function: mse
eye_multiplier: 3
mouth_multiplier: 2
penalized_mask_loss: True
mask_type: extended
mask_blur_kernel: 3
mask_threshold: 4
learn_mask: False
[model.dfaker]
output_size: 128
[model.dfl_h128]
lowmem: False
[model.dfl_sae]
input_size: 128
architecture: df
autoencoder_dims: 0
encoder_dims: 42
decoder_dims: 21
multiscale_decoder: False
[model.dlight]
features: best
details: good
output_size: 256
[model.original]
lowmem: False
[model.phaze_a]
output_size: 128
shared_fc: None
enable_gblock: True
split_fc: True
split_gblock: False
split_decoders: False
enc_architecture: fs_original
enc_scaling: 7
enc_load_weights: True
bottleneck_type: dense
bottleneck_norm: None
bottleneck_size: 1024
bottleneck_in_encoder: True
fc_depth: 1
fc_min_filters: 1024
fc_max_filters: 1024
fc_dimensions: 4
fc_filter_slope: -0.5
fc_dropout: 0.0
fc_upsampler: upsample2d
fc_upsamples: 1
fc_upsample_filters: 512
fc_gblock_depth: 3
fc_gblock_min_nodes: 512
fc_gblock_max_nodes: 512
fc_gblock_filter_slope: -0.5
fc_gblock_dropout: 0.0
dec_upscale_method: subpixel
dec_upscales_in_fc: 0
dec_norm: None
dec_min_filters: 64
dec_max_filters: 512
dec_slope_mode: full
dec_filter_slope: -0.45
dec_res_blocks: 1
dec_output_kernel: 5
dec_gaussian: True
dec_skip_last_residual: True
freeze_layers: keras_encoder
load_layers: encoder
fs_original_depth: 4
fs_original_min_filters: 128
fs_original_max_filters: 1024
fs_original_use_alt: False
mobilenet_width: 1.0
mobilenet_depth: 1
mobilenet_dropout: 0.001
mobilenet_minimalistic: False
[model.realface]
input_size: 64
output_size: 128
dense_nodes: 1536
complexity_encoder: 128
complexity_decoder: 512
[model.unbalanced]
input_size: 128
lowmem: False
nodes: 1024
complexity_encoder: 128
complexity_decoder_a: 384
complexity_decoder_b: 512
[model.villain]
lowmem: False
[trainer.original]
preview_images: 14
zoom_amount: 5
rotation_range: 10
shift_range: 5
flip_chance: 50
color_lightness: 30
color_ab: 8
color_clahe_chance: 50
color_clahe_max_size: 4