Bug: Updated to Latest faceswap version now crashing when starting training

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

Just upgraded to the latest version of faceswap, now getting the below when beginning a training session.

Code: Select all

05/06/2022 16:37:52 INFO     Log level set to: INFO
05/06/2022 16:37:57 INFO     Model A Directory: 'C:\Convert AI\LVR2\Training Set' (10560 images)
05/06/2022 16:37:57 INFO     Model B Directory: 'C:\Convert AI\L Work Folder\Brand New Set 512' (726 images)
05/06/2022 16:37:57 INFO     Training data directory: C:\Convert AI\LVR2\test
05/06/2022 16:37:57 INFO     ===================================================
05/06/2022 16:37:57 INFO       Starting
05/06/2022 16:37:57 INFO     ===================================================
05/06/2022 16:37:58 INFO     Loading data, this may take a while...
05/06/2022 16:37:58 INFO     Loading Model from Phaze_A plugin...
05/06/2022 16:37:58 INFO     No existing state file found. Generating.
05/06/2022 16:38:02 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "C:\Convert\faceswap\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "C:\Convert\faceswap\scripts\train.py", line 189, in process
    err = self._monitor(thread)
  File "C:\Convert\faceswap\scripts\train.py", line 395, in _monitor
    if keypress.kbhit():
  File "C:\Convert\faceswap\lib\keypress.py", line 90, in kbhit
    if self.is_gui and os.name != "nt" or not sys.stdout.isatty():
NameError: name 'sys' is not defined
05/06/2022 16:38:02 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Convert\faceswap\crash_report.2022.05.06.163757415186.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Process exited.

User avatar
torzdf
Posts: 1775
Joined: Fri Jul 12, 2019 12:53 am
Answers: 135
Has thanked: 76 times
Been thanked: 357 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by torzdf »

Yeah. It was a major tensorflow update. Unfortunately, because Conda are dragging their heels so much on Tensorflow updates, it has made 'auto-update' for these kinds of upgrades non-feasible/broken.

If you delete faceswap + re-run the installer, all should work again. I will try to put some kind of messaging in place next time there is a major backend update.


Edit: Actually there was a bug in there too. Just pushed an update.

My word is final


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

Oh yes, we were aware of the tensorflow 2.8 manual upgrade - we did it inside miniconda already so we could start using EfficientNetV2. That all worked fine. It seems there was a commit today or yesterday that caused the issue described above.

However, all fixed now after just updating - thank you :)

P.s. instead of the irregular PayPal donations we made, we have now subscribed on Patreon.

Thanks for the project!


User avatar
torzdf
Posts: 1775
Joined: Fri Jul 12, 2019 12:53 am
Answers: 135
Has thanked: 76 times
Been thanked: 357 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by torzdf »

Glad it's all working again. And many thanks for your support, it is hugely appreciated :)

My word is final


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

No problem :D

One new bug we are noticing in the latest versions is when switching from preview back to the session graph on an active training session, we get a complete crash and this is output in the terminal:

crash-example.jpg
crash-example.jpg (100.48 KiB) Viewed 845 times

It's consistent and seems to keep happening - doesn't seem to be a one-off.


User avatar
torzdf
Posts: 1775
Joined: Fri Jul 12, 2019 12:53 am
Answers: 135
Has thanked: 76 times
Been thanked: 357 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by torzdf »

I have literally never seen this before... looks like it's being generated from TF itself, which switching around in the GUI doesn't directly impact. Would be useful to get the full text of the error please.

My word is final


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

No problem.

We just switched from preview back to session graph on an active session and reproduced a crash. It seemed to log a report so here it is:

Code: Select all

05/07/2022 22:22:09 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['03579.png', '06427.png', '01761.png', '06144.png', '01268.png']
05/07/2022 22:22:11 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['09530.png', '04870.png', '03438.png', '07545.png', '01785.png']
05/07/2022 22:22:14 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['02406.png', '03829.png', '09482.png', '05399.png', '01876.png']
05/07/2022 22:22:16 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['02428.png', '10602.png', '00239.png', '08793.png', '08451.png']
05/07/2022 22:22:19 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['00478.png', '08664.png', '04416.png', '09345.png', '00448.png']
05/07/2022 22:22:22 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['09560.png', '03496.png', '09380.png', '05842.png', '03877.png']
05/07/2022 22:22:24 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['05337.png', '08500.png', '04145.png', '05222.png', '03419.png']
05/07/2022 22:22:27 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['09503.png', '08846.png', '06926.png', '03326.png', '05017.png']
05/07/2022 22:22:30 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['00942.png', '03173.png', '09885.png', '10417.png', '10565.png']
05/07/2022 22:22:32 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['02884.png', '03842.png', '09246.png', '04563.png', '04737.png']
05/07/2022 22:22:35 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['07989.png', '03885.png', '10616.png', '07268.png', '00270.png']
05/07/2022 22:22:38 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['07308.png', '05281.png', '08401.png', '09281.png', '08685.png']
05/07/2022 22:22:40 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['10378.png', '05292.png', '07052.png', '00539.png', '07737.png']
05/07/2022 22:22:43 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['07573.png', '08968.png', '00856.png', '00640.png', '01667.png']
05/07/2022 22:22:46 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['08268.png', '00400.png', '08811.png', '01895.png', '00550.png']
05/07/2022 22:22:48 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['02878.png', '09182.png', '08688.png', '01811.png', '10277.png']
05/07/2022 22:22:51 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['06369.png', '04020.png', '10585.png', '02178.png', '09142.png']
05/07/2022 22:22:54 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['04492.png', '01282.png', '06344.png', '03188.png', '02644.png']
05/07/2022 22:22:57 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['05381.png', '04707.png', '10261.png', '04729.png', '09365.png']
05/07/2022 22:22:59 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['03918.png', '05473.png', '09662.png', '05705.png', '02001.png']
05/07/2022 22:23:01 MainProcess     _run_0                         generator       cache_metadata                 VERBOSE  Cache filled: 'C:\Convert AI\LVR2\Training Set'
05/07/2022 22:29:14 MainProcess     _training_0                    _base           generate_preview               DEBUG    Generating preview
05/07/2022 22:29:14 MainProcess     _training_0                    _base           compile_sample                 DEBUG    Compiling samples: (side: 'a', samples: 14)
05/07/2022 22:29:14 MainProcess     _training_0                    _base           compile_sample                 DEBUG    Compiling samples: (side: 'b', samples: 14)
05/07/2022 22:29:14 MainProcess     _training_0                    _base           show_sample                    DEBUG    Showing sample
05/07/2022 22:29:14 MainProcess     _training_0                    _base           _get_predictions               DEBUG    Getting Predictions
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _get_predictions               DEBUG    Returning predictions: {'a_a': (14, 384, 384, 3), 'b_b': (14, 384, 384, 3), 'a_b': (14, 384, 384, 3), 'b_a': (14, 384, 384, 3)}
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _to_full_frame                 DEBUG    side: 'a', number of sample arrays: 3, prediction.shapes: [(14, 384, 384, 3), (14, 384, 384, 3)])
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _process_full                  DEBUG    full_size: 384, prediction_size: 384, color: (0, 0, 255)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 384, 384, 3), target_size: 438, scale: 1.140625)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'a' shape: (14, 438, 438, 3))
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _process_full                  DEBUG    Overlayed background. Shape: (14, 438, 438, 3)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _compile_masked                DEBUG    masked shapes: [(14, 384, 384, 3), (14, 384, 384, 3), (14, 384, 384, 3)]
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 438, 438, 3)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 438, 438, 3)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 438, 438, 3)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 438, 438, 3), target_size: 328, scale: 0.7488584474885844)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'a' shape: (14, 328, 328, 3))
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 438, 438, 3), target_size: 328, scale: 0.7488584474885844)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'a' shape: (14, 328, 328, 3))
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 438, 438, 3), target_size: 328, scale: 0.7488584474885844)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'a' shape: (14, 328, 328, 3))
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _get_headers                   DEBUG    side: 'a', width: 328
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _get_headers                   DEBUG    height: 72, total_width: 984
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _get_headers                   DEBUG    texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(183, 23), (296, 23), (259, 23)], text_x: [72, 344, 690], text_y: 47
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _get_headers                   DEBUG    header_box.shape: (72, 984, 3)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _to_full_frame                 DEBUG    side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 384, 384, 3), (14, 384, 384, 3)])
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _process_full                  DEBUG    full_size: 384, prediction_size: 384, color: (0, 0, 255)
05/07/2022 22:29:16 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 384, 384, 3), target_size: 438, scale: 1.140625)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'b' shape: (14, 438, 438, 3))
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _process_full                  DEBUG    Overlayed background. Shape: (14, 438, 438, 3)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _compile_masked                DEBUG    masked shapes: [(14, 384, 384, 3), (14, 384, 384, 3), (14, 384, 384, 3)]
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 438, 438, 3)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 438, 438, 3)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 438, 438, 3)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 438, 438, 3), target_size: 328, scale: 0.7488584474885844)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'b' shape: (14, 328, 328, 3))
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 438, 438, 3), target_size: 328, scale: 0.7488584474885844)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'b' shape: (14, 328, 328, 3))
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 438, 438, 3), target_size: 328, scale: 0.7488584474885844)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'b' shape: (14, 328, 328, 3))
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _get_headers                   DEBUG    side: 'b', width: 328
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _get_headers                   DEBUG    height: 72, total_width: 984
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _get_headers                   DEBUG    texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(150, 23), (222, 23), (259, 23)], text_x: [89, 381, 690], text_y: 47
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _get_headers                   DEBUG    header_box.shape: (72, 984, 3)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _duplicate_headers             DEBUG    side: a header.shape: (72, 984, 3)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _duplicate_headers             DEBUG    side: b header.shape: (72, 984, 3)
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _stack_images                  DEBUG    Stack images
05/07/2022 22:29:17 MainProcess     _training_0                    _base           get_transpose_axes             DEBUG    Even number of images to stack
05/07/2022 22:29:17 MainProcess     _training_0                    _base           _stack_images                  DEBUG    Stacked images
05/07/2022 22:29:17 MainProcess     _training_0                    _base           show_sample                    DEBUG    Compiled sample
05/07/2022 22:29:18 MainProcess     _training_0                    train           _show                          DEBUG    Updating preview: (name: Training - 'S': Save Now. 'R': Refresh Preview. 'M': Toggle Mask. 'ENTER': Save and Quit)
05/07/2022 22:29:18 MainProcess     _training_0                    train           _show                          DEBUG    Generating preview for GUI
05/07/2022 22:29:18 MainProcess     _training_0                    train           _show                          DEBUG    Generated preview for GUI: '.gui_training_preview.jpg'
05/07/2022 22:29:18 MainProcess     _training_0                    train           _show                          DEBUG    Generating preview for display: 'Training - 'S': Save Now. 'R': Refresh Preview. 'M': Toggle Mask. 'ENTER': Save and Quit'
05/07/2022 22:29:18 MainProcess     _training_0                    train           _show                          DEBUG    Generated preview for display: 'Training - 'S': Save Now. 'R': Refresh Preview. 'M': Toggle Mask. 'ENTER': Save and Quit'
05/07/2022 22:29:18 MainProcess     _training_0                    train           _show                          DEBUG    Updated preview: (name: Training - 'S': Save Now. 'R': Refresh Preview. 'M': Toggle Mask. 'ENTER': Save and Quit)
05/07/2022 22:29:18 MainProcess     _training_0                    train           _run_training_cycle            DEBUG    Save Iteration: (iteration: 4500
05/07/2022 22:29:18 MainProcess     _training_0                    _base           _save                          DEBUG    Backing up and saving models
05/07/2022 22:29:18 MainProcess     _training_0                    _base           _get_save_averages             DEBUG    Getting save averages
05/07/2022 22:29:18 MainProcess     _training_0                    _base           _get_save_averages             DEBUG    Average losses since last save: [0.054676631107926366, 0.05488332705199719]
05/07/2022 22:29:18 MainProcess     _training_0                    _base           _should_backup                 DEBUG    Updated lowest historical save iteration averages from: {'a': 0.05644378334283829, 'b': 0.05529949029535055} to: {'a': 0.054676631107926366, 'b': 0.05488332705199719}
05/07/2022 22:29:18 MainProcess     _training_0                    _base           _should_backup                 DEBUG    Should backup: True
05/07/2022 22:29:18 MainProcess     _training_0                    backup_restore  backup_model                   VERBOSE  Backing up: 'C:\Convert AI\LVR2\Model\phaze_a.h5' to 'C:\Convert AI\LVR2\Model\phaze_a.h5.bk'
05/07/2022 22:29:18 MainProcess     _training_0                    backup_restore  backup_model                   VERBOSE  Backing up: 'C:\Convert AI\LVR2\Model\phaze_a_state.json' to 'C:\Convert AI\LVR2\Model\phaze_a_state.json.bk'
05/07/2022 22:29:22 MainProcess     _training_0                    _base           save                           DEBUG    Saving State
05/07/2022 22:29:22 MainProcess     _training_0                    serializer      save                           DEBUG    filename: C:\Convert AI\LVR2\Model\phaze_a_state.json, data type: <class 'dict'>
05/07/2022 22:29:22 MainProcess     _training_0                    serializer      _check_extension               DEBUG    Original filename: 'C:\Convert AI\LVR2\Model\phaze_a_state.json', final filename: 'C:\Convert AI\LVR2\Model\phaze_a_state.json'
05/07/2022 22:29:22 MainProcess     _training_0                    serializer      marshal                        DEBUG    data type: <class 'dict'>
05/07/2022 22:29:22 MainProcess     _training_0                    serializer      marshal                        DEBUG    returned data type: <class 'bytes'>
05/07/2022 22:29:22 MainProcess     _training_0                    _base           save                           DEBUG    Saved State
05/07/2022 22:29:22 MainProcess     _training_0                    _base           _save                          INFO     [Saved models] - Average loss since last save: face_a: 0.05468, face_b: 0.05488
05/07/2022 22:34:03 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): [Errno 22] Invalid argument
05/07/2022 22:34:05 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
05/07/2022 22:34:05 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
05/07/2022 22:34:05 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
05/07/2022 22:34:05 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
05/07/2022 22:34:05 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
05/07/2022 22:34:05 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
05/07/2022 22:34:05 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "C:\Convert\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "C:\Convert\scripts\train.py", line 190, in process
    self._end_thread(thread, err)
  File "C:\Convert\scripts\train.py", line 230, in _end_thread
    thread.join()
  File "C:\Convert\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Convert\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Convert\scripts\train.py", line 252, in _training
    raise err
  File "C:\Convert\scripts\train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Convert\scripts\train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "C:\Convert\plugins\train\trainer\_base.py", line 225, in train_one_step
    self._print_loss(loss)
  File "C:\Convert\plugins\train\trainer\_base.py", line 314, in _print_loss
    print(f"\r{output}", end="")
OSError: [Errno 22] Invalid argument

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         a046248 BugFix - lib.keypress. 60f95bb fix: PhazeA - Use correct name for EffNetV2 freezing
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce RTX 3090, GPU_1: NVIDIA GeForce RTX 2080 Ti
gpu_devices_active:  GPU_0
gpu_driver:          512.15
gpu_vram:            GPU_0: 24576MB, GPU_1: 11264MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.22000-SP0
os_release:          10
py_command:          C:\Convert\faceswap.py train -A C:/Convert AI/LVR2/Training Set -B C:/Convert AI/L Work Folder/Brand New Set 512 -m C:/Convert AI/LVR2/Model -t phaze-a -bs 5 -it 1000000 -s 500 -ss 25000 -ps 75 -p -wl -X 1 -L INFO -gui
py_conda_version:    conda 4.12.0
py_implementation:   CPython
py_version:          3.8.13
py_virtual_env:      True
sys_cores:           48
sys_processor:       Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
sys_ram:             Total: 130718MB, Available: 117856MB, Used: 12861MB, Free: 117856MB

=============== Pip Packages ===============


============== Conda Packages ==============
# packages in environment at C:\Users\		\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   1.0.0                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
ca-certificates           2021.10.8            h5b45459_0    conda-forge
cachetools                5.0.0                    pypi_0    pypi
certifi                   2021.10.8        py38haa244fe_2    conda-forge
charset-normalizer        2.0.12                   pypi_0    pypi
colorama                  0.4.4              pyhd3eb1b0_0  
cudatoolkit               11.2.2              h933977f_10    conda-forge
cudnn                     8.1.0.77             h3e0f4f4_0    conda-forge
cycler                    0.11.0             pyhd3eb1b0_0  
fastcluster               1.2.6            py38hcc40339_1    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
flatbuffers               2.0                      pypi_0    pypi
freetype                  2.10.4               hd328e21_0  
gast                      0.5.3                    pypi_0    pypi
git                       2.34.1               haa95532_0  
google-auth               2.6.6                    pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.46.0                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
icc_rt                    2019.0.0             h0cc432a_1  
icu                       58.2                 ha925a31_3  
idna                      3.3                      pypi_0    pypi
imageio                   2.9.0              pyhd3eb1b0_0  
imageio-ffmpeg            0.4.7              pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.3                   pypi_0    pypi
intel-openmp              2021.4.0          haa95532_3556  
joblib                    1.1.0              pyhd3eb1b0_0  
jpeg                      9e                   h2bbff1b_0  
keras                     2.8.0                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.3.2            py38hd77b12b_0  
libclang                  14.0.1                   pypi_0    pypi
libpng                    1.6.37               h2a8f88b_0  
libtiff                   4.2.0                hd0e1b90_0  
libwebp                   1.2.2                h2bbff1b_0  
lz4-c                     1.9.3                h2bbff1b_1  
markdown                  3.3.7                    pypi_0    pypi
matplotlib                3.2.2                         0  
matplotlib-base           3.2.2            py38h64f37c6_0  
mkl                       2021.4.0           haa95532_640  
mkl-service               2.4.0            py38h2bbff1b_0  
mkl_fft                   1.3.1            py38h277e83a_0  
mkl_random                1.2.2            py38hf11a4ad_0  
numpy                     1.21.5           py38h7a0a035_2  
numpy-base                1.21.5           py38hca35cd5_2  
nvidia-ml-py              11.510.69                pypi_0    pypi
oauthlib                  3.2.0                    pypi_0    pypi
opencv-python             4.5.5.64                 pypi_0    pypi
openssl                   1.1.1o               h8ffe710_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
pillow                    9.0.1            py38hdc2b20a_0  
pip                       21.2.2           py38haa95532_0  
protobuf                  3.20.1                   pypi_0    pypi
psutil                    5.8.0            py38h2bbff1b_1  
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 3.0.4              pyhd3eb1b0_0  
pyqt                      5.9.2            py38hd77b12b_6  
python                    3.8.13               h6244533_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python_abi                3.8                      2_cp38    conda-forge
pywin32                   302              py38h2bbff1b_2  
qt                        5.9.7            vc14h73c81de_0  
requests                  2.27.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.8                      pypi_0    pypi
scikit-learn              1.0.2            py38hf11a4ad_1  
scipy                     1.7.3            py38h0a974cb_0  
setuptools                61.2.0           py38haa95532_0  
sip                       4.19.13          py38hd77b12b_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.38.3               h2bbff1b_0  
tensorboard               2.8.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow-gpu            2.8.0                    pypi_0    pypi
tensorflow-io-gcs-filesystem 0.25.0                   pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tf-estimator-nightly      2.8.0.dev2021122109          pypi_0    pypi
threadpoolctl             2.2.0              pyh0d69192_0  
tk                        8.6.11               h2bbff1b_0  
tornado                   6.1              py38h2bbff1b_0  
tqdm                      4.64.0           py38haa95532_0  
typing-extensions         4.2.0                    pypi_0    pypi
urllib3                   1.26.9                   pypi_0    pypi
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
werkzeug                  2.1.2                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py38haa95532_2  
wrapt                     1.14.1                   pypi_0    pypi
xz                        5.2.5                h8cc25b3_1  
zipp                      3.8.0                    pypi_0    pypi
zlib                      1.2.12               h8cc25b3_2  
zstd                      1.4.9                h19a0ad4_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               50
batch-size:               4

[mask.bisenet_fp]
batch-size:               8
weights:                  faceswap
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    False

--------- train.ini ---------

[global]
centering:                face
coverage:                 87.5
icnr_init:                False
conv_aware_init:          True
optimizer:                adam
learning_rate:            4e-05
epsilon_exponent:         -5
reflect_padding:          False
allow_growth:             False
mixed_precision:          True
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
mask_loss_function:       mse
l2_reg_term:              100
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                bisenet-fp_face
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.phaze_a]
output_size:              384
shared_fc:                None
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         efficientnet_v2_l
enc_scaling:              80
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          None
bottleneck_size:          512
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1280
fc_max_filters:           1280
fc_dimensions:            8
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      1280
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       resize_images
dec_norm:                 None
dec_min_filters:          160
dec_max_filters:          640
dec_filter_slope:         -0.33
dec_res_blocks:           1
dec_output_kernel:        3
dec_gaussian:             True
dec_skip_last_residual:   False
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

User avatar
torzdf
Posts: 1775
Joined: Fri Jul 12, 2019 12:53 am
Answers: 135
Has thanked: 76 times
Been thanked: 357 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by torzdf »

I have seen this before and I don't understand it. A print statement should not be raising an OS Error :/

My word is final


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

Just a quick one and for some advice really...

We're playing with EfficientNetV2 as mentioned. We've also modified Phaze A on top of StoJo preset:

Screen Shot 2022-05-10 at 13.38.16.png
Screen Shot 2022-05-10 at 13.38.16.png (135.48 KiB) Viewed 746 times

When starting a brand new model, sometimes we get white boxes in the previews from the get-go which never develop into previews. However, it seems to depend on the dataset.

Example from another user's forum post:

Image

What could be causing this? Should we modify the learning rate &/or EE to start the session and then revert back just to get it going? We are running a small batch size (6) which could be the issue too of course.

Here are our settings:

Screen Shot 2022-05-10 at 13.40.22.png
Screen Shot 2022-05-10 at 13.40.22.png (84.13 KiB) Viewed 746 times

Thank you for any suggestions and tips in advance. :)


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

Interestingly, we just reverted back to the default StoJo preset with a BS of 21 and the white preview box issue is still present.

Strange. Must be something to do with TF 2.8.

Any ideas are greatly appreciated. I can also output a verbose log of a new training session if it helps.


User avatar
Hanrahahanrahan
Posts: 5
Joined: Sat Aug 22, 2020 1:44 pm
Has thanked: 3 times
Been thanked: 1 time

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by Hanrahahanrahan »

I sometimes have the same issue with EfficentNetV2_s and Tensorflow 2.8, what worked for me is slowing the learning rate down a bit. With the "s" variant (50 enc scaling, 256 output) I have now 4.7e-5 as oppose the default 5e-5. Maybe with the "l" (80 enc scaling/ 384 output) it's too overwhelmed even with your rate set to 4e-5?


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

Thanks for the reply @Hanrahahanrahan

However, as mentioned in my last post above - we reverted back to StoJo default (256, 60%, env1b4) and still reproduced the model crash (white box preview).

It doesn't even make it past the first model save/preview.


User avatar
Hanrahahanrahan
Posts: 5
Joined: Sat Aug 22, 2020 1:44 pm
Has thanked: 3 times
Been thanked: 1 time

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by Hanrahahanrahan »

The instant white on the first save I witnessed too, since I save every 1000 intervals (which takes ~30 minutes on my machine), I refresh now after 100-200 intervals early on to see how it's going. Maybe the learning rate has nothing to do with it and it was just coincidence that it didn't collapse that early.

When you play around with the settings, do you delete/change the model(training) directory?


User avatar
torzdf
Posts: 1775
Joined: Fri Jul 12, 2019 12:53 am
Answers: 135
Has thanked: 76 times
Been thanked: 357 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by torzdf »

Honestly, my suspicion would be learning rate too. Why this happens on TF 2,8, I'm not sure. Just to rule something out, does changing the loss to MAE also cause this issue? I think it's highly unlikely, but as part of the TF 2.8 update I had to change the SSIM loss implementation. I actually changed it to a stock Tensorflow solution, which is why I think this is unlikely to be the culprit but worth checking.

If none of the above work, it would be very useful it you could share your command line used (with the generate button) and also the model.ini file from the faceswap .config folder.

Also the output of Help > Output System Info will also be useful

My word is final


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

Yes, of course - completely delete the corrupt model folder and try again with different settings.

Strange thing is that we managed to get one model going after several attempts (albeit with a different dataset).

But now, no matter what we try, we can't get a model fired up.


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

torzdf wrote: Wed May 11, 2022 9:37 am

Honestly, my suspicion would be learning rate too. Why this happens on TF 2,8, I'm not sure. Just to rule something out, does changing the loss to `MAE` also cause this issue? I think it's highly unlikely, but as part of the TF 2.8 update I had to change the SSIM loss implementation. I actually changed it to a stock Tensorflow solution, which is why I think this is unlikely to be the culprit but worth checking.

If none of the above work, it would be very useful it you could share your command line used (with the generate button) and also the `model.ini` file from the faceswap `.config` folder.

Also the output of Help > Output System Info will also be useful

Thank you for the response.

Even with a learning rate as low as 3e-5 just tested now - white boxes straight away.

Just tested MAE. It works! A just-recognizable face being put together in a preview after just 500 iterations.

Training command used:

Code: Select all

"C:\Users\		\MiniConda3\envs\faceswap\python.exe" C:\Convert\faceswap.py train -A "C:/Convert AI/LVR2/Training Set" -B "C:/Convert AI/L Work Folder/Brand New Set 512" -m "C:/Convert AI/LVR2/Model" -t phaze-a -bs 5 -it 1000000 -s 500 -ss 25000 -ps 40 -p -wl -X 1 -L INFO

model.ini file:
Unless I'm losing my mind (or looking in the wrong location) - we don't have a model.ini file?

Screen Shot 2022-05-11 at 11.28.03.png
Screen Shot 2022-05-11 at 11.28.03.png (104.52 KiB) Viewed 670 times

System Info:

Code: Select all

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         b2cd8eb bugfix: Windows - Stop training when terminate button pressed. 8ab085f bugfix: gui - settings popup. Always reload config. 71c2025 bugfix: Preview Tool, ensure all config items are written. adb5975 Graph popup - Always open in same position. d933965 bisenet-fp - Add face-centered trained weights
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce RTX 3090, GPU_1: NVIDIA GeForce RTX 2080 Ti
gpu_devices_active:  GPU_0, GPU_1
gpu_driver:          512.59
gpu_vram:            GPU_0: 24576MB, GPU_1: 11264MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.22000-SP0
os_release:          10
py_command:          C:\Convert/faceswap.py gui
py_conda_version:    conda 4.12.0
py_implementation:   CPython
py_version:          3.8.13
py_virtual_env:      True
sys_cores:           48
sys_processor:       Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
sys_ram:             Total: 130718MB, Available: 116862MB, Used: 13855MB, Free: 116862MB

=============== Pip Packages ===============


============== Conda Packages ==============
# packages in environment at C:\Users\		\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   1.0.0                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
ca-certificates           2021.10.8            h5b45459_0    conda-forge
cachetools                5.0.0                    pypi_0    pypi
certifi                   2021.10.8        py38haa244fe_2    conda-forge
charset-normalizer        2.0.12                   pypi_0    pypi
colorama                  0.4.4              pyhd3eb1b0_0  
cudatoolkit               11.2.2              h933977f_10    conda-forge
cudnn                     8.1.0.77             h3e0f4f4_0    conda-forge
cycler                    0.11.0             pyhd3eb1b0_0  
fastcluster               1.2.6            py38hcc40339_1    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
flatbuffers               2.0                      pypi_0    pypi
freetype                  2.10.4               hd328e21_0  
gast                      0.5.3                    pypi_0    pypi
git                       2.34.1               haa95532_0  
google-auth               2.6.6                    pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.46.0                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
icc_rt                    2019.0.0             h0cc432a_1  
icu                       58.2                 ha925a31_3  
idna                      3.3                      pypi_0    pypi
imageio                   2.9.0              pyhd3eb1b0_0  
imageio-ffmpeg            0.4.7              pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.3                   pypi_0    pypi
intel-openmp              2021.4.0          haa95532_3556  
joblib                    1.1.0              pyhd3eb1b0_0  
jpeg                      9e                   h2bbff1b_0  
keras                     2.8.0                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.3.2            py38hd77b12b_0  
libclang                  14.0.1                   pypi_0    pypi
libpng                    1.6.37               h2a8f88b_0  
libtiff                   4.2.0                hd0e1b90_0  
libwebp                   1.2.2                h2bbff1b_0  
lz4-c                     1.9.3                h2bbff1b_1  
markdown                  3.3.7                    pypi_0    pypi
matplotlib                3.2.2                         0  
matplotlib-base           3.2.2            py38h64f37c6_0  
mkl                       2021.4.0           haa95532_640  
mkl-service               2.4.0            py38h2bbff1b_0  
mkl_fft                   1.3.1            py38h277e83a_0  
mkl_random                1.2.2            py38hf11a4ad_0  
numpy                     1.21.5           py38h7a0a035_2  
numpy-base                1.21.5           py38hca35cd5_2  
nvidia-ml-py              11.510.69                pypi_0    pypi
oauthlib                  3.2.0                    pypi_0    pypi
opencv-python             4.5.5.64                 pypi_0    pypi
openssl                   1.1.1o               h8ffe710_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
pillow                    9.0.1            py38hdc2b20a_0  
pip                       21.2.2           py38haa95532_0  
protobuf                  3.20.1                   pypi_0    pypi
psutil                    5.8.0            py38h2bbff1b_1  
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 3.0.4              pyhd3eb1b0_0  
pyqt                      5.9.2            py38hd77b12b_6  
python                    3.8.13               h6244533_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python_abi                3.8                      2_cp38    conda-forge
pywin32                   302              py38h2bbff1b_2  
qt                        5.9.7            vc14h73c81de_0  
requests                  2.27.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.8                      pypi_0    pypi
scikit-learn              1.0.2            py38hf11a4ad_1  
scipy                     1.7.3            py38h0a974cb_0  
setuptools                61.2.0           py38haa95532_0  
sip                       4.19.13          py38hd77b12b_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.38.3               h2bbff1b_0  
tensorboard               2.8.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow-gpu            2.8.0                    pypi_0    pypi
tensorflow-io-gcs-filesystem 0.25.0                   pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tf-estimator-nightly      2.8.0.dev2021122109          pypi_0    pypi
threadpoolctl             2.2.0              pyh0d69192_0  
tk                        8.6.11               h2bbff1b_0  
tornado                   6.1              py38h2bbff1b_0  
tqdm                      4.64.0           py38haa95532_0  
typing-extensions         4.2.0                    pypi_0    pypi
urllib3                   1.26.9                   pypi_0    pypi
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
werkzeug                  2.1.2                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py38haa95532_2  
wrapt                     1.14.1                   pypi_0    pypi
xz                        5.2.5                h8cc25b3_1  
zipp                      3.8.0                    pypi_0    pypi
zlib                      1.2.12               h8cc25b3_2  
zstd                      1.4.9                h19a0ad4_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               50
batch-size:               4

[mask.bisenet_fp]
batch-size:               8
weights:                  faceswap
include_ears:             False
include_hair:             False
include_glasses:          False

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    False

--------- train.ini ---------

[global]
centering:                face
coverage:                 87.5
icnr_init:                False
conv_aware_init:          True
optimizer:                adam
learning_rate:            4e-05
epsilon_exponent:         -5
reflect_padding:          False
allow_growth:             False
mixed_precision:          True
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            mae
mask_loss_function:       mse
l2_reg_term:              100
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                bisenet-fp_face
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.phaze_a]
output_size:              384
shared_fc:                None
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         efficientnet_v2_l
enc_scaling:              80
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          None
bottleneck_size:          512
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1280
fc_max_filters:           1280
fc_dimensions:            8
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      1280
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       resize_images
dec_norm:                 None
dec_min_filters:          160
dec_max_filters:          640
dec_filter_slope:         -0.33
dec_res_blocks:           1
dec_output_kernel:        3
dec_gaussian:             True
dec_skip_last_residual:   False
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4



User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

For the sake of an accurate test, I did it all again...

Switched back to SSIM. Started a new model and got the white preview boxes straight away with no masks visible and they continued into the first save and going forward and never developed.

Switched over to MAE. Started a new model and within 500 iterations I could see the shapes of faces along with masks.

Your suspicion re. the SSIM change seems to be correct, or at least on the right path.

I don't know if there is anything we can change on this end but if there is a fix/workaround that would be great.

I'll also be around to provide feedback, reports, tests, and whatever else you may need to reproduce/resolve.

Thansk, @torzdf ! :)


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

Well, Mr. @torzdf, we just updated to your latest commits (today) and managed to fire up a model straight away with recognizable previews appearing after 1,000 iterations.

No more white boxes and corrupt models it seems.

Thank you!

P.s. what was the issue? It'd be interesting to learn what happened here.


User avatar
torzdf
Posts: 1775
Joined: Fri Jul 12, 2019 12:53 am
Answers: 135
Has thanked: 76 times
Been thanked: 357 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by torzdf »

Honestly, nothing I updated yesterday would have impacted this. The commits were basically a refactoring of some of the code so that we can (in the future) support Apple Silicon.

I guess it was just one of those things, but all's well that ends well!

My word is final


User avatar
ianstephens
Posts: 88
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times
Been thanked: 5 times

Re: Bug: Updated to Latest faceswap version now crashing when starting training

Post by ianstephens »

How strange. We fired up a model right away with no issues.

However, I haven't tried to start any new models since I got that one started so it's possible it'll reappear in the future.

I'll keep you updated. However, your suggestion of using MAE did fix the issue. But we've managed to get the model running with SSIM now (perhaps a lucky one-off).


Post Reply