[Outdated] Training Using Google Colab

Want to use Faceswap in The Cloud? This is not directly supported by the Devs, but you may find community support here


Forum rules

Read the FAQs and search the forum before posting a new topic.

NB: The Devs do not directly support using Cloud based services, but you can find community support here.

Please mark any answers that fixed your problems so others can find the solutions.

User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Re: [Resource] Training Using Google Colab

Post by aolvera27 »

I couldn't fit everything in just one post, but here's the first part of crash report:

Code: Select all

01/27/2022 11:39:46 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_headers                   DEBUG    side: 'a', width: 150
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_headers                   DEBUG    height: 33, total_width: 450
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_headers                   DEBUG    texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(84, 11), (136, 11), (119, 11)], text_x: [33, 157, 315], text_y: 22
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_headers                   DEBUG    header_box.shape: (33, 450, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _to_full_frame                 DEBUG    side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 128, 128, 3), (14, 128, 128, 3)])
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _process_full                  DEBUG    full_size: 384, prediction_size: 128, color: (0, 0, 255)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 384, 384, 3), target_size: 150, scale: 0.390625)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'b' shape: (14, 150, 150, 3))
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _process_full                  DEBUG    Overlayed background. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _compile_masked                DEBUG    masked shapes: [(14, 128, 128, 3), (14, 128, 128, 3), (14, 128, 128, 3)]
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_headers                   DEBUG    side: 'b', width: 150
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_headers                   DEBUG    height: 33, total_width: 450
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_headers                   DEBUG    texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(69, 11), (102, 11), (119, 11)], text_x: [40, 174, 315], text_y: 22
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_headers                   DEBUG    header_box.shape: (33, 450, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _duplicate_headers             DEBUG    side: a header.shape: (33, 450, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _duplicate_headers             DEBUG    side: b header.shape: (33, 450, 3)
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _stack_images                  DEBUG    Stack images
01/27/2022 11:39:46 MainProcess     _training_0                    _base           get_transpose_axes             DEBUG    Even number of images to stack
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _stack_images                  DEBUG    Stacked images
01/27/2022 11:39:46 MainProcess     _training_0                    _base           show_sample                    DEBUG    Compiled sample
01/27/2022 11:39:46 MainProcess     _training_0                    _base           output_timelapse               DEBUG    Created time-lapse: '/content/drive/My Drive/colab_files/faceswap/output/timelapse/1643305186.jpg'
01/27/2022 11:39:46 MainProcess     _training_0                    train           _run_training_cycle            DEBUG    Save Iteration: (iteration: 5000
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _save                          DEBUG    Backing up and saving models
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_save_averages             DEBUG    Getting save averages
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _get_save_averages             DEBUG    Average losses since last save: [0.010751839741133154, 0.01970715313591063]
01/27/2022 11:39:46 MainProcess     _training_0                    _base           _should_backup                 DEBUG    Should backup: False
01/27/2022 11:39:53 MainProcess     _training_0                    _base           save                           DEBUG    Saving State
01/27/2022 11:39:53 MainProcess     _training_0                    serializer      save                           DEBUG    filename: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json, data type: <class 'dict'>
01/27/2022 11:39:53 MainProcess     _training_0                    serializer      _check_extension               DEBUG    Original filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json', final filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json'
01/27/2022 11:39:53 MainProcess     _training_0                    serializer      marshal                        DEBUG    data type: <class 'dict'>
01/27/2022 11:39:53 MainProcess     _training_0                    serializer      marshal                        DEBUG    returned data type: <class 'bytes'>
01/27/2022 11:39:53 MainProcess     _training_0                    _base           save                           DEBUG    Saved State
01/27/2022 11:39:53 MainProcess     _training_0                    _base           _save                          INFO     [Saved models] - Average loss since last save: face_a: 0.01075, face_b: 0.01971
01/27/2022 12:17:11 MainProcess     _training_0                    _base           output_timelapse               DEBUG    Ouputting time-lapse
01/27/2022 12:17:11 MainProcess     _training_0                    _base           output_timelapse               DEBUG    Getting time-lapse samples
01/27/2022 12:17:11 MainProcess     _training_0                    _base           compile_sample                 DEBUG    Compiling samples: (side: 'a', samples: 14)
01/27/2022 12:17:11 MainProcess     _training_0                    _base           compile_sample                 DEBUG    Compiling samples: (side: 'b', samples: 14)
01/27/2022 12:17:11 MainProcess     _training_0                    _base           output_timelapse               DEBUG    Got time-lapse samples: {'a': 3, 'b': 3}
01/27/2022 12:17:11 MainProcess     _training_0                    _base           show_sample                    DEBUG    Showing sample
01/27/2022 12:17:11 MainProcess     _training_0                    _base           _get_predictions               DEBUG    Getting Predictions
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _get_predictions               DEBUG    Returning predictions: {'a_a': (14, 128, 128, 3), 'b_b': (14, 128, 128, 3), 'a_b': (14, 128, 128, 3), 'b_a': (14, 128, 128, 3)}
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _to_full_frame                 DEBUG    side: 'a', number of sample arrays: 3, prediction.shapes: [(14, 128, 128, 3), (14, 128, 128, 3)])
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _process_full                  DEBUG    full_size: 384, prediction_size: 128, color: (0, 0, 255)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 384, 384, 3), target_size: 150, scale: 0.390625)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'a' shape: (14, 150, 150, 3))
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _process_full                  DEBUG    Overlayed background. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _compile_masked                DEBUG    masked shapes: [(14, 128, 128, 3), (14, 128, 128, 3), (14, 128, 128, 3)]
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _get_headers                   DEBUG    side: 'a', width: 150
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _get_headers                   DEBUG    height: 33, total_width: 450
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _get_headers                   DEBUG    texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(84, 11), (136, 11), (119, 11)], text_x: [33, 157, 315], text_y: 22
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _get_headers                   DEBUG    header_box.shape: (33, 450, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _to_full_frame                 DEBUG    side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 128, 128, 3), (14, 128, 128, 3)])
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _process_full                  DEBUG    full_size: 384, prediction_size: 128, color: (0, 0, 255)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 384, 384, 3), target_size: 150, scale: 0.390625)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _resize_sample                 DEBUG    Resized sample: (side: 'b' shape: (14, 150, 150, 3))
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _process_full                  DEBUG    Overlayed background. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _compile_masked                DEBUG    masked shapes: [(14, 128, 128, 3), (14, 128, 128, 3), (14, 128, 128, 3)]
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _overlay_foreground            DEBUG    Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _get_headers                   DEBUG    side: 'b', width: 150
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _get_headers                   DEBUG    height: 33, total_width: 450
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _get_headers                   DEBUG    texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(69, 11), (102, 11), (119, 11)], text_x: [40, 174, 315], text_y: 22
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _get_headers                   DEBUG    header_box.shape: (33, 450, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _duplicate_headers             DEBUG    side: a header.shape: (33, 450, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _duplicate_headers             DEBUG    side: b header.shape: (33, 450, 3)
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _stack_images                  DEBUG    Stack images
01/27/2022 12:17:13 MainProcess     _training_0                    _base           get_transpose_axes             DEBUG    Even number of images to stack
01/27/2022 12:17:13 MainProcess     _training_0                    _base           _stack_images                  DEBUG    Stacked images
01/27/2022 12:17:13 MainProcess     _training_0                    _base           show_sample                    DEBUG    Compiled sample
01/27/2022 12:17:14 MainProcess     _training_0                    _base           output_timelapse               DEBUG    Created time-lapse: '/content/drive/My Drive/colab_files/faceswap/output/timelapse/1643307433.jpg'
01/27/2022 12:17:14 MainProcess     _training_0                    train           _run_training_cycle            DEBUG    Save Iteration: (iteration: 6000
01/27/2022 12:17:14 MainProcess     _training_0                    _base           _save                          DEBUG    Backing up and saving models
01/27/2022 12:17:14 MainProcess     _training_0                    _base           _get_save_averages             DEBUG    Getting save averages
01/27/2022 12:17:14 MainProcess     _training_0                    _base           _get_save_averages             DEBUG    Average losses since last save: [0.010746411305852235, 0.019705730494111777]
01/27/2022 12:17:14 MainProcess     _training_0                    _base           _should_backup                 DEBUG    Should backup: False
01/27/2022 12:17:20 MainProcess     _training_0                    _base           save                           DEBUG    Saving State
01/27/2022 12:17:20 MainProcess     _training_0                    serializer      save                           DEBUG    filename: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json, data type: <class 'dict'>
01/27/2022 12:17:20 MainProcess     _training_0                    serializer      _check_extension               DEBUG    Original filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json', final filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json'
01/27/2022 12:17:20 MainProcess     _training_0                    serializer      marshal                        DEBUG    data type: <class 'dict'>
01/27/2022 12:17:20 MainProcess     _training_0                    serializer      marshal                        DEBUG    returned data type: <class 'bytes'>
01/27/2022 12:17:20 MainProcess     _training_0                    _base           save                           DEBUG    Saved State
01/27/2022 12:17:20 MainProcess     _training_0                    _base           _save                          INFO     [Saved models] - Average loss since last save: face_a: 0.01075, face_b: 0.01971
01/27/2022 12:28:18 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): 2 root error(s) found.\n  (0) FAILED_PRECONDITION:  /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected\n	Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2\n	Could not flush events file.\n	 [[node batch_decoder_b_loss\n (defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)\n]]\n	 [[GroupCrossDeviceControlEdges_0/NoOp/_91]]\n  (1) FAILED_PRECONDITION:  /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected\n	Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2\n	Could not flush events file.\n	 [[node batch_decoder_b_loss\n (defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)\n]]\n0 successful operations.\n0 derived errors ignored. [Op:__inference_train_function_14947]\n\nErrors may have originated from an input operation.\nInput Source operations connected to node batch_decoder_b_loss:\nIn[0] batch_decoder_b_loss/writer:	\nIn[1] batch_decoder_b_loss/ReadVariableOp:	\nIn[2] batch_decoder_b_loss/tag:	\nIn[3] batch_decoder_b_loss/Identity:\n\nOperation defined at: (most recent call last)\n>>>   File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap\n>>>     self._bootstrap_inner()\n>>> \n>>>   File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner\n>>>     self.run()\n>>> \n>>>   File "/content/faceswap/lib/multithreading.py", line 37, in run\n>>>     self._target(*self._args, **self._kwargs)\n>>> \n>>>   File "/content/faceswap/scripts/train.py", line 242, in _training\n>>>     self._run_training_cycle(model, trainer)\n>>> \n>>>   File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle\n>>>     trainer.train_one_step(viewer, timelapse)\n>>> \n>>>   File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step\n>>>     loss = self._model.model.train_on_batch(model_inputs, y=model_targets)\n>>> \n>>>   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch\n>>>     logs = self.train_function(iterator)\n>>> \n>>>   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function\n>>>     return step_function(self, iterator)\n>>> \n>>>   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function\n>>>     write_scalar_summaries(outputs, step=model._train_counter)  # pylint: disable=protected-access\n>>> \n>>>   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries\n>>>     summary_ops_v2.scalar('batch_' + name, value, step=step)\n>>> \n\nInput Source operations connected to node batch_decoder_b_loss:\nIn[0] batch_decoder_b_loss/writer:	\nIn[1] batch_decoder_b_loss/ReadVariableOp:	\nIn[2] batch_decoder_b_loss/tag:	\nIn[3] batch_decoder_b_loss/Identity:\n\nOperation defined at: (most recent call last)\n>>>   File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap\n>>>     self._bootstrap_inner()\n>>> \n>>>   File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner\n>>>     self.run()\n>>> \n>>>   File "/content/faceswap/lib/multithreading.py", line 37, in run\n>>>     self._target(*self._args, **self._kwargs)\n>>> \n>>>   File "/content/faceswap/scripts/train.py", line 242, in _training\n>>>     self._run_training_cycle(model, trainer)\n>>> \n>>>   File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle\n>>>     trainer.train_one_step(viewer, timelapse)\n>>> \n>>>   File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step\n>>>     loss = self._model.model.train_on_batch(model_inputs, y=model_targets)\n>>> \n>>>   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch\n>>>     logs = self.train_function(iterator)\n>>> \n>>>   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function\n>>>     return step_function(self, iterator)\n>>> \n>>>   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function\n>>>     write_scalar_summaries(outputs, step=model._train_counter)  # pylint: disable=protected-access\n>>> \n>>>   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries\n>>>     summary_ops_v2.scalar('batch_' + name, value, step=step)\n>>> \n\nFunction call stack:\ntrain_function -> cond_2_true_14925 -> train_function -> cond_2_true_14925\n
01/27/2022 12:28:18 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
01/27/2022 12:28:18 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
01/27/2022 12:28:18 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
01/27/2022 12:28:18 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
01/27/2022 12:28:18 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
01/27/2022 12:28:18 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
01/27/2022 12:28:18 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
    process.process()
  File "/content/faceswap/scripts/train.py", line 190, in process
    self._end_thread(thread, err)
  File "/content/faceswap/scripts/train.py", line 230, in _end_thread
    thread.join()
  File "/content/faceswap/lib/multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/content/faceswap/scripts/train.py", line 252, in _training
    raise err
  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
    logs = self.train_function(iterator)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
  (0) FAILED_PRECONDITION:  /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected
	Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2
	Could not flush events file.
	 [[node batch_decoder_b_loss
 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)
]]
	 [[GroupCrossDeviceControlEdges_0/NoOp/_91]]
  (1) FAILED_PRECONDITION:  /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected
	Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2
	Could not flush events file.
	 [[node batch_decoder_b_loss
 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)
]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14947]

Errors may have originated from an input operation.
Input Source operations connected to node batch_decoder_b_loss:
In[0] batch_decoder_b_loss/writer:	
In[1] batch_decoder_b_loss/ReadVariableOp:	
In[2] batch_decoder_b_loss/tag:	
In[3] batch_decoder_b_loss/Identity:

Operation defined at: (most recent call last)
  File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()

  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()

  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)

  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)

  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)

  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
    logs = self.train_function(iterator)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function
    return step_function(self, iterator)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function
    write_scalar_summaries(outputs, step=model._train_counter)  # pylint: disable=protected-access

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries
    summary_ops_v2.scalar('batch_' + name, value, step=step)


Input Source operations connected to node batch_decoder_b_loss:
In[0] batch_decoder_b_loss/writer:	
In[1] batch_decoder_b_loss/ReadVariableOp:	
In[2] batch_decoder_b_loss/tag:	
In[3] batch_decoder_b_loss/Identity:

Operation defined at: (most recent call last)
  File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()

  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()

  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)

  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)

  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)

  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
    logs = self.train_function(iterator)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function
    return step_function(self, iterator)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function
    write_scalar_summaries(outputs, step=model._train_counter)  # pylint: disable=protected-access

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries
    summary_ops_v2.scalar('batch_' + name, value, step=step)


Function call stack:
train_function -> cond_2_true_14925 -> train_function -> cond_2_true_14925


============ System Information ============
encoding:            UTF-8
git_branch:          Not Found
git_commits:         Not Found
gpu_cuda:            11.1
gpu_cudnn:           No global version found
gpu_devices:         GPU_0: Tesla K80
gpu_devices_active:  GPU_0
gpu_driver:          460.32.03
gpu_vram:            GPU_0: 11441MB
os_machine:          x86_64
os_platform:         Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
os_release:          5.4.144+
py_command:          faceswap/faceswap.py train -A face_a -B face_b -m /content/drive/My Drive/colab_files/faceswap/models/JdKQry -t villain -bs 8 -it 250000 -s 1000 -ss 50000 -tia face_a -tib face_b -to /content/drive/My Drive/colab_files/faceswap/output/timelapse -nf
py_conda_version:    N/A
py_implementation:   CPython
py_version:          3.7.12
py_virtual_env:      False
sys_cores:           2
sys_processor:       x86_64
sys_ram:             Total: 12991MB, Available: 10007MB, Used: 3928MB, Free: 967MB

=============== Pip Packages ===============
absl-py==1.0.0
alabaster==0.7.12
albumentations==0.1.12
altair==4.2.0
appdirs==1.4.4
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arviz==0.11.4
astor==0.8.1
astropy==4.3.1
astunparse==1.6.3
atari-py==0.2.9
atomicwrites==1.4.0
attrs==21.4.0
audioread==2.1.9
autograd==1.3
Babel==2.9.1
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==4.1.0
blis==0.4.1
bokeh==2.3.3
Bottleneck==1.3.2
branca==0.4.2
bs4==0.0.1
CacheControl==0.12.10
cached-property==1.5.2
cachetools==4.2.4
catalogue==1.0.0
certifi==2021.10.8
cffi==1.15.0
cftime==1.5.2
chardet==3.0.4
charset-normalizer==2.0.10
click==7.1.2
cloudpickle==1.3.0
cmake==3.12.0
cmdstanpy==0.9.5
colorcet==3.0.0
colorlover==0.3.0
community==1.0.0b1
contextlib2==0.5.5
convertdate==2.4.0
coverage==3.7.1
coveralls==0.5
crcmod==1.7
cufflinks==0.17.3
cupy-cuda111==9.4.0
cvxopt==1.2.7
cvxpy==1.0.31
cycler==0.11.0
cymem==2.0.6
Cython==0.29.26
daft==0.0.4
dask==2.12.0
datascience==0.10.6
debugpy==1.0.0
decorator==4.4.2
defusedxml==0.7.1
descartes==1.1.0
dill==0.3.4
distributed==1.25.3
dlib @ file:///dlib-19.18.0-cp37-cp37m-linux_x86_64.whl
dm-tree==0.1.6
docopt==0.6.2
docutils==0.17.1
dopamine-rl==1.0.5
earthengine-api==0.1.295
easydict==1.9
ecos==2.0.10
editdistance==0.5.3
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.5/en_core_web_sm-2.2.5.tar.gz
entrypoints==0.3
ephem==4.1.3
et-xmlfile==1.1.0
fa2==0.3.5
fastai==1.0.61
fastcluster==1.2.4
fastdtw==0.3.4
fastprogress==1.0.0
fastrlock==0.8
fbprophet==0.7.1
feather-format==0.4.1
ffmpy==0.2.3
filelock==3.4.2
firebase-admin==4.4.0
fix-yahoo-finance==0.0.22
Flask==1.1.4
flatbuffers==2.0
folium==0.8.3
future==0.16.0
gast==0.4.0
GDAL==2.2.2
gdown==3.6.4
gensim==3.6.0
geographiclib==1.52
geopy==1.17.0
gin-config==0.5.0
glob2==0.7
google==2.0.3
google-api-core==1.26.3
google-api-python-client==1.12.10
google-auth==1.35.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.6
google-cloud-bigquery==1.21.0
google-cloud-bigquery-storage==1.1.0
google-cloud-core==1.0.3
google-cloud-datastore==1.8.0
google-cloud-firestore==1.7.0
google-cloud-language==1.2.0
google-cloud-storage==1.18.1
google-cloud-translate==1.5.0
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz
google-pasta==0.2.0
google-resumable-media==0.4.1
googleapis-common-protos==1.54.0
googledrivedownloader==0.4
GPUtil==1.4.0
graphviz==0.10.1
greenlet==1.1.2
grpcio==1.43.0
gspread==3.4.2
gspread-dataframe==3.0.8
gym==0.17.3
h5py==3.1.0
HeapDict==1.0.1
hijri-converter==2.2.2
holidays==0.10.5.2
holoviews==1.14.7
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.17.4
httplib2shim==0.0.3
humanize==0.5.1
hyperopt==0.1.2
ideep4py==2.0.0.post3
idna==2.10
imageio==2.14.1
imageio-ffmpeg==0.4.5
imagesize==1.3.0
imbalanced-learn==0.8.1
imblearn==0.0
imgaug==0.2.9
importlib-metadata==4.10.1
importlib-resources==5.4.0
imutils==0.5.4
inflect==2.1.0
iniconfig==1.1.1
intel-openmp==2022.0.2
intervaltree==2.1.0
ipykernel==4.10.1
ipython==5.5.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.6.5
itsdangerous==1.1.0
jax==0.2.25
jaxlib @ https://storage.googleapis.com/jax-releases/cuda111/jaxlib-0.1.71+cuda111-cp37-none-manylinux2010_x86_64.whl
jdcal==1.4.1
jedi==0.18.1
jieba==0.42.1
Jinja2==2.11.3
joblib==1.1.0
jpeg4py==0.1.4
jsonschema==4.3.3
jupyter==1.0.0
jupyter-client==5.3.5
jupyter-console==5.2.0
jupyter-core==4.9.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.2
kaggle==1.5.12
kapre==0.3.7
keras==2.7.0
Keras-Preprocessing==1.1.2
keras-vis==0.4.1
kiwisolver==1.3.2
korean-lunar-calendar==0.2.1
libclang==12.0.0
librosa==0.8.1
lightgbm==2.2.3
llvmlite==0.34.0
lmdb==0.99
LunarCalendar==0.0.9
lxml==4.2.6
Markdown==3.3.6
MarkupSafe==2.0.1
matplotlib==3.2.2
matplotlib-inline==0.1.3
matplotlib-venn==0.11.6
missingno==0.5.0
mistune==0.8.4
mizani==0.6.0
mkl==2019.0
mlxtend==0.14.0
more-itertools==8.12.0
moviepy==0.2.3.5
mpmath==1.2.1
msgpack==1.0.3
multiprocess==0.70.12.2
multitasking==0.0.10
murmurhash==1.0.6
music21==5.5.0
natsort==5.5.0
nbclient==0.5.10
nbconvert==5.6.1
nbformat==5.1.3
nest-asyncio==1.5.4
netCDF4==1.5.8
networkx==2.6.3
nibabel==3.0.2
nltk==3.2.5
notebook==5.3.1
numba==0.51.2
numexpr==2.8.1
numpy==1.19.5
nvidia-ml-py==11.495.46
nvidia-ml-py3==7.352.0
oauth2client==4.1.3
oauthlib==3.1.1
okgrade==0.4.3
opencv-contrib-python==4.1.2.30
opencv-python==4.5.5.62
openpyxl==2.5.9
opt-einsum==3.3.0
osqp==0.6.2.post0
packaging==21.3
palettable==3.3.0
pandas==1.1.5
pandas-datareader==0.9.0
pandas-gbq==0.13.3
pandas-profiling==1.4.1
pandocfilters==1.5.0
panel==0.12.1
param==1.12.0
parso==0.8.3
pathlib==1.0.1
patsy==0.5.2
pep517==0.12.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.0.0
pip-tools==6.2.0
plac==1.1.3
plotly==5.5.0
plotnine==0.6.0
pluggy==0.7.1
pooch==1.6.0
portpicker==1.3.9
prefetch-generator==1.0.1
preshed==3.0.6
prettytable==3.0.0
progressbar2==3.38.0
prometheus-client==0.12.0
promise==2.3
prompt-toolkit==1.0.18
protobuf==3.17.3
psutil==5.9.0
psycopg2==2.7.6.1
ptyprocess==0.7.0
py==1.11.0
pyarrow==3.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.4
pycparser==2.21
pyct==0.4.8
pydata-google-auth==1.3.0
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyerfa==2.0.0.1
pyglet==1.5.0
Pygments==2.6.1
pygobject==3.26.1
pymc3==3.11.4
PyMeeus==0.5.11
pymongo==4.0.1
pymystem3==0.2.0
PyOpenGL==3.1.5
pyparsing==3.0.7
pyrsistent==0.18.1
pysndfile==1.3.8
PySocks==1.7.1
pystan==2.19.1.1
pytest==3.6.4
python-apt==0.0.0
python-chess==0.23.11
python-dateutil==2.8.2
python-louvain==0.15
python-slugify==5.0.2
python-utils==3.1.0
pytz==2018.9
pyviz-comms==2.1.0
PyWavelets==1.2.0
PyYAML==3.13
pyzmq==22.3.0
qdldl==0.1.5.post0
qtconsole==5.2.2
QtPy==2.0.0
regex==2019.12.20
requests==2.23.0
requests-oauthlib==1.3.0
resampy==0.2.2
rpy2==3.4.5
rsa==4.8
scikit-image==0.18.3
scikit-learn==1.0.2
scipy==1.4.1
screen-resolution-extra==0.0.0
scs==3.1.0
seaborn==0.11.2
semver==2.13.0
Send2Trash==1.8.0
setuptools-git==1.2
Shapely==1.8.0
simplegeneric==0.8.1
six==1.15.0
sklearn==0.0
sklearn-pandas==1.8.0
smart-open==5.2.1
snowballstemmer==2.2.0
sortedcontainers==2.4.0
SoundFile==0.10.3.post1
spacy==2.2.4
Sphinx==1.8.6
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib-websupport==1.2.4
SQLAlchemy==1.4.31
sqlparse==0.4.2
srsly==1.0.5
statsmodels==0.10.2
sympy==1.7.1
tables==3.4.4
tabulate==0.8.9
tblib==1.7.0
tenacity==8.0.1
tensorboard==2.7.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow @ file:///tensorflow-2.7.0-cp37-cp37m-linux_x86_64.whl
tensorflow-datasets==4.0.1
tensorflow-estimator==2.7.0
tensorflow-gcs-config==2.7.0
tensorflow-hub==0.12.0
tensorflow-io-gcs-filesystem==0.23.1
tensorflow-metadata==1.6.0
tensorflow-probability==0.15.0
termcolor==1.1.0
terminado==0.12.1
testpath==0.5.0
text-unidecode==1.3
textblob==0.15.3
Theano-PyMC==1.1.2
thinc==7.4.0
threadpoolctl==3.0.0
tifffile==2021.11.2
toml==0.10.2
tomli==2.0.0
toolz==0.11.2
torch @ https://download.pytorch.org/whl/cu111/torch-1.10.0%2Bcu111-cp37-cp37m-linux_x86_64.whl
torchaudio @ https://download.pytorch.org/whl/cu111/torchaudio-0.10.0%2Bcu111-cp37-cp37m-linux_x86_64.whl
torchsummary==1.5.1
torchtext==0.11.0
torchvision @ https://download.pytorch.org/whl/cu111/torchvision-0.11.1%2Bcu111-cp37-cp37m-linux_x86_64.whl
tornado==5.1.1
tqdm==4.62.3
traitlets==5.1.1
tweepy==3.10.0
typeguard==2.7.1
typing-extensions==3.10.0.2
tzlocal==1.5.1
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.9.0
wasabi==0.9.0
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.2
wordcloud==1.5.0
wrapt==1.13.3
xarray==0.18.2
xgboost==0.90
xkit==0.0.0
xlrd==1.1.0
xlwt==1.3.0
yellowbrick==1.3.post1
zict==2.0.0
zipp==3.7.0

Tags:
User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Re: [Resource] Training Using Google Colab

Post by aolvera27 »

And here's the second part of the crash report:

Code: Select all

=============== State File =================
{
  "name": "villain",
  "sessions": {
    "1": {
      "timestamp": 1641934344.543849,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 51,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "2": {
      "timestamp": 1641934623.014452,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 10,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "3": {
      "timestamp": 1641934768.9540205,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 221,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "4": {
      "timestamp": 1641947575.8845406,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "5": {
      "timestamp": 1641954699.6672215,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 167,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "6": {
      "timestamp": 1641955327.2494335,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 3200,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "7": {
      "timestamp": 1642000981.0195663,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 39,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "8": {
      "timestamp": 1642001409.712794,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 403,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "9": {
      "timestamp": 1642010609.8734589,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 2568,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "10": {
      "timestamp": 1642173293.5962155,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 4,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "11": {
      "timestamp": 1642174851.483579,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 5,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "12": {
      "timestamp": 1642311664.1887724,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 631,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "13": {
      "timestamp": 1642312856.8077965,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 7000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "14": {
      "timestamp": 1642388467.9061952,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1501,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "15": {
      "timestamp": 1642390893.315852,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 49,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "16": {
      "timestamp": 1642391024.4992113,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 2706,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "17": {
      "timestamp": 1642395200.0481231,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 4464,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "18": {
      "timestamp": 1642402275.408796,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1547,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "19": {
      "timestamp": 1642404758.0418355,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 883,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "20": {
      "timestamp": 1642406265.8939164,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 51,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "21": {
      "timestamp": 1642428718.068633,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 4355,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "22": {
      "timestamp": 1642435712.9913034,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 2290,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "23": {
      "timestamp": 1642439393.9256265,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 226,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "24": {
      "timestamp": 1642443270.867618,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 849,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "25": {
      "timestamp": 1642446988.2322323,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 2879,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "26": {
      "timestamp": 1642451852.8665605,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1467,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "27": {
      "timestamp": 1642463927.643077,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 4633,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "28": {
      "timestamp": 1642479362.051146,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 5869,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "29": {
      "timestamp": 1642489251.8226745,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "30": {
      "timestamp": 1642517320.5381777,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 2581,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "31": {
      "timestamp": 1642521574.2323477,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 2000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "32": {
      "timestamp": 1642525392.1788833,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1582,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "33": {
      "timestamp": 1642528166.7455914,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 3568,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "34": {
      "timestamp": 1642534259.9923055,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 4351,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "35": {
      "timestamp": 1642601548.2177327,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 229,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "36": {
      "timestamp": 1642602424.9744818,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 5622,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "37": {
      "timestamp": 1642612239.5765386,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 83,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "38": {
      "timestamp": 1642612469.9669256,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 8000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "39": {
      "timestamp": 1642639530.4240417,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1915,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "40": {
      "timestamp": 1642642650.6493409,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 5692,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "41": {
      "timestamp": 1642651725.0207484,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 5128,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "42": {
      "timestamp": 1642660007.4126034,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 380,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "43": {
      "timestamp": 1642660757.283941,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 6000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "44": {
      "timestamp": 1642697681.6425247,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 8800,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "45": {
      "timestamp": 1642712259.4185376,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 1600,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "46": {
      "timestamp": 1642725332.1631973,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 21900,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "47": {
      "timestamp": 1642784753.5974474,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 15000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "48": {
      "timestamp": 1642823971.201501,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 6601,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "49": {
      "timestamp": 1642835376.511773,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 8000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "50": {
      "timestamp": 1642855231.9796786,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 4149,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "51": {
      "timestamp": 1642862084.321354,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 3070,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "52": {
      "timestamp": 1643039771.70297,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 10180,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "53": {
      "timestamp": 1643056310.7085412,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 2000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "54": {
      "timestamp": 1643119841.0648844,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 1000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "55": {
      "timestamp": 1643126050.79449,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 8200,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "56": {
      "timestamp": 1643204255.8205197,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 6750,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "57": {
      "timestamp": 1643220198.5405877,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 500,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "58": {
      "timestamp": 1643222067.9398222,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "59": {
      "timestamp": 1643243663.408284,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 7050,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "60": {
      "timestamp": 1643254403.6218565,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 12000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "61": {
      "timestamp": 1643293853.5289521,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 6000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    }
  },
  "lowest_avg_loss": {
    "a": 0.006357450038194656,
    "b": 0.01431506872177124
  },
  "iterations": 220000,
  "config": {
    "centering": "face",
    "coverage": 85.0,
    "optimizer": "adam",
    "learning_rate": 5e-05,
    "epsilon_exponent": -7,
    "allow_growth": false,
    "mixed_precision": false,
    "nan_protection": true,
    "convert_batchsize": 16,
    "loss_function": "ssim",
    "mask_loss_function": "mse",
    "l2_reg_term": 100,
    "eye_multiplier": 3,
    "mouth_multiplier": 2,
    "penalized_mask_loss": true,
    "mask_type": "extended",
    "mask_blur_kernel": 3,
    "mask_threshold": 4,
    "learn_mask": false,
    "lowmem": false
  }
}

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

--------- extract.ini ---------

[global]
allow_growth:             False

[detect.cv2_dnn]
confidence:               50

[detect.s3fd]
confidence:               70
batch-size:               4

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[align.fan]
batch-size:               12

[mask.vgg_clear]
batch-size:               6

[mask.bisenet_fp]
batch-size:               8
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.vgg_obstructed]
batch-size:               2

[mask.unet_dfl]
batch-size:               8

--------- train.ini ---------

[global]
centering:                face
coverage:                 85.0
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
mask_loss_function:       mse
l2_reg_term:              100
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfl_h128]
lowmem:                   False

[model.phaze_a]
output_size:              128
shared_fc:                none
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              40
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          none
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_norm:                 none
dec_min_filters:          64
dec_max_filters:          512
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.villain]
lowmem:                   False

[model.original]
lowmem:                   False

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.dfaker]
output_size:              128

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4
User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: [Resource] Training Using Google Colab

Post by torzdf »

This looks like Out Of Memory.. Try lowering batch size.

My word is final

User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Re: [Resource] Training Using Google Colab

Post by aolvera27 »

You're probably right. It works again now.

User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Re: [Resource] Training Using Google Colab

Post by aolvera27 »

This stopped working again, which makes me wonder if they're purposefully trying to prevent people from using their GPU resources on Faceswap. =p

The error seems pretty easy. When trying to run training, it quickly tells me that Tensorflow should be downgraded from 2.8 to 2.7, and also this:

Code: Select all

module compiled against API version 0xe but this version of numpy is 0xd

Indeed, I noticed this message after running !pip install -r faceswap/_requirements_base.txt:

Code: Select all

Ignoring pywin32: markers 'sys_platform == "win32"' don't match your environment faceswap
Ignoring pynvx: markers 'sys_platform == "darwin"' don't match your environment

So I proceed to do this, thinking it would be just as simple:

Code: Select all

!pip install numpy --upgrade
!pip install tensorflow==2.7
!pip install tensorflow-gpu==2.7

But it just makes it worse. I guess it's time for me to really learn what I'm doing? Do you have advice on this?

User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Re: [Resource] Training Using Google Colab

Post by aolvera27 »

Sorry for replying to myself. I checked Andentze's way of setting up Faceswap in Colab, which he shares hereand I updated my Notebook accordingly.

Code: Select all

!pip install -r faceswap/_requirements_base.txt
!git clone https://github.com/andentze/facecolab_requirements
!pip uninstall -y tensorflow
!pip install -r "/content/facecolab_requirements/requirements_nvidia.txt"

It is working now, although I don't know for how long or whether I'm breaking it somewhere else.

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: [Outdated] Training Using Google Colab

Post by torzdf »

A new Notebook has been setup at the following location: viewtopic.php?f=23&t=1886 Please use that thread for further colab discussions.

I would like to thank the original Notebook author and contributors for their efforts so far.

My word is final

Locked