I ran into a problem this evening!
I have been fit training a model to nearly 300k iterations on realface.
This evening I manually aligned the faces from the video I actually wish to convert, loaded them onto the trainer and trained over 1000 iterations.
I saved the project and noticed faceswap needed an update, so I updated and restarted faceswap.
After restarting, I tried to continue training but kept getting this error.
Any idea what happened??
Tried completely deleting and reinstalling faceswap and still the same.
Code: Select all
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_007065_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_003471_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002746_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003409_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009461_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003673_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_006586_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_006586_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_007065_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003564_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Mask already generated. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010720_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Mask already generated. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010699_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Mask already generated. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009550_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_007065_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010441_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_003471_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002746_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_006586_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000201_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009461_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009461_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000201_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_009766_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000201_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003392_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003564_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_009766_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003564_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_001772_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004766_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010441_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_009766_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002997_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010441_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004766_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009739_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003392_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003392_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_001186_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004766_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_000947_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_001772_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004896_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001956_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001846_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_001772_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_001186_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004896_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001956_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001846_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002997_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002971_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_001186_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002997_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_007801_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004896_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001956_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001846_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009739_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009739_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_007801_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002461_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_008748_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_007801_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_000947_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_006922_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_000947_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002461_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003828_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002461_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000261_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002971_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000261_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004596_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002431_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004596_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_008748_0.png'
09/08/2020 02:10:50 MainProcess _run_0 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_005456_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002431_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_005456_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_006922_0.png'
09/08/2020 02:10:50 MainProcess _run_1 training_data _expand_partials DEBUG Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003828_0.png'
09/08/2020 02:10:51 MainProcess _training_0 losses_tf call DEBUG Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A6CF9AEE0>, weight: 1.0, mask_channel: -1)
09/08/2020 02:10:51 MainProcess _training_0 losses_tf _apply_mask DEBUG No mask to apply
09/08/2020 02:10:51 MainProcess _training_0 losses_tf call DEBUG Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A5BC31C70>, weight: 1.0, mask_channel: -1)
09/08/2020 02:10:51 MainProcess _training_0 losses_tf _apply_mask DEBUG No mask to apply
09/08/2020 02:10:51 MainProcess _training_0 losses_tf call DEBUG Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A6C7D3430>, weight: 12.0, mask_channel: 3)
09/08/2020 02:10:51 MainProcess _training_0 losses_tf _apply_mask DEBUG Applying mask from channel 3
09/08/2020 02:10:51 MainProcess _training_0 losses_tf call DEBUG Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A6C723D60>, weight: 1.0, mask_channel: 1)
09/08/2020 02:10:51 MainProcess _training_0 losses_tf _apply_mask DEBUG Applying mask from channel 1
09/08/2020 02:10:51 MainProcess _training_0 losses_tf call DEBUG Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A5BA4CCA0>, weight: 8.0, mask_channel: 4)
09/08/2020 02:10:51 MainProcess _training_0 losses_tf _apply_mask DEBUG Applying mask from channel 4
09/08/2020 02:10:51 MainProcess _training_0 multithreading run DEBUG Error in thread (_training_0): in user code:\n\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:571 train_function *\n outputs = self.distribute_strategy.run(\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:951 run **\n return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2290 call_for_each_replica\n return self._call_for_each_replica(fn, args, kwargs)\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2649 _call_for_each_replica\n return fn(*args, **kwargs)\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:532 train_step **\n loss = self.compiled_loss(\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\compile_utils.py:205 __call__\n loss_value = loss_obj(y_t, y_p, sample_weight=sw)\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\losses.py:143 __call__\n losses = self.call(y_true, y_pred)\n I:\FS\faceswap\lib\model\losses_tf.py:559 call\n n_true, n_pred = self._apply_mask(y_true, y_pred, mask_channel)\n I:\FS\faceswap\lib\model\losses_tf.py:589 _apply_mask\n mask = K.expand_dims(y_true[..., mask_channel], axis=-1)\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:973 _slice_helper\n return strided_slice(\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:1140 strided_slice\n op = gen_array_ops.strided_slice(\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_array_ops.py:10174 strided_slice\n _, _, _op, _outputs = _op_def_library._apply_op_helper(\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\op_def_library.py:742 _apply_op_helper\n op = g._create_op_internal(op_type_name, inputs, dtypes=None,\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py:593 _create_op_internal\n return super(FuncGraph, self)._create_op_internal( # pylint: disable=protected-access\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:3319 _create_op_internal\n ret = Operation(\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1816 __init__\n self._c_op = _create_c_op(self._graph, node_def, inputs,\n C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1657 _create_c_op\n raise ValueError(str(e))\n\n ValueError: slice index 4 of dimension 3 out of bounds. for '{{node LossWrapper/strided_slice_22}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=1, end_mask=0, new_axis_mask=0, shrink_axis_mask=2](IteratorGetNext:2, LossWrapper/strided_slice_22/stack, LossWrapper/strided_slice_22/stack_1, LossWrapper/strided_slice_22/stack_2)' with input shapes: [16,128,128,4], [2], [2], [2] and with computed input tensors: input[1] = <0 4>, input[2] = <0 5>, input[3] = <1 1>.\n
09/08/2020 02:10:51 MainProcess MainThread train _monitor DEBUG Thread error detected
09/08/2020 02:10:51 MainProcess MainThread train _monitor DEBUG Closed Monitor
09/08/2020 02:10:51 MainProcess MainThread train _end_thread DEBUG Ending Training thread
09/08/2020 02:10:51 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...
09/08/2020 02:10:51 MainProcess MainThread multithreading join DEBUG Joining Threads: '_training'
09/08/2020 02:10:51 MainProcess MainThread multithreading join DEBUG Joining Thread: '_training_0'
09/08/2020 02:10:51 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training_0'
Traceback (most recent call last):
File "I:\FS\faceswap\lib\cli\launcher.py", line 156, in execute_script
process.process()
File "I:\FS\faceswap\scripts\train.py", line 165, in process
self._end_thread(thread, err)
File "I:\FS\faceswap\scripts\train.py", line 205, in _end_thread
thread.join()
File "I:\FS\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "I:\FS\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "I:\FS\faceswap\scripts\train.py", line 227, in _training
raise err
File "I:\FS\faceswap\scripts\train.py", line 217, in _training
self._run_training_cycle(model, trainer)
File "I:\FS\faceswap\scripts\train.py", line 298, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "I:\FS\faceswap\plugins\train\trainer\_base.py", line 211, in train_one_step
loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1348, in train_on_batch
logs = train_function(iterator)
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__
result = self._call(*args, **kwds)
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 627, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 505, in _initialize
self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2446, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2777, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2657, in _create_graph_function
func_graph_module.func_graph_from_py_func(
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py", line 981, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 441, in wrapped_fn
return weak_wrapped_fn().__wrapped__(*args, **kwds)
File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py", line 968, in wrapper
raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:571 train_function *
outputs = self.distribute_strategy.run(
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:951 run **
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2290 call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2649 _call_for_each_replica
return fn(*args, **kwargs)
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:532 train_step **
loss = self.compiled_loss(
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\compile_utils.py:205 __call__
loss_value = loss_obj(y_t, y_p, sample_weight=sw)
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\losses.py:143 __call__
losses = self.call(y_true, y_pred)
I:\FS\faceswap\lib\model\losses_tf.py:559 call
n_true, n_pred = self._apply_mask(y_true, y_pred, mask_channel)
I:\FS\faceswap\lib\model\losses_tf.py:589 _apply_mask
mask = K.expand_dims(y_true[..., mask_channel], axis=-1)
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:973 _slice_helper
return strided_slice(
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:1140 strided_slice
op = gen_array_ops.strided_slice(
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_array_ops.py:10174 strided_slice
_, _, _op, _outputs = _op_def_library._apply_op_helper(
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\op_def_library.py:742 _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py:593 _create_op_internal
return super(FuncGraph, self)._create_op_internal( # pylint: disable=protected-access
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:3319 _create_op_internal
ret = Operation(
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1816 __init__
self._c_op = _create_c_op(self._graph, node_def, inputs,
C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1657 _create_c_op
raise ValueError(str(e))
ValueError: slice index 4 of dimension 3 out of bounds. for '{{node LossWrapper/strided_slice_22}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=1, end_mask=0, new_axis_mask=0, shrink_axis_mask=2](IteratorGetNext:2, LossWrapper/strided_slice_22/stack, LossWrapper/strided_slice_22/stack_1, LossWrapper/strided_slice_22/stack_2)' with input shapes: [16,128,128,4], [2], [2], [2] and with computed input tensors: input[1] = <0 4>, input[2] = <0 5>, input[3] = <1 1>.
============ System Information ============
encoding: cp1252
git_branch: master
git_commits: 5530346 Bugfix - Pre-caching masks when warp-to-landmarks is disabled. 49f86e1 training - Add config item to disable warp. cd3fc81 Training - Cache eye and mouth masks during first epoch rather than at startup. a2f1cb4 Bugfix: Preview Tool. Fix blank swapped faces issue. 10da0c4 Loss multipliers: - Lower defaults. - Set to updateable
gpu_cuda: No global version found. Check Conda packages for Conda Cuda
gpu_cudnn: No global version found. Check Conda packages for Conda cuDNN
gpu_devices: GPU_0: GeForce RTX 2060 SUPER
gpu_devices_active: GPU_0
gpu_driver: 452.06
gpu_vram: GPU_0: 8192MB
os_machine: AMD64
os_platform: Windows-10-10.0.18362-SP0
os_release: 10
py_command: I:\FS\faceswap\faceswap.py train -A I:/Faceswap Resources/cl1/FaceSwap/InputA/Extracted/finnn -ala I:/Faceswap Resources/cl1/FaceSwap/InputA/Video/Untitled Projectccc_alignments.fsa -B I:/Faceswap Resources/cl1/FaceSwap/InputB/Extracted -alb I:/Faceswap Resources/cl1/FaceSwap/InputB/Video/alignments_merged_20200902_215938.fsa -m I:/Faceswap Resources/cl1/FaceSwap/Models -t realface -bs 16 -it 20000 -s 200 -ss 40000 -ps 50 -L INFO -gui
py_conda_version: conda 4.8.4
py_implementation: CPython
py_version: 3.8.5
py_virtual_env: True
sys_cores: 4
sys_processor: AMD64 Family 23 Model 17 Stepping 0, AuthenticAMD
sys_ram: Total: 32715MB, Available: 26808MB, Used: 5906MB, Free: 26808MB
=============== Pip Packages ===============
absl-py==0.9.0
astunparse==1.6.3
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.3.3
google-auth==1.20.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.31.0
h5py==2.10.0
idna==2.10
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1589202782679/work
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
Markdown==3.2.2
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.1.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.4.0.40
opt-einsum==3.3.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1594298230227/work
protobuf==3.13.0
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
pywin32==227
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-learn @ file:///C:/ci/scikit-learn_1592853510272/work
scipy==1.4.1
sip==4.19.13
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tensorflow-estimator==2.2.0
tensorflow-gpu==2.2.0
tensorflow-gpu-estimator==2.2.0
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1596810128862/work
urllib3==1.25.10
Werkzeug==1.0.1
wincertstore==0.2
wrapt==1.12.1
============== Conda Packages ==============
# packages in environment at C:\Users\Matt\MiniConda3\envs\faceswap:
#
# Name Version Build Channel
absl-py 0.9.0 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
blas 1.0 mkl
ca-certificates 2020.6.24 0
cachetools 4.1.1 pypi_0 pypi
certifi 2020.6.20 py38_0
chardet 3.0.4 pypi_0 pypi
cudatoolkit 10.1.243 h74a9793_0
cudnn 7.6.5 cuda10.1_0
cycler 0.10.0 py38_0
fastcluster 1.1.26 py38hbe40bda_1 conda-forge
ffmpeg 4.3.1 ha925a31_0 conda-forge
ffmpy 0.2.3 pypi_0 pypi
freetype 2.10.2 hd328e21_0
gast 0.3.3 pypi_0 pypi
git 2.23.0 h6bb4b03_0
google-auth 1.20.1 pypi_0 pypi
google-auth-oauthlib 0.4.1 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.31.0 pypi_0 pypi
h5py 2.10.0 pypi_0 pypi
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 pypi_0 pypi
imageio 2.9.0 py_0
imageio-ffmpeg 0.4.2 py_0 conda-forge
intel-openmp 2020.1 216
joblib 0.16.0 py_0
jpeg 9b hb83a4c4_2
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.2.0 py38h74a9793_0
libpng 1.6.37 h2a8f88b_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 h62dcd97_1
markdown 3.2.2 pypi_0 pypi
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38h64f37c6_0
mkl 2020.1 216
mkl-service 2.3.0 py38hb782905_0
mkl_fft 1.1.0 py38h45dec08_0
mkl_random 1.1.1 py38h47e9c7a_0
numpy 1.19.1 py38h5510c5b_0
numpy-base 1.19.1 py38ha3acd2a_0
nvidia-ml-py3 7.352.1 pypi_0 pypi
oauthlib 3.1.0 pypi_0 pypi
olefile 0.46 py_0
opencv-python 4.4.0.40 pypi_0 pypi
openssl 1.1.1g he774522_1
opt-einsum 3.3.0 pypi_0 pypi
pathlib 1.0.1 py_1
pillow 7.2.0 py38hcc1f983_0
pip 20.2.2 py38_0
protobuf 3.13.0 pypi_0 pypi
psutil 5.7.0 py38he774522_0
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pyparsing 2.4.7 py_0
pyqt 5.9.2 py38ha925a31_4
python 3.8.5 he1778fa_0
python-dateutil 2.8.1 py_0
python_abi 3.8 1_cp38 conda-forge
pywin32 227 py38he774522_1
qt 5.9.7 vc14h73c81de_0
requests 2.24.0 pypi_0 pypi
requests-oauthlib 1.3.0 pypi_0 pypi
rsa 4.6 pypi_0 pypi
scikit-learn 0.23.1 py38h25d0782_0
scipy 1.4.1 pypi_0 pypi
setuptools 49.6.0 py38_0
sip 4.19.13 py38ha925a31_0
six 1.15.0 py_0
sqlite 3.32.3 h2a8f88b_0
tensorboard 2.2.2 pypi_0 pypi
tensorboard-plugin-wit 1.7.0 pypi_0 pypi
tensorflow-estimator 2.2.0 pypi_0 pypi
tensorflow-gpu 2.2.0 pypi_0 pypi
tensorflow-gpu-estimator 2.2.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
tornado 6.0.4 py38he774522_1
tqdm 4.48.2 py_0
urllib3 1.25.10 pypi_0 pypi
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_3
werkzeug 1.0.1 pypi_0 pypi
wheel 0.34.2 py38_0
wincertstore 0.2 py38_0
wrapt 1.12.1 pypi_0 pypi
xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.5 h04227a9_0
=============== State File =================
{
"name": "realface",
"sessions": {
"1": {
"timestamp": 1599115396.6223,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 12,
"iterations": 24,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"2": {
"timestamp": 1599155754.10659,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 12,
"iterations": 3143,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"3": {
"timestamp": 1599160104.3641214,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 12,
"iterations": 722,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"4": {
"timestamp": 1599165347.106994,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 12,
"iterations": 108393,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"5": {
"timestamp": 1599241490.780011,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 12,
"iterations": 190,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"6": {
"timestamp": 1599241750.2682142,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 46134,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"7": {
"timestamp": 1599259533.2947102,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 87784,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"8": {
"timestamp": 1599324558.7970932,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 15801,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"9": {
"timestamp": 1599330730.2586458,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 35648,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"10": {
"timestamp": 1599526126.6474314,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 825,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
},
"11": {
"timestamp": 1599527226.4353244,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 110,
"config": {
"learning_rate": 5e-05,
"allow_growth": false,
"convert_batchsize": 16
}
}
},
"lowest_avg_loss": {
"a": 0.05607394129037857,
"b": 0.07404948770999908
},
"iterations": 298774,
"training_size": 256,
"config": {
"coverage": 86.49,
"optimizer": "adam",
"learning_rate": 5e-05,
"allow_growth": false,
"mixed_precision": true,
"convert_batchsize": 16,
"loss_function": "ssim",
"mask_loss_function": "mse",
"l2_reg_term": 100,
"eye_multiplier": 12,
"mouth_multiplier": 8,
"penalized_mask_loss": false,
"mask_type": "extended",
"mask_blur_kernel": 3,
"mask_threshold": 4,
"learn_mask": false,
"input_size": 64,
"output_size": 128,
"dense_nodes": 1536,
"complexity_encoder": 128,
"complexity_decoder": 512
}
}
================= Configs ==================
--------- .faceswap ---------
backend: nvidia
--------- convert.ini ---------
[color.color_transfer]
clip: True
preserve_paper: True
[color.manual_balance]
colorspace: HSV
balance_1: 0.0
balance_2: 0.0
balance_3: 0.0
contrast: 0.0
brightness: 0.0
[color.match_hist]
threshold: 99.0
[mask.box_blend]
type: gaussian
distance: 11.0
radius: 5.0
passes: 1
[mask.mask_blend]
type: gaussian
kernel_size: 3
passes: 4
threshold: 4
erosion: 0.0
[scaling.sharpen]
method: gaussian
amount: 150
radius: 0.3
threshold: 5.0
[writer.ffmpeg]
container: mp4
codec: libx264
crf: 23
preset: medium
tune: none
profile: auto
level: auto
skip_mux: False
[writer.gif]
fps: 25
loop: 0
palettesize: 256
subrectangles: False
[writer.opencv]
format: png
draw_transparent: False
jpg_quality: 75
png_compress_level: 3
[writer.pillow]
format: png
draw_transparent: False
optimize: False
gif_interlace: True
jpg_quality: 75
png_compress_level: 3
tif_compression: tiff_deflate
--------- extract.ini ---------
[global]
allow_growth: False
[align.fan]
batch-size: 12
[detect.cv2_dnn]
confidence: 50
[detect.mtcnn]
minsize: 20
threshold_1: 0.6
threshold_2: 0.7
threshold_3: 0.7
scalefactor: 0.709
batch-size: 8
[detect.s3fd]
confidence: 70
batch-size: 4
[mask.unet_dfl]
batch-size: 8
[mask.vgg_clear]
batch-size: 6
[mask.vgg_obstructed]
batch-size: 4
--------- gui.ini ---------
[global]
fullscreen: True
tab: extract
options_panel_width: 30
console_panel_height: 20
icon_size: 14
font: default
font_size: 9
autosave_last_session: prompt
timeout: 120
auto_load_model_stats: True
--------- train.ini ---------
[global]
coverage: 81.85
icnr_init: False
conv_aware_init: False
optimizer: adam
learning_rate: 5e-05
reflect_padding: False
allow_growth: False
mixed_precision: False
convert_batchsize: 16
[global.loss]
loss_function: ssim
mask_loss_function: mse
l2_reg_term: 100
eye_multiplier: 6
mouth_multiplier: 4
penalized_mask_loss: True
mask_type: extended
mask_blur_kernel: 3
mask_threshold: 4
learn_mask: False
[model.dfl_h128]
lowmem: False
[model.dfl_sae]
input_size: 128
clipnorm: True
architecture: df
autoencoder_dims: 0
encoder_dims: 42
decoder_dims: 21
multiscale_decoder: False
[model.dlight]
features: best
details: good
output_size: 256
[model.original]
lowmem: False
[model.realface]
input_size: 64
output_size: 128
dense_nodes: 1536
complexity_encoder: 128
complexity_decoder: 512
[model.unbalanced]
input_size: 128
lowmem: False
clipnorm: True
nodes: 1024
complexity_encoder: 128
complexity_decoder_a: 384
complexity_decoder_b: 512
[model.villain]
lowmem: False
[trainer.original]
preview_images: 14
zoom_amount: 5
rotation_range: 10
shift_range: 5
flip_chance: 50
disable_warp: False
color_lightness: 30
color_ab: 8
color_clahe_chance: 50
color_clahe_max_size: 4