Could not train after updating

Training your model
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Post Reply
User avatar
Antipope1
Posts: 33
Joined: Thu Jul 02, 2020 10:51 am
Has thanked: 6 times
Been thanked: 2 times

Could not train after updating

Post by Antipope1 »

I ran into a problem this evening!
I have been fit training a model to nearly 300k iterations on realface.
This evening I manually aligned the faces from the video I actually wish to convert, loaded them onto the trainer and trained over 1000 iterations.
I saved the project and noticed faceswap needed an update, so I updated and restarted faceswap.
After restarting, I tried to continue training but kept getting this error.
Any idea what happened??
Tried completely deleting and reinstalling faceswap and still the same.

Code: Select all

 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_007065_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_003471_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002746_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003409_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003673_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_006586_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_006586_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_007065_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003564_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010720_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010699_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009550_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_007065_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010441_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_003471_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002746_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_006586_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000201_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000201_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_009766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000201_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003392_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003564_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_009766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003564_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_001772_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010441_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_009766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002997_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010441_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009739_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003392_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003392_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_001186_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_000947_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_001772_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004896_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001956_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001846_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_001772_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_001186_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004896_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001956_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001846_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002997_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002971_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_001186_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002997_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_007801_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004896_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001956_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001846_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009739_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009739_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_007801_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_008748_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_007801_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_000947_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_006922_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_000947_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003828_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000261_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002971_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000261_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004596_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002431_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004596_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_008748_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_005456_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002431_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_005456_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_006922_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003828_0.png'
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A6CF9AEE0>, weight: 1.0, mask_channel: -1)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    No mask to apply
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A5BC31C70>, weight: 1.0, mask_channel: -1)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    No mask to apply
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A6C7D3430>, weight: 12.0, mask_channel: 3)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    Applying mask from channel 3
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A6C723D60>, weight: 1.0, mask_channel: 1)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    Applying mask from channel 1
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A5BA4CCA0>, weight: 8.0, mask_channel: 4)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    Applying mask from channel 4
 09/08/2020 02:10:51 MainProcess     _training_0     multithreading  run                       DEBUG    Error in thread (_training_0): in user code:\n\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:571 train_function  *\n        outputs = self.distribute_strategy.run(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:951 run  **\n        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2290 call_for_each_replica\n        return self._call_for_each_replica(fn, args, kwargs)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2649 _call_for_each_replica\n        return fn(*args, **kwargs)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:532 train_step  **\n        loss = self.compiled_loss(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\compile_utils.py:205 __call__\n        loss_value = loss_obj(y_t, y_p, sample_weight=sw)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\losses.py:143 __call__\n        losses = self.call(y_true, y_pred)\n    I:\FS\faceswap\lib\model\losses_tf.py:559 call\n        n_true, n_pred = self._apply_mask(y_true, y_pred, mask_channel)\n    I:\FS\faceswap\lib\model\losses_tf.py:589 _apply_mask\n        mask = K.expand_dims(y_true[..., mask_channel], axis=-1)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:973 _slice_helper\n        return strided_slice(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:1140 strided_slice\n        op = gen_array_ops.strided_slice(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_array_ops.py:10174 strided_slice\n        _, _, _op, _outputs = _op_def_library._apply_op_helper(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\op_def_library.py:742 _apply_op_helper\n        op = g._create_op_internal(op_type_name, inputs, dtypes=None,\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py:593 _create_op_internal\n        return super(FuncGraph, self)._create_op_internal(  # pylint: disable=protected-access\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:3319 _create_op_internal\n        ret = Operation(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1816 __init__\n        self._c_op = _create_c_op(self._graph, node_def, inputs,\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1657 _create_c_op\n        raise ValueError(str(e))\n\n    ValueError: slice index 4 of dimension 3 out of bounds. for '{{node LossWrapper/strided_slice_22}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=1, end_mask=0, new_axis_mask=0, shrink_axis_mask=2](IteratorGetNext:2, LossWrapper/strided_slice_22/stack, LossWrapper/strided_slice_22/stack_1, LossWrapper/strided_slice_22/stack_2)' with input shapes: [16,128,128,4], [2], [2], [2] and with computed input tensors: input[1] = <0 4>, input[2] = <0 5>, input[3] = <1 1>.\n
 09/08/2020 02:10:51 MainProcess     MainThread      train           _monitor                  DEBUG    Thread error detected
 09/08/2020 02:10:51 MainProcess     MainThread      train           _monitor                  DEBUG    Closed Monitor
 09/08/2020 02:10:51 MainProcess     MainThread      train           _end_thread               DEBUG    Ending Training thread
 09/08/2020 02:10:51 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
 09/08/2020 02:10:51 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Threads: '_training'
 09/08/2020 02:10:51 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Thread: '_training_0'
 09/08/2020 02:10:51 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
 Traceback (most recent call last):
   File "I:\FS\faceswap\lib\cli\launcher.py", line 156, in execute_script
     process.process()
   File "I:\FS\faceswap\scripts\train.py", line 165, in process
     self._end_thread(thread, err)
   File "I:\FS\faceswap\scripts\train.py", line 205, in _end_thread
     thread.join()
   File "I:\FS\faceswap\lib\multithreading.py", line 121, in join
     raise thread.err[1].with_traceback(thread.err[2])
   File "I:\FS\faceswap\lib\multithreading.py", line 37, in run
     self._target(*self._args, **self._kwargs)
   File "I:\FS\faceswap\scripts\train.py", line 227, in _training
     raise err
   File "I:\FS\faceswap\scripts\train.py", line 217, in _training
     self._run_training_cycle(model, trainer)
   File "I:\FS\faceswap\scripts\train.py", line 298, in _run_training_cycle
     trainer.train_one_step(viewer, timelapse)
   File "I:\FS\faceswap\plugins\train\trainer\_base.py", line 211, in train_one_step
     loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1348, in train_on_batch
     logs = train_function(iterator)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__
     result = self._call(*args, **kwds)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 627, in _call
     self._initialize(args, kwds, add_initializers_to=initializers)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 505, in _initialize
     self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2446, in _get_concrete_function_internal_garbage_collected
     graph_function, _, _ = self._maybe_define_function(args, kwargs)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2777, in _maybe_define_function
     graph_function = self._create_graph_function(args, kwargs)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2657, in _create_graph_function
     func_graph_module.func_graph_from_py_func(
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py", line 981, in func_graph_from_py_func
     func_outputs = python_func(*func_args, **func_kwargs)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 441, in wrapped_fn
     return weak_wrapped_fn().__wrapped__(*args, **kwds)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py", line 968, in wrapper
     raise e.ag_error_metadata.to_exception(e)
 ValueError: in user code:
      C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:571 train_function  *
         outputs = self.distribute_strategy.run(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:951 run  **
         return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2290 call_for_each_replica
         return self._call_for_each_replica(fn, args, kwargs)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2649 _call_for_each_replica
         return fn(*args, **kwargs)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:532 train_step  **
         loss = self.compiled_loss(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\compile_utils.py:205 __call__
         loss_value = loss_obj(y_t, y_p, sample_weight=sw)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\losses.py:143 __call__
         losses = self.call(y_true, y_pred)
     I:\FS\faceswap\lib\model\losses_tf.py:559 call
         n_true, n_pred = self._apply_mask(y_true, y_pred, mask_channel)
     I:\FS\faceswap\lib\model\losses_tf.py:589 _apply_mask
         mask = K.expand_dims(y_true[..., mask_channel], axis=-1)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:973 _slice_helper
         return strided_slice(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:1140 strided_slice
         op = gen_array_ops.strided_slice(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_array_ops.py:10174 strided_slice
         _, _, _op, _outputs = _op_def_library._apply_op_helper(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\op_def_library.py:742 _apply_op_helper
         op = g._create_op_internal(op_type_name, inputs, dtypes=None,
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py:593 _create_op_internal
         return super(FuncGraph, self)._create_op_internal(  # pylint: disable=protected-access
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:3319 _create_op_internal
         ret = Operation(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1816 __init__
         self._c_op = _create_c_op(self._graph, node_def, inputs,
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1657 _create_c_op
         raise ValueError(str(e))
      ValueError: slice index 4 of dimension 3 out of bounds. for '{{node LossWrapper/strided_slice_22}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=1, end_mask=0, new_axis_mask=0, shrink_axis_mask=2](IteratorGetNext:2, LossWrapper/strided_slice_22/stack, LossWrapper/strided_slice_22/stack_1, LossWrapper/strided_slice_22/stack_2)' with input shapes: [16,128,128,4], [2], [2], [2] and with computed input tensors: input[1] = <0 4>, input[2] = <0 5>, input[3] = <1 1>.
   ============ System Information ============
 encoding:            cp1252
 git_branch:          master
 git_commits:         5530346 Bugfix - Pre-caching masks when warp-to-landmarks is disabled. 49f86e1 training - Add config item to disable warp. cd3fc81 Training - Cache eye and mouth masks during first epoch rather than at startup. a2f1cb4 Bugfix: Preview Tool. Fix blank swapped faces issue. 10da0c4 Loss multipliers:   - Lower defaults.   - Set to updateable
 gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
 gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
 gpu_devices:         GPU_0: GeForce RTX 2060 SUPER
 gpu_devices_active:  GPU_0
 gpu_driver:          452.06
 gpu_vram:            GPU_0: 8192MB
 os_machine:          AMD64
 os_platform:         Windows-10-10.0.18362-SP0
 os_release:          10
 py_command:          I:\FS\faceswap\faceswap.py train -A I:/Faceswap Resources/cl1/FaceSwap/InputA/Extracted/finnn -ala I:/Faceswap Resources/cl1/FaceSwap/InputA/Video/Untitled Projectccc_alignments.fsa -B I:/Faceswap Resources/cl1/FaceSwap/InputB/Extracted -alb I:/Faceswap Resources/cl1/FaceSwap/InputB/Video/alignments_merged_20200902_215938.fsa -m I:/Faceswap Resources/cl1/FaceSwap/Models -t realface -bs 16 -it 20000 -s 200 -ss 40000 -ps 50 -L INFO -gui
 py_conda_version:    conda 4.8.4
 py_implementation:   CPython
 py_version:          3.8.5
 py_virtual_env:      True
 sys_cores:           4
 sys_processor:       AMD64 Family 23 Model 17 Stepping 0, AuthenticAMD
 sys_ram:             Total: 32715MB, Available: 26808MB, Used: 5906MB, Free: 26808MB
  =============== Pip Packages ===============
 absl-py==0.9.0
 astunparse==1.6.3
 cachetools==4.1.1
 certifi==2020.6.20
 chardet==3.0.4
 cycler==0.10.0
 fastcluster==1.1.26
 ffmpy==0.2.3
 gast==0.3.3
 google-auth==1.20.1
 google-auth-oauthlib==0.4.1
 google-pasta==0.2.0
 grpcio==1.31.0
 h5py==2.10.0
 idna==2.10
 imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
 imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1589202782679/work
 joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
 Keras-Preprocessing==1.1.2
 kiwisolver==1.2.0
 Markdown==3.2.2
 matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
 mkl-fft==1.1.0
 mkl-random==1.1.1
 mkl-service==2.3.0
 numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
 nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
 oauthlib==3.1.0
 olefile==0.46
 opencv-python==4.4.0.40
 opt-einsum==3.3.0
 pathlib==1.0.1
 Pillow @ file:///C:/ci/pillow_1594298230227/work
 protobuf==3.13.0
 psutil==5.7.0
 pyasn1==0.4.8
 pyasn1-modules==0.2.8
 pyparsing==2.4.7
 python-dateutil==2.8.1
 pywin32==227
 requests==2.24.0
 requests-oauthlib==1.3.0
 rsa==4.6
 scikit-learn @ file:///C:/ci/scikit-learn_1592853510272/work
 scipy==1.4.1
 sip==4.19.13
 six==1.15.0
 tensorboard==2.2.2
 tensorboard-plugin-wit==1.7.0
 tensorflow-estimator==2.2.0
 tensorflow-gpu==2.2.0
 tensorflow-gpu-estimator==2.2.0
 termcolor==1.1.0
 threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
 tornado==6.0.4
 tqdm @ file:///tmp/build/80754af9/tqdm_1596810128862/work
 urllib3==1.25.10
 Werkzeug==1.0.1
 wincertstore==0.2
 wrapt==1.12.1
  ============== Conda Packages ==============
 # packages in environment at C:\Users\Matt\MiniConda3\envs\faceswap:
 #
 # Name                    Version                   Build  Channel
 absl-py                   0.9.0                    pypi_0    pypi
 astunparse                1.6.3                    pypi_0    pypi
 blas                      1.0                         mkl  
 ca-certificates           2020.6.24                     0  
 cachetools                4.1.1                    pypi_0    pypi
 certifi                   2020.6.20                py38_0  
 chardet                   3.0.4                    pypi_0    pypi
 cudatoolkit               10.1.243             h74a9793_0  
 cudnn                     7.6.5                cuda10.1_0  
 cycler                    0.10.0                   py38_0  
 fastcluster               1.1.26           py38hbe40bda_1    conda-forge
 ffmpeg                    4.3.1                ha925a31_0    conda-forge
 ffmpy                     0.2.3                    pypi_0    pypi
 freetype                  2.10.2               hd328e21_0  
 gast                      0.3.3                    pypi_0    pypi
 git                       2.23.0               h6bb4b03_0  
 google-auth               1.20.1                   pypi_0    pypi
 google-auth-oauthlib      0.4.1                    pypi_0    pypi
 google-pasta              0.2.0                    pypi_0    pypi
 grpcio                    1.31.0                   pypi_0    pypi
 h5py                      2.10.0                   pypi_0    pypi
 icc_rt                    2019.0.0             h0cc432a_1  
 icu                       58.2                 ha925a31_3  
 idna                      2.10                     pypi_0    pypi
 imageio                   2.9.0                      py_0  
 imageio-ffmpeg            0.4.2                      py_0    conda-forge
 intel-openmp              2020.1                      216  
 joblib                    0.16.0                     py_0  
 jpeg                      9b                   hb83a4c4_2  
 keras-preprocessing       1.1.2                    pypi_0    pypi
 kiwisolver                1.2.0            py38h74a9793_0  
 libpng                    1.6.37               h2a8f88b_0  
 libtiff                   4.1.0                h56a325e_1  
 lz4-c                     1.9.2                h62dcd97_1  
 markdown                  3.2.2                    pypi_0    pypi
 matplotlib                3.2.2                         0  
 matplotlib-base           3.2.2            py38h64f37c6_0  
 mkl                       2020.1                      216  
 mkl-service               2.3.0            py38hb782905_0  
 mkl_fft                   1.1.0            py38h45dec08_0  
 mkl_random                1.1.1            py38h47e9c7a_0  
 numpy                     1.19.1           py38h5510c5b_0  
 numpy-base                1.19.1           py38ha3acd2a_0  
 nvidia-ml-py3             7.352.1                  pypi_0    pypi
 oauthlib                  3.1.0                    pypi_0    pypi
 olefile                   0.46                       py_0  
 opencv-python             4.4.0.40                 pypi_0    pypi
 openssl                   1.1.1g               he774522_1  
 opt-einsum                3.3.0                    pypi_0    pypi
 pathlib                   1.0.1                      py_1  
 pillow                    7.2.0            py38hcc1f983_0  
 pip                       20.2.2                   py38_0  
 protobuf                  3.13.0                   pypi_0    pypi
 psutil                    5.7.0            py38he774522_0  
 pyasn1                    0.4.8                    pypi_0    pypi
 pyasn1-modules            0.2.8                    pypi_0    pypi
 pyparsing                 2.4.7                      py_0  
 pyqt                      5.9.2            py38ha925a31_4  
 python                    3.8.5                he1778fa_0  
 python-dateutil           2.8.1                      py_0  
 python_abi                3.8                      1_cp38    conda-forge
 pywin32                   227              py38he774522_1  
 qt                        5.9.7            vc14h73c81de_0  
 requests                  2.24.0                   pypi_0    pypi
 requests-oauthlib         1.3.0                    pypi_0    pypi
 rsa                       4.6                      pypi_0    pypi
 scikit-learn              0.23.1           py38h25d0782_0  
 scipy                     1.4.1                    pypi_0    pypi
 setuptools                49.6.0                   py38_0  
 sip                       4.19.13          py38ha925a31_0  
 six                       1.15.0                     py_0  
 sqlite                    3.32.3               h2a8f88b_0  
 tensorboard               2.2.2                    pypi_0    pypi
 tensorboard-plugin-wit    1.7.0                    pypi_0    pypi
 tensorflow-estimator      2.2.0                    pypi_0    pypi
 tensorflow-gpu            2.2.0                    pypi_0    pypi
 tensorflow-gpu-estimator  2.2.0                    pypi_0    pypi
 termcolor                 1.1.0                    pypi_0    pypi
 threadpoolctl             2.1.0              pyh5ca1d4c_0  
 tk                        8.6.10               he774522_0  
 tornado                   6.0.4            py38he774522_1  
 tqdm                      4.48.2                     py_0  
 urllib3                   1.25.10                  pypi_0    pypi
 vc                        14.1                 h0510ff6_4  
 vs2015_runtime            14.16.27012          hf0eaf9b_3  
 werkzeug                  1.0.1                    pypi_0    pypi
 wheel                     0.34.2                   py38_0  
 wincertstore              0.2                      py38_0  
 wrapt                     1.12.1                   pypi_0    pypi
 xz                        5.2.5                h62dcd97_0  
 zlib                      1.2.11               h62dcd97_4  
 zstd                      1.4.5                h04227a9_0  
  =============== State File =================
 {
   "name": "realface",
   "sessions": {
     "1": {
       "timestamp": 1599115396.6223,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 12,
       "iterations": 24,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "2": {
       "timestamp": 1599155754.10659,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 12,
       "iterations": 3143,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "3": {
       "timestamp": 1599160104.3641214,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 12,
       "iterations": 722,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "4": {
       "timestamp": 1599165347.106994,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 12,
       "iterations": 108393,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "5": {
       "timestamp": 1599241490.780011,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 12,
       "iterations": 190,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "6": {
       "timestamp": 1599241750.2682142,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 16,
       "iterations": 46134,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "7": {
       "timestamp": 1599259533.2947102,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 16,
       "iterations": 87784,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "8": {
       "timestamp": 1599324558.7970932,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 16,
       "iterations": 15801,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "9": {
       "timestamp": 1599330730.2586458,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 16,
       "iterations": 35648,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "10": {
       "timestamp": 1599526126.6474314,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 16,
       "iterations": 825,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     },
     "11": {
       "timestamp": 1599527226.4353244,
       "no_logs": false,
       "loss_names": [
         "total",
         "face_a",
         "face_b"
       ],
       "batchsize": 16,
       "iterations": 110,
       "config": {
         "learning_rate": 5e-05,
         "allow_growth": false,
         "convert_batchsize": 16
       }
     }
   },
   "lowest_avg_loss": {
     "a": 0.05607394129037857,
     "b": 0.07404948770999908
   },
   "iterations": 298774,
   "training_size": 256,
   "config": {
     "coverage": 86.49,
     "optimizer": "adam",
     "learning_rate": 5e-05,
     "allow_growth": false,
     "mixed_precision": true,
     "convert_batchsize": 16,
     "loss_function": "ssim",
     "mask_loss_function": "mse",
     "l2_reg_term": 100,
     "eye_multiplier": 12,
     "mouth_multiplier": 8,
     "penalized_mask_loss": false,
     "mask_type": "extended",
     "mask_blur_kernel": 3,
     "mask_threshold": 4,
     "learn_mask": false,
     "input_size": 64,
     "output_size": 128,
     "dense_nodes": 1536,
     "complexity_encoder": 128,
     "complexity_decoder": 512
   }
 }
  ================= Configs ==================
 --------- .faceswap ---------
 backend:                  nvidia
  --------- convert.ini ---------
  [color.color_transfer]
 clip:                     True
 preserve_paper:           True
  [color.manual_balance]
 colorspace:               HSV
 balance_1:                0.0
 balance_2:                0.0
 balance_3:                0.0
 contrast:                 0.0
 brightness:               0.0
  [color.match_hist]
 threshold:                99.0
  [mask.box_blend]
 type:                     gaussian
 distance:                 11.0
 radius:                   5.0
 passes:                   1
  [mask.mask_blend]
 type:                     gaussian
 kernel_size:              3
 passes:                   4
 threshold:                4
 erosion:                  0.0
  [scaling.sharpen]
 method:                   gaussian
 amount:                   150
 radius:                   0.3
 threshold:                5.0
  [writer.ffmpeg]
 container:                mp4
 codec:                    libx264
 crf:                      23
 preset:                   medium
 tune:                     none
 profile:                  auto
 level:                    auto
 skip_mux:                 False
  [writer.gif]
 fps:                      25
 loop:                     0
 palettesize:              256
 subrectangles:            False
  [writer.opencv]
 format:                   png
 draw_transparent:         False
 jpg_quality:              75
 png_compress_level:       3
  [writer.pillow]
 format:                   png
 draw_transparent:         False
 optimize:                 False
 gif_interlace:            True
 jpg_quality:              75
 png_compress_level:       3
 tif_compression:          tiff_deflate
  --------- extract.ini ---------
  [global]
 allow_growth:             False
  [align.fan]
 batch-size:               12
  [detect.cv2_dnn]
 confidence:               50
  [detect.mtcnn]
 minsize:                  20
 threshold_1:              0.6
 threshold_2:              0.7
 threshold_3:              0.7
 scalefactor:              0.709
 batch-size:               8
  [detect.s3fd]
 confidence:               70
 batch-size:               4
  [mask.unet_dfl]
 batch-size:               8
  [mask.vgg_clear]
 batch-size:               6
  [mask.vgg_obstructed]
 batch-size:               4
  --------- gui.ini ---------
  [global]
 fullscreen:               True
 tab:                      extract
 options_panel_width:      30
 console_panel_height:     20
 icon_size:                14
 font:                     default
 font_size:                9
 autosave_last_session:    prompt
 timeout:                  120
 auto_load_model_stats:    True
  --------- train.ini ---------
  [global]
 coverage:                 81.85
 icnr_init:                False
 conv_aware_init:          False
 optimizer:                adam
 learning_rate:            5e-05
 reflect_padding:          False
 allow_growth:             False
 mixed_precision:          False
 convert_batchsize:        16
  [global.loss]
 loss_function:            ssim
 mask_loss_function:       mse
 l2_reg_term:              100
 eye_multiplier:           6
 mouth_multiplier:         4
 penalized_mask_loss:      True
 mask_type:                extended
 mask_blur_kernel:         3
 mask_threshold:           4
 learn_mask:               False
  [model.dfl_h128]
 lowmem:                   False
  [model.dfl_sae]
 input_size:               128
 clipnorm:                 True
 architecture:             df
 autoencoder_dims:         0
 encoder_dims:             42
 decoder_dims:             21
 multiscale_decoder:       False
  [model.dlight]
 features:                 best
 details:                  good
 output_size:              256
  [model.original]
 lowmem:                   False
  [model.realface]
 input_size:               64
 output_size:              128
 dense_nodes:              1536
 complexity_encoder:       128
 complexity_decoder:       512
  [model.unbalanced]
 input_size:               128
 lowmem:                   False
 clipnorm:                 True
 nodes:                    1024
 complexity_encoder:       128
 complexity_decoder_a:     384
 complexity_decoder_b:     512
  [model.villain]
 lowmem:                   False
  [trainer.original]
 preview_images:           14
 zoom_amount:              5
 rotation_range:           10
 shift_range:              5
 flip_chance:              50
 disable_warp:             False
 color_lightness:          30
 color_ab:                 8
 color_clahe_chance:       50
 color_clahe_max_size:     4
  
The beginning of the end.

User avatar
Antipope1
Posts: 33
Joined: Thu Jul 02, 2020 10:51 am
Has thanked: 6 times
Been thanked: 2 times

Re: Could not train after updating

Post by Antipope1 »

I managed to get this training again by restoring the model and using the model generated from that, however when starting that it has no logs and looks like I haven't done any training at all.
Hane I lost the 300k iterations I done one this model??
The beginning of the end.

User avatar
bryanlyon
Site Admin
Posts: 468
Joined: Fri Jul 12, 2019 12:49 am
Answers: 39
Location: San Francisco
Has thanked: 3 times
Been thanked: 114 times
Contact:

Re: Could not train after updating

Post by bryanlyon »

No,. your training isn't lost, just the logs. The format has changed due to the move to Tensorflow 2.0 but the training should be intact.

User avatar
Antipope1
Posts: 33
Joined: Thu Jul 02, 2020 10:51 am
Has thanked: 6 times
Been thanked: 2 times

Re: Could not train after updating

Post by Antipope1 »

Thanks, I was starting to worry!
So I'm guessing I can't use my original model (now archived), only my restored model?
When I now start the training the preview faces are completely blurry as if I have only just started training from scratch, which is what gave me the impression I've lost all my training. will this eventually show more detailed images as it builds a log?

Edit: I don't think the update I applied when the problem occurred was the big switch to 2.0 update, I think it was only a small update released in the last day or two.
The beginning of the end.

User avatar
pimpim
Posts: 2
Joined: Sun Sep 13, 2020 5:13 pm

Re: Could not train after updating

Post by pimpim »

I had this issue after update, and I was unable to train my previous model. Still I was able to use it to convertand automatically the model was converted to the new format (this model was using Dlight).

Here is the error from the log file:

Code: Select all

  File "Y:\faceswap2\lib\training_data.py", line 605, in _separate_target_mask
    masks=target_batch[-1][..., 3][..., None])
IndexError: index 3 is out of bounds for axis 3 with size 3
I found a temporary solution (a bad one I suppose) without searching to understand more the program, I just modified this line and everything is working nicely now, so line 605 in training_data.py I've modified:

Code: Select all

        retval = dict(targets=[batch[..., :3] for batch in target_batch],
                      masks=target_batch[-1][..., 2][..., None])
#replace this line                      masks=target_batch[-1][..., 3][..., None])
3 out of bounds? let's put 2 :D
but if anyone has a clean solution it may be better...

And, contrary to the previous answer, there was no backup file in my model folder, even in the snapshots (the restore function couldn't work...)

User avatar
bryanlyon
Site Admin
Posts: 468
Joined: Fri Jul 12, 2019 12:49 am
Answers: 39
Location: San Francisco
Has thanked: 3 times
Been thanked: 114 times
Contact:

Re: Could not train after updating

Post by bryanlyon »

pimpim wrote: Sun Sep 13, 2020 5:25 pm I had this issue after update, and I was unable to train my previous model. Still I was able to use it to convertand automatically the model was converted to the new format (this model was using Dlight).

Here is the error from the log file:

Code: Select all

  File "Y:\faceswap2\lib\training_data.py", line 605, in _separate_target_mask
    masks=target_batch[-1][..., 3][..., None])
IndexError: index 3 is out of bounds for axis 3 with size 3
I found a temporary solution (a bad one I suppose) without searching to understand more the program, I just modified this line and everything is working nicely now, so line 605 in training_data.py I've modified:

Code: Select all

        retval = dict(targets=[batch[..., :3] for batch in target_batch],
                      masks=target_batch[-1][..., 2][..., None])
#replace this line                      masks=target_batch[-1][..., 3][..., None])
3 out of bounds? let's put 2 :D
but if anyone has a clean solution it may be better...

And, contrary to the previous answer, there was no backup file in my model folder, even in the snapshots (the restore function couldn't work...)
Yeah, don't do that. You're just breaking the model by giving it incorrect data.

User avatar
pimpim
Posts: 2
Joined: Sun Sep 13, 2020 5:13 pm

Re: Could not train after updating

Post by pimpim »

bryanlyon wrote: Mon Sep 14, 2020 5:12 pm
pimpim wrote: Sun Sep 13, 2020 5:25 pm ...
Yeah, don't do that. You're just breaking the model by giving it incorrect data.
it was either that or no more use of the model... this can be useful for someone in my case before any better solution
so now everything from training to conversion worked fine with this mod
+ i can imagine that's only incorrect model data for the masks (see the code masks=...)
+ I did not use masks in my model(so storing an incorrect mask is not a problem, and as i understood if you started a model without mask you couldn't continue training with a mask, but i still use mask for convert without pb)

so at the end i have a working script for this model and if i start a new one (with a mask the next one) i will just switch back the modified line...

at the end that would be better to have this kind of test within the code:
if no model_mask_option =none then masks = none or sthing like that...

User avatar
torzdf
Posts: 948
Joined: Fri Jul 12, 2019 12:53 am
Answers: 126
Has thanked: 26 times
Been thanked: 184 times

Re: Could not train after updating

Post by torzdf »

These look there might be a genuine issue.....

Your old model is not lost. There is an archived version which still works in Faceswap 1.0, It's a bit of an inconvenience, but in the meantime you can download and install that version to continue training/convert.

To help me find the issue(s), and implement a fix, it would be really useful if anyone who hits one of these (either the mask issue, or the faces going back to blurry issue) provide me with the <model>_state.json files from both their archived folder and their newly created folder.

My word is final


Post Reply