Could not train after updating

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Antipope1
Posts: 35
Joined: Thu Jul 02, 2020 10:51 am
Has thanked: 6 times
Been thanked: 3 times

Could not train after updating

Post by Antipope1 »

I ran into a problem this evening!
I have been fit training a model to nearly 300k iterations on realface.
This evening I manually aligned the faces from the video I actually wish to convert, loaded them onto the trainer and trained over 1000 iterations.
I saved the project and noticed faceswap needed an update, so I updated and restarted faceswap.
After restarting, I tried to continue training but kept getting this error.
Any idea what happened??
Tried completely deleting and reinstalling faceswap and still the same.

Code: Select all

 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_007065_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_003471_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002746_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003409_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003673_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_006586_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_006586_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_007065_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003564_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010720_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010699_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009550_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_007065_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010441_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_003471_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002746_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_006586_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000201_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000201_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_009766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000201_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003392_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003564_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_009766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003564_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_001772_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010441_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_009766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002997_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_010441_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009739_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003392_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003392_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_001186_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004766_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_000947_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_001772_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004896_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001956_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001846_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_001772_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_001186_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004896_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001956_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001846_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002997_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002971_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_001186_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002997_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_007801_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004896_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Mask already generated. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001956_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_001846_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009739_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_009739_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_007801_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_008748_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_007801_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_000947_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_006922_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_000947_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003828_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002461_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000261_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_002971_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_000261_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004596_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002431_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_004596_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_008748_0.png'
 09/08/2020 02:10:50 MainProcess     _run_0          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_005456_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\VID_20200901_115123_002431_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'b', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputB\Extracted\clall_005456_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_006922_0.png'
 09/08/2020 02:10:50 MainProcess     _run_1          training_data   _expand_partials          DEBUG    Generating mask. side: 'a', filename: 'I:\Faceswap Resources\cl1\FaceSwap\InputA\Extracted\finnn\Untitled Projectccc_003828_0.png'
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A6CF9AEE0>, weight: 1.0, mask_channel: -1)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    No mask to apply
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A5BC31C70>, weight: 1.0, mask_channel: -1)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    No mask to apply
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A6C7D3430>, weight: 12.0, mask_channel: 3)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    Applying mask from channel 3
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A6C723D60>, weight: 1.0, mask_channel: 1)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    Applying mask from channel 1
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       call                      DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x0000016A5BA4CCA0>, weight: 8.0, mask_channel: 4)
 09/08/2020 02:10:51 MainProcess     _training_0     losses_tf       _apply_mask               DEBUG    Applying mask from channel 4
 09/08/2020 02:10:51 MainProcess     _training_0     multithreading  run                       DEBUG    Error in thread (_training_0): in user code:\n\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:571 train_function  *\n        outputs = self.distribute_strategy.run(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:951 run  **\n        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2290 call_for_each_replica\n        return self._call_for_each_replica(fn, args, kwargs)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2649 _call_for_each_replica\n        return fn(*args, **kwargs)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:532 train_step  **\n        loss = self.compiled_loss(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\compile_utils.py:205 __call__\n        loss_value = loss_obj(y_t, y_p, sample_weight=sw)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\losses.py:143 __call__\n        losses = self.call(y_true, y_pred)\n    I:\FS\faceswap\lib\model\losses_tf.py:559 call\n        n_true, n_pred = self._apply_mask(y_true, y_pred, mask_channel)\n    I:\FS\faceswap\lib\model\losses_tf.py:589 _apply_mask\n        mask = K.expand_dims(y_true[..., mask_channel], axis=-1)\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:973 _slice_helper\n        return strided_slice(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:1140 strided_slice\n        op = gen_array_ops.strided_slice(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_array_ops.py:10174 strided_slice\n        _, _, _op, _outputs = _op_def_library._apply_op_helper(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\op_def_library.py:742 _apply_op_helper\n        op = g._create_op_internal(op_type_name, inputs, dtypes=None,\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py:593 _create_op_internal\n        return super(FuncGraph, self)._create_op_internal(  # pylint: disable=protected-access\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:3319 _create_op_internal\n        ret = Operation(\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1816 __init__\n        self._c_op = _create_c_op(self._graph, node_def, inputs,\n    C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1657 _create_c_op\n        raise ValueError(str(e))\n\n    ValueError: slice index 4 of dimension 3 out of bounds. for '{{node LossWrapper/strided_slice_22}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=1, end_mask=0, new_axis_mask=0, shrink_axis_mask=2](IteratorGetNext:2, LossWrapper/strided_slice_22/stack, LossWrapper/strided_slice_22/stack_1, LossWrapper/strided_slice_22/stack_2)' with input shapes: [16,128,128,4], [2], [2], [2] and with computed input tensors: input[1] = <0 4>, input[2] = <0 5>, input[3] = <1 1>.\n
 09/08/2020 02:10:51 MainProcess     MainThread      train           _monitor                  DEBUG    Thread error detected
 09/08/2020 02:10:51 MainProcess     MainThread      train           _monitor                  DEBUG    Closed Monitor
 09/08/2020 02:10:51 MainProcess     MainThread      train           _end_thread               DEBUG    Ending Training thread
 09/08/2020 02:10:51 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
 09/08/2020 02:10:51 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Threads: '_training'
 09/08/2020 02:10:51 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Thread: '_training_0'
 09/08/2020 02:10:51 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
 Traceback (most recent call last):
   File "I:\FS\faceswap\lib\cli\launcher.py", line 156, in execute_script
     process.process()
   File "I:\FS\faceswap\scripts\train.py", line 165, in process
     self._end_thread(thread, err)
   File "I:\FS\faceswap\scripts\train.py", line 205, in _end_thread
     thread.join()
   File "I:\FS\faceswap\lib\multithreading.py", line 121, in join
     raise thread.err[1].with_traceback(thread.err[2])
   File "I:\FS\faceswap\lib\multithreading.py", line 37, in run
     self._target(*self._args, **self._kwargs)
   File "I:\FS\faceswap\scripts\train.py", line 227, in _training
     raise err
   File "I:\FS\faceswap\scripts\train.py", line 217, in _training
     self._run_training_cycle(model, trainer)
   File "I:\FS\faceswap\scripts\train.py", line 298, in _run_training_cycle
     trainer.train_one_step(viewer, timelapse)
   File "I:\FS\faceswap\plugins\train\trainer\_base.py", line 211, in train_one_step
     loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1348, in train_on_batch
     logs = train_function(iterator)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__
     result = self._call(*args, **kwds)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 627, in _call
     self._initialize(args, kwds, add_initializers_to=initializers)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 505, in _initialize
     self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2446, in _get_concrete_function_internal_garbage_collected
     graph_function, _, _ = self._maybe_define_function(args, kwargs)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2777, in _maybe_define_function
     graph_function = self._create_graph_function(args, kwargs)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2657, in _create_graph_function
     func_graph_module.func_graph_from_py_func(
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py", line 981, in func_graph_from_py_func
     func_outputs = python_func(*func_args, **func_kwargs)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 441, in wrapped_fn
     return weak_wrapped_fn().__wrapped__(*args, **kwds)
   File "C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py", line 968, in wrapper
     raise e.ag_error_metadata.to_exception(e)
 ValueError: in user code:
      C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:571 train_function  *
         outputs = self.distribute_strategy.run(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:951 run  **
         return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2290 call_for_each_replica
         return self._call_for_each_replica(fn, args, kwargs)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2649 _call_for_each_replica
         return fn(*args, **kwargs)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:532 train_step  **
         loss = self.compiled_loss(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\compile_utils.py:205 __call__
         loss_value = loss_obj(y_t, y_p, sample_weight=sw)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\losses.py:143 __call__
         losses = self.call(y_true, y_pred)
     I:\FS\faceswap\lib\model\losses_tf.py:559 call
         n_true, n_pred = self._apply_mask(y_true, y_pred, mask_channel)
     I:\FS\faceswap\lib\model\losses_tf.py:589 _apply_mask
         mask = K.expand_dims(y_true[..., mask_channel], axis=-1)
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:973 _slice_helper
         return strided_slice(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\array_ops.py:1140 strided_slice
         op = gen_array_ops.strided_slice(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_array_ops.py:10174 strided_slice
         _, _, _op, _outputs = _op_def_library._apply_op_helper(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\op_def_library.py:742 _apply_op_helper
         op = g._create_op_internal(op_type_name, inputs, dtypes=None,
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py:593 _create_op_internal
         return super(FuncGraph, self)._create_op_internal(  # pylint: disable=protected-access
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:3319 _create_op_internal
         ret = Operation(
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1816 __init__
         self._c_op = _create_c_op(self._graph, node_def, inputs,
     C:\Users\Matt\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:1657 _create_c_op
         raise ValueError(str(e))
      ValueError: slice index 4 of dimension 3 out of bounds. for '{{node LossWrapper/strided_slice_22}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=1, end_mask=0, new_axis_mask=0, shrink_axis_mask=2](IteratorGetNext:2, LossWrapper/strided_slice_22/stack, LossWrapper/strided_slice_22/stack_1, LossWrapper/strided_slice_22/stack_2)' with input shapes: [16,128,128,4], [2], [2], [2] and with computed input tensors: input[1] = <0 4>, input[2] = <0 5>, input[3] = <1 1>.
   ============ System Information ============
 encoding:            cp1252
 git_branch:          master
 git_commits:         5530346 Bugfix - Pre-caching masks when warp-to-landmarks is disabled. 49f86e1 training - Add config item to disable warp. cd3fc81 Training - Cache eye and mouth masks during first epoch rather than at startup. a2f1cb4 Bugfix: Preview Tool. Fix blank swapped faces issue. 10da0c4 Loss multipliers:   - Lower defaults.   - Set to updateable
 gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
 gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
 gpu_devices:         GPU_0: GeForce RTX 2060 SUPER
 gpu_devices_active:  GPU_0
 gpu_driver:          452.06
 gpu_vram:            GPU_0: 8192MB
 os_machine:          AMD64
 os_platform:         Windows-10-10.0.18362-SP0
 os_release:          10
 py_command:          I:\FS\faceswap\faceswap.py train -A I:/Faceswap Resources/cl1/FaceSwap/InputA/Extracted/finnn -ala I:/Faceswap Resources/cl1/FaceSwap/InputA/Video/Untitled Projectccc_alignments.fsa -B I:/Faceswap Resources/cl1/FaceSwap/InputB/Extracted -alb I:/Faceswap Resources/cl1/FaceSwap/InputB/Video/alignments_merged_20200902_215938.fsa -m I:/Faceswap Resources/cl1/FaceSwap/Models -t realface -bs 16 -it 20000 -s 200 -ss 40000 -ps 50 -L INFO -gui
 py_conda_version:    conda 4.8.4
 py_implementation:   CPython
 py_version:          3.8.5
 py_virtual_env:      True
 sys_cores:           4
 sys_processor:       AMD64 Family 23 Model 17 Stepping 0, AuthenticAMD
 sys_ram:             Total: 32715MB, Available: 26808MB, Used: 5906MB, Free: 26808MB
  =============== Pip Packages ===============
 absl-py==0.9.0
 astunparse==1.6.3
 cachetools==4.1.1
 certifi==2020.6.20
 chardet==3.0.4
 cycler==0.10.0
 fastcluster==1.1.26
 ffmpy==0.2.3
 gast==0.3.3
 google-auth==1.20.1
 google-auth-oauthlib==0.4.1
 google-pasta==0.2.0
 grpcio==1.31.0
 h5py==2.10.0
 idna==2.10
 imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
 imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1589202782679/work
 joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
 Keras-Preprocessing==1.1.2
 kiwisolver==1.2.0
 Markdown==3.2.2
 matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
 mkl-fft==1.1.0
 mkl-random==1.1.1
 mkl-service==2.3.0
 numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
 nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
 oauthlib==3.1.0
 olefile==0.46
 opencv-python==4.4.0.40
 opt-einsum==3.3.0
 pathlib==1.0.1
 Pillow @ file:///C:/ci/pillow_1594298230227/work
 protobuf==3.13.0
 psutil==5.7.0
 pyasn1==0.4.8
 pyasn1-modules==0.2.8
 pyparsing==2.4.7
 python-dateutil==2.8.1
 pywin32==227
 requests==2.24.0
 requests-oauthlib==1.3.0
 rsa==4.6
 scikit-learn @ file:///C:/ci/scikit-learn_1592853510272/work
 scipy==1.4.1
 sip==4.19.13
 six==1.15.0
 tensorboard==2.2.2
 tensorboard-plugin-wit==1.7.0
 tensorflow-estimator==2.2.0
 tensorflow-gpu==2.2.0
 tensorflow-gpu-estimator==2.2.0
 termcolor==1.1.0
 threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
 tornado==6.0.4
 tqdm @ file:///tmp/build/80754af9/tqdm_1596810128862/work
 urllib3==1.25.10
 Werkzeug==1.0.1
 wincertstore==0.2
 wrapt==1.12.1
  ============== Conda Packages ==============
 # packages in environment at C:\Users\Matt\MiniConda3\envs\faceswap:
 #
 # Name                    Version                   Build  Channel
 absl-py                   0.9.0                    pypi_0    pypi
 astunparse                1.6.3                    pypi_0    pypi
 blas                      1.0                         mkl  
ca-certificates 2020.6.24 0
cachetools 4.1.1 pypi_0 pypi certifi 2020.6.20 py38_0
chardet 3.0.4 pypi_0 pypi cudatoolkit 10.1.243 h74a9793_0
cudnn 7.6.5 cuda10.1_0
cycler 0.10.0 py38_0
fastcluster 1.1.26 py38hbe40bda_1 conda-forge ffmpeg 4.3.1 ha925a31_0 conda-forge ffmpy 0.2.3 pypi_0 pypi freetype 2.10.2 hd328e21_0
gast 0.3.3 pypi_0 pypi git 2.23.0 h6bb4b03_0
google-auth 1.20.1 pypi_0 pypi google-auth-oauthlib 0.4.1 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.31.0 pypi_0 pypi h5py 2.10.0 pypi_0 pypi icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 pypi_0 pypi imageio 2.9.0 py_0
imageio-ffmpeg 0.4.2 py_0 conda-forge intel-openmp 2020.1 216
joblib 0.16.0 py_0
jpeg 9b hb83a4c4_2
keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.2.0 py38h74a9793_0
libpng 1.6.37 h2a8f88b_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 h62dcd97_1
markdown 3.2.2 pypi_0 pypi matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38h64f37c6_0
mkl 2020.1 216
mkl-service 2.3.0 py38hb782905_0
mkl_fft 1.1.0 py38h45dec08_0
mkl_random 1.1.1 py38h47e9c7a_0
numpy 1.19.1 py38h5510c5b_0
numpy-base 1.19.1 py38ha3acd2a_0
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 pypi_0 pypi olefile 0.46 py_0
opencv-python 4.4.0.40 pypi_0 pypi openssl 1.1.1g he774522_1
opt-einsum 3.3.0 pypi_0 pypi pathlib 1.0.1 py_1
pillow 7.2.0 py38hcc1f983_0
pip 20.2.2 py38_0
protobuf 3.13.0 pypi_0 pypi psutil 5.7.0 py38he774522_0
pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pyparsing 2.4.7 py_0
pyqt 5.9.2 py38ha925a31_4
python 3.8.5 he1778fa_0
python-dateutil 2.8.1 py_0
python_abi 3.8 1_cp38 conda-forge pywin32 227 py38he774522_1
qt 5.9.7 vc14h73c81de_0
requests 2.24.0 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rsa 4.6 pypi_0 pypi scikit-learn 0.23.1 py38h25d0782_0
scipy 1.4.1 pypi_0 pypi setuptools 49.6.0 py38_0
sip 4.19.13 py38ha925a31_0
six 1.15.0 py_0
sqlite 3.32.3 h2a8f88b_0
tensorboard 2.2.2 pypi_0 pypi tensorboard-plugin-wit 1.7.0 pypi_0 pypi tensorflow-estimator 2.2.0 pypi_0 pypi tensorflow-gpu 2.2.0 pypi_0 pypi tensorflow-gpu-estimator 2.2.0 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
tornado 6.0.4 py38he774522_1
tqdm 4.48.2 py_0
urllib3 1.25.10 pypi_0 pypi vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_3
werkzeug 1.0.1 pypi_0 pypi wheel 0.34.2 py38_0
wincertstore 0.2 py38_0
wrapt 1.12.1 pypi_0 pypi xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.5 h04227a9_0
=============== State File ================= { "name": "realface", "sessions": { "1": { "timestamp": 1599115396.6223, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 12, "iterations": 24, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "2": { "timestamp": 1599155754.10659, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 12, "iterations": 3143, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "3": { "timestamp": 1599160104.3641214, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 12, "iterations": 722, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "4": { "timestamp": 1599165347.106994, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 12, "iterations": 108393, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "5": { "timestamp": 1599241490.780011, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 12, "iterations": 190, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "6": { "timestamp": 1599241750.2682142, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 16, "iterations": 46134, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "7": { "timestamp": 1599259533.2947102, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 16, "iterations": 87784, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "8": { "timestamp": 1599324558.7970932, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 16, "iterations": 15801, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "9": { "timestamp": 1599330730.2586458, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 16, "iterations": 35648, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "10": { "timestamp": 1599526126.6474314, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 16, "iterations": 825, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } }, "11": { "timestamp": 1599527226.4353244, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 16, "iterations": 110, "config": { "learning_rate": 5e-05, "allow_growth": false, "convert_batchsize": 16 } } }, "lowest_avg_loss": { "a": 0.05607394129037857, "b": 0.07404948770999908 }, "iterations": 298774, "training_size": 256, "config": { "coverage": 86.49, "optimizer": "adam", "learning_rate": 5e-05, "allow_growth": false, "mixed_precision": true, "convert_batchsize": 16, "loss_function": "ssim", "mask_loss_function": "mse", "l2_reg_term": 100, "eye_multiplier": 12, "mouth_multiplier": 8, "penalized_mask_loss": false, "mask_type": "extended", "mask_blur_kernel": 3, "mask_threshold": 4, "learn_mask": false, "input_size": 64, "output_size": 128, "dense_nodes": 1536, "complexity_encoder": 128, "complexity_decoder": 512 } } ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: gaussian kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: gaussian amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 70 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 4 --------- gui.ini --------- [global] fullscreen: True tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] coverage: 81.85 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 reflect_padding: False allow_growth: False mixed_precision: False convert_batchsize: 16 [global.loss] loss_function: ssim mask_loss_function: mse l2_reg_term: 100 eye_multiplier: 6 mouth_multiplier: 4 penalized_mask_loss: True mask_type: extended mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 disable_warp: False color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
The beginning of the end.
User avatar
Antipope1
Posts: 35
Joined: Thu Jul 02, 2020 10:51 am
Has thanked: 6 times
Been thanked: 3 times

Re: Could not train after updating

Post by Antipope1 »

I managed to get this training again by restoring the model and using the model generated from that, however when starting that it has no logs and looks like I haven't done any training at all.
Hane I lost the 300k iterations I done one this model??

The beginning of the end.
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Could not train after updating

Post by bryanlyon »

No,. your training isn't lost, just the logs. The format has changed due to the move to Tensorflow 2.0 but the training should be intact.

User avatar
Antipope1
Posts: 35
Joined: Thu Jul 02, 2020 10:51 am
Has thanked: 6 times
Been thanked: 3 times

Re: Could not train after updating

Post by Antipope1 »

Thanks, I was starting to worry!
So I'm guessing I can't use my original model (now archived), only my restored model?
When I now start the training the preview faces are completely blurry as if I have only just started training from scratch, which is what gave me the impression I've lost all my training. will this eventually show more detailed images as it builds a log?

Edit: I don't think the update I applied when the problem occurred was the big switch to 2.0 update, I think it was only a small update released in the last day or two.

The beginning of the end.
User avatar
pimpim
Posts: 2
Joined: Sun Sep 13, 2020 5:13 pm

Re: Could not train after updating

Post by pimpim »

I had this issue after update, and I was unable to train my previous model. Still I was able to use it to convertand automatically the model was converted to the new format (this model was using Dlight).

Here is the error from the log file:

Code: Select all

  File "Y:\faceswap2\lib\training_data.py", line 605, in _separate_target_mask
    masks=target_batch[-1][..., 3][..., None])
IndexError: index 3 is out of bounds for axis 3 with size 3

I found a temporary solution (a bad one I suppose) without searching to understand more the program, I just modified this line and everything is working nicely now, so line 605 in training_data.py I've modified:

Code: Select all

        retval = dict(targets=[batch[..., :3] for batch in target_batch],
                      masks=target_batch[-1][..., 2][..., None])
#replace this line                      masks=target_batch[-1][..., 3][..., None])

3 out of bounds? let's put 2 :D
but if anyone has a clean solution it may be better...

And, contrary to the previous answer, there was no backup file in my model folder, even in the snapshots (the restore function couldn't work...)

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Could not train after updating

Post by bryanlyon »

pimpim wrote: Sun Sep 13, 2020 5:25 pm

I had this issue after update, and I was unable to train my previous model. Still I was able to use it to convertand automatically the model was converted to the new format (this model was using Dlight).

Here is the error from the log file:

Code: Select all

  File "Y:\faceswap2\lib\training_data.py", line 605, in _separate_target_mask
    masks=target_batch[-1][..., 3][..., None])
IndexError: index 3 is out of bounds for axis 3 with size 3

I found a temporary solution (a bad one I suppose) without searching to understand more the program, I just modified this line and everything is working nicely now, so line 605 in training_data.py I've modified:

Code: Select all

        retval = dict(targets=[batch[..., :3] for batch in target_batch],
                      masks=target_batch[-1][..., 2][..., None])
#replace this line                      masks=target_batch[-1][..., 3][..., None])

3 out of bounds? let's put 2 :D
but if anyone has a clean solution it may be better...

And, contrary to the previous answer, there was no backup file in my model folder, even in the snapshots (the restore function couldn't work...)

Yeah, don't do that. You're just breaking the model by giving it incorrect data.

User avatar
pimpim
Posts: 2
Joined: Sun Sep 13, 2020 5:13 pm

Re: Could not train after updating

Post by pimpim »

bryanlyon wrote: Mon Sep 14, 2020 5:12 pm
pimpim wrote: Sun Sep 13, 2020 5:25 pm

...

Yeah, don't do that. You're just breaking the model by giving it incorrect data.

it was either that or no more use of the model... this can be useful for someone in my case before any better solution
so now everything from training to conversion worked fine with this mod

  • i can imagine that's only incorrect model data for the masks (see the code masks=...)
  • I did not use masks in my model(so storing an incorrect mask is not a problem, and as i understood if you started a model without mask you couldn't continue training with a mask, but i still use mask for convert without pb)

so at the end i have a working script for this model and if i start a new one (with a mask the next one) i will just switch back the modified line...

at the end that would be better to have this kind of test within the code:
if no model_mask_option =none then masks = none or sthing like that...

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Could not train after updating

Post by torzdf »

These look there might be a genuine issue.....

Your old model is not lost. There is an archived version which still works in Faceswap 1.0, It's a bit of an inconvenience, but in the meantime you can download and install that version to continue training/convert.

To help me find the issue(s), and implement a fix, it would be really useful if anyone who hits one of these (either the mask issue, or the faces going back to blurry issue) provide me with the <model>_state.json files from both their archived folder and their newly created folder.

My word is final

Locked