This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.
Please mark any answers that fixed your problems so others can find the solutions.
I have selected Bisenet-Fp for the extraction and when I start training, the training settings report an error whether I select Bisenet-Fp Face or Bisenet-Fp Head. I don't understand which step is wrong. But when I turn off "Penalized Mask Loss" I can start training again. But there is a warning message.
"01/04/2022 18:28:44 WARNING You have selected eye/mouth loss multipliers greater than 1x, but Penalized Mask Loss is disabled. Disabling all multipliers."
Loading...
Setting Faceswap backend to NVIDIA
01/04/2022 18:19:35 INFO Log level set to: INFO
01/04/2022 18:19:38 INFO Model A Directory: 'C:\Users\robinchen\Documents\face\FC2\1\1' (650 images)
01/04/2022 18:19:38 INFO Model B Directory: 'C:\Users\robinchen\Documents\face\WBB\B\B' (2436 images)
01/04/2022 18:19:38 INFO Training data directory: C:\Users\robinchen\Documents\face\FC2\3
01/04/2022 18:19:38 INFO ===================================================
01/04/2022 18:19:38 INFO Starting
01/04/2022 18:19:38 INFO Press 'Stop' to save and quit
01/04/2022 18:19:38 INFO ===================================================
01/04/2022 18:19:39 INFO Loading data, this may take a while...
01/04/2022 18:19:39 INFO Loading Model from Original plugin...
01/04/2022 18:19:39 INFO No existing state file found. Generating.
01/04/2022 18:19:39 INFO Enabling Mixed Precision Training.
01/04/2022 18:19:39 INFO Mixed precision compatibility check (mixed_float16): OK\nYour GPU will likely run quickly with dtype policy mixed_float16 as it has compute capability of at least 7.0. Your GPU: NVIDIA GeForce RTX 3080 Ti, compute capability 8.6
01/04/2022 18:19:40 INFO Loading Trainer from Original plugin...
01/04/2022 18:19:42 CRITICAL Error caught! Exiting...
01/04/2022 18:19:42 ERROR Caught exception in thread: '_training_0'
01/04/2022 18:19:42 ERROR You have selected the mask type 'bisenet-fp_face' but at least one face does not contain the selected mask.
01/04/2022 18:19:42 ERROR The face that failed was: 'C:\Users\robinchen\Documents\face\WBB\B\B\B_005052_0.png'
01/04/2022 18:19:42 ERROR The masks that exist for this face are: ['components', 'extended']
Process exited.
That error may happen, if you have added a new alignment / face using the manual tool and then done extraction from alignment file. And that face does not have a mask in bisenet-fp_face type, it just has the two default ones... while rest of the files have that mask.
You have two options, if above is the case:
run tools -> masks -> add mask bisenet-fp_face for that single file
or you can delete that single face from the training set. I usually do the latter, the ones that are left undetected by original extraction are usually low quality or "difficult" faces.
I search these errors with following command in the face directory: grep -L "bisenet" .
That will return a list of all files that do not have the bisenet mask, and you can then fix them or remove them from the training set instead of trial and error.
That error may happen, if you have added a new alignment / face using the manual tool and then done extraction from alignment file. And that face does not have a mask in bisenet-fp_face type, it just has the two default ones... while rest of the files have that mask.
You have two options, if above is the case:
run tools -> masks -> add mask bisenet-fp_face for that single file
or you can delete that single face from the training set. I usually do the latter, the ones that are left undetected by original extraction are usually low quality or "difficult" faces.
I search these errors with following command in the face directory: grep -L "bisenet" .
That will return a list of all files that do not have the bisenet mask, and you can then fix them or remove them from the training set instead of trial and error.
I am having the same issue but I'm not using any other masks other than the default ones. It doesn't give me an error but it says its disabling the penalized mask loss. So I'm still able to train but says it's disabled? I checked the Loss area of the training options and it is checked so I'm not sure why its not working.
Loading...
Setting Faceswap backend to NVIDIA
02/08/2022 07:34:05 INFO Log level set to: INFO
02/08/2022 07:34:06 INFO Model A Directory: 'E:\newfiles\PS\Rainy\working\Sort_A' (5592 images)
02/08/2022 07:34:06 INFO Model B Directory: 'E:\newfiles\PS\Rainy\working\Align_A' (1543 images)
02/08/2022 07:34:06 INFO Training data directory: E:\newfiles\PS\Rainy\working\Sfx
02/08/2022 07:34:07 INFO ===================================================
02/08/2022 07:34:07 INFO Starting
02/08/2022 07:34:07 INFO Press 'Stop' to save and quit
02/08/2022 07:34:07 INFO ===================================================
02/08/2022 07:34:08 INFO Loading data, this may take a while...
02/08/2022 07:34:08 INFO Loading Model from Dfl_Sae plugin...
02/08/2022 07:34:08 INFO Using configuration saved in state file
02/08/2022 07:34:08 INFO Enabling Mixed Precision Training.
02/08/2022 07:34:09 INFO Loaded model from disk: 'E:\newfiles\PS\Rainy\working\Sfx\dfl_sae.h5'
02/08/2022 07:34:09 WARNING Clipnorm has been selected, but is unsupported when using distributed or mixed_precision training, so has been disabled. If you wish to enable clipnorm, then you must disable these options.
02/08/2022 07:34:10 WARNING You have selected eye/mouth loss multipliers greater than 1x, but Penalized Mask Loss is disabled. Disabling all multipliers.
02/08/2022 07:34:10 INFO Loading Trainer from Original plugin...
02/08/2022 07:34:27 INFO [Saved models] - Average loss since last save: face_a: 1.34029, face_b: 0.70108