Clarificaton on Training options: Mask selection

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
363LS2GTO
Posts: 30
Joined: Fri Jul 08, 2022 7:06 pm
Has thanked: 1 time
Been thanked: 4 times

Clarificaton on Training options: Mask selection

Post by 363LS2GTO »

I have had a few bugs here and there (the delete button on the keyboard does work now in the manual tool where it would not work at first for example) and I realized that due to one of these glitches I don't fully understand the mask option in the training settings.

First, the guides say to always enable penalized mask loss and provides a good explanation. Next, if this is enabled you have to select a mask. I believe 'extended' is selected by default.

After that there is an option for "learn mask."
The guide says that "learn mask" will eat VRAM and may or may not help. The mask selected in training will be learned and can be applied at convert using the "preferred" mask setting.

If you choose a non landmarks based mask, all of your training data must have been extracted with the mask you selected in this section. If not the training program will not run until all of the offending images are removed.

What I don't understand is why there is an option to select a non landmarks based mask if you are only selecting the penalized loss function. It seems that there is nothing to gain and only something to loose.

Is there any benefit to selecting a mask other than components or extended unless you plan on using the learn mask feature for convert?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Clarificaton on Training options: Mask selection

Post by torzdf »

Yes, there are huge benefits for multiple reasons. Off the top of my head:

1) Landmarks based masks are only as good as the landmarks. It really requires very precise landmark alignments for the mask to be effective. For face alignment, only the core landmarks are used (all landmarks, excluding the jawline). For a landmarks based mask, all landmarks are used, so if landmarks are wrong, the mask will be wrong.

2) Landmarks based masks only go up to the top of the eyebrows (these are the highest landmarks that are calculated). You will automatically be masking out the forehead if you use a landmarks based mask. This may, or may not, be important to you.

3) Neural networks masks (particularly Bisenet-FP with the faceswap weights) will mask the whole face, and will also handle face obstructions and hair. These will not be handled at all by landmarks based masks.

My word is final

User avatar
363LS2GTO
Posts: 30
Joined: Fri Jul 08, 2022 7:06 pm
Has thanked: 1 time
Been thanked: 4 times

Re: Clarificaton on Training options: Mask selection

Post by 363LS2GTO »

Thank you for the reply.

I have come the realize the benefits of using a NN mask.

I am still confused as to how selecting this feature in the training options without selecting 'learn mask' works.

How does selecting a specific mask for training affect the end results vs. just applying any mask that has been applied to the alignments file at the convert stage?

Does this mean that if I train with Bisenet (but not with learn mask so preferred is not an option) it will look better if I then select Bisenet at convert versus training with a landmark based mask and then selecting Bisenet at convert?

How would this affect the final output by training with Bisenet but then selecting Unet at convert?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Clarificaton on Training options: Mask selection

Post by torzdf »

The easiest way to see it is visually. Look at the "red" area in the training previews. These are masked out. Features are badly learned in the masked out area (this is the point of masking without 'learn mask enabled', it tells the NN which parts of the image it is interested in, and which it isn't).

If you train with a Landmark masked, and convert with a Bisenet mask you will get bad results. Specifically, the forehead will be blotchy/unlearned as you masked that area out with the Landmarks mask.

My word is final

User avatar
363LS2GTO
Posts: 30
Joined: Fri Jul 08, 2022 7:06 pm
Has thanked: 1 time
Been thanked: 4 times

Re: Clarificaton on Training options: Mask selection

Post by 363LS2GTO »

That answers my question.

I have been training with the Bisenet mask and will continue to do so as it seems to provide the best results after Unet.

Thank you.

Locked