Masking for training?

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Post Reply
User avatar
wulfrik
Posts: 3
Joined: Sat Mar 30, 2024 3:43 am
Has thanked: 1 time
Been thanked: 1 time

Masking for training?

Post by wulfrik »

Apologies if this is a stupid set of questions or posted in the wrong place. I'm new and still learning.

So I extracted frames from various sources for both A and B. Some video sources and some from random photography. However, during the extraction phase I checked "custom" masker for both. I thought it didn't matter for the training, and all I needed was the faces aligned. I've run the resulting output images through 100k training iterations and the results are looking sharp! However, I noticed the masks in the preview thumbs are often...bad. For instance, in many frames the source is wearing a headband. It doesn't necessarily obscure the face, but the masker left some of that headband in and it's leaked into the model. Other masks are just not tightly constrained to the face. But the results look pretty good!

Is this an issue, or is it working as intended and the masking isn't a big deal?

Will the model learn OK regardless, or will the masking issues lead to downstream problems?

If the faulty masks are a problem, what's the remedy? For instance, how do I fix the headband issue for, say 1500 extracted images? Do I have to re-extract from the source and address each on manually? What tool do I use to see the masks before training?

Thanks for any input or feedback.

User avatar
torzdf
Posts: 2667
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Masking for training?

Post by torzdf »

wulfrik wrote: Sat Mar 30, 2024 4:13 am

However, during the extraction phase I checked "custom" masker for both.

Custom masker doesn't really do anything. It just generates an empty mask for editing elsewhere. It doesn't hurt to have it, but it won't be helping you any.

wulfrik wrote: Sat Mar 30, 2024 4:13 am

I thought it didn't matter for the training, and all I needed was the faces aligned. I've run the resulting output images through 100k training iterations and the results are looking sharp! However, I noticed the masks in the preview thumbs are often...bad. For instance, in many frames the source is wearing a headband. It doesn't necessarily obscure the face, but the masker left some of that headband in and it's leaked into the model. Other masks are just not tightly constrained to the face. But the results look pretty good!

Most likely you are using the default 'extended' mask. This is based entirely on the face landmark points.... Its fine, but it doesn't handle obstructions (as you've seen), it doesn't cover the forehead, and, if the alignments are wrong, it won't fit the face properly (as you have also seen).

wulfrik wrote: Sat Mar 30, 2024 4:13 am

Is this an issue, or is it working as intended and the masking isn't a big deal?

It is working as intended, but it isn't working the best it could. You should use the BiSeNet-FP mask. You can generate additional masks for facesets/alignment files using the mask tool (no need to completely re-extract again)

wulfrik wrote: Sat Mar 30, 2024 4:13 am

Will the model learn OK regardless, or will the masking issues lead to downstream problems?

It will be 'ok'. It won't be great. Unfortunately the masker is locked to the model, so you will need to restart training once the new mask has been generated, and ensure that you have selected the correct mask in training settings.

All of this is covered in the guides, so make sure you are following them:

Extract Guide (https://forum.faceswap.dev/viewtopic.php?t=27) wrote:

Each masker has different pros and cons. If you wish to use a Neural Network based mask for training or converting your frames then you can choose a mask here. The bisenet-fp mask is easily the best performing mask

Training guide (https://forum.faceswap.dev/viewtopic.php?t=146#config_loss) wrote:
  • Mask Type - The type of mask to be used for training. To use a mask you must have added the mask you require to the original alignments file that generated your training set. You can add/update masks with the mask tool. See viewtopic.php?f=5&t=27#extract for a thorough description of each of the masks.

My word is final

User avatar
wulfrik
Posts: 3
Joined: Sat Mar 30, 2024 3:43 am
Has thanked: 1 time
Been thanked: 1 time

Re: Masking for training?

Post by wulfrik »

Thank you for the reply. I appreciate the attention and help.

So I re-ran the masker on the extracted files, in the Tools section (choosing BiSeNet-FP). It doesn't seem to have done a lot better. For instance, it's still picking up a lot of the headband fabric. I guess what I feel like I'm missing is where I can see these masks before I start training? And moreover, is there a guide for how to manually edit these masks? I'd love a grid of thumbnails where I can quickly scan for weird masks and fix them. I can't tell if there's something like that within the application or if it's another process in, say, Photoshop (or similar).

User avatar
torzdf
Posts: 2667
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Masking for training?

Post by torzdf »

Most likely you are either continuing training the same model, in which case the mask you originally selected will be used (default is Extended), or you have started a new model but have neglected to select the BiSeNet-FP mask in training settings so the default is used (Extended).

A screengrab would be able to clear that up fairly quickly.

You can view/edit masks in the Manual tool.

BiSeNet will handle most obstructions (including hairbands) just fine:

Image

Last edited by torzdf on Sun Mar 31, 2024 6:24 pm, edited 1 time in total.

My word is final

User avatar
wulfrik
Posts: 3
Joined: Sat Mar 30, 2024 3:43 am
Has thanked: 1 time
Been thanked: 1 time

Re: Masking for training?

Post by wulfrik »

or you have started a new model but have neglected to select the BiSeNet-FP mask in training settings so the default is used (Extended).

Oh wow. This was it!

Thank you so much for suggesting that. I didn't see that in the docs and I don't know if I would have thought to look in Settings -> Train -> Global -> Loss -> Mask Type to match the masker to the trainer.

Image

So to reiterate and better describe my issue for the sake of posterity and any wayward travelers with a similar issue:

  • I chose "custom" masker, wrongly believing that masking wasn't critical for training

  • The training worked well (so far as looking good and sharp), but the masking was poor, and I figured I'd reach out to the forum for help.

  • The fix was to use Tools -> Mask -> Bisenet-FP, and leave the alignments field blank to re-mask the source and target folders of extracted faces

  • This, however, was not sufficient because it was still training off of the default "Extended" mask.

  • It wasn't until I checked the corresponding box in Settings -> Train -> Global -> Loss -> Mask Type that it worked.

Thank you again for the expertise and timely assistance.

Post Reply