Masking for training?

[wulfrik]

Apologies if this is a stupid set of questions or posted in the wrong place. I'm new and still learning.

So I extracted frames from various sources for both A and B. Some video sources and some from random photography. However, during the extraction phase I checked "custom" masker for both. I thought it didn't matter for the training, and all I needed was the faces aligned. I've run the resulting output images through 100k training iterations and the results are looking sharp! However, I noticed the masks in the preview thumbs are often...bad. For instance, in many frames the source is wearing a headband. It doesn't necessarily obscure the face, but the masker left some of that headband in and it's leaked into the model. Other masks are just not tightly constrained to the face. But the results look pretty good!

Is this an issue, or is it working as intended and the masking isn't a big deal?

Will the model learn OK regardless, or will the masking issues lead to downstream problems?

If the faulty masks are a problem, what's the remedy? For instance, how do I fix the headband issue for, say 1500 extracted images? Do I have to re-extract from the source and address each on manually? What tool do I use to see the masks before training?

Thanks for any input or feedback.

[torzdf]

However, during the extraction phase I checked "custom" masker for both.

Custom masker doesn't really do anything. It just generates an empty mask for editing elsewhere. It doesn't hurt to have it, but it won't be helping you any.

I thought it didn't matter for the training, and all I needed was the faces aligned. I've run the resulting output images through 100k training iterations and the results are looking sharp! However, I noticed the masks in the preview thumbs are often...bad. For instance, in many frames the source is wearing a headband. It doesn't necessarily obscure the face, but the masker left some of that headband in and it's leaked into the model. Other masks are just not tightly constrained to the face. But the results look pretty good!

Most likely you are using the default 'extended' mask. This is based entirely on the face landmark points.... Its fine, but it doesn't handle obstructions (as you've seen), it doesn't cover the forehead, and, if the alignments are wrong, it won't fit the face properly (as you have also seen).

Is this an issue, or is it working as intended and the masking isn't a big deal?

It is working as intended, but it isn't working the best it could. You should use the BiSeNet-FP mask. You can generate additional masks for facesets/alignment files using the mask tool (no need to completely re-extract again)

Will the model learn OK regardless, or will the masking issues lead to downstream problems?

It will be 'ok'. It won't be great. Unfortunately the masker is locked to the model, so you will need to restart training once the new mask has been generated, and ensure that you have selected the correct mask in training settings.

All of this is covered in the guides, so make sure you are following them:

Each masker has different pros and cons. If you wish to use a Neural Network based mask for training or converting your frames then you can choose a mask here. The bisenet-fp mask is easily the best performing mask

• Mask Type - The type of mask to be used for training. To use a mask you must have added the mask you require to the original alignments file that generated your training set. You can add/update masks with the mask tool. See viewtopic.php?f=5&t=27#extract for a thorough description of each of the masks.

[wulfrik]

Thank you for the reply. I appreciate the attention and help.

So I re-ran the masker on the extracted files, in the Tools section (choosing BiSeNet-FP). It doesn't seem to have done a lot better. For instance, it's still picking up a lot of the headband fabric. I guess what I feel like I'm missing is where I can see these masks before I start training? And moreover, is there a guide for how to manually edit these masks? I'd love a grid of thumbnails where I can quickly scan for weird masks and fix them. I can't tell if there's something like that within the application or if it's another process in, say, Photoshop (or similar).

[torzdf]

Most likely you are either continuing training the same model, in which case the mask you originally selected will be used (default is Extended), or you have started a new model but have neglected to select the BiSeNet-FP mask in training settings so the default is used (Extended).

A screengrab would be able to clear that up fairly quickly.

You can view/edit masks in the Manual tool.

BiSeNet will handle most obstructions (including hairbands) just fine:

[wulfrik]

or you have started a new model but have neglected to select the BiSeNet-FP mask in training settings so the default is used (Extended).

Oh wow. This was it!

Thank you so much for suggesting that. I didn't see that in the docs and I don't know if I would have thought to look in Settings -> Train -> Global -> Loss -> Mask Type to match the masker to the trainer.

So to reiterate and better describe my issue for the sake of posterity and any wayward travelers with a similar issue:

• I chose "custom" masker, wrongly believing that masking wasn't critical for training

• The training worked well (so far as looking good and sharp), but the masking was poor, and I figured I'd reach out to the forum for help.

• The fix was to use Tools -> Mask -> Bisenet-FP, and leave the alignments field blank to re-mask the source and target folders of extracted faces

• This, however, was not sufficient because it was still training off of the default "Extended" mask.

• It wasn't until I checked the corresponding box in Settings -> Train -> Global -> Loss -> Mask Type that it worked.

Thank you again for the expertise and timely assistance.