Faceswap Forum

1) If you have a sizable dataset, but certain angles or expressions are very rare, is it beneficial to make multiple copies of the rare angles or expressions in the dataset so that they get more weight?

2) If a face goes off the edge of the picture a bit, but it is properly recognized and aligned during extraction, has the software masked off the empty (black) areas as not important? And is it beneficial to include it if it contains a rare angle or expression? See the following two images for an example:

1) This is one way to do it. I have plans to add a "pose bias" option to the trainer, that will assign weights to images based on pose, but this is a fair way off.

2) My general approach to training with cropped images like that, is if it isn't eating too much into the face area, then I will include it, Whether all the black is masked out depends on the masker being used (for example, the landmarks based maskers have no idea what the edge of a frame, nor an obstruction is, so it will attempt to include parts of the face that go out of frame. The NN based masks may have more success in masking out areas that go out of frame).

1) Any rules of thumb on how many duplicates to add in proportion to the rest of the dataset? I assume there is some type of balance that you need to achieve. Add too few, and the rare angle/expression might not make a difference. Add too many, and other parts of the duplicated image might be too much of an influence (i.e., you're trying to get a frown in the dataset, but by adding too many duplicates, now the eyebrows always look a certain way, etc.).

2) I'm confused about configuring masks. The guide shows the UI looking a bit different. In the Extract panel, the Masker section only lists Unset-Dfl, Vgg-Clear, and Vgg-Obstructed. But in the Tools->Mask panel, it also lists Components and Extended. What's the right way to do this?

In my opinion you'll be better off getting a sub-dataset that just has those rare details you want, and then using it as part of your fit training. That'll work better than forcing it to appear multiple times (sicne the hash based system will probably prevent most duplicates you try to force.

JustJohnny wrote: ↑Fri Jun 19, 2020 1:40 am

2) I'm confused about configuring masks. The guide shows the UI looking a bit different. In the Extract panel, the Masker section only lists Unset-Dfl, Vgg-Clear, and Vgg-Obstructed. But in the Tools->Mask panel, it also lists Components and Extended. What's the right way to do this?

The components and extended masks are generated from landmark alignment data, so they are always available. Those listed in extract are additional masks that can be used in training and convert if you wish,

bryanlyon wrote: ↑Fri Jun 19, 2020 1:46 am

In my opinion you'll be better off getting a sub-dataset that just has those rare details you want, and then using it as part of your fit training. That'll work better than forcing it to appear multiple times (sicne the hash based system will probably prevent most duplicates you try to force.

So are you suggesting training the model without the rare expressions, and then at the end make a small training set with just the rare expressions for fit training? Does it make a difference if the rare expressions belong to A or B? In my case it would be B.

No, just keep them to add to your fit training set. That way they get extra attention during that time. But if they're only in your B set, you don't really need to worry about it since they'll not really be used directly since the A set will be the one that receives the B face.