Faceswap Forum

I'm doing a conversion after training my model (dfaker) and I'm consistently getting very distracting black spots around the mouth, nose, and eyes whenever the face is partially blocked or off of the screen. Even though the face is visible from the nose down, for example, the mouth will be partially or all black and the rest of the face is blurry. When the face is fully visible, the swap looks decent, though still missing details like teeth gaps. I checked the landmarks with the Manual tool and they all look good for those particular frames (lips picked up correctly, nose, etc.). Anything else I might be missing that's leading to this behavior?

That effect sounds like you have Seamless Clone enabled. Like the convert guide says:

torzdf wrote: ↑Tue Oct 20, 2020 12:28 am

Seamless Clone - This method is good for single images, but terrible for videos, No matter how good the result may look, if you are creating a video, do not use this method.

If it's not that, please post your logs and/or some example images so we can diagnose your issue.

I have had similar outcome in two cases, both my own fault:

1) the training dataset had consistent masking issue where obstructed part of face was still masked as part of face. Eg. if the face had a hat, that had cast a shadow to forehead. The shadow was so consistently present that it became part of the face, causing all swap faces to have "black shadow hue" on their forehead. The issue got corrected when I went through the dataset and fixed the masking (by removing the shadowy parts & replaced the training images where the shadow was present).

2) the conversion target had masking issues similar to case 1. Eg. the target face had a facemask on, or hand moved across the face, and that mask/hand was part of face masking... The issue got corrected when I (painstakingly) went through all the target video frames and corrected the masks. Maybe using a specific masking algorithm (vgg-obstructed) could give better results. It is not very good with profile faces. On other hand the one I usually use (bisenet-fp) very often masks also obstructions as part of face.

Hope this helps...

I'm not using seamless clone, but you gave me an idea to tinker with my color adjustment options and that is having an impact. I was using "match-hist", which leads to the folowing kind of behavior:

If I switch to "avg-color", it gets better, but still not completely gone:

Using "color-transfer" looks the same as far as the blotches go, but overall the face looks more gray:

Note that everything above the top of these pictures is off screen. As far as training goes, I removed all partial faces prior to training, meaning both eyes, nose, and mouth had to be visible to be included, so I don't think it's a masking issue in the training data.