Black spots on swapped face

Got questions or tips about the Conversion process? This is the place to discuss them.


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Converting faces from your trained model.

If you are having issues with the Convert process not working as you would expect, then you should post in the Convert Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
master_of_disaster
Posts: 5
Joined: Sat Apr 16, 2022 4:28 pm

Black spots on swapped face

Post by master_of_disaster »

I'm doing a conversion after training my model (dfaker) and I'm consistently getting very distracting black spots around the mouth, nose, and eyes whenever the face is partially blocked or off of the screen. Even though the face is visible from the nose down, for example, the mouth will be partially or all black and the rest of the face is blurry. When the face is fully visible, the swap looks decent, though still missing details like teeth gaps. I checked the landmarks with the Manual tool and they all look good for those particular frames (lips picked up correctly, nose, etc.). Anything else I might be missing that's leading to this behavior?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Black spots on swapped face

Post by bryanlyon »

That effect sounds like you have Seamless Clone enabled. Like the convert guide says:

torzdf wrote: Tue Oct 20, 2020 12:28 am

Seamless Clone - This method is good for single images, but terrible for videos, No matter how good the result may look, if you are creating a video, do not use this method.

If it's not that, please post your logs and/or some example images so we can diagnose your issue.

User avatar
acaint
Posts: 9
Joined: Sat Nov 06, 2021 3:04 pm
Answers: 1
Has thanked: 2 times

Re: Black spots on swapped face

Post by acaint »

I have had similar outcome in two cases, both my own fault:

1) the training dataset had consistent masking issue where obstructed part of face was still masked as part of face. Eg. if the face had a hat, that had cast a shadow to forehead. The shadow was so consistently present that it became part of the face, causing all swap faces to have "black shadow hue" on their forehead. The issue got corrected when I went through the dataset and fixed the masking (by removing the shadowy parts & replaced the training images where the shadow was present).

2) the conversion target had masking issues similar to case 1. Eg. the target face had a facemask on, or hand moved across the face, and that mask/hand was part of face masking... The issue got corrected when I (painstakingly) went through all the target video frames and corrected the masks. Maybe using a specific masking algorithm (vgg-obstructed) could give better results. It is not very good with profile faces. On other hand the one I usually use (bisenet-fp) very often masks also obstructions as part of face.

Hope this helps...

User avatar
master_of_disaster
Posts: 5
Joined: Sat Apr 16, 2022 4:28 pm

Re: Black spots on swapped face

Post by master_of_disaster »

I'm not using seamless clone, but you gave me an idea to tinker with my color adjustment options and that is having an impact. I was using "match-hist", which leads to the folowing kind of behavior:

Image

If I switch to "avg-color", it gets better, but still not completely gone:

Image

Using "color-transfer" looks the same as far as the blotches go, but overall the face looks more gray:

Image

Note that everything above the top of these pictures is off screen. As far as training goes, I removed all partial faces prior to training, meaning both eyes, nose, and mouth had to be visible to be included, so I don't think it's a masking issue in the training data.

Locked