I'm doing a conversion after training my model (dfaker) and I'm consistently getting very distracting black spots around the mouth, nose, and eyes whenever the face is partially blocked or off of the screen. Even though the face is visible from the nose down, for example, the mouth will be partially or all black and the rest of the face is blurry. When the face is fully visible, the swap looks decent, though still missing details like teeth gaps. I checked the landmarks with the Manual tool and they all look good for those particular frames (lips picked up correctly, nose, etc.). Anything else I might be missing that's leading to this behavior?
Black spots on swapped face
Read the FAQs and search the forum before posting a new topic.
This forum is for discussing tips and understanding the process involved with Converting faces from your trained model.
If you are having issues with the Convert process not working as you would expect, then you should post in the Convert Support forum.
Please mark any answers that fixed your problems so others can find the solutions.
- bryanlyon
- Site Admin
- Posts: 793
- Joined: Fri Jul 12, 2019 12:49 am
- Location: San Francisco
- Has thanked: 4 times
- Been thanked: 218 times
- Contact:
Re: Black spots on swapped face
That effect sounds like you have Seamless Clone enabled. Like the convert guide says:
If it's not that, please post your logs and/or some example images so we can diagnose your issue.
Re: Black spots on swapped face
I have had similar outcome in two cases, both my own fault:
1) the training dataset had consistent masking issue where obstructed part of face was still masked as part of face. Eg. if the face had a hat, that had cast a shadow to forehead. The shadow was so consistently present that it became part of the face, causing all swap faces to have "black shadow hue" on their forehead. The issue got corrected when I went through the dataset and fixed the masking (by removing the shadowy parts & replaced the training images where the shadow was present).
2) the conversion target had masking issues similar to case 1. Eg. the target face had a facemask on, or hand moved across the face, and that mask/hand was part of face masking... The issue got corrected when I (painstakingly) went through all the target video frames and corrected the masks. Maybe using a specific masking algorithm (vgg-obstructed) could give better results. It is not very good with profile faces. On other hand the one I usually use (bisenet-fp) very often masks also obstructions as part of face.
Hope this helps...
- master_of_disaster
- Posts: 5
- Joined: Sat Apr 16, 2022 4:28 pm
Re: Black spots on swapped face
I'm not using seamless clone, but you gave me an idea to tinker with my color adjustment options and that is having an impact. I was using "match-hist", which leads to the folowing kind of behavior:
If I switch to "avg-color", it gets better, but still not completely gone:
Using "color-transfer" looks the same as far as the blotches go, but overall the face looks more gray:
Note that everything above the top of these pictures is off screen. As far as training goes, I removed all partial faces prior to training, meaning both eyes, nose, and mouth had to be visible to be included, so I don't think it's a masking issue in the training data.