Faceswap Forum

Occasionally this happens: The face is recognized (apparently) correctly -if I look at the landmarks-... however it is presented (and extracted) upside-down.

is this a glitch ? Does this behaviour have a particular meaning?

(Note: seen only on profile faces so far..)

Basically, the landmarks that aren't visible on the face are given random values, this causes our alignments to get weird as we can't ignore those values and the average causes the face to get lost.

I've given up trying to get decent results on anything where both eyes (landmarks) are not visible for just this reason.
No matter how many different images of sideways you have, you cannot train out the randomised values for the missing landmarks in both training and swap. The resultant swap is always somewhat poor in comparison to straight on faces where all landmarks are visible.

Too bad.

I noticed that results on profile faces are below average. .. I Was wandering if it would not be possible to handle this forcing a different recognition (with a different landmark pattern or similar)
Even if this means duplicating the model.

I have had this problem as well, and not just on full profiles. It happens even in clear 3/4 views sometimes. You'll see seemingly identical face detected, and then for some reason some will flip upside down even with the face registered perfectly. If you need the flipped faces you CAN get them to flip back using the Manual Tool, just by giggling the extract box a bit, but the question is still why, when the landmarks (including hidden lines in 3/4 views) look great. Giggling the box doesn't appear to change anything except reversing the extract bounding box. Keep in mind that any manual tool changes currently do not change the actual extracted faces on the disk - only the landmarks in the alignments file. So if you flip or otherwise adjust faces, you need to use the Alignment Tool's Extract function to rewrite adjusted files out of the frames. This is very fast since the exact coordinates of the exract box is already known.

I believe the new Manual Tool will change the faces on disk when modified?

I use the manual tool a lot, since faces often get "skewed" especially when there are difficult light conditions.

However it is not so intuitive, and also the behaviour when shrinking / enlarging/ moving the selection box is farely difficult to understand.

I believe a new tool is under testing... let's see what happens. I really would like some freedom here, like manually adjusting the position of the landmarks, masking, maybe some filtering and basically "forcing" a face recognition even in a difficult situation.

let's see what happens...

However yes, you are right!
Whenever you correct an alignment both the original face and the relative mask are lost: a second extraction is necessary AND a re-masking operation is needed.... however the extraction is faster, since you tell the SW exatly where to look at.

I believe the flipping is in the Umeyama code. You are welcome to have a look to diagnose a problem/find a fix:

Umeyama:
https://github.com/deepfakes/faceswap/b ... umeyama.py

Alignment:
https://github.com/deepfakes/faceswap/b ... aligner.py

Hi,

Why does the extraction process rotate many faces at extreme angles or even upside-down?

Thanks.

This happens when the face alignment model can't find a good face. Sometimes some manual work in the manual tool can fix it, but if it's not part of your convert video, I recommend just deleting them.