Flipped face on extreme profiles

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Grassone
Posts: 45
Joined: Sun Apr 19, 2020 7:32 pm
Has thanked: 6 times
Been thanked: 2 times

Flipped face on extreme profiles

Post by Grassone »

Occasionally this happens: The face is recognized (apparently) correctly -if I look at the landmarks-... however it is presented (and extracted) upside-down.

is this a glitch ? Does this behaviour have a particular meaning?

(Note: seen only on profile faces so far..)

Cattura flipover.JPG
Cattura flipover.JPG (16.65 KiB) Viewed 5351 times
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Flipped face..

Post by bryanlyon »

Basically, the landmarks that aren't visible on the face are given random values, this causes our alignments to get weird as we can't ignore those values and the average causes the face to get lost.

User avatar
djandg
Posts: 43
Joined: Mon Dec 09, 2019 7:00 pm
Has thanked: 4 times
Been thanked: 2 times

Re: Flipped face..

Post by djandg »

I've given up trying to get decent results on anything where both eyes (landmarks) are not visible for just this reason.
No matter how many different images of sideways you have, you cannot train out the randomised values for the missing landmarks in both training and swap. The resultant swap is always somewhat poor in comparison to straight on faces where all landmarks are visible.

User avatar
Grassone
Posts: 45
Joined: Sun Apr 19, 2020 7:32 pm
Has thanked: 6 times
Been thanked: 2 times

Re: Flipped face..

Post by Grassone »

Too bad.

I noticed that results on profile faces are below average. .. I Was wandering if it would not be possible to handle this forcing a different recognition (with a different landmark pattern or similar)
Even if this means duplicating the model.

User avatar
Tekniklee
Posts: 37
Joined: Fri Jan 31, 2020 6:03 pm
Has thanked: 7 times
Been thanked: 3 times

Re: Flipped face..

Post by Tekniklee »

I have had this problem as well, and not just on full profiles. It happens even in clear 3/4 views sometimes. You'll see seemingly identical face detected, and then for some reason some will flip upside down even with the face registered perfectly. If you need the flipped faces you CAN get them to flip back using the Manual Tool, just by giggling the extract box a bit, but the question is still why, when the landmarks (including hidden lines in 3/4 views) look great. Giggling the box doesn't appear to change anything except reversing the extract bounding box. Keep in mind that any manual tool changes currently do not change the actual extracted faces on the disk - only the landmarks in the alignments file. So if you flip or otherwise adjust faces, you need to use the Alignment Tool's Extract function to rewrite adjusted files out of the frames. This is very fast since the exact coordinates of the exract box is already known.

I believe the new Manual Tool will change the faces on disk when modified?

User avatar
Grassone
Posts: 45
Joined: Sun Apr 19, 2020 7:32 pm
Has thanked: 6 times
Been thanked: 2 times

Re: Flipped face..

Post by Grassone »

I use the manual tool a lot, since faces often get "skewed" especially when there are difficult light conditions.

However it is not so intuitive, and also the behaviour when shrinking / enlarging/ moving the selection box is farely difficult to understand.

I believe a new tool is under testing... let's see what happens. I really would like some freedom here, like manually adjusting the position of the landmarks, masking, maybe some filtering and basically "forcing" a face recognition even in a difficult situation.

let's see what happens...

However yes, you are right!
Whenever you correct an alignment both the original face and the relative mask are lost: a second extraction is necessary AND a re-masking operation is needed.... however the extraction is faster, since you tell the SW exatly where to look at.

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Flipped face..

Post by torzdf »

I believe the flipping is in the Umeyama code. You are welcome to have a look to diagnose a problem/find a fix:

Umeyama:
https://github.com/deepfakes/faceswap/b ... umeyama.py

Alignment:
https://github.com/deepfakes/faceswap/b ... aligner.py

My word is final

User avatar
bb46970
Posts: 2
Joined: Mon Sep 07, 2020 9:35 pm

Face rotation

Post by bb46970 »

Hi,

Why does the extraction process rotate many faces at extreme angles or even upside-down?

Thanks.

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Face rotation

Post by bryanlyon »

This happens when the face alignment model can't find a good face. Sometimes some manual work in the manual tool can fix it, but if it's not part of your convert video, I recommend just deleting them.

Locked