Extracting zoomed out frames? Faces too small per frame

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Post Reply
User avatar
cosmico
Posts: 94
Joined: Sat Jan 18, 2020 6:32 pm
Has thanked: 13 times
Been thanked: 28 times

Extracting zoomed out frames? Faces too small per frame

Post by cosmico »

So I was trying to teach my buddy faceswap when he ran into this problem where when he extracted faces they came out too small/the frames were too zoomed out. Here's a couple pics as examples.
Image
Image

So I tried extracting the video and I got frames like you would expect
.
.
Image

He's running an up to date faceswap, mines at least 1-2 updates old.
These are the settings he's running:
Image
Image
And my setting are the exact same except for re-feed which mine doesnt even have and he has his min size on 20 while mine is on 0.
He did say he updated his drivers so he can play video games the night before, ......but strangely after I extracted for him and gave him alignments, he had no issue training on gpu. ......So maybe his nvidia drivers weren't up to date?
Also it might be related; he had another issue with extraction that randomly went away. We were extracting a 1 minute video that should be about 1200 frames, because that what my working faceswap ends up with. despite him being on the same detector, aligner, masker as me, with the same confidence of 70 he ended up with 5300 pictures. my first thought was to check his confidence since that happens when you lower it to like 5, but it was at 70 just like mine. <this issue only seemed to happen once or twice than just randomly stopped and his FS started working normal again.

by torzdf » Thu Jan 14, 2021 6:26 pm

The zoomed out version is correct.

Our default extract size is now 512px (This means that the same facial area from your version within the new extract is 308px, so the sizes are actually comparable)

Why did we do this? As you can see from your crop, the forehead is nearly always cropped out, and sometimes so is the chin.

The new extract brings out the entire head, which gives us scope in the future for different masking and head based swaps (which may or may not be possible, but either way needs a bigger extract).

We do not want multiple extract types, because it's confusing.

When you train a model, you can still train with "legacy" centering. This is basically exactly the same crop as you see in your zoomed in version.

If you look at the PR I linked to above, the boxes show what is fed into the model for "legacy" or "face" centering. At this moment in time the full extracted image is never fed into the model, rather a sub-crop from the extract is fed in.

Go to full post

User avatar
bryanlyon
Site Admin
Posts: 596
Joined: Fri Jul 12, 2019 12:49 am
Answers: 40
Location: San Francisco
Has thanked: 3 times
Been thanked: 154 times
Contact:

Re: Extracting zoomed out frames? Faces too small per frame

Post by bryanlyon »

This is expected operation. See https://github.com/deepfakes/faceswap/c ... 9868b0b662

You're running an old version while he has the current version is all.


User avatar
torzdf
Posts: 1267
Joined: Fri Jul 12, 2019 12:53 am
Answers: 120
Has thanked: 39 times
Been thanked: 252 times

Re: Extracting zoomed out frames? Faces too small per frame

Post by torzdf »

More specifically:

https://github.com/deepfakes/faceswap/pull/1095

Don't know about the 2nd issue. Sounds like a glitch.

My word is final


User avatar
cosmico
Posts: 94
Joined: Sat Jan 18, 2020 6:32 pm
Has thanked: 13 times
Been thanked: 28 times

Re: Extracting zoomed out frames? Faces too small per frame

Post by cosmico »

I'm don't think either of you understand the issue. Perhaps I didn't write it clearly.

The issue is that his newer version is the one messed up one and giving faces that are way too zoomed out, while my older version is giving the correct appropriate sized frames. And as far as I can tell, every single one of our settings related to extraction are the same.


User avatar
bryanlyon
Site Admin
Posts: 596
Joined: Fri Jul 12, 2019 12:49 am
Answers: 40
Location: San Francisco
Has thanked: 3 times
Been thanked: 154 times
Contact:

Re: Extracting zoomed out frames? Faces too small per frame

Post by bryanlyon »

If they look like the samples given in those links, then your friend's extract is correct. If they're more zoomed out than that, please post some samples so we can see what you mean.


User avatar
cosmico
Posts: 94
Joined: Sat Jan 18, 2020 6:32 pm
Has thanked: 13 times
Been thanked: 28 times

Re: Extracting zoomed out frames? Faces too small per frame

Post by cosmico »

Hey guys, torzdf and bryanlyon. Thanks for both being so great and willing to help everyone out. You two really what makes faceswap work.

I just want absolute, crystal clear, 1000%, beyond any doubt, understanding on what your saying.
.
.
So you are saying that his update to date program that is extracting 256 sized image,with 77% coverage, and results in an frames like these two below were the faces are really zoomed out, is correct?
Image Image
.
.
And that my slightly older version, when extracting the same exact video with the same settings, and gives a frame that the face takes up the majority of the frame is out of date? like the one below?
Image

I think the reason I'm kind of confused and asking for an ridiculous amount of confirmation is for several reasons. I've ran faceswap for close to a 1.25 years now, lots of updates and new versions, and faceswap always extracted frames like mine currently is rather than his zoomed out one, so this seemed more like an error than a new change. Another reason why it seems like an error is ....the face being the majority of the frame seems like a good thing.....does it not? If I'm trying to make a swap on a trainer where the input/output are both say 172 pixels, and I'm using 256px extracted frames, ...well that's totally fine because the actual face takes up like 176pixels in that 256px frame, where as in his, it feels like his face is only 128 pixels big in 256 sized framed....which seems like it would be worse and thus this seems like an error. Any chance you could quell my worries and give me more understanding? Sorry for being so high maintenance about this issue.


User avatar
bryanlyon
Site Admin
Posts: 596
Joined: Fri Jul 12, 2019 12:49 am
Answers: 40
Location: San Francisco
Has thanked: 3 times
Been thanked: 154 times
Contact:

Re: Extracting zoomed out frames? Faces too small per frame

Post by bryanlyon »

Yes, this is deliberate. See the link that Torzdf posted as to why.


User avatar
torzdf
Posts: 1267
Joined: Fri Jul 12, 2019 12:53 am
Answers: 120
Has thanked: 39 times
Been thanked: 252 times

Re: Extracting zoomed out frames? Faces too small per frame

Post by torzdf »

The zoomed out version is correct.

Our default extract size is now 512px (This means that the same facial area from your version within the new extract is 308px, so the sizes are actually comparable)

Why did we do this? As you can see from your crop, the forehead is nearly always cropped out, and sometimes so is the chin.

The new extract brings out the entire head, which gives us scope in the future for different masking and head based swaps (which may or may not be possible, but either way needs a bigger extract).

We do not want multiple extract types, because it's confusing.

When you train a model, you can still train with "legacy" centering. This is basically exactly the same crop as you see in your zoomed in version.

If you look at the PR I linked to above, the boxes show what is fed into the model for "legacy" or "face" centering. At this moment in time the full extracted image is never fed into the model, rather a sub-crop from the extract is fed in.

My word is final


User avatar
cosmico
Posts: 94
Joined: Sat Jan 18, 2020 6:32 pm
Has thanked: 13 times
Been thanked: 28 times

Re: Extracting zoomed out frames? Faces too small per frame

Post by cosmico »

Thank you, this explains everything perfectly, and I even understand you reasoning for doing it. Unfortunately I'm only able to mark one of your comments as the answer, but thank you to both of you.


Post Reply