Page 1 of 1

Extracting zoomed out frames? Faces too small per frame

Posted: Mon Jan 11, 2021 5:51 pm
by cosmico

So I was trying to teach my buddy faceswap when he ran into this problem where when he extracted faces they came out too small/the frames were too zoomed out. Here's a couple pics as examples.
Image
Image

So I tried extracting the video and I got frames like you would expect
.
.
Image

He's running an up to date faceswap, mines at least 1-2 updates old.
These are the settings he's running:
Image
Image
And my setting are the exact same except for re-feed which mine doesnt even have and he has his min size on 20 while mine is on 0.
He did say he updated his drivers so he can play video games the night before, ......but strangely after I extracted for him and gave him alignments, he had no issue training on gpu. ......So maybe his nvidia drivers weren't up to date?
Also it might be related; he had another issue with extraction that randomly went away. We were extracting a 1 minute video that should be about 1200 frames, because that what my working faceswap ends up with. despite him being on the same detector, aligner, masker as me, with the same confidence of 70 he ended up with 5300 pictures. my first thought was to check his confidence since that happens when you lower it to like 5, but it was at 70 just like mine. <this issue only seemed to happen once or twice than just randomly stopped and his FS started working normal again.


Re: Extracting zoomed out frames? Faces too small per frame

Posted: Mon Jan 11, 2021 5:55 pm
by bryanlyon

This is expected operation. See https://github.com/deepfakes/faceswap/c ... 9868b0b662

You're running an old version while he has the current version is all.


Re: Extracting zoomed out frames? Faces too small per frame

Posted: Mon Jan 11, 2021 6:11 pm
by torzdf

More specifically:

https://github.com/deepfakes/faceswap/pull/1095

Don't know about the 2nd issue. Sounds like a glitch.


Re: Extracting zoomed out frames? Faces too small per frame

Posted: Tue Jan 12, 2021 4:37 pm
by cosmico

I'm don't think either of you understand the issue. Perhaps I didn't write it clearly.

The issue is that his newer version is the one messed up one and giving faces that are way too zoomed out, while my older version is giving the correct appropriate sized frames. And as far as I can tell, every single one of our settings related to extraction are the same.


Re: Extracting zoomed out frames? Faces too small per frame

Posted: Tue Jan 12, 2021 4:38 pm
by bryanlyon

If they look like the samples given in those links, then your friend's extract is correct. If they're more zoomed out than that, please post some samples so we can see what you mean.


Re: Extracting zoomed out frames? Faces too small per frame

Posted: Thu Jan 14, 2021 6:20 pm
by cosmico

Hey guys, torzdf and bryanlyon. Thanks for both being so great and willing to help everyone out. You two really what makes faceswap work.

I just want absolute, crystal clear, 1000%, beyond any doubt, understanding on what your saying.
.
.
So you are saying that his update to date program that is extracting 256 sized image,with 77% coverage, and results in an frames like these two below were the faces are really zoomed out, is correct?
Image Image
.
.
And that my slightly older version, when extracting the same exact video with the same settings, and gives a frame that the face takes up the majority of the frame is out of date? like the one below?
Image

I think the reason I'm kind of confused and asking for an ridiculous amount of confirmation is for several reasons. I've ran faceswap for close to a 1.25 years now, lots of updates and new versions, and faceswap always extracted frames like mine currently is rather than his zoomed out one, so this seemed more like an error than a new change. Another reason why it seems like an error is ....the face being the majority of the frame seems like a good thing.....does it not? If I'm trying to make a swap on a trainer where the input/output are both say 172 pixels, and I'm using 256px extracted frames, ...well that's totally fine because the actual face takes up like 176pixels in that 256px frame, where as in his, it feels like his face is only 128 pixels big in 256 sized framed....which seems like it would be worse and thus this seems like an error. Any chance you could quell my worries and give me more understanding? Sorry for being so high maintenance about this issue.


Re: Extracting zoomed out frames? Faces too small per frame

Posted: Thu Jan 14, 2021 6:25 pm
by bryanlyon

Yes, this is deliberate. See the link that Torzdf posted as to why.


Re: Extracting zoomed out frames? Faces too small per frame

Posted: Thu Jan 14, 2021 6:26 pm
by torzdf

The zoomed out version is correct.

Our default extract size is now 512px (This means that the same facial area from your version within the new extract is 308px, so the sizes are actually comparable)

Why did we do this? As you can see from your crop, the forehead is nearly always cropped out, and sometimes so is the chin.

The new extract brings out the entire head, which gives us scope in the future for different masking and head based swaps (which may or may not be possible, but either way needs a bigger extract).

We do not want multiple extract types, because it's confusing.

When you train a model, you can still train with "legacy" centering. This is basically exactly the same crop as you see in your zoomed in version.

If you look at the PR I linked to above, the boxes show what is fed into the model for "legacy" or "face" centering. At this moment in time the full extracted image is never fed into the model, rather a sub-crop from the extract is fed in.


Re: Extracting zoomed out frames? Faces too small per frame

Posted: Thu Jan 14, 2021 10:30 pm
by cosmico

Thank you, this explains everything perfectly, and I even understand you reasoning for doing it. Unfortunately I'm only able to mark one of your comments as the answer, but thank you to both of you.