Phaze-A - 256px Demo

Display post links to interesting swaps


Locked
User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Phaze-A - 256px Demo

Post by torzdf »

Above is a demonstration of Phaze-A trained at 256px on consumer hardware (1x 2080Ti). The source is We Bought a Zoo, replacing Scarlett Johansson with Emma Stone. No VFX have been applied to this swap. This is raw Faceswap output. The only post-processing applied was to place in side-by-side shots of swap vs original.

This video does not demonstrate the best of what this model is capable of. It is a demonstration of some quickly chosen settings on 2 face-sets I happened to have kicking around (I'm not planning on going viral here!). The best settings for this model have definitely not been discovered yet, so there are definitely improvements to be had for this GPU.

This model took a long time to train (2m iters over about 2 weeks). Good results take time.

I have decided to keep this video quite long, as I don't want to be too selective with the "best scenes", so you can see the good parts, and other areas where it hasn't worked quite so well.

The actual convert is done at 720p. 256px is probably the absolute minimum to get away with close-ups at this resolution, and it would probably benefit from being trained at a slightly higher resolution.

The settings for this model will be placed in the presets section of the Phaze-A thread.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Phaze-A - 256px Demo

Post by ianstephens »

Looks mighty fine I must say :D

We're still playing with your StoJo model.

We've installed our 3090 FE now and were wondering now with all of that extra VRAM...

You mentioned,

256px is probably the absolute minimum to get away with close-ups at this resolution, and it would probably benefit from being trained at a slightly higher resolution.

With 512px training images for both sides - how would one go about correctly training at a higher resolution?

Is this simple change below all it takes to train at 384px for example (or is there more to it)?

stojo.png
stojo.png (58.04 KiB) Viewed 33880 times

Excuse my ignorance - still learning here :)

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Phaze-A - 256px Demo

Post by ianstephens »

Quick question [mention]torzdf[/mention] -

Did you use "no warp" at the end of training for this video? Or was it simply a run of 2M iterations with the StoJo preset model?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 215 times
Contact:

Re: Phaze-A - 256px Demo

Post by bryanlyon »

ianstephens wrote: Mon Jul 12, 2021 8:31 pm

Quick question @torzdf -

Did you use "no warp" at the end of training for this video? Or was it simply a run of 2M iterations with the StoJo preset model?

Torzdf has said in the past that he regularly turns off warp for the last part of training. I would guess that he did the same here.

torzdf wrote: Tue Jan 26, 2021 12:27 am

Yeah, disabling warp late in train is proper "secret sauce".

I may move it to the main window at some point in the not too distant future.

Disabling warp from the very beginning is enough to totally kill most models though. Think of it as baking the cake (warp on) then icing the cake (warp off).

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: Phaze-A - 256px Demo

Post by torzdf »

ianstephens wrote: Tue Jul 06, 2021 1:52 pm

Is this simple change below all it takes to train at 384px for example (or is there more to it)?

Short answer. You could just do that, but there's more to it.

Long answer: A lot of maths.... I will come back and articulate at some point in the (hopefully) not too distant future.

My word is final

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: Phaze-A - 256px Demo

Post by torzdf »

ianstephens wrote: Mon Jul 12, 2021 8:31 pm

Quick question @torzdf -

Did you use "no warp" at the end of training for this video? Or was it simply a run of 2M iterations with the StoJo preset model?

Yes. Can't remember when I disabled. Probably about halfway through

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Phaze-A - 256px Demo

Post by ianstephens »

Curious question re. input images (when extracting for example)...

I know we are training at 256px with the StoJo model. Would there be any benefit of using 512px or even 1024px face training images (i.e. the size we specify when extracting sources)?

We always thought that input images needed to be the same as what the model was using - we've been using 256px face images for as long as we can remember but noticed we had better results with the latest training sets (under StoJo) when using 512px. Would 1024 fair any better?

And if you will, what is the science behind this?

Thank you in advance :)

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 215 times
Contact:

Re: Phaze-A - 256px Demo

Post by bryanlyon »

When using legacy faces, the face is extracted from the full image at that percentage then scaled to for the model. In the case of a 256x face at 100% coverage, then the while 256x256 image would be scaled and used to train. If you're using a 256x256 model with less than 100% coverage you may need higher resolution extracts to get the full resolution.

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: Phaze-A - 256px Demo

Post by torzdf »

I'll add to the above, it's not just legacy faces.

The input resolution to the model is separate from the size of the extracted faces

Try not to think of the extracted faces as the training images, rather, they contain the training images. Whether you use Face or legacy centering, the actual images fed to the model will be a sub-crop from the extracted faces. Therefore, the extracted faces should always be of a higher resolution than the model input.

This gives a visualization of the sub-crops:
https://github.com/deepfakes/faceswap/pull/1095

Bear in mind that this is with 100% coverage. Sub-crop will shrink further with lower centering.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Phaze-A - 256px Demo

Post by ianstephens »

Thank you both [mention]torzdf[/mention] and [mention]bryanlyon[/mention] for your replies and explanations.

They both make much sense and I can see how the images presented to the trainer are used now - up to speed.

So essentially a 512px image presented to a 256px trainer should indeed be adequate and no need for 1024px (unless perhaps training a 512px model).

So really, we were going wrong with these 256px training images and it's understandable why we saw such a difference with the 512px versions with the 256px StoJo model.

I suppose if we do ever train at 512px (it's now plausible with the new hardware) we would then perhaps use something like a 768px input image.

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Phaze-A - 256px Demo

Post by ianstephens »

torzdf wrote: Tue Jul 13, 2021 12:01 am
ianstephens wrote: Tue Jul 06, 2021 1:52 pm

Is this simple change below all it takes to train at 384px for example (or is there more to it)?

Short answer. You could just do that, but there's more to it.

Long answer: A lot of maths.... I will come back and articulate at some point in the (hopefully) not too distant future.

[mention]torzdf[/mention] - just an open request with no expectations but would it be possible for you to send over (or make available in the repo for the next update) a version of StoJo that can train at 512px (correctly, "...there's more to it"?) It'd be really interesting to do such and play with what's possible.

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Phaze-A - 256px Demo

Post by ianstephens »

torzdf wrote: Tue Jul 13, 2021 12:01 am
ianstephens wrote: Tue Jul 06, 2021 1:52 pm

Is this simple change below all it takes to train at 384px for example (or is there more to it)?

Short answer. You could just do that, but there's more to it.

Long answer: A lot of maths.... I will come back and articulate at some point in the (hopefully) not too distant future.

[mention]torzdf[/mention] - any advice or tips for training StoJo at 512px are also greatly appreciated :D

Still learning/researching this - it's still very new to me :)

Would make a fun project for sure.

User avatar
dodo
Posts: 1
Joined: Sun Jun 07, 2020 9:35 pm
Has thanked: 1 time

Re: Phaze-A - 256px Demo

Post by dodo »

Hi [mention]torzdf[/mention]
Can you give us some hints about that dataset used for this video?
How many images per identity, how many different sources, did you use photos or only video frames etc.
Thanks.

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: Phaze-A - 256px Demo

Post by torzdf »

i pulled ScaJo set from 21 movies and Emma Stone from 13 movies, all 1080p+ sources. I can't remember how many images... probably more than I needed.

The most important part was that I spent a not inconsiderable amount of time making sure that I had a decent quality training set for each side.

My word is final

User avatar
Dartagnian
Posts: 2
Joined: Sat Jan 29, 2022 4:46 am

Re: Phaze-A - 256px Demo

Post by Dartagnian »

Could you show your convert settings for this as well? I know convert is the most simple part of the 3 main stages, but it's the one I have the most difficulty with. I'd love to see it if you still have it.

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: Phaze-A - 256px Demo

Post by torzdf »

Ooof, I doubt I still have those around. I tend to just play with the Preview Tool until I'm happy with the output, then roll with those settings.

My word is final

Locked