[Guide] Introducing - Phaze-A

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

User avatar
Scrapemist
Posts: 16
Joined: Sat Nov 05, 2022 2:26 pm
Has thanked: 8 times

Re: [Guide] Introducing - Phaze-A

Post by Scrapemist »

Wow, very interesting!
Thnx for showcasing the results.

Ryzen1988 wrote: Wed Nov 23, 2022 2:50 am

So my default was a simple network with just split decoders, Adabelief, Layer norm and subpixel.
""...""
A version with separate fully connected & shared FC with a single decoder.

Its surprizing how little difference there is between these two (The first and last in the second row)


Tags:
User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: [Guide] Introducing - Phaze-A

Post by ianstephens »

Am just about to play with the sym384 model.

The DNY 1024 is a great model but a very slow burner due to low batch size meaning crazily high iterations and time are needed (over a month+ to get clarity with our tested datasets) and I want to play with something that can run on efficientnetv2 as I've had fantastic results when it comes to learning speed when testing with the StoJo model.

What would you guys go with in terms of the learning rate for the sym384 model?

We're currently set at 2e-5 from the DNY 1024 model but I can't imagine it needing it this low.

Thank you for your suggestions in advance.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: [Guide] Introducing - Phaze-A

Post by torzdf »

If you can get away with not using mixed precision, do, as I had NaN issues with SYM-384 using MP. It resolves nice and quickly but NaNs earlier than I would have liked. Otherwise, I would say start with a fairly low lr. Maybe 3.5e-5, may 2e-5. Can't give more guidance beyond that sadly.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: [Guide] Introducing - Phaze-A

Post by ianstephens »

torzdf wrote: Sat Nov 26, 2022 11:23 am

If you can get away with not using mixed precision, do, as I had NaN issues with SYM-384 using MP. It resolves nice and quickly but NaNs earlier than I would have liked. Otherwise, I would say start with a fairly low lr. Maybe 3.5e-5, may 2e-5. Can't give more guidance beyond that sadly.

Thanks for the reply @torzdf. To be honest, I've currently shelved MP as I got sick of the NaNs. So for me, it will always be FP provided I am able with the current hardware.

I'll settle in the middle at 3e-5 and report back :D

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: [Guide] Introducing - Phaze-A

Post by ianstephens »

I, for the life of me, cannot get a SYM-384 model started.

After the first model save (500its) the preview always turns out like this:

Screenshot 2022-11-28 at 10.27.49.png
Screenshot 2022-11-28 at 10.27.49.png (38.54 KiB) Viewed 20096 times

I've tried lowering the learning rate as low as 1e-5 and still the same after the first model save (500its).

I've also tried starting the model with ICNR Init and Conv Aware Init disabled but still the same.

Any tips/advice?

Thank you in advance.

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: [Guide] Introducing - Phaze-A

Post by MaxHunter »

I've currently shelved MP as I got sick of the NaNs. So for me, it will always be FP provided I am able with the current hardware

@ianstephens I know you have a big rig, but out of curiosity what is your batch size and how long has it taken you to train a 512 on FP?

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: [Guide] Introducing - Phaze-A

Post by ianstephens »

MaxHunter wrote: Mon Nov 28, 2022 5:25 pm

I've currently shelved MP as I got sick of the NaNs. So for me, it will always be FP provided I am able with the current hardware

@ianstephens I know you have a big rig, but out of curiosity what is your batch size and how long has it taken you to train a 512 on FP?

@MaxHunter:

I've not played with the DNY 512 model yet - I have been working with the DNY 1024 model.

For the DNY 1024 model, we managed to get BS 2-3 at FP. However, it was more stable at BS 2 as sometimes it hit OOM.

We ran the model with ~ 6000 faces on both sides A/B.

It took over four weeks of non-stop training (minus a few hiccups and pauses) and several million iterations to start seeing clarity with the previews. However, on a test convert and using a full 1024+ px (face taking up most of the frame) swap it was clear it still had a long way to go. I imagine once I switch off warp it'll really bring it together but we never got that far and it's not quite ready for that yet. I have the model saved and may resume training and finally training with nowarp once I get some time.

But now I'm playing with the SYM-384 model (finally managed to get it started after some tweaking) and when I woke up this morning after running it through the night I am amazed at how the previews look already. I really do love efficientnetv2. We run this at BS 12.

Hope this helps!

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: [Guide] Introducing - Phaze-A

Post by torzdf »

ianstephens wrote: Mon Nov 28, 2022 10:32 am

I, for the life of me, cannot get a SYM-384 model started.

After the first model save (500its) the preview always turns out like this:

I've tried lowering the learning rate as low as 1e-5 and still the same after the first model save (500its).

I've also tried starting the model with ICNR Init and Conv Aware Init disabled but still the same.

Any tips/advice?

Thank you in advance.

Sadly not. I have heard of 1 other person this has happened to, and I have not been able to replicate, so I wonder whether it is in some way hardware or data-set related.

My word is final

User avatar
Icarus
Posts: 8
Joined: Mon Aug 15, 2022 9:18 pm
Has thanked: 10 times
Been thanked: 8 times

Re: [Guide] Introducing - Phaze-A

Post by Icarus »

Ryzen1988 wrote: Wed Nov 23, 2022 2:50 am

Thats like me driving a toyota and he comes driving up next to me in a monster truck :geek: makes you question things in life :ugeek:

This made me smile in a warm and fuzzy kinda way. :)

These observations and your observations a few replies up are truly insightful and really made me question a few things in turn. Especially the identity bleed part in relation to upscales in the fc. I didn't realize that was result.

After reading through both of your posts, I realized a couple of things. The main thing being that so much comes down to how different your A / B data is. I would prefer 0 identity bleed, however it's actually much needed if A / B shapes are very different. I did an experiment where I was actually pleasantly surprised by how the id leakage made the swap much better.

Your thoughts on the gblock having a positive impact on the split fc approach are also interesting and just ran a model without it and did notice the quality drop significantly. I guess it might be a neccessary evil. As far as how deep to make the gblock, honestly I'm at a loss. I haven't noticed any significant changes from 3 to 9 layers. I think you're correct in that one would need to run 2 identical models side by side with only changes to gbock to see. However I'm still too wrapped up in trying to get a handle on the fc :?

Thanks again for your thoughtful response and observations. They made me change the setup for a new experiment I'm starting (upscale in fc). Sorry for the delayed reply, Just seeing this today.

User avatar
Icarus
Posts: 8
Joined: Mon Aug 15, 2022 9:18 pm
Has thanked: 10 times
Been thanked: 8 times

Re: [Guide] Introducing - Phaze-A

Post by Icarus »

ianstephens wrote: Mon Nov 28, 2022 10:32 am

I, for the life of me, cannot get a SYM-384 model started.

After the first model save (500its) the preview always turns out like this:

Screenshot 2022-11-28 at 10.27.49.png

I've tried lowering the learning rate as low as 1e-5 and still the same after the first model save (500its).

I've also tried starting the model with ICNR Init and Conv Aware Init disabled but still the same.

Any tips/advice?

Thank you in advance.

Make sure you lower the epsilon to 4. I've been using MP without NAN's for almost a year now after reading the Nvidia guide to MP on their blog. You can most likely get away with an lr of 2.5 as long as your epsilon is 4.

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: [Guide] Introducing - Phaze-A

Post by MaxHunter »

If Icarus's advice doesn't work (1e-5/-4 doesn't always work for me,) I'm convinced lowering your iterations and restarting the cycle does the trick. See the recent post about my experiment on this.

viewtopic.php?f=27&t=2409

(I don't know why, and it's just a guess, but I suspect it has something to do with VRAM issues.)

...and @Icarus if you're able to I bet you could raise your epsilon, gain that extra detail, and avoid NaNs if you run multiple smaller cycles. 🙂😉

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: [Guide] Introducing - Phaze-A

Post by ianstephens »

Icarus wrote: Fri Dec 16, 2022 2:19 am
ianstephens wrote: Mon Nov 28, 2022 10:32 am

I, for the life of me, cannot get a SYM-384 model started.

After the first model save (500its) the preview always turns out like this:

Screenshot 2022-11-28 at 10.27.49.png

I've tried lowering the learning rate as low as 1e-5 and still the same after the first model save (500its).

I've also tried starting the model with ICNR Init and Conv Aware Init disabled but still the same.

Any tips/advice?

Thank you in advance.

Make sure you lower the epsilon to 4. I've been using MP without NAN's for almost a year now after reading the Nvidia guide to MP on their blog. You can most likely get away with an lr of 2.5 as long as your epsilon is 4.

I'm not using MP - I only run FP. And this issue has nothing to do with hitting NaNs. Secondly, I don't like the idea of lowering the epsilon too much due to a less accurate model.

I have a workaround I use for this particular issue. I simply delete the model and start again from scratch after running a forced Faceswap update (random I know). I mentioned this to @torzdf in the past so perhaps there is something in the codebase that can be changed. Aside, I (eventually) manage to get a model going but it can be frustrating to begin with.

User avatar
MattB
Posts: 22
Joined: Fri Aug 19, 2022 4:54 pm
Been thanked: 5 times

Re: [Guide] Introducing - Phaze-A

Post by MattB »

I'm trying to tweak models for the highest possible resolution result. But given it takes me a week to run a model, it's a painful process. There are just too darned many options, and tweaking a setting, testing, then tweaking again is exhausting. But it sounds like you may have done the heavy lifting for me!

With that in mind... <he gets his tin cup out> is there any chance you can share your .ini files with your "optimal" settings?

Yep! I'm lazy enough to ask. :)

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: [Guide] Introducing - Phaze-A

Post by torzdf »

@ianstephens random question. I have a pretty far-fetched theory given that I have only known this to happen to 2 people, so am trying to look for common factors.

Is your data 'real' data (ie, it is real human faces, not avatars etc,). Also, are the faces on a green screen/solid background or do the backgrounds very a lot?

My word is final

User avatar
Icarus
Posts: 8
Joined: Mon Aug 15, 2022 9:18 pm
Has thanked: 10 times
Been thanked: 8 times

Re: [Guide] Introducing - Phaze-A

Post by Icarus »

Sorry, I may have jumped to the wrong conclusion. You're right, this isn't a NaN issue, it's just the model crashing during creation (possibly due to VRAM).

MaxHunter wrote: Fri Dec 16, 2022 3:12 am

...and @Icarus if you're able to I bet you could raise your epsilon, gain that extra detail, and avoid NaNs if you run multiple smaller cycles. 🙂😉

Like shorter duration training cycles (ex: run training for 6hrs, stop and restart vs continuous training over multiple days)? This actually reduces NaN's with mixed precision?

Also, I use Adam as an optimizer (just to make sure no one thinks I'm recommending a 4 epsilon value with ada!). It would obviously have to be much lower for adabelief...probably 12 if I'm understanding the math that Nvidia is doing on their blog about this.

https://docs.nvidia.com/deeplearning/pe ... l#training

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: [Guide] Introducing - Phaze-A

Post by MaxHunter »

Like shorter duration training cycles (ex: run training for 6hrs, stop and restart vs continuous training over multiple days)? This actually reduces NaN's with mixed precision?

That's exactly what I'm talking about. 🙂

I've noticed that this helps tremendously! It's not perfect because I'm just starting to run into issues.

I started doing this experiment because 1e4 wasn't working for me for some reason. But I've stopped my experiment because I bought a 3090 to compliment my 3080 ti so I can't tell what my problems are related to.

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: [Guide] Introducing - Phaze-A

Post by MaxHunter »

I'd like a little clarity, please.

Can an encoder like efficienet v2 that doesn't naturally support 512, be forced to output 512?

I was under the impression it can not, but reading @adam_macchiato post (for the umpteenth time 😆) suggests it can.

Is it a fools errand to try?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: [Guide] Introducing - Phaze-A

Post by torzdf »

Encoder doesn't output at image size. It outputs a feature map at reduced spatial dimension, but increased depth.

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: [Guide] Introducing - Phaze-A

Post by MaxHunter »

I'm assuming that answer means, "Yes. Efficienetv2 B0 will work with DNY512, but was designed for smaller models, but scale at 100%.".

😁

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: [Guide] Introducing - Phaze-A

Post by torzdf »

The encoder is compressing an image down to a smaller representation. the Decoder enlarges the representation back up to a reconstructed image.

I think you are probably thinking more about input size vs output size, in which case, encoder scaling is there to make those kinds of balances if you so wish

My word is final

Post Reply