"Disable Warp", not recommended but mandatory

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
tokafondo
Posts: 32
Joined: Mon Dec 16, 2019 1:43 pm
Has thanked: 10 times
Been thanked: 5 times

"Disable Warp", not recommended but mandatory

Post by tokafondo »

I was tired of not getting sharp results from my training sessions.

After hours past creating photosets, aligning marks, refining masks, and doing several tests with several models, resolutions, batch sizes and settings, I just was getting frustrated with this. "How do they do it?" I wondered. "How people get those creations with cards about the same as mine?" And I'm talking about a GTX1070 with 8GB, and I mean real as linux says, 8GB instead of 6Gb because of the Windows 10 feature of reserving 2GB of your VRAM -- or more.

Well, I had read that disabling warping in the last stage of traing could potentially improve sharpness and detail. And boy, that changed death to life, night to day and sadness to hapiness!!

So my current attempt has been training a dataset up to 100K iterations with unbalanced model, 128 pixels, batch size 16 (although I could use 24 but I read somewhere that I should use only 2 ^ X values).

And from there, I disabled warping and the loss drop considerably, and now I'm getting details in the pictures while reaching 200K iterations, that I simply hadn't got in my >600K, a long week, sessions!!

I still have a lot to learn and to improve, but this is just encouraging. A whole 180 degree turn from my previous experience.

So as rule of thumb, with well prepared and quality datasets and after enough training, "disable warp" is mandatory instead of recommended -- at least for me.

Thanks all for reading and the team for the software.

by torzdf » Tue Jan 26, 2021 12:27 am

Yeah, disabling warp late in train is proper "secret sauce".

I may move it to the main window at some point in the not too distant future.

Disabling warp from the very beginning is enough to totally kill most models though. Think of it as baking the cake (warp on) then icing the cake (warp off).

As a general rule, I will leave warping on until I am getting distinct teeth/eye glare. At this point I will stop training and enable the no-warp option.

Go to full post
User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: "Disable Warp", not recommended but mandatory

Post by torzdf »

Yeah, disabling warp late in train is proper "secret sauce".

I may move it to the main window at some point in the not too distant future.

Disabling warp from the very beginning is enough to totally kill most models though. Think of it as baking the cake (warp on) then icing the cake (warp off).

As a general rule, I will leave warping on until I am getting distinct teeth/eye glare. At this point I will stop training and enable the no-warp option.

My word is final

User avatar
AtlasRedux
Posts: 9
Joined: Tue Feb 16, 2021 11:05 pm
Has thanked: 3 times
Been thanked: 2 times

Re: "Disable Warp", not recommended but mandatory

Post by AtlasRedux »

torzdf wrote: Tue Jan 26, 2021 12:27 am

Yeah, disabling warp late in train is proper "secret sauce".

I may move it to the main window at some point in the not too distant future.

Disabling warp from the very beginning is enough to totally kill most models though. Think of it as baking the cake (warp on) then icing the cake (warp off).

What do you consider late? Or should we just take it on a "feel" when improvement seems to grind to a halt?
BTW, running my first test now, getting around 30K iterations per hour on a 2080Ti with Threadripper with default settings. Quite happy with that.

EDIT : guess can just backup the training and experiment with when to turn off warping

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: "Disable Warp", not recommended but mandatory

Post by torzdf »

AtlasRedux wrote: Wed Feb 17, 2021 2:33 am

What do you consider late? Or should we just take it on a "feel" when improvement seems to grind to a halt?

Pretty much this, yes. When you feel the model has gone about as far as it will go, this is the time to turn it off.

I would always take a back up prior to disengaging warp though.

My word is final

User avatar
AtlasRedux
Posts: 9
Joined: Tue Feb 16, 2021 11:05 pm
Has thanked: 3 times
Been thanked: 2 times

Re: "Disable Warp", not recommended but mandatory

Post by AtlasRedux »

Do you think there is anything to gain from doing a little bit of no warp in between "normal" training?
IE : run for 10 hours normally (in my case, almost exactly 10K iterations with Dlf-H128 and current settings) , turn off warp for a couple of hours to force in some quick unprecise clarity, then turn on again?
Actually testing now, but since this is my first project, I have no references -.-

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 215 times
Contact:

Re: "Disable Warp", not recommended but mandatory

Post by bryanlyon »

No, turning it off and on is actually just recreating what we already do. Not every image is warped, but at the end, turning all warping off help reduce the warped effects on the convert.

User avatar
helpimbeing
Posts: 6
Joined: Mon May 04, 2020 2:22 am
Has thanked: 1 time

Re: "Disable Warp", not recommended but mandatory

Post by helpimbeing »

Thanks for sharing this! How should we approach this if we are also doing fit training? Warp on > Warp off > Fit or Warp on > Warp off AND Fit simultaneously?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 215 times
Contact:

Re: "Disable Warp", not recommended but mandatory

Post by bryanlyon »

Doing warp off while fit training is fine. Turning off warp is just another way of fitting the data.

User avatar
deepfakesweb
Posts: 6
Joined: Mon Mar 15, 2021 6:25 pm
Has thanked: 3 times
Been thanked: 1 time

Re: "Disable Warp", not recommended but mandatory

Post by deepfakesweb »

Is there a general rule of thumb of how many iterations of No Warp might yield best results? i.e 10% of total iterations of regular training? Would too big of a % lead to overtraining?

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: "Disable Warp", not recommended but mandatory

Post by torzdf »

Not really. "Until it looks good" is generally the approach. Feel free to report back with findings though.

My word is final

Locked