Faceswap Forum

I was tired of not getting sharp results from my training sessions.

After hours past creating photosets, aligning marks, refining masks, and doing several tests with several models, resolutions, batch sizes and settings, I just was getting frustrated with this. "How do they do it?" I wondered. "How people get those creations with cards about the same as mine?" And I'm talking about a GTX1070 with 8GB, and I mean real as linux says, 8GB instead of 6Gb because of the Windows 10 feature of reserving 2GB of your VRAM -- or more.

Well, I had read that disabling warping in the last stage of traing could potentially improve sharpness and detail. And boy, that changed death to life, night to day and sadness to hapiness!!

So my current attempt has been training a dataset up to _100K iterations with unbalanced model, 128 pixels, batch size 16 (although I could use 24 but I read somewhere that I should use only 2 ^ X values).

And from there, I disabled warping and the loss drop considerably, and now I'm getting details in the pictures while reaching 200K iterations, that I simply hadn't got in my >600K, a long week, sessions!!

I still have a lot to learn and to improve, but this is just encouraging. A whole 180 degree turn from my previous experience.

So as rule of thumb, with well prepared and quality datasets and after enough training, "disable warp" is mandatory instead of recommended -- at least for me.

Thanks all for reading and the team for the software.

Yeah, disabling warp late in train is proper "secret sauce".

I may move it to the main window at some point in the not too distant future.

Disabling warp from the very beginning is enough to totally kill most models though. Think of it as baking the cake (warp on) then icing the cake (warp off).

As a general rule, I will leave warping on until I am getting distinct teeth/eye glare. At this point I will stop training and enable the no-warp option.

torzdf wrote: ↑Tue Jan 26, 2021 12:27 am

Yeah, disabling warp late in train is proper "secret sauce".

I may move it to the main window at some point in the not too distant future.

Disabling warp from the very beginning is enough to totally kill most models though. Think of it as baking the cake (warp on) then icing the cake (warp off).

What do you consider late? Or should we just take it on a "feel" when improvement seems to grind to a halt?
BTW, running my first test now, getting around 30K iterations per hour on a 2080Ti with Threadripper with default settings. Quite happy with that.

EDIT : guess can just backup the training and experiment with when to turn off warping

AtlasRedux wrote: ↑Wed Feb 17, 2021 2:33 am

What do you consider late? Or should we just take it on a "feel" when improvement seems to grind to a halt?

Pretty much this, yes. When you feel the model has gone about as far as it will go, this is the time to turn it off.

I would always take a back up prior to disengaging warp though.

Do you think there is anything to gain from doing a little bit of no warp in between "normal" training?
IE : run for 10 hours normally (in my case, almost exactly 10K iterations with Dlf-H128 and current settings) , turn off warp for a couple of hours to force in some quick unprecise clarity, then turn on again?
Actually testing now, but since this is my first project, I have no references -.-

No, turning it off and on is actually just recreating what we already do. Not every image is warped, but at the end, turning all warping off help reduce the warped effects on the convert.

Thanks for sharing this! How should we approach this if we are also doing fit training? Warp on > Warp off > Fit or Warp on > Warp off AND Fit simultaneously?

Doing warp off while fit training is fine. Turning off warp is just another way of fitting the data.

Is there a general rule of thumb of how many iterations of No Warp might yield best results? i.e 10% of total iterations of regular training? Would too big of a % lead to overtraining?

Not really. "Until it looks good" is generally the approach. Feel free to report back with findings though.