Page 1 of 1

Screendoor like image when training

Posted: Wed Mar 23, 2022 2:20 am
by pilipinoguy

Hi,

Can you have a look at my screenshot below. There is a screendoor like red effect on top of my image when training. I invested a lot of hours(700k iterations) on this dfaker model. Is this model dead. Thank you


Re: Screendoor like image when training

Posted: Fri Mar 25, 2022 11:45 am
by torzdf

That looks to me like the model is collapsing. DLight?

If so, then there isn't much you can do to save the current version of the model. You;d need to roll back to significantly prior to these artefacts showing, lower the learning rate and continue.


Re: Screendoor like image when training

Posted: Fri Mar 25, 2022 9:13 pm
by pilipinoguy

Houston we got a problem. It Nan'ed eventually.

Will be doing your suggestion, good thing it autosaves every 50k for me. Rolling back to 350k. Lowering the training rate and TRAINNN!!!

Thanks