Page 1 of 1
Screendoor like image when training
Posted: Wed Mar 23, 2022 2:20 am
by pilipinoguy
Hi,
Can you have a look at my screenshot below. There is a screendoor like red effect on top of my image when training. I invested a lot of hours(700k iterations) on this dfaker model. Is this model dead. Thank you
Re: Screendoor like image when training
Posted: Fri Mar 25, 2022 11:45 am
by torzdf
That looks to me like the model is collapsing. DLight?
If so, then there isn't much you can do to save the current version of the model. You;d need to roll back to significantly prior to these artefacts showing, lower the learning rate and continue.
Re: Screendoor like image when training
Posted: Fri Mar 25, 2022 9:13 pm
by pilipinoguy
Houston we got a problem. It Nan'ed eventually.
Will be doing your suggestion, good thing it autosaves every 50k for me. Rolling back to 350k. Lowering the training rate and TRAINNN!!!
Thanks