Hi,
Can you have a look at my screenshot below. There is a screendoor like red effect on top of my image when training. I invested a lot of hours(700k iterations) on this dfaker model. Is this model dead. Thank you
Read the FAQs and search the forum before posting a new topic.
This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.
Please mark any answers that fixed your problems so others can find the solutions.
Hi,
Can you have a look at my screenshot below. There is a screendoor like red effect on top of my image when training. I invested a lot of hours(700k iterations) on this dfaker model. Is this model dead. Thank you
That looks to me like the model is collapsing. DLight?
If so, then there isn't much you can do to save the current version of the model. You;d need to roll back to significantly prior to these artefacts showing, lower the learning rate and continue.
That looks to me like the model is collapsing. DLight?
If so, then there isn't much you can do to save the current version of the model. You;d need to roll back to significantly prior to these artefacts showing, lower the learning rate and continue.
My word is final
Houston we got a problem. It Nan'ed eventually.
Will be doing your suggestion, good thing it autosaves every 50k for me. Rolling back to 350k. Lowering the training rate and TRAINNN!!!
Thanks