Screendoor like image when training

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
pilipinoguy
Posts: 9
Joined: Tue Mar 08, 2022 5:14 am
Has thanked: 8 times

Screendoor like image when training

Post by pilipinoguy »

Hi,

Can you have a look at my screenshot below. There is a screendoor like red effect on top of my image when training. I invested a lot of hours(700k iterations) on this dfaker model. Is this model dead. Thank you

by torzdf » Fri Mar 25, 2022 11:45 am

That looks to me like the model is collapsing. DLight?

If so, then there isn't much you can do to save the current version of the model. You;d need to roll back to significantly prior to these artefacts showing, lower the learning rate and continue.

Go to full post
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Screendoor like image when training

Post by torzdf »

That looks to me like the model is collapsing. DLight?

If so, then there isn't much you can do to save the current version of the model. You;d need to roll back to significantly prior to these artefacts showing, lower the learning rate and continue.

My word is final

User avatar
pilipinoguy
Posts: 9
Joined: Tue Mar 08, 2022 5:14 am
Has thanked: 8 times

Re: Screendoor like image when training

Post by pilipinoguy »

Houston we got a problem. It Nan'ed eventually.

Will be doing your suggestion, good thing it autosaves every 50k for me. Rolling back to 350k. Lowering the training rate and TRAINNN!!!

Thanks

Locked