FaceB got distorted with red static

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Endmypainpls
Posts: 2
Joined: Sun Apr 25, 2021 10:57 am

FaceB got distorted with red static

Post by Endmypainpls »

Thats a thumbnail from my current preview, for the first like 200k iterations it was fine then it started appearing with this distorting red static, anyone know why/how to fix? Can't work out how to load the project at a prior state which would be useful here

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: FaceB got distorted with red static

Post by torzdf »

This is model corruption (but not corruption like I have seen before).

For this kind of corruption, I would roll back to an earlier snapshot and possibly lower the learning rate a bit.

To roll back to an ealier snapshot, just delete your model folder, copy one of your snapshot folders, and rename the copy to the original model folder name (i.e. remove the _snapshot_xxxxxx from the end of the folder name) and then continue training.

My word is final

User avatar
Endmypainpls
Posts: 2
Joined: Sun Apr 25, 2021 10:57 am

Re: FaceB got distorted with red static

Post by Endmypainpls »

Thank you so much mate, really appreciate that!!

Locked