After 177 hours of training, I woke up to....

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
NOOCH
Posts: 1
Joined: Thu Mar 09, 2023 12:55 pm

After 177 hours of training, I woke up to....

Post by NOOCH »

I've been training for 177 hours. This morning I woke up to a message that it was unable to save (presumably in the same folder where I've been saving it). I checked the folder memory and it has over 300 GB remaining.
Also had to unplug my computer to get it to shut down/reboot -- everything was stuck.
I tried to restart the project and it fails. I even tried reloading all the folders and same result.
I'm at a loss here and I have no idea what to do. I'd rather not start over after 177 hours (in fact, I think I would just throw in the towel on this ever working).
Is there any way to recover or reset to the previous day/iteration?
Help!

Last edited by NOOCH on Thu Mar 09, 2023 1:30 pm, edited 1 time in total.
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: After 177 hours of training, I woke up to....

Post by torzdf »

Looks like model file corruption for some reason.

See here to restore from backup/snapshot and get back on your way:
viewtopic.php?p=8361#p8361

My word is final

Locked