Page 1 of 1

is my training process frozen?

Posted: Fri Apr 03, 2020 12:31 am
by nerdfrombrazil

I started training this morning and at one point he stopped at 314 iterations. My last preview was updated at 13:00 pm. Process time is stopped at 02:59:03 but the green progress bar keeps going from left to right and I didn't get any error or crash messages. I don't know if the program is stuck or if it just hasn't updated the training. Has this ever happened to any of you? I don't want to stop progress now because sometimes when I try to continue training from before, the program has a lack of memory.


Re: is my training process frozen?

Posted: Fri Apr 03, 2020 10:13 am
by torzdf

If you go to your model folder, and copy the <model_name>_state.json file somewhere, then open the copy, it will tell you how many iterations it has done, so you should be able to tell if it is still incrementing...

Make sure you take a copy rather than open the original (whilst training)