I started training this morning and at one point he stopped at 314 iterations. My last preview was updated at 13:00 pm. Process time is stopped at 02:59:03 but the green progress bar keeps going from left to right and I didn't get any error or crash messages. I don't know if the program is stuck or if it just hasn't updated the training. Has this ever happened to any of you? I don't want to stop progress now because sometimes when I try to continue training from before, the program has a lack of memory.
is my training process frozen?
Forum rules
Read the FAQs and search the forum before posting a new topic.
This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.
Please mark any answers that fixed your problems so others can find the solutions.
-
nerdfrombrazil
- Posts: 2
- Joined: Sat Mar 28, 2020 3:48 pm
- Has thanked: 2 times
is my training process frozen?
If you go to your model folder, and copy the <model_name>_state.json file somewhere, then open the copy, it will tell you how many iterations it has done, so you should be able to tell if it is still incrementing...
Make sure you take a copy rather than open the original (whilst training)
Re: is my training process frozen?
If you go to your model folder, and copy the <model_name>_state.json file somewhere, then open the copy, it will tell you how many iterations it has done, so you should be able to tell if it is still incrementing...
Make sure you take a copy rather than open the original (whilst training)
My word is final