Page 1 of 1

Crash When Resuming Training

Posted: Fri Jan 10, 2020 3:42 am
by chancerisker

I have a model that i trained over 100000its, now when i try to resume training it crashes with a memory error. Does resuming training take more memory than just training a model from the start? Sometimes it goes for a few cycles then throws an exception. Any help is appreciated, thanks.


Re: Crash When Resuming Training

Posted: Sun Jan 12, 2020 5:27 am
by bryanlyon

It sounds like you were marginally capable at that BS. Sometimes just by luck you run at a batch size once that wont work again later. Try a lower BS or close some applications that may be using vram (Browsers and Video Players are good examples).