I have a model that i trained over 100000its, now when i try to resume training it crashes with a memory error. Does resuming training take more memory than just training a model from the start? Sometimes it goes for a few cycles then throws an exception. Any help is appreciated, thanks.
Crash When Resuming Training
Forum rules
Read the FAQs and search the forum before posting a new topic.
This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.
Please mark any answers that fixed your problems so others can find the solutions.
-
bryanlyon
- Site Admin
- Posts: 793
- Joined: Fri Jul 12, 2019 12:49 am
- Location: San Francisco
- Has thanked: 4 times
- Been thanked: 218 times
- Contact:
Re: Crash When Resuming Training
It sounds like you were marginally capable at that BS. Sometimes just by luck you run at a batch size once that wont work again later. Try a lower BS or close some applications that may be using vram (Browsers and Video Players are good examples).