Search found 3 matches

by mgolvach
Mon Jun 21, 2021 1:45 pm
Forum: Training Support
Topic: Training exits with no error message - Max iterations not reached
Replies: 2
Views: 7521

Re: Training exits with no error message - Max iterations not reached

I feel silly for having written, but hopefully this will help someone out. I'll have to reboot or just stop and start faceswap more often on my setup, I think. It ended up accumulating too much memory and not releasing it over time. The main issue was python resource exhaustion (virtual memory used ...
by mgolvach
Mon Jun 21, 2021 1:33 pm
Forum: Training Support
Topic: Training exits with no error message - Max iterations not reached
Replies: 2
Views: 7521

Training exits with no error message - Max iterations not reached

Hi, I looked around and couldnt' find this anywhere. Probably nothing to worry about, but I was curious. Last night (I train overnight), my training stopped for no reason I can figure out. The end of the faceswap.log looks like the end of the output of my training: 06/21/2021 06:38:52 MainProcess _t...
by mgolvach
Sun May 17, 2020 2:07 am
Forum: Convert Support
Topic: OOM out of memory during convert but not training
Replies: 14
Views: 17669

Re: OOM out of memory during convert but not training

Just in case it helps, I had a similar situation. Training with DFL-SAE at 128px (max I could do) was working fine, but conversion gave me the error: Resource exhausted: OOM when allocating tensor with shape[16,130,130,126] and type float... I had turned on "allow growth" for conversion, b...