OOM out of memory during convert but not training

Converting source frames into swapped frames
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Post Reply
User avatar
superjj
Posts: 2
Joined: Sat Apr 04, 2020 9:59 pm

OOM out of memory during convert but not training

Post by superjj » Tue Apr 14, 2020 10:27 pm

I've trained my model to about 91k iterations with no crashes. But when I try to convert, faceswap crashes with an OOM error in the logs. Conversion seems to work when I select a small range of frames to convert, maybe 15 frames at a time. But that's barely half a second of video.

Has anyone trained fine, but end up with OOM crashes during conversion?

I'm using a GTX 1650 Super 4gb, and training on the Dlight model with resource-saving options turned on.

Thanks!

User avatar
torzdf
Posts: 554
Joined: Fri Jul 12, 2019 12:53 am
Answers: 86
Has thanked: 17 times
Been thanked: 121 times

Re: OOM out of memory during convert but not training

Post by torzdf » Wed Apr 15, 2020 9:28 am

I have never seen this before, sadly.
My word is final

User avatar
bryanlyon
Site Admin
Posts: 275
Joined: Fri Jul 12, 2019 12:49 am
Answers: 20
Location: San Francisco
Has thanked: 3 times
Been thanked: 78 times
Contact:

Re: OOM out of memory during convert but not training

Post by bryanlyon » Wed Apr 15, 2020 7:59 pm

Were you having to use Allow Growth during training? In which case, you might be running into a weird issue we've noticed on some people's setups.

User avatar
PLAY-911
Posts: 6
Joined: Mon Apr 13, 2020 6:52 pm
Has thanked: 1 time

Re: OOM out of memory during convert but not training

Post by PLAY-911 » Wed Apr 15, 2020 9:08 pm

superjj wrote:
Tue Apr 14, 2020 10:27 pm
I've trained my model to about 91k iterations with no crashes. But when I try to convert, faceswap crashes with an OOM error in the logs. Conversion seems to work when I select a small range of frames to convert, maybe 15 frames at a time. But that's barely half a second of video.

Has anyone trained fine, but end up with OOM crashes during conversion?

I'm using a GTX 1650 Super 4gb, and training on the Dlight model with resource-saving options turned on.

Thanks!
Are you in Windows? I had problems with virtual memory assigned by windows

User avatar
superjj
Posts: 2
Joined: Sat Apr 04, 2020 9:59 pm

Re: OOM out of memory during convert but not training

Post by superjj » Thu Apr 16, 2020 12:14 am

bryanlyon wrote:
Wed Apr 15, 2020 7:59 pm
Were you having to use Allow Growth during training? In which case, you might be running into a weird issue we've noticed on some people's setups.
Yes I had Allow Growth turned on during training.

User avatar
torzdf
Posts: 554
Joined: Fri Jul 12, 2019 12:53 am
Answers: 86
Has thanked: 17 times
Been thanked: 121 times

Re: OOM out of memory during convert but not training

Post by torzdf » Thu Apr 16, 2020 9:52 am

Make sure you select "Allow Growth" for convert too
My word is final

User avatar
mgolvach
Posts: 1
Joined: Sun May 17, 2020 2:01 am
Has thanked: 1 time

Re: OOM out of memory during convert but not training

Post by mgolvach » Sun May 17, 2020 2:07 am

Just in case it helps, I had a similar situation. Training with DFL-SAE at 128px (max I could do) was working fine, but conversion gave me the error:

Resource exhausted: OOM when allocating tensor with shape[16,130,130,126] and type float...

I had turned on "allow growth" for conversion, but found I did not have "allow growth" checked for training. Though it seemed counterintuitive, I turned off (unchecked) "allow growth" for conversion, and that solved the problem.

I think, essentially, with regard to the "allow growth" option, you need to be consistent with training and conversion. If you train with it on (or off), you must do the same for conversion.

This may not be the case for everyone. I'm certain more GPU power would probably solve the problem as well ;)

Thanks for this board's wealth of information and help!

Mike

User avatar
bryanlyon
Site Admin
Posts: 275
Joined: Fri Jul 12, 2019 12:49 am
Answers: 20
Location: San Francisco
Has thanked: 3 times
Been thanked: 78 times
Contact:

Re: OOM out of memory during convert but not training

Post by bryanlyon » Mon May 18, 2020 6:36 pm

Allow_growth does not affect your model in anyway, it only changes how Tensorflow allocates memory. You are likely running into a different issue. But we recommend leaving allow_growth off unless it's absolutely necessary to getting Faceswap running on your system.

Post Reply