can't run two processes simultaneously

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
korrupt78
Posts: 50
Joined: Wed Jan 29, 2020 1:34 am
Has thanked: 2 times
Been thanked: 1 time

can't run two processes simultaneously

Post by korrupt78 »

Just got a fancy new 3060 w/12GB RAM, and everything's working fine - but I can't do two things at the same time.

If I run a training process, then (in another tab) start the GUI to work on cleaning up some alignment files (Tools -> Manual), the second task crashes with tensorflow.python.framework.errors_impl.ResourceExhaustedError.

If I do the opposite - start the GUI work first, then start training job via CLI - same thing. The second task I start fails to run.

Is this normal? Am I only supposed to have one thing using my CUDA cores at a time?

I tried playing with the more obvious options in config/*.ini, like allow_growth and convert_batchsize, but it made no difference.

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: can't run two processes simultaneously

Post by torzdf »

It is doable, it's not straightforward though as Tensorflow likes to reserve all the VRAM it can find.

My word is final

User avatar
PooOnBirbs
Posts: 4
Joined: Thu Oct 07, 2021 6:41 pm

Re: can't run two processes simultaneously

Post by PooOnBirbs »

I also wanted to train while I Tools>Manual cleaned alignments (the .FSA file). I attempted to run 2 instances of FaceSwap. This obviously didn't work.
Next I created a new dummy user profile, installed FaceSwap under that, started training, and then returned to my main account to finish cleaning the FSA (FaceSwap was open and running during the install to the dummy account). This seemed to work fine until it didn't. Somehow the directory pointers (environment variables?) bled over from the other account into my main account. If I remember correctly - this only happened after a restart/reboot. I actually ended up having to reinstall FaceSwap on my main account just to get the directories (again, environment variables?) straightened back out.
I think this way has potential to allow you to have a second instance of FaceSwap even by simply using the "Run as..." feature of Windows... if you can sort out the environment variables issues I ran into.

WIN10
FaceSwap version updated 26JUN2022

Locked