Page 1 of 1

Why hasn't it started training 2 hours after reading data and alignments

Posted: Wed Aug 05, 2020 2:02 pm
by Lanyu

The status showed that the training process was still running. This status had last for nearly 2 hours. I was confused whether it is normal? Thanks much for your time.

Info:
……
08/05/2020 20:36:56 INFO Reading alignments from: 'XXX\alignments.fsa'
08/05/2020 20:36:56 INFO Reading alignments from: 'XXXX\alignments.fsa'
08/05/2020 20:37:07 INFO Enabled TensorBoard Logging
(No more info)

Status:Executing - train.py


Re: Why hasn't it started training 2 hours after reading data and alignments

Posted: Wed Aug 05, 2020 3:01 pm
by abigflea

Curious. Let's start with the basic.
What video card and vram?
What batch size did you use?


Re: Why hasn't it started training 2 hours after reading data and alignments

Posted: Thu Aug 06, 2020 5:48 am
by Lanyu
abigflea wrote: Wed Aug 05, 2020 3:01 pm

Curious. Let's start with the basic.
What video card and vram?
What batch size did you use?

Thank you for your reply. I'm sure here is a problem but I can't figure it out…

Video card: GeForce GTX 1050Ti
Vram: 8G
Batch size: 16


Re: Why hasn't it started training 2 hours after reading data and alignments

Posted: Thu Aug 06, 2020 6:32 am
by Lanyu

I have figured out this problem. The reason might be that my video card dosen't have enough memory.

When I didn't check all vram saving strategies, the process would be interrupted.
When I checked only "Memory saving gradients", then I met the problem I described.
When I checked both "Memory saving gradients" and "Optimizer Savings", the training process started soon.

Thanks for abigflea's reply.


Re: Why hasn't it started training 2 hours after reading data and alignments

Posted: Thu Aug 06, 2020 6:50 am
by torzdf

An 8GB card should be more than enough to run Faceswap, but it is model dependent. It also doesn't normally just 'hang', it will usually error out if you don't have enough VRAM.

Glad you got it working though,


Re: Why hasn't it started training 2 hours after reading data and alignments

Posted: Thu Aug 06, 2020 3:17 pm
by abigflea

Thought the 1050ti was 4gb.
There is a 8gb version?