Why hasn't it started training 2 hours after reading data and alignments

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Lanyu
Posts: 3
Joined: Wed Aug 05, 2020 1:35 pm

Why hasn't it started training 2 hours after reading data and alignments

Post by Lanyu »

The status showed that the training process was still running. This status had last for nearly 2 hours. I was confused whether it is normal? Thanks much for your time.

Info:
……
08/05/2020 20:36:56 INFO Reading alignments from: 'XXX\alignments.fsa'
08/05/2020 20:36:56 INFO Reading alignments from: 'XXXX\alignments.fsa'
08/05/2020 20:37:07 INFO Enabled TensorBoard Logging
(No more info)

Status:Executing - train.py

User avatar
abigflea
Posts: 182
Joined: Sat Feb 22, 2020 10:59 pm
Answers: 2
Has thanked: 20 times
Been thanked: 62 times

Re: Why hasn't it started training 2 hours after reading data and alignments

Post by abigflea »

Curious. Let's start with the basic.
What video card and vram?
What batch size did you use?

:o I dunno what I'm doing :shock:
2X RTX 3090 : RTX 3080 : RTX: 2060 : 2x RTX 2080 Super : Ghetto 1060

User avatar
Lanyu
Posts: 3
Joined: Wed Aug 05, 2020 1:35 pm

Re: Why hasn't it started training 2 hours after reading data and alignments

Post by Lanyu »

abigflea wrote: Wed Aug 05, 2020 3:01 pm

Curious. Let's start with the basic.
What video card and vram?
What batch size did you use?

Thank you for your reply. I'm sure here is a problem but I can't figure it out…

Video card: GeForce GTX 1050Ti
Vram: 8G
Batch size: 16

User avatar
Lanyu
Posts: 3
Joined: Wed Aug 05, 2020 1:35 pm

Re: Why hasn't it started training 2 hours after reading data and alignments

Post by Lanyu »

I have figured out this problem. The reason might be that my video card dosen't have enough memory.

When I didn't check all vram saving strategies, the process would be interrupted.
When I checked only "Memory saving gradients", then I met the problem I described.
When I checked both "Memory saving gradients" and "Optimizer Savings", the training process started soon.

Thanks for abigflea's reply.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Why hasn't it started training 2 hours after reading data and alignments

Post by torzdf »

An 8GB card should be more than enough to run Faceswap, but it is model dependent. It also doesn't normally just 'hang', it will usually error out if you don't have enough VRAM.

Glad you got it working though,

My word is final

User avatar
abigflea
Posts: 182
Joined: Sat Feb 22, 2020 10:59 pm
Answers: 2
Has thanked: 20 times
Been thanked: 62 times

Re: Why hasn't it started training 2 hours after reading data and alignments

Post by abigflea »

Thought the 1050ti was 4gb.
There is a 8gb version?

:o I dunno what I'm doing :shock:
2X RTX 3090 : RTX 3080 : RTX: 2060 : 2x RTX 2080 Super : Ghetto 1060

Locked