empty face

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
damingxing
Posts: 23
Joined: Mon Jun 13, 2022 8:55 pm
Has thanked: 6 times

empty face

Post by damingxing »

I came across an issue: the face of the output video is empty. Does anyone happen to know why and how we can mitigate this from happening? I think this happened in the training process since the images from the extract are fine. Is it possible that there is a vram leakage? (not an expert, just guessing)

I have used the same hyperparameters and settings to successfully create the same videos (source and target) before. The issue did not appear.

I am attaching an example from the output video and the last preview.

Thank you!

example from the output video
example from the output video
image.png (213.77 KiB) Viewed 1340 times
example from the last preview
example from the last preview
empty face1.jpg (39.65 KiB) Viewed 1340 times
User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: empty face

Post by torzdf »

Corrupted model.

Lower learning rate + rollback 50k

My word is final

User avatar
damingxing
Posts: 23
Joined: Mon Jun 13, 2022 8:55 pm
Has thanked: 6 times

Re: empty face

Post by damingxing »

Do you happen to know what may cause this? This recently has been happening quite often on my side. Thanks.

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: empty face

Post by torzdf »

It's model collapse. It happens in AI sadly.

My word is final

User avatar
damingxing
Posts: 23
Joined: Mon Jun 13, 2022 8:55 pm
Has thanked: 6 times

Re: empty face

Post by damingxing »

thanks for the response. I realized there are times the model folder was empty, and there is 'model-nf" folder has .h5 and state status files. Do you happen to know why that is? thanks

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: empty face

Post by torzdf »

No, I've never heard of that before, I'm afraid. Sounds very strange indeed.

Another thing to check for (re: model corruption) is to make sure there are no overclocks (including factory overclocks) on your GPU, as this can cause corruption.

My word is final

Locked