Page 1 of 1
empty face
Posted: Mon Aug 08, 2022 4:01 pm
by damingxing
I came across an issue: the face of the output video is empty. Does anyone happen to know why and how we can mitigate this from happening? I think this happened in the training process since the images from the extract are fine. Is it possible that there is a vram leakage? (not an expert, just guessing)
I have used the same hyperparameters and settings to successfully create the same videos (source and target) before. The issue did not appear.
I am attaching an example from the output video and the last preview.
Thank you!
- example from the output video
- image.png (213.77 KiB) Viewed 1764 times
- example from the last preview
- empty face1.jpg (39.65 KiB) Viewed 1764 times
Re: empty face
Posted: Mon Aug 08, 2022 5:48 pm
by torzdf
Corrupted model.
Lower learning rate + rollback 50k
Re: empty face
Posted: Fri Aug 12, 2022 6:49 pm
by damingxing
Do you happen to know what may cause this? This recently has been happening quite often on my side. Thanks.
Re: empty face
Posted: Sun Aug 14, 2022 9:28 am
by torzdf
It's model collapse. It happens in AI sadly.
Re: empty face
Posted: Sun Aug 14, 2022 1:14 pm
by damingxing
thanks for the response. I realized there are times the model folder was empty, and there is 'model-nf" folder has .h5 and state status files. Do you happen to know why that is? thanks
Re: empty face
Posted: Sun Aug 14, 2022 1:16 pm
by torzdf
No, I've never heard of that before, I'm afraid. Sounds very strange indeed.
Another thing to check for (re: model corruption) is to make sure there are no overclocks (including factory overclocks) on your GPU, as this can cause corruption.