Page 1 of 1
Villian Training always reaches a point where the output just becomes white face
Posted: Sat Sep 21, 2024 7:59 am
by queenkong
I've been trying to train a swap using villain but the model after a few thousand iterations just converges to outputting a completely white face.
Any tips to stop this? Thanks!
Re: Villian Training always reaches a point where the output just becomes white face
Posted: Tue Oct 01, 2024 3:25 pm
by torzdf
This is not an issue I have seen, but it sounds like the model has collapsed.
Roll back 50k iterations and lower the learning rate
Re: Villian Training always reaches a point where the output just becomes white face
Posted: Tue Oct 22, 2024 1:21 pm
by queenkong
Thanks for the reply!
I restarted the whole thing and tried rolling back learning rate to 1e-6 and it still converges to white. I reduced my batch size to 4 as well.
Do you have any other tips? Thanks!
Re: Villian Training always reaches a point where the output just becomes white face
Posted: Sat Nov 16, 2024 3:30 am
by torzdf
I do not.
At this point, it may be a hardware or data related issue.
Make sure that any overclocks (including stock overclocks) are disabled on your GPU.
Beyond that, I could not say, sorry.