Page 1 of 1

Villian Training always reaches a point where the output just becomes white face

Posted: Sat Sep 21, 2024 7:59 am
by queenkong

I've been trying to train a swap using villain but the model after a few thousand iterations just converges to outputting a completely white face.

Any tips to stop this? Thanks!


Re: Villian Training always reaches a point where the output just becomes white face

Posted: Tue Oct 01, 2024 3:25 pm
by torzdf

This is not an issue I have seen, but it sounds like the model has collapsed.

Roll back 50k iterations and lower the learning rate


Re: Villian Training always reaches a point where the output just becomes white face

Posted: Tue Oct 22, 2024 1:21 pm
by queenkong

Thanks for the reply!

I restarted the whole thing and tried rolling back learning rate to 1e-6 and it still converges to white. I reduced my batch size to 4 as well.

Do you have any other tips? Thanks!


Re: Villian Training always reaches a point where the output just becomes white face

Posted: Sat Nov 16, 2024 3:30 am
by torzdf

I do not.

At this point, it may be a hardware or data related issue.

Make sure that any overclocks (including stock overclocks) are disabled on your GPU.

Beyond that, I could not say, sorry.