Villian Training always reaches a point where the output just becomes white face

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Post Reply
User avatar
queenkong
Posts: 3
Joined: Wed Sep 18, 2024 12:56 pm

Villian Training always reaches a point where the output just becomes white face

Post by queenkong »

I've been trying to train a swap using villain but the model after a few thousand iterations just converges to outputting a completely white face.

Any tips to stop this? Thanks!

User avatar
torzdf
Posts: 2761
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 141 times
Been thanked: 643 times

Re: Villian Training always reaches a point where the output just becomes white face

Post by torzdf »

This is not an issue I have seen, but it sounds like the model has collapsed.

Roll back 50k iterations and lower the learning rate

My word is final

User avatar
queenkong
Posts: 3
Joined: Wed Sep 18, 2024 12:56 pm

Re: Villian Training always reaches a point where the output just becomes white face

Post by queenkong »

Thanks for the reply!

I restarted the whole thing and tried rolling back learning rate to 1e-6 and it still converges to white. I reduced my batch size to 4 as well.

Do you have any other tips? Thanks!

User avatar
torzdf
Posts: 2761
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 141 times
Been thanked: 643 times

Re: Villian Training always reaches a point where the output just becomes white face

Post by torzdf »

I do not.

At this point, it may be a hardware or data related issue.

Make sure that any overclocks (including stock overclocks) are disabled on your GPU.

Beyond that, I could not say, sorry.

My word is final

Post Reply