Hi. I trying to learning Villaing model on Nvidia 3090. But after 200k iterations faces still gray. Then I tryied restart this model and another models but have the same result.
What I do wrong or how I can fix this?
Thanks.
Gray faces after 200k iterations
Read the FAQs and search the forum before posting a new topic.
This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.
Please mark any answers that fixed your problems so others can find the solutions.
Gray faces after 200k iterations
Re: Gray faces after 200k iterations
This is model corruption/model not learning. It will be an issue with your specific setup as this is not a pattern I have seen before. Check that you have no overclocks (even factory overclocks) on your GPU.
[mention]bryanlyon[/mention] may have other suggestions.
My word is final
- bryanlyon
- Site Admin
- Posts: 793
- Joined: Fri Jul 12, 2019 12:49 am
- Location: San Francisco
- Has thanked: 4 times
- Been thanked: 218 times
- Contact:
Re: Gray faces after 200k iterations
Corruption can happen randomly (though overclocks make it more likely). Your only choice is to revert to an older version and continue training from there.
Re: Gray faces after 200k iterations
I tried to use CPU instead of GPU. And it work. On GPU faces are gray always: on 1st , 2nd, 3rd... iteration. On all variations of settings and algorithm. Looks like problem somewhere on GPU.