Page 1 of 1

Gray faces after 200k iterations

Posted: Sun Oct 17, 2021 7:46 pm
by ludka

Hi. I trying to learning Villaing model on Nvidia 3090. But after 200k iterations faces still gray. Then I tryied restart this model and another models but have the same result.
What I do wrong or how I can fix this?
Thanks.


Re: Gray faces after 200k iterations

Posted: Tue Oct 19, 2021 11:00 am
by torzdf

This is model corruption/model not learning. It will be an issue with your specific setup as this is not a pattern I have seen before. Check that you have no overclocks (even factory overclocks) on your GPU.

[mention]bryanlyon[/mention] may have other suggestions.


Re: Gray faces after 200k iterations

Posted: Tue Oct 19, 2021 1:22 pm
by ludka

I haven't overclock videocard.


Re: Gray faces after 200k iterations

Posted: Tue Oct 19, 2021 6:09 pm
by bryanlyon

Corruption can happen randomly (though overclocks make it more likely). Your only choice is to revert to an older version and continue training from there.


Re: Gray faces after 200k iterations

Posted: Tue Oct 19, 2021 6:10 pm
by torzdf

I think the issue here is it never isn't grey... which would suggest O/C issues imho.


Re: Gray faces after 200k iterations

Posted: Tue Oct 19, 2021 6:39 pm
by ludka

I tried to use CPU instead of GPU. And it work. On GPU faces are gray always: on 1st , 2nd, 3rd... iteration. On all variations of settings and algorithm. Looks like problem somewhere on GPU.