Gray faces after 200k iterations

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
ludka
Posts: 3
Joined: Sun Oct 17, 2021 7:38 pm

Gray faces after 200k iterations

Post by ludka »

Hi. I trying to learning Villaing model on Nvidia 3090. But after 200k iterations faces still gray. Then I tryied restart this model and another models but have the same result.
What I do wrong or how I can fix this?
Thanks.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Gray faces after 200k iterations

Post by torzdf »

This is model corruption/model not learning. It will be an issue with your specific setup as this is not a pattern I have seen before. Check that you have no overclocks (even factory overclocks) on your GPU.

[mention]bryanlyon[/mention] may have other suggestions.

My word is final

User avatar
ludka
Posts: 3
Joined: Sun Oct 17, 2021 7:38 pm

Re: Gray faces after 200k iterations

Post by ludka »

I haven't overclock videocard.

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Gray faces after 200k iterations

Post by bryanlyon »

Corruption can happen randomly (though overclocks make it more likely). Your only choice is to revert to an older version and continue training from there.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Gray faces after 200k iterations

Post by torzdf »

I think the issue here is it never isn't grey... which would suggest O/C issues imho.

My word is final

User avatar
ludka
Posts: 3
Joined: Sun Oct 17, 2021 7:38 pm

Re: Gray faces after 200k iterations

Post by ludka »

I tried to use CPU instead of GPU. And it work. On GPU faces are gray always: on 1st , 2nd, 3rd... iteration. On all variations of settings and algorithm. Looks like problem somewhere on GPU.

Locked