Loss briefly spiked, then recovered. What happened?

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Surrogator
Posts: 13
Joined: Sun Jul 14, 2019 11:39 pm
Has thanked: 2 times
Been thanked: 2 times

Loss briefly spiked, then recovered. What happened?

Post by Surrogator »

I just started training for the first time. It was running for about 15 minutes, and I had stepped away from the computer for a moment. When I came back I saw this.

collapse_and_recover.png
collapse_and_recover.png (130.98 KiB) Viewed 3393 times

I think this means that my model collapsed, but it also recovered on its own. I'm pretty sure I would've terminated training if I was present during this 200 iterations.

So, what happened? Did my model collapse and recover? Is this a bad omen?

by torzdf » Tue Jul 16, 2019 10:52 pm

This has happened to me before. I don't like it when I see it, but it should be ok.

To be honest, if it's early in the train I tend to kill my model and start again, because I'm super paranoid.

Go to full post
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Loss briefly spiked, then recovered. What happened?

Post by bryanlyon »

This is usually a sign of an overclocked GPU. The other option is exploding gradients that recovered -- which is far more rare. The fact it was uniform for A and B tells me the GPU is more likely the culprit.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Loss briefly spiked, then recovered. What happened?

Post by torzdf »

This has happened to me before. I don't like it when I see it, but it should be ok.

To be honest, if it's early in the train I tend to kill my model and start again, because I'm super paranoid.

My word is final

User avatar
jhsy1209021
Posts: 4
Joined: Thu Feb 27, 2020 2:26 pm

Peaks on the graph.

Post by jhsy1209021 »

I noticed a interesting phenomenon. When I was training the model, there were some peaks appeared on the graph. Although it just happened several times and had no impact on the model, but its is still wierd. Have anyone known or encountered the same situation before?

Train settings:
Model : Dlight
Mask : Unet-Dfl
Coverage : 62.5%
Conv Aware Init : true

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Peaks on the graph.

Post by bryanlyon »

Perfectly normal and nothing to worry about.

Locked