First attempt: Weird learning graph

[dococt]

Hello guys!

First, please forgive me if the question is stupid, but I'm a total newbie.

I tried to do my best, I extracted the faces, ordered and cleared all of them
Now, after 295.000 iteration, this is my graph:
(https://pasteboard.co/JAnpVUv.png)

I believe all the peaks I have are because my model is not good (I dunno why, I cleaned it as much as I can).

But what I really don't understand is the last part of the graph, where the red graph is not at the bottom anymore...

Any idea about what that means?

Last but now least, suddenly both the "Original > Original" columns and the "Swap > Original" contains just a red filled box in place of the real face.

What is happening? Until some iteration ago I could see the face, even if blurred.

Thank you!

[torzdf]

This is model corruption and is, sadly, unrecoverable.

You can try to use the restore tool to restore from the last good backup, or continue training from a previous snapshot.

[wentdot]

This is model corruption and is, sadly, unrecoverable.

You can try to use the restore tool to restore from the last good backup, or continue training from a previous snapshot.

It looks like what happened is the huge fluctuations in the loss function knocked it into some stable local minimum. Is this the correct interpretation?

[bryanlyon]

No, the float has gotten "stuck" in a representation of infinity that's used for error states. Local minimums can recover, infinity cannot.