Page 1 of 1
First attempt: Weird learning graph
Posted: Sat Nov 14, 2020 9:32 pm
by dococt
Hello guys!
First, please forgive me if the question is stupid, but I'm a total newbie.
I tried to do my best, I extracted the faces, ordered and cleared all of them
Now, after 295.000 iteration, this is my graph:
(https://pasteboard.co/JAnpVUv.png)
I believe all the peaks I have are because my model is not good (I dunno why, I cleaned it as much as I can).
But what I really don't understand is the last part of the graph, where the red graph is not at the bottom anymore...
Any idea about what that means?
Last but now least, suddenly both the "Original > Original" columns and the "Swap > Original" contains just a red filled box in place of the real face.
What is happening? Until some iteration ago I could see the face, even if blurred.
Thank you!
Re: First attempt: Weird learning graph
Posted: Mon Nov 16, 2020 11:01 am
by torzdf
This is model corruption and is, sadly, unrecoverable.
You can try to use the restore tool to restore from the last good backup, or continue training from a previous snapshot.
Re: First attempt: Weird learning graph
Posted: Mon Nov 16, 2020 1:32 pm
by wentdot
torzdf wrote: ↑Mon Nov 16, 2020 11:01 am
This is model corruption and is, sadly, unrecoverable.
You can try to use the restore tool to restore from the last good backup, or continue training from a previous snapshot.
It looks like what happened is the huge fluctuations in the loss function knocked it into some stable local minimum. Is this the correct interpretation?
Re: First attempt: Weird learning graph
Posted: Mon Nov 16, 2020 3:54 pm
by bryanlyon
No, the float has gotten "stuck" in a representation of infinity that's used for error states. Local minimums can recover, infinity cannot.