" Average loss since last save: 282 octillion" or some times "nan" ???

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
cosmico
Posts: 95
Joined: Sat Jan 18, 2020 6:32 pm
Answers: 0
Has thanked: 13 times
Been thanked: 35 times

" Average loss since last save: 282 octillion" or some times "nan" ???

Post by cosmico »

.
.
My training seems to be going pretty well but I couldn't help but notice this in the lower log:

09/26/2020 15:22:00 INFO [Saved models] - Average loss since last save: face_a: 0.12294, face_b: 0.18499

09/26/2020 15:24:16 INFO [Saved models] - Average loss since last save: face_a: 0.12187, face_b: 0.18332

09/26/2020 15:26:31 INFO [Saved models] - Average loss since last save: face_a: 28232528235788282727801815040.00000, face_b: 0.18421

09/26/2020 15:28:48 INFO [Saved models] - Average loss since last save: face_a: 0.12402, face_b: 0.18483

09/26/2020 15:31:03 INFO [Saved models] - Average loss since last save: face_a: 0.12242, face_b: 0.18711

When you look at the graph, theres a massive spike corresponding with that 282 octillion. Sometimes I also get something like this:

09/26/2020 15:58:01 INFO [Saved models] - Average loss since last save: face_a: 0.12202, face_b: 0.18423

09/26/2020 16:00:16 INFO [Saved models] - Average loss since last save: face_a: 0.12158, face_b: 0.18298

[16:02:12] [#91980] Loss A: nan, Loss B: 0.19720
09/26/2020 16:02:31 INFO [Saved models] - Average loss since last save: face_a: nan, face_b: 0.18399

09/26/2020 16:04:45 INFO [Saved models] - Average loss since last save: face_a: 0.12137, face_b: 0.18484

09/26/2020 16:06:59 INFO [Saved models] - Average loss since last save: face_a: 0.12248, face_b: 0.18268

Any Idea what this could be or what it could mean? My training does seem to becoming along just fine, however since I have updated a couple a days ago, faceswap has been pretty buggy and thinking alot of my safe previously made models are corrupted

Top
User avatar
cosmico
Posts: 95
Joined: Sat Jan 18, 2020 6:32 pm
Answers: 0
Has thanked: 13 times
Been thanked: 35 times

Re: " Average loss since last save: 282 octillion" or some times "nan" ???

Post by cosmico »

Ummmmmmmmmmmmm
Also I have sessions 1, 4 ,and 6 two times. and those 2 session number 6's start at the same exact time but by looking at the eg's and iterations, they appear to be different sessions. Also One of them has a eg/s of 140 which is real nice, but I'm 10,000% sure the best my computer can do on this model size and settings is 70-80 eg/s. Since taking this screenshot and in the middle of writing this post, I checked again and a third session 6 appear with unrealistic numbers.
....Now theres 4.
.
Oh boy.
Oh Jeez.
I think my faceswap is about to go critical mass.

Image
training previews still look pretty good though

Top
Locked

Return to “Training Support”