Yet, Another Loss Value Question 😁

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Post Reply
User avatar
MaxHunter
Posts: 47
Joined: Thu May 26, 2022 6:02 am
Has thanked: 42 times
Been thanked: 6 times

Yet, Another Loss Value Question 😁

Post by MaxHunter »

I'm at 975,000it's and I can't get the loss value below .071. In over 150+K it's been working in the .07s. It can't have reached convergence yet, can it? It just seems like it should be down in the threes, or the fours before reaching that point. Is this when I freeze the weights? Use No Warp?

And, while we're at it...

The previews are still grey/blueish. Is that normal?

Open to opinions?

Using a Phase A Stojo/Icarus Mix 256 Model


User avatar
torzdf
Posts: 1979
Joined: Fri Jul 12, 2019 12:53 am
Answers: 139
Has thanked: 90 times
Been thanked: 433 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

MaxHunter wrote: Tue Sep 06, 2022 5:28 am

I'm at 975,000it's and I can't get the loss value below .071. In over 150+K it's been working in the .07s. It can't have reached convergence yet, can it? It just seems like it should be down in the threes, or the fours before reaching that point. Is this when I freeze the weights? Use No Warp?

Loss values are meaningless. All that matters is that they are going down. Don't look at the raw numbers

The previews are still grey/blueish. Is that normal?

Not sure what you mean by this. Are they all a solid color? If so, you model is not working.

My word is final


User avatar
MaxHunter
Posts: 47
Joined: Thu May 26, 2022 6:02 am
Has thanked: 42 times
Been thanked: 6 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

This is what I'm talking about. The image in the left is the training reconstruction, the image on the right is the swap preview. All of my preview images are like this. I just figured it had something to do with one of the loss functions. It doesn't seem to affect the actual swap though.

Also, since I have your attention. Has anyone considered having a "convergence alert"? Maybe an automatic alert that warns the users that loss hasn't dropped in "x" amount of iterations? I mean, if you had something like that think of all the problems and questions about loss values that would cease from the noobs like me. 😆😁

Attachments
Preview_20220906_204141_edited.jpg
Preview_20220906_204141_edited.jpg (102.08 KiB) Viewed 473 times
Last edited by MaxHunter on Wed Sep 07, 2022 7:24 pm, edited 1 time in total.

User avatar
torzdf
Posts: 1979
Joined: Fri Jul 12, 2019 12:53 am
Answers: 139
Has thanked: 90 times
Been thanked: 433 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

I think this is an RGB/BGR color order issue with the previews. I will investigate

My word is final


User avatar
torzdf
Posts: 1979
Joined: Fri Jul 12, 2019 12:53 am
Answers: 139
Has thanked: 90 times
Been thanked: 433 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

This should be fixed in latest commit. Please update

My word is final


User avatar
MaxHunter
Posts: 47
Joined: Thu May 26, 2022 6:02 am
Has thanked: 42 times
Been thanked: 6 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

I updated and it seems to still be there. Maybe the changes will take effect on a new model? I'll assume I shouldn't worry too much about it because it hasn't been an issue so far when swapping. I think a question I do have is, will this affect the color of the model?


User avatar
torzdf
Posts: 1979
Joined: Fri Jul 12, 2019 12:53 am
Answers: 139
Has thanked: 90 times
Been thanked: 433 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

No. It is just the preview window, it's not model connected. Bit odd that an update didn't fix it, I will have to double check.

My word is final


User avatar
MaxHunter
Posts: 47
Joined: Thu May 26, 2022 6:02 am
Has thanked: 42 times
Been thanked: 6 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

I started training w/DNY512 and encoder fs original, rather than v2_l and the Stojo/Icarus Model. But I also turned off the learn Mask option. I'm beginning to wonder if the learn Mask is causing a lot of the problems, and one of the problems I think it's causing is this discoloration. Everything else is the same, loss functions, etc.

I haven't had any problems with ooms, nans, or this discoloration that I was having using the Stojo setting. Is it possible the Learn Mask could be causing problems?


User avatar
torzdf
Posts: 1979
Joined: Fri Jul 12, 2019 12:53 am
Answers: 139
Has thanked: 90 times
Been thanked: 433 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

It shouldn't make a difference.

The reason for the color weirdness is that most models are in BGR color order (openCVs color format) but some are in RGB order (most keras encoders), so we have to display the previews correctly (as I said, the model is fine, this just impact previews). Basically, in those examples you give above, the red and blue channels have been switched on the background and in the NN generated face. The image which shows the original face is in the correct channel order.

My word is final


User avatar
MaxHunter
Posts: 47
Joined: Thu May 26, 2022 6:02 am
Has thanked: 42 times
Been thanked: 6 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

Oh my gosh, thanks for explaining that fully. I know it's hard and tedious to answer all these questions, but when you explain what's happening "under the hood," it makes it so much easier to understand other questions not asked.


Post Reply