I'm at 975,000it's and I can't get the loss value below .071. In over 150+K it's been working in the .07s. It can't have reached convergence yet, can it? It just seems like it should be down in the threes, or the fours before reaching that point. Is this when I freeze the weights? Use No Warp?
And, while we're at it...
The previews are still grey/blueish. Is that normal?
Open to opinions?
Using a Phase A Stojo/Icarus Mix 256 Model
Last edited by torzdf on Tue Feb 14, 2023 1:10 pm, edited 2 times in total.
I'm at 975,000it's and I can't get the loss value below .071. In over 150+K it's been working in the .07s. It can't have reached convergence yet, can it? It just seems like it should be down in the threes, or the fours before reaching that point. Is this when I freeze the weights? Use No Warp?
Loss values are meaningless. All that matters is that they are going down. Don't look at the raw numbers
The previews are still grey/blueish. Is that normal?
Not sure what you mean by this. Are they all a solid color? If so, you model is not working.
This is what I'm talking about. The image in the left is the training reconstruction, the image on the right is the swap preview. All of my preview images are like this. I just figured it had something to do with one of the loss functions. It doesn't seem to affect the actual swap though.
Also, since I have your attention. Has anyone considered having a "convergence alert"? Maybe an automatic alert that warns the users that loss hasn't dropped in "x" amount of iterations? I mean, if you had something like that think of all the problems and questions about loss values that would cease from the noobs like me.
Attachments
Preview_20220906_204141_edited.jpg (102.08 KiB) Viewed 5196 times
Last edited by MaxHunter on Wed Sep 07, 2022 7:24 pm, edited 1 time in total.
I updated and it seems to still be there. Maybe the changes will take effect on a new model? I'll assume I shouldn't worry too much about it because it hasn't been an issue so far when swapping. I think a question I do have is, will this affect the color of the model?
I started training w/DNY512 and encoder fs original, rather than v2_l and the Stojo/Icarus Model. But I also turned off the learn Mask option. I'm beginning to wonder if the learn Mask is causing a lot of the problems, and one of the problems I think it's causing is this discoloration. Everything else is the same, loss functions, etc.
I haven't had any problems with ooms, nans, or this discoloration that I was having using the Stojo setting. Is it possible the Learn Mask could be causing problems?
The reason for the color weirdness is that most models are in BGR color order (openCVs color format) but some are in RGB order (most keras encoders), so we have to display the previews correctly (as I said, the model is fine, this just impact previews). Basically, in those examples you give above, the red and blue channels have been switched on the background and in the NN generated face. The image which shows the original face is in the correct channel order.
Oh my gosh, thanks for explaining that fully. I know it's hard and tedious to answer all these questions, but when you explain what's happening "under the hood," it makes it so much easier to understand other questions not asked.
Hate to add another post on this, as we dealt with this at an earlier time but I can't find the old post.
I started a new model based on STOJO, at 512, using efficienetv2 -L (a 23Gb model .) It seems the color scale hasn't been fixed. I brought this up months ago but moved on to the DNY512 model. Now that I'm back I was wondering if anyone can fix it because it makes it really hard to figure out when it's done.
Hate to add another post on this, as we dealt with this at an earlier time but I can't find the old post.
I started a new model based on STOJO, at 512, using efficienetv2 -L (a 23Gb model .) It seems the color scale hasn't been fixed. I brought this up months ago but moved on to the DNY512 model. Now that I'm back I was wondering if anyone can fix it because it makes it really hard to figure out when it's done.
This issue was fixed, quite some time ago.
Are you up to date? If so please provide new screengrabs
Yep. Up to date.
Here you go...
(Also, in case anyone cares, this is STOJO - with a few modifications - at 512, 100K in. I'm shocked at how good it looks already!)
Attachments
Screenshot 2023-02-01 203607.png (66.2 KiB) Viewed 4430 times
Last edited by MaxHunter on Thu Feb 02, 2023 1:41 am, edited 3 times in total.
I don't know if this helps, but I just noticed there's a difference between these pics and previous pics. You'll notice only the face is off this time, where as before the entire photo was off. Not sure if that helps. Also, here are my parameters (And if you see anything you'd change please let me know.):