Yet, Another Loss Value Question 😁

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Yet, Another Loss Value Question 😁

Post by MaxHunter »

I'm at 975,000it's and I can't get the loss value below .071. In over 150+K it's been working in the .07s. It can't have reached convergence yet, can it? It just seems like it should be down in the threes, or the fours before reaching that point. Is this when I freeze the weights? Use No Warp?

And, while we're at it...

The previews are still grey/blueish. Is that normal?

Open to opinions?

Using a Phase A Stojo/Icarus Mix 256 Model

Last edited by torzdf on Tue Feb 14, 2023 1:10 pm, edited 2 times in total.
User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

MaxHunter wrote: Tue Sep 06, 2022 5:28 am

I'm at 975,000it's and I can't get the loss value below .071. In over 150+K it's been working in the .07s. It can't have reached convergence yet, can it? It just seems like it should be down in the threes, or the fours before reaching that point. Is this when I freeze the weights? Use No Warp?

Loss values are meaningless. All that matters is that they are going down. Don't look at the raw numbers

The previews are still grey/blueish. Is that normal?

Not sure what you mean by this. Are they all a solid color? If so, you model is not working.

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

This is what I'm talking about. The image in the left is the training reconstruction, the image on the right is the swap preview. All of my preview images are like this. I just figured it had something to do with one of the loss functions. It doesn't seem to affect the actual swap though.

Also, since I have your attention. Has anyone considered having a "convergence alert"? Maybe an automatic alert that warns the users that loss hasn't dropped in "x" amount of iterations? I mean, if you had something like that think of all the problems and questions about loss values that would cease from the noobs like me. 😆😁

Attachments
Preview_20220906_204141_edited.jpg
Preview_20220906_204141_edited.jpg (102.08 KiB) Viewed 4155 times
Last edited by MaxHunter on Wed Sep 07, 2022 7:24 pm, edited 1 time in total.
User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

I think this is an RGB/BGR color order issue with the previews. I will investigate

My word is final

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

This should be fixed in latest commit. Please update

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

I updated and it seems to still be there. Maybe the changes will take effect on a new model? I'll assume I shouldn't worry too much about it because it hasn't been an issue so far when swapping. I think a question I do have is, will this affect the color of the model?

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

No. It is just the preview window, it's not model connected. Bit odd that an update didn't fix it, I will have to double check.

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

I started training w/DNY512 and encoder fs original, rather than v2_l and the Stojo/Icarus Model. But I also turned off the learn Mask option. I'm beginning to wonder if the learn Mask is causing a lot of the problems, and one of the problems I think it's causing is this discoloration. Everything else is the same, loss functions, etc.

I haven't had any problems with ooms, nans, or this discoloration that I was having using the Stojo setting. Is it possible the Learn Mask could be causing problems?

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

It shouldn't make a difference.

The reason for the color weirdness is that most models are in BGR color order (openCVs color format) but some are in RGB order (most keras encoders), so we have to display the previews correctly (as I said, the model is fine, this just impact previews). Basically, in those examples you give above, the red and blue channels have been switched on the background and in the NN generated face. The image which shows the original face is in the correct channel order.

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

Oh my gosh, thanks for explaining that fully. I know it's hard and tedious to answer all these questions, but when you explain what's happening "under the hood," it makes it so much easier to understand other questions not asked.

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Bug - RGB/BRG

Post by MaxHunter »

Hate to add another post on this, as we dealt with this at an earlier time but I can't find the old post.

I started a new model based on STOJO, at 512, using efficienetv2 -L (a 23Gb model 😯.) It seems the color scale hasn't been fixed. I brought this up months ago but moved on to the DNY512 model. Now that I'm back I was wondering if anyone can fix it because it makes it really hard to figure out when it's done.

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Bug - RGB/BRG

Post by torzdf »

MaxHunter wrote: Wed Feb 01, 2023 11:33 pm

Hate to add another post on this, as we dealt with this at an earlier time but I can't find the old post.

I started a new model based on STOJO, at 512, using efficienetv2 -L (a 23Gb model 😯.) It seems the color scale hasn't been fixed. I brought this up months ago but moved on to the DNY512 model. Now that I'm back I was wondering if anyone can fix it because it makes it really hard to figure out when it's done.

This issue was fixed, quite some time ago.

Are you up to date? If so please provide new screengrabs

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

Yep. Up to date.
Here you go...
(Also, in case anyone cares, this is STOJO - with a few modifications - at 512, 100K in. I'm shocked at how good it looks already!)

Attachments
Screenshot 2023-02-01 203607.png
Screenshot 2023-02-01 203607.png (66.2 KiB) Viewed 3389 times
Last edited by MaxHunter on Thu Feb 02, 2023 1:41 am, edited 3 times in total.
User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

I don't know if this helps, but I just noticed there's a difference between these pics and previous pics. You'll notice only the face is off this time, where as before the entire photo was off. Not sure if that helps. 🙂 Also, here are my parameters (And if you see anything you'd change please let me know.):

Code: Select all

================= Configs ==================

--------- train.ini ---------

[global]
centering:                face
coverage:                 87.5
icnr_init:                True
conv_aware_init:          True
optimizer:                adam
learning_rate:            1e-05
epsilon_exponent:         -7
autoclip:                 True
reflect_padding:          False
allow_growth:             False
mixed_precision:          True
nan_protection:           True
convert_batchsize:        20

[global.loss]
loss_function:            ms_ssim
loss_function_2:          logcosh
loss_weight_2:            50
loss_function_3:          lpips_vgg16
loss_weight_3:            35
loss_function_4:          ffl
loss_weight_4:            100
mask_loss_function:       mse
eye_multiplier:           3
mouth_multiplier:         4
penalized_mask_loss:      True
mask_type:                bisenet-fp_face
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               True

[model.phaze_a]
output_size:              512
shared_fc:                none
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           True
enc_architecture:         efficientnet_v2_l
enc_scaling:              100
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          none
bottleneck_size:          512
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1280
fc_max_filters:           1280
fc_dimensions:            8
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upscale_hybrid
fc_upsamples:             1
fc_upsample_filters:      1280
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       upscale_hybrid
dec_upscales_in_fc:       0
dec_norm:                 none
dec_min_filters:          160
dec_max_filters:          640
dec_slope_mode:           full
dec_filter_slope:         -0.33
dec_res_blocks:           1
dec_output_kernel:        3
dec_gaussian:             True
dec_skip_last_residual:   False
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
fs_original_use_alt:      False
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4
Last edited by torzdf on Sat Feb 04, 2023 2:56 am, edited 3 times in total.
User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

Added bug tag, Will look over the weekend

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: Yet, Another Loss Value Question 😁

Post by MaxHunter »

Just in case this was supposed to be fixed in the past couple updates. Colors are still messed up. 🙂

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

Not checked yet. Will update this thread when I do.

My word is final

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Yet, Another Loss Value Question 😁

Post by torzdf »

Fixed.

My word is final

Locked