Strange Red Area/Artifact Creeping into Model

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

Hi there,

We're playing with the StoJo model preset and after reaching over 150,000 iterations - we are noticing some strange red/black artifacts entering the previews.

Here are examples:

ex4.jpg
ex4.jpg (4.87 KiB) Viewed 15179 times
e3.jpg
e3.jpg (3.92 KiB) Viewed 15179 times
ex2.jpg
ex2.jpg (4.37 KiB) Viewed 15179 times

Do not be mistaken - these previews do not have any masks overlaid - the red/black spot is the preview we get. The red areas do seem to be outside the mask area at the moment but could be creeping in as the iterations increase.

Do we need to modify anything - learning rate, epsilon, etc, and rollback?

Any help, suggestions, and advice are greatly appreciated.

Last edited by ianstephens on Fri Aug 13, 2021 6:50 pm, edited 1 time in total.
User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Artifact Creeping into Model

Post by ianstephens »

Also, worth noting (and apologies for the separate post - I could only attach 3 images per post)...

We did notice this anomaly in the graphs:

graph.jpg
graph.jpg (106.02 KiB) Viewed 15178 times

This is a single session graph (not the whole training batch).

It seemed to begin after that anomaly.

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

I should also note, we've not had any NaN reports and do have NaN detection enabled.

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

This is also interesting...

Screen Shot 2021-08-13 at 21.55.45.png
Screen Shot 2021-08-13 at 21.55.45.png (126.12 KiB) Viewed 15172 times
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Strange Red Area/Artifact Creeping into Model

Post by torzdf »

Jumps like that happen sometimes. As long as preview is still improving and it continues to trend down, I wouldn't worry about it too much.

That red area is weird. Not seen it before. It may be related to the jump, it may not. I'm hoping that it doesn't mean your model is dead, but I guess only time will tell. I assume you are training with a mask, and that appears in the masked out area?

As an aside, I pushed an update recently which should decrease the possibility of NaNs in that model. It will only come into effect for new models, so do make sure you are up to date.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

Thanks for the reply [mention]torzdf[/mention].

So after reading your post we decided to let the model run a bit longer. The strange red areas have now turned fluorescent green and have started eating into certain places of the main masked area. We are noticing that these green artifacts are mainly eating into areas that are completely dark/black on the input images. If the face has a white background and is bright it's not as noticeable.

I do presume the model is dead.

What would have caused this? We do have NaN protection enabled. Would it be wise to adjust anything (learning rate etc) before we attempt to start from scratch?

As an aside, I pushed an update recently which should decrease the possibility of NaNs in that model. It will only come into effect for new models, so do make sure you are up to date.

Thank you for that info - we started training this model on the 8th of August - was the NaN update after that? We usually update FS before we start a new model.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Strange Red Area/Artifact Creeping into Model

Post by torzdf »

It may be data driven, may be a combination of images in the batch, may be a power spike... honestly, I really don't know what would have caused the issue.

You can probably still rescue the model by rolling back to a snapshot pre "spike" (i.e. back to c. 150k iters) and continuing from there, monitoring whether the issue happens again.

The update was pushed on 4th August, so you're probably covered from that end. FWIW, I currentlly have a StoJo model training with Mixed Precision at 5e-5 all the way. I am at 1.75m iters (BS 6) and have had no issues at all (which is more than I can say for my original showcase model).

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

Thanks for the reply.

Yes, it is a strange one. Not experienced this before with any of the original models.

We're going to roll back to 150k and see if it spikes again.

We're using the same learning rate (5e-5) and also have mixed-precision enabled. The only difference on our end is that we are running at BS 21.

Worth noting: We do notice this now when resuming a training session (I don't remember this notice popping up when running the initial training session - and possibly resuming sessions after that):

Code: Select all

Config item: 'freeze_layers' has been updated from '['efficientnetb4']' to '['keras_encoder']'
User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

Tried rolling back to 150k twice now and it happens at the exact point every time:

Screen Shot 2021-08-14 at 13.42.37.png
Screen Shot 2021-08-14 at 13.42.37.png (203.91 KiB) Viewed 15074 times

No idea what we can do... any advice is greatly appreciated :)

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Strange Red Area/Artifact Creeping into Model

Post by torzdf »

I'm honestly at a loss. It may be that the issue in the model has already occurred prior to 150k, it just does not represent in the loss until later. I would start the model again and pray. Not much help, I know. I have just never hit this kind of issue before. [mention]bryanlyon[/mention] may have a better idea.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

Thanks [mention]torzdf[/mention] - we will start from scratch and see what happens this time.

Did you see my note above about:

Code: Select all

Config item: 'freeze_layers' has been updated from '['efficientnetb4']' to '['keras_encoder']'

We've just started a brand-new model and this notice didn't appear this time. Just wanted to ensure that this wasn't the problem.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Strange Red Area/Artifact Creeping into Model

Post by torzdf »

Yeah, sorry. That's not an issue and can be ignored. It basically appears because of the way Phaze-A is constructed (multiple encoders) and not wanting to include every possible encoder as a configuration option, hence we store "keras-encoder" in the configuration file, which has to be rewritten to the actual encoder used at runtime.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

Thanks for that [mention]torzdf[/mention] - just wanted to confirm.

Re. the dead model - if I recall correctly when connecting to the machine one morning I noticed it had been restarted. Windows performed an update overnight and took it on itself to restart the machine during the training session. We've disabled auto restart now.

I'm not sure how the model was closed by the automatic restart but I would have presumed it would have been fine. The in-memory data would have been lost and when resumed it would have gone from the last save point.

If that's how it works then it shouldn't have been an issue - but who knows - it could have restarted and killed the process right on a save point.

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

torzdf wrote: Sun Aug 15, 2021 10:01 am

I'm honestly at a loss. It may be that the issue in the model has already occurred prior to 150k, it just does not represent in the loss until later. I would start the model again and pray. Not much help, I know. I have just never hit this kind of issue before. @bryanlyon may have a better idea.

[mention]torzdf[/mention] - just curious to know what mask you used for training? We used the newer Bisenet-FP Face for this particular model with the corruption. I am predicting you used extended?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Strange Red Area/Artifact Creeping into Model

Post by torzdf »

Mask won't cause that kind of issue. However, I used vgg-clear (bisenet-fp had not yet been implemented). Since then I have trained with bisenet-fp just fine.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

torzdf wrote: Tue Aug 17, 2021 9:23 am

Mask won't cause that kind of issue. However, I used vgg-clear (bisenet-fp had not yet been implemented). Since then I have trained with bisenet-fp just fine.

Thank you for that info [mention]torzdf[/mention].

When you trained your StoJo model - at what point did you decide to switch to nowarp? When there were no more consistent losses visible in the stats? When the previews looked good (shape-wise)? Or just a random figure of 1M (halfway)?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Strange Red Area/Artifact Creeping into Model

Post by torzdf »

Honestly, can't remember. My general approach is to disable warp when I'm getting clearly defined teeth and eye glare.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

torzdf wrote: Tue Aug 17, 2021 9:23 am

Mask won't cause that kind of issue. However, I used vgg-clear (bisenet-fp had not yet been implemented). Since then I have trained with bisenet-fp just fine.

Thank you for that info [mention]torzdf[/mention]. And may I ask, what mask do you convert with? :)

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Strange Red Area/Artifact Creeping into Model

Post by torzdf »

The same mask I train with. Normally fixed up in manual tool.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Red Area/Artifact Creeping into Model

Post by ianstephens »

Started afresh and after 150k+ iterations the same again. Fluorescent green slowly creeping into the data.

Screen Shot 2021-08-28 at 09.33.23.png
Screen Shot 2021-08-28 at 09.33.23.png (87.45 KiB) Viewed 11939 times

Do you think it's worth turning mixed-precision off? Or lowering the learning rate?

Any tips/tricks are greatly appreciated :)

Locked