Dlight worked to about 6000 its, then flatlined

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
jay2infiniti
Posts: 2
Joined: Sun Aug 07, 2022 1:31 am
Has thanked: 2 times

Dlight worked to about 6000 its, then flatlined

Post by jay2infiniti »

Hi folks,

First post, so first off, big thanks to the software creator (and probable responder to this post). I've found the software very easy to use and reliable given my modest hardware (Nvidea 1660Ti 6GB) after spending some time getting to know it. I'm also impressed by your patience in answering questions from people here who don't seem to grasp that AI is doing something far more complex and versatile than simply swapping image A-B faces. Okay - hopefully you're buttered up. But really, thanks - I'll try to support monetarily as time goes on.

I have had some pretty good success with Dfaker and Dfl-H128, at least I'm getting close to the end stages of training based on previews of a couple models between 180-300k iterations going 64/128 and 128/128 with batch sizes generally around 12. For the longest training model, that represents just over 41 hours of training. Yesterday, I set up a new model to try Dlight. I am using 3-4k images from at least 5-10 video sources and I think the face sets are respectable and very similar to my other models. Should be capable of producing decent results.

Question: What does it mean for Dlight to start fine, get to the blurry face stage at around 6000 its, then flatline in the graphs indicating that the model is not learning and stay there up past 100k its? No errors were shown. Nan protection is on. My settings are very close to default. Should I just try it again and monitor it? Does the learning rate need to be adjusted down since it hit an impasse? I can certainly try a couple of things, but I wonder if you might have seen this behavior before since it doesn't seem to match other problems I've read about here.

Thanks!

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Dlight worked to about 6000 its, then flatlined

Post by torzdf »

It's been a long time since I used DLight, but this is not an issue I have seen before.

Generally that model needs a lower LR (3.5e-5) as it tends to collapse otherwise.

That may solve your issue, it may not.

My word is final

User avatar
jay2infiniti
Posts: 2
Joined: Sun Aug 07, 2022 1:31 am
Has thanked: 2 times

Re: Dlight worked to about 6000 its, then flatlined - addt'l questions

Post by jay2infiniti »

Thanks, I may play with the training rate, then. Other posts seemed to like this model, so I wanted to try it. Villain seems to be the favorite of many if you have the resources, but I'm not confident I do. Is there another favorite [balanced] model of yours?

Unrelated - maybe some of this will help as you update the guide over time. My biggest confusion was when the alignments file was needed and what it contains. After seeing that training didn't require an alignments file as an input, I got ahead of myself and ended up extracting sources directly to a folder that I also use for training. I didn't see the purpose of cleaning the alignments files for training since I was not going to use those sources for conversion. That leaves me with a jumbled folder of source faces from which I remove unwanted faces for training and alignments files that exist but which are mostly from the original extract (extract n= usually 15-40 depending on fps, length and quality of video) and which would be difficult to tie back to specific faces.

I also didn't understand until later that the header can contain alignment information, but the guide says that the alignments files are what contain the generated masks. So, when training with a Bisenet mask that I added when extracting, is that type of mask being regenerated anew for training automatically? Or is that also in the header? I can't imagine it is getting it from the alignments file, since even though they are in the default location, I was not consistent while still setting up my workflow. I would have expected an error if it needed to reference that file.

I know alignments files can be used to regenerate face sets, so they have other uses before conversion, but am I missing some function of a clean alignments file during training by ignoring your good advice? In other words, ignoring backup-ability, am I safe but just need to regenerate an n=1 alignments file and clean it with masks and the manual tool for conversion or have I tainted the training process?

One last thing, sorry for the length. When you recommend "Fit training", do you mean that a model trained on 4000 images should then be trained additionally with the new source faces (source for conversion) alone, say a folder with only the 500 new images? Or should those images be added to the original training folder - making 4500?

Thank you! Fun stuff.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Dlight worked to about 6000 its, then flatlined - addt'l questions

Post by torzdf »

jay2infiniti wrote: Mon Aug 08, 2022 4:17 pm

That leaves me with a jumbled folder of source faces from which I remove unwanted faces for training and alignments files that exist but which are mostly from the original extract (extract n= usually 15-40 depending on fps, length and quality of video) and which would be difficult to tie back to specific faces.

This is fine. You just lose the ability to fix things up/add new data to them. However, you can regenerate the alignments file from faces folder (with the alignments tool) if you require

I also didn't understand until later that the header can contain alignment information, but the guide says that the alignments files are what contain the generated masks. So, when training with a Bisenet mask that I added when extracting, is that type of mask being regenerated anew for training automatically? Or is that also in the header? I can't imagine it is getting it from the alignments file, since even though they are in the default location, I was not consistent while still setting up my workflow. I would have expected an error if it needed to reference that file.

The PNG header contains a copy of its entry in the alignments file, so yes, masks are stored in the PNG headers.

I know alignments files can be used to regenerate face sets, so they have other uses before conversion, but am I missing some function of a clean alignments file during training by ignoring your good advice? In other words, ignoring backup-ability, am I safe but just need to regenerate an n=1 alignments file and clean it with masks and the manual tool for conversion or have I tainted the training process?

Yes, that's fine. I don't generally spend too much time cleaning up training sets myself.

One last thing, sorry for the length. When you recommend "Fit training", do you mean that a model trained on 4000 images should then be trained additionally with the new source faces (source for conversion) alone, say a folder with only the 500 new images? Or should those images be added to the original training folder - making 4500?

I don't generally do fit training, but others swear by it. More info here: viewtopic.php?t=74#fittrain

My word is final

Locked