Dlight worked to about 6000 its, then flatlined
Hi folks,
First post, so first off, big thanks to the software creator (and probable responder to this post). I've found the software very easy to use and reliable given my modest hardware (Nvidea 1660Ti 6GB) after spending some time getting to know it. I'm also impressed by your patience in answering questions from people here who don't seem to grasp that AI is doing something far more complex and versatile than simply swapping image A-B faces. Okay - hopefully you're buttered up. But really, thanks - I'll try to support monetarily as time goes on.
I have had some pretty good success with Dfaker and Dfl-H128, at least I'm getting close to the end stages of training based on previews of a couple models between 180-300k iterations going 64/128 and 128/128 with batch sizes generally around 12. For the longest training model, that represents just over 41 hours of training. Yesterday, I set up a new model to try Dlight. I am using 3-4k images from at least 5-10 video sources and I think the face sets are respectable and very similar to my other models. Should be capable of producing decent results.
Question: What does it mean for Dlight to start fine, get to the blurry face stage at around 6000 its, then flatline in the graphs indicating that the model is not learning and stay there up past 100k its? No errors were shown. Nan protection is on. My settings are very close to default. Should I just try it again and monitor it? Does the learning rate need to be adjusted down since it hit an impasse? I can certainly try a couple of things, but I wonder if you might have seen this behavior before since it doesn't seem to match other problems I've read about here.
Thanks!