Hello.
I tried to change the trainer from DFL-H128 to DLIGHT , but I got this instability. It does not seem to be able to recover.
- diverge.JPG (85.84 KiB) Viewed 2618 times
Any advice?
DLIGHT Diverged....
Forum rules
Read the FAQs and search the forum before posting a new topic.
This forum is for discussing tips and understanding the process involved with Training a Faceswap model.
If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.
Please mark any answers that fixed your problems so others can find the solutions.
-
bryanlyon
- Site Admin
- Posts: 793
- Joined: Fri Jul 12, 2019 12:49 am
- Location: San Francisco
- Has thanked: 4 times
- Been thanked: 218 times
- Contact:
Re: DLIGHT Diverged....
You can't change trainers mid train (No, like it's actually impossible, you're using the same trainer) but what you're really getting here is probably one of the corruptions that sometimes randomly occurs. You can use the restore tool to restore to the last known good model.
Re: DLIGHT Diverged....
Hi...
I explained myself poorly.
I made a video using DFL h128 and then I RESTARTED from scratch using DLIGHT with the same sets/alignment.
DHL 128 was not bad... but I wanted to compare it with something else.
Things went well at the beginning... then everything exploded.
Just wandering if is there any parameter that can prevent this mess....
-
bryanlyon
- Site Admin
- Posts: 793
- Joined: Fri Jul 12, 2019 12:49 am
- Location: San Francisco
- Has thanked: 4 times
- Been thanked: 218 times
- Contact:
Re: DLIGHT Diverged....
Yes, that's reasonable. These "blow ups" do just happen randomly sometimes. Thats why we have the restore tool. Just restore to the last good version and continue training.
If it keeps happening, you might want to reduce your Learning Rate in the train settings.
Re: DLIGHT Diverged....
I was going to add that you might want to double check where your A set is pointing. I got a very similar looking problem when I accidentially changed to set things up on a different project, and then didn't change one of my sets back properly. But then I noticed that the error is increasing, and I think mine decreased. In any event, the ultimate solution is the same, which is use the Restore to bring things back, and if that doesn't work you can also just delete everything in your model folder and copy the contents of the nearest major "25K save" into it.