Thoughts on Auto-Training

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Thoughts on Auto-Training

Post by MaxHunter »

I was wondering if there is an easy way to develop an auto-trainer (auto-pilot) for Faceswap for when it runs into NaN warnings.

The way I imagine it is:

Two sliders representing:
1) Rollback - This represents the amount to roll back when hitting a NaN warning.
2) Learning Rate (Decay) - The amount to tune down (-1,-.5,-.1, etc.)

As an example: if the operator starts a model at the default of 5e-5 and it hits a NaN warning, the machine will automatically roll back 50k and tune down to 4e-5. If it hits another NaN warning, it again rolls back and tunes down to 3e-5, and so forth.

This simple fix allows the Faceswap trainer to run for extended periods of time while the operator is away from the machine (work, sleeping, etc.,) without the machine shutting down and wasting valuable training time to NaN warnings.

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Thoughts on Auto-Training

Post by bryanlyon »

Yes, this is possible, and the mechanics of it are quite easy. The hard part is the heuristics. When you roll back, how much you roll back, how much you lower LR and all of that are completely unknowable and so we leave it as an exercise to the user.

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: Thoughts on Auto-Training

Post by MaxHunter »

I understand about the "exercise" of it. In fact my original idea was three sliders adding a slider for attempts. So after three rollbacks, or warnings, it shuts down, in order to force the user to re-evaluate their settings.

These sliders would be tuned by the user to be more detailed or less detailed depending on the users knowledge. Just like an auto-pilot in a plane, it's not there to fly the plane, but there to assist the pilot of the plane when he momentarily leaves the cockpit or is doing other things. Therefore, I think it should be noted (if you were to develop something like this) that it's not there to be a crutch for good practice, but there to assist user in bursts when the user is unavailable.

It's just so frustrating when I go to work or sleep and leave my machine running, only to come back and find the model got a NaN warning and shut down because the learning rate was too high. When all the machine had to do is roll back 50k, or 100k (or whatever the user prefers by tuning the slider) and lower the learning rate (by whatever degree they choose,) which is ultimately what I will do anyway.

Just my two cents. I hope you reconsider. 🙂

Edit: Hell, you could offer Auto-Pilot as a Patron feature for ten dollars or more. I think the freedom of walking away is worth 10 bucks.

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Thoughts on Auto-Training

Post by bryanlyon »

We welcome PRs, as like I said, the mechanics are easy but this is unlikely to be done by us since we don't know how we'd set up the options.

Locked