Faceswap Forum

I was wondering if there is an easy way to develop an auto-trainer (auto-pilot) for Faceswap for when it runs into NaN warnings.

The way I imagine it is:

Two sliders representing:
1) Rollback - This represents the amount to roll back when hitting a NaN warning.
2) Learning Rate (Decay) - The amount to tune down (-1,-.5,-.1, etc.)

As an example: if the operator starts a model at the default of 5e-5 and it hits a NaN warning, the machine will automatically roll back 50k and tune down to 4e-5. If it hits another NaN warning, it again rolls back and tunes down to 3e-5, and so forth.

This simple fix allows the Faceswap trainer to run for extended periods of time while the operator is away from the machine (work, sleeping, etc.,) without the machine shutting down and wasting valuable training time to NaN warnings.

Yes, this is possible, and the mechanics of it are quite easy. The hard part is the heuristics. When you roll back, how much you roll back, how much you lower LR and all of that are completely unknowable and so we leave it as an exercise to the user.

I understand about the "exercise" of it. In fact my original idea was three sliders adding a slider for attempts. So after three rollbacks, or warnings, it shuts down, in order to force the user to re-evaluate their settings.

These sliders would be tuned by the user to be more detailed or less detailed depending on the users knowledge. Just like an auto-pilot in a plane, it's not there to fly the plane, but there to assist the pilot of the plane when he momentarily leaves the cockpit or is doing other things. Therefore, I think it should be noted (if you were to develop something like this) that it's not there to be a crutch for good practice, but there to assist user in bursts when the user is unavailable.

It's just so frustrating when I go to work or sleep and leave my machine running, only to come back and find the model got a NaN warning and shut down because the learning rate was too high. When all the machine had to do is roll back 50k, or 100k (or whatever the user prefers by tuning the slider) and lower the learning rate (by whatever degree they choose,) which is ultimately what I will do anyway.

Just my two cents. I hope you reconsider.

Edit: Hell, you could offer Auto-Pilot as a Patron feature for ten dollars or more. I think the freedom of walking away is worth 10 bucks.

We welcome PRs, as like I said, the mechanics are easy but this is unlikely to be done by us since we don't know how we'd set up the options.