I was wondering if there is an easy way to develop an auto-trainer (auto-pilot) for Faceswap for when it runs into NaN warnings.
The way I imagine it is:
Two sliders representing:
1) Rollback - This represents the amount to roll back when hitting a NaN warning.
2) Learning Rate (Decay) - The amount to tune down (-1,-.5,-.1, etc.)
As an example: if the operator starts a model at the default of 5e-5 and it hits a NaN warning, the machine will automatically roll back 50k and tune down to 4e-5. If it hits another NaN warning, it again rolls back and tunes down to 3e-5, and so forth.
This simple fix allows the Faceswap trainer to run for extended periods of time while the operator is away from the machine (work, sleeping, etc.,) without the machine shutting down and wasting valuable training time to NaN warnings.