Page 1 of 1

Cyclical learning rates and learning rate finder?

Posted: Fri Feb 21, 2020 2:31 pm
by blackomen

Does anyone know if the gradient descent algorithm used by faceswap is using cyclical learning rates?

Also, can we implement a learning rate finder (similar to the one in the Fastai library)?


Re: Cyclical learning rates and learning rate finder?

Posted: Fri Feb 21, 2020 3:27 pm
by bryanlyon

We do not support cyclical learning rates. We've decided to favor stability over optimization. We don't want a training to fail due to poorly optimized learning rates (unless the user enters them himself, which means its his own fault).

A learning rate finder (like the one in Fast.ai) only works for a small number of iterations before it's changed and needs to be adjusted. For pretrained models that you're retraining into classifiers that's fine since you'll probably only train for 5-20 iterations. But for FS where you're going to train for 100,000 iterations the whole meaning is lost. Instead we recommend just keeping the default LR and keeping an eye on the general flow of training.