Cyclical learning rates and learning rate finder?

Talk about AI and Deep Learning


Locked
User avatar
blackomen
Posts: 9
Joined: Fri Feb 21, 2020 2:14 pm
Has thanked: 2 times
Been thanked: 1 time

Cyclical learning rates and learning rate finder?

Post by blackomen »

Does anyone know if the gradient descent algorithm used by faceswap is using cyclical learning rates?

Also, can we implement a learning rate finder (similar to the one in the Fastai library)?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Cyclical learning rates and learning rate finder?

Post by bryanlyon »

We do not support cyclical learning rates. We've decided to favor stability over optimization. We don't want a training to fail due to poorly optimized learning rates (unless the user enters them himself, which means its his own fault).

A learning rate finder (like the one in Fast.ai) only works for a small number of iterations before it's changed and needs to be adjusted. For pretrained models that you're retraining into classifiers that's fine since you'll probably only train for 5-20 iterations. But for FS where you're going to train for 100,000 iterations the whole meaning is lost. Instead we recommend just keeping the default LR and keeping an eye on the general flow of training.

Locked