How was the default value of save_interval selected?

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
obicons
Posts: 2
Joined: Mon May 01, 2023 1:54 pm

How was the default value of save_interval selected?

Post by obicons »

Hello,

I'm hoping someone can help answer this question:

How was the default value of save_interval selected? For example, https://github.com/deepfakes/faceswap/b ... s.py#L1063.

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: How was the default value of save_interval selected?

Post by bryanlyon »

Low enough that you don't lose too much time if it crashes, high enough that you don't lose too much time with excessive saves.

It's probably no longer the best choice, because it was picked when GPUs and storage were slower.

Last edited by bryanlyon on Mon May 01, 2023 3:43 pm, edited 1 time in total.
User avatar
obicons
Posts: 2
Joined: Mon May 01, 2023 1:54 pm

Re: How was the default value of save_interval selected?

Post by obicons »

Thanks for the reply, @bryanlyon.

Was there a methodology used to manage these tradeoffs in the first place? Or was it selected based on developer experiences?

Last edited by obicons on Mon May 01, 2023 5:32 pm, edited 1 time in total.
User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: How was the default value of save_interval selected?

Post by torzdf »

Just based on experience. It used to be 100 iterations. I bumped it to 250 when we moved to Tensorflow 2.x

My word is final

Locked