Any tips on securing yourself from possible crash?

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
kruszkush
Posts: 9
Joined: Thu May 07, 2020 10:36 pm
Answers: 0
Has thanked: 3 times

Any tips on securing yourself from possible crash?

Post by kruszkush »

I got weird problem with my graphics card which gives blue screen, it is possible that card itself got hardware problem because nothing really worked here, but I don't want to get into that, let's assume it is physical thing and cannot be done right now.

I can get blue screen, it is very rare but just happened once while I was extracting one file, didn't lose much maybe 40 minutes as it was almost done. Saved nothing.

As I will be about to train it, is there any way to make it save every like 2 hours for example, is there any precauction setting I could apply? Any tips I might do? Thanks!

ps. Any tip for upside-down extraction? I get maybe like 5% of pictures upside-down or tilted 60 degrees

User avatar
bryanlyon
Site Admin
Posts: 805
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 224 times
Contact:

Re: Any tips on securing yourself from possible crash?

Post by bryanlyon »

It does automatically save (by default) every 100 iterations and saves a backup snapshot every 25,000 iterations. You can freely change those to any numbers you want.

However, you should consider removing any factory applied overclock on your GPU as that is the cause for about 99% of crashes like you're seeing.

Locked