Page 1 of 1

Crashing after 70 iterations

Posted: Tue Sep 28, 2021 11:45 am
by jebanke

Hello,
I'm trying to train my model, but it keeps crashing after approximately 70 iterations.
My setup is as follows : phaze-a stojo preset, at 384px output size, mixed precision, batch size of 10, in "distributed" mode (I have 5x RTX 2060S)

Here is the crash log generated by faceswap : https://pastebin.com/k7PJA2m0

I did check to see if I was using the latest version of faceswap, and it says I am.

Any help is appreciated !


Re: Crashing after 70 iterations

Posted: Mon Oct 04, 2021 6:30 pm
by bryanlyon

This is a network/remote storage error. It's saying that your network data is becoming unavailable and causing faceswap to be unable to access the files. Unfortunately this is nothing to do with Faceswap and is entirely due to your setup. Either get your network storage more reliable or move the files locally to train.