Page 1 of 1

CRITICAL unexpected crash has occurred

Posted: Sun Nov 10, 2019 12:34 am
by EXHYST

Hello,

I keep getting this error every like 20m of Training at this moment.

I attached the error crash report.

Thanks for the help.

11/09/2019 23:57:24 MainProcess training_0 _base save_models INFO [Saved models] - Average since last save: total_loss_A: 0.04119, total_loss_B: 0.03918
11/10/2019 00:01:23 MainProcess training_0 multithreading run DEBUG Error in thread (training_0): GPU sync failed
11/10/2019 00:01:24 MainProcess MainThread train monitor DEBUG Thread error detected
11/10/2019 00:01:24 MainProcess MainThread train monitor DEBUG Closed Monitor
11/10/2019 00:01:24 MainProcess MainThread train end_thread DEBUG Ending Training thread
11/10/2019 00:01:24 MainProcess MainThread train end_thread CRITICAL Error caught! Exiting...
11/10/2019 00:01:24 MainProcess MainThread multithreading join DEBUG Joining Threads: 'training'
11/10/2019 00:01:24 MainProcess MainThread multithreading join DEBUG Joining Thread: 'training_0'
11/10/2019 00:01:24 MainProcess MainThread multithreading join ERROR Caught exception in thread: 'training_0'


Re: CRITICAL unexpected crash has occurred

Posted: Sun Nov 10, 2019 7:38 pm
by torzdf

InternalError: GPU sync failed

This is a hardware/driver issue. Most likely power or overclocking related.

https://www.google.com/search?q=Interna ... ync+failed


Re: CRITICAL unexpected crash has occurred

Posted: Tue Nov 12, 2019 4:58 pm
by bryanlyon

I agree with Torzdf, but I'd like to point out that many GPUs have a factory overclock, this is fine for games, but I HIGHLY recommend you remove any factory overclocks for Faceswap as it's far more likely to cause serious problems in Cuda applications.