CRITICAL unexpected crash has occurred

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
EXHYST
Posts: 3
Joined: Sat Nov 09, 2019 9:51 am

CRITICAL unexpected crash has occurred

Post by EXHYST »

Hello,

I keep getting this error every like 20m of Training at this moment.

I attached the error crash report.

Thanks for the help.

11/09/2019 23:57:24 MainProcess training_0 _base save_models INFO [Saved models] - Average since last save: total_loss_A: 0.04119, total_loss_B: 0.03918
11/10/2019 00:01:23 MainProcess training_0 multithreading run DEBUG Error in thread (training_0): GPU sync failed
11/10/2019 00:01:24 MainProcess MainThread train monitor DEBUG Thread error detected
11/10/2019 00:01:24 MainProcess MainThread train monitor DEBUG Closed Monitor
11/10/2019 00:01:24 MainProcess MainThread train end_thread DEBUG Ending Training thread
11/10/2019 00:01:24 MainProcess MainThread train end_thread CRITICAL Error caught! Exiting...
11/10/2019 00:01:24 MainProcess MainThread multithreading join DEBUG Joining Threads: 'training'
11/10/2019 00:01:24 MainProcess MainThread multithreading join DEBUG Joining Thread: 'training_0'
11/10/2019 00:01:24 MainProcess MainThread multithreading join ERROR Caught exception in thread: 'training_0'

Attachments
crash_report.2019.11.10.000126827594.rar
(6.59 KiB) Downloaded 217 times
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: CRITICAL unexpected crash has occurred

Post by torzdf »

InternalError: GPU sync failed

This is a hardware/driver issue. Most likely power or overclocking related.

https://www.google.com/search?q=Interna ... ync+failed

My word is final

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: CRITICAL unexpected crash has occurred

Post by bryanlyon »

I agree with Torzdf, but I'd like to point out that many GPUs have a factory overclock, this is fine for games, but I HIGHLY recommend you remove any factory overclocks for Faceswap as it's far more likely to cause serious problems in Cuda applications.

Locked