Hello,
I keep getting this error every like 20m of Training at this moment.
I attached the error crash report.
Thanks for the help.
11/09/2019 23:57:24 MainProcess training_0 _base save_models INFO [Saved models] - Average since last save: total_loss_A: 0.04119, total_loss_B: 0.03918
11/10/2019 00:01:23 MainProcess training_0 multithreading run DEBUG Error in thread (training_0): GPU sync failed
11/10/2019 00:01:24 MainProcess MainThread train monitor DEBUG Thread error detected
11/10/2019 00:01:24 MainProcess MainThread train monitor DEBUG Closed Monitor
11/10/2019 00:01:24 MainProcess MainThread train end_thread DEBUG Ending Training thread
11/10/2019 00:01:24 MainProcess MainThread train end_thread CRITICAL Error caught! Exiting...
11/10/2019 00:01:24 MainProcess MainThread multithreading join DEBUG Joining Threads: 'training'
11/10/2019 00:01:24 MainProcess MainThread multithreading join DEBUG Joining Thread: 'training_0'
11/10/2019 00:01:24 MainProcess MainThread multithreading join ERROR Caught exception in thread: 'training_0'