Train can't Run - E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
vt0822
Posts: 1
Joined: Tue Dec 08, 2020 5:25 am

Train can't Run - E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)

Post by vt0822 »

There is critical error when I start the train process, the following is the log:

Code: Select all

12/07/2020 21:24:21 MainProcess     MainThread      logger          log_setup                 INFO     Log level set to: INFO
12/07/2020 21:24:21 MainProcess     MainThread      gpu_stats       _log                      WARNING  There was an error reading from the Nvidia Machine Learning Library. Either you do not have an Nvidia GPU (in which case this warning can be ignored) or the most likely cause is incorrectly installed drivers. If this is the case, Please remove and reinstall your Nvidia drivers before reporting.Original Error: NVML Shared Library Not Found
12/07/2020 21:24:21 MainProcess     MainThread      gpu_stats       _log                      WARNING  No GPU detected. Switching to CPU mode
12/07/2020 21:24:24 MainProcess     MainThread      train           _get_images               INFO     Model A Directory: C:\DF\Extract\EX_AVM1
12/07/2020 21:24:24 MainProcess     MainThread      train           _get_images               INFO     Model B Directory: C:\DF\Extract\EX_Chalortte
12/07/2020 21:24:24 MainProcess     MainThread      train           process                   INFO     Training data directory: C:\DF\Train\T_CHL_AV1
12/07/2020 21:24:24 MainProcess     MainThread      train           _monitor                  INFO     ===================================================
12/07/2020 21:24:24 MainProcess     MainThread      train           _monitor                  INFO       Starting
12/07/2020 21:24:24 MainProcess     MainThread      train           _monitor                  INFO       Press 'Stop' to save and quit
12/07/2020 21:24:24 MainProcess     MainThread      train           _monitor                  INFO     ===================================================
12/07/2020 21:24:25 MainProcess     _training_0     train           _training                 INFO     Loading data, this may take a while...
12/07/2020 21:24:25 MainProcess     _training_0     plugin_loader   _import                   INFO     Loading Model from Original plugin...
12/07/2020 21:24:25 MainProcess     _training_0     _base           _load                     INFO     No existing state file found. Generating.
12/07/2020 21:24:28 MainProcess     _training_0     plugin_loader   _import                   INFO     Loading Trainer from Original plugin...
12/07/2020 21:24:28 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
12/07/2020 21:24:28 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
12/07/2020 21:24:28 MainProcess     MainThread      launcher        execute_script 

One more error message in the Faceswap application:
12/07/2020 21:24:25 INFO     No existing state file found. Generating.
2020-12-07 21:24:25.896794: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)

Any idea to fix it?
Thanks

User avatar
torzdf
Posts: 2672
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Train can't Run - E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)

Post by torzdf »

You can search this error (search.php?keywords=failed+call+to+cuIn ... +%28303%29)

If you can't find a solution then please post the crash report from your faceswap folder.

My word is final

Locked