Extraction hangs with error: CUDA driver version is insufficient for CUDA runtime version

The Extraction process failing on you, and you aren't getting an error back with clear instructions? Tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Extraction process. If you want to get tips, or better understand the Extract process, then you should look in the Extract Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
jayekaye
Posts: 7
Joined: Mon Jul 15, 2019 6:27 pm
Been thanked: 1 time

Extraction hangs with error: CUDA driver version is insufficient for CUDA runtime version

Post by jayekaye »

Multiple people have reported their extraction workflow never progresses past initialization. This is a thread dedicated to that problem: post solutions here :)

[mod user=torzdf]
The specific cause of the process getting stuck is this error message appearing in the log file:

Code: Select all

tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version

[/mod]

by torzdf » Mon Jul 15, 2019 8:47 pm

You need to update your graphics driver.

Code: Select all

gpu_driver:          397.64

This is insufficient for Cuda 10.

From Nvidia website:

CUDA 10.1 (10.1.105) >= 418.39
CUDA 10.0 (10.0.130) >= 410.48

Get the latest Driver available on Nvidia's website .

Make sure you reboot after installing the drivers

Thanks

Go to full post
Last edited by torzdf on Sat Jul 20, 2019 2:20 am, edited 1 time in total.
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Extraction hangs on "Starting, this may take a while..."

Post by bryanlyon »

We've seen several reports where extract says it's intelligent but then just doesn't nice any farther. If you run into this issue know we're aware of it and trying to solve it,.

I'd you have a problem but it doesn't match this exactly, please post a separate message.

We welcome any new information on this failure to be posted in this thread.

User avatar
jayekaye
Posts: 7
Joined: Mon Jul 15, 2019 6:27 pm
Been thanked: 1 time

Re: Extraction hangs on "Starting, this may take a while..."

Post by jayekaye »

I used the Windows installer and the GUI and let it run for an hour with no progress at all. Tried terminating and restarting a few times. I just switched to command line and realized I needed to add python and conda to the PATH and then install numpy. Now I've progressed to "initializing face alignment network" and I'm running into other errors, ultimately crashing the program.

Last edited by jayekaye on Mon Jul 15, 2019 8:36 pm, edited 1 time in total.
User avatar
jayekaye
Posts: 7
Joined: Mon Jul 15, 2019 6:27 pm
Been thanked: 1 time

Re: Extraction hangs on "Starting, this may take a while..."

Post by jayekaye »

So I've made it past the "starting..." but now get this:

07/15/2019 13:33:43 INFO Loading Align from Fan plugin...
07/15/2019 13:33:43 INFO Starting, this may take a while...
07/15/2019 13:33:45 INFO Initializing Face Alignment Network...
07/15/2019 13:33:49 ERROR Caught exception in child process: 1724

followed by a traceback, followed by:

'C:\Users\Test' is not recognized as an internal or external command,
operable program or batch file.
07/15/2019 13:33:51 ERROR Got Exception on main handler:

followed by another traceback, followed by :

07/15/2019 13:33:51 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\Test Machine\faceswap\crash_report.2019.07.15.133351430277.log'. Please verify you are running the latest version of faceswap before reporting

any ideas?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Extraction hangs on "Starting, this may take a while..."

Post by torzdf »

Hi Jakeeye

Please post the contents of the crash log listed:

Code: Select all

C:\Users\Test Machine\faceswap\crash_report.2019.07.15.133351430277.log

My word is final

User avatar
jayekaye
Posts: 7
Joined: Mon Jul 15, 2019 6:27 pm
Been thanked: 1 time

Re: Extraction hangs on "Starting, this may take a while..."

Post by jayekaye »

Code: Select all

07/15/2019 13:33:43 MainProcess     MainThread      multithreading  start                     DEBUG    Started all threads 'save_faces': 1
07/15/2019 13:33:43 MainProcess     MainThread      extract         process_item_count        DEBUG    Items already processed: 0
07/15/2019 13:33:44 MainProcess     MainThread      extract         process_item_count        DEBUG    Items to be Processed: 78938
07/15/2019 13:33:44 MainProcess     MainThread      pipeline        launch                    DEBUG    Launching aligner and detector
07/15/2019 13:33:44 MainProcess     MainThread      pipeline        launch_aligner            DEBUG    Launching Aligner
07/15/2019 13:33:44 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initializing SpawnProcess: (target: 'Aligner.run', args: (), kwargs: {})
07/15/2019 13:33:44 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initialized SpawnProcess: 'Aligner.run'
07/15/2019 13:33:44 MainProcess     MainThread      multithreading  start                     DEBUG    Spawning Process: (name: 'Aligner.run', args: (), kwargs: {'event': <multiprocessing.synchronize.Event object at 0x000001BF35617D30>, 'error': <multiprocessing.synchronize.Event object at 0x000001BF356487F0>, 'log_init': <function set_root_logger at 0x000001BF2C472950>, 'log_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1bf2c76f3c8>, 'log_level': 10, 'in_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1bf35617828>, 'out_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1bf355e3940>}, daemon: True)
07/15/2019 13:33:44 MainProcess     MainThread      multithreading  start                     DEBUG    Spawned Process: (name: 'Aligner.run', PID: 1724)
07/15/2019 13:33:45 Aligner.run     MainThread      _base           initialize                DEBUG    _base initialize Align: (PID: 1724, args: (), kwargs: {'event': <multiprocessing.synchronize.Event object at 0x000001F9F067FC88>, 'error': <multiprocessing.synchronize.Event object at 0x000001F9F3077588>, 'log_init': <function set_root_logger at 0x000001F9F09371E0>, 'log_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1f9f96c8160>, 'log_level': 10, 'in_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1f9f96c8278>, 'out_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1f9f96c82e8>})
07/15/2019 13:33:45 Aligner.run     MainThread      fan             initialize                INFO     Initializing Face Alignment Network...
07/15/2019 13:33:45 Aligner.run     MainThread      fan             initialize                DEBUG    fan initialize: (args: () kwargs: {'event': <multiprocessing.synchronize.Event object at 0x000001F9F067FC88>, 'error': <multiprocessing.synchronize.Event object at 0x000001F9F3077588>, 'log_init': <function set_root_logger at 0x000001F9F09371E0>, 'log_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1f9f96c8160>, 'log_level': 10, 'in_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1f9f96c8278>, 'out_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1f9f96c82e8>})
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       __init__                  DEBUG    Initializing GPUStats
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       initialize                DEBUG    OS is not macOS. Using pynvml
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_device_count          DEBUG    GPU Device count: 1
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_active_devices        DEBUG    Active GPU Devices: [0]
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_handles               DEBUG    GPU Handles found: 1
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_driver                DEBUG    GPU Driver: 397.64
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_devices               DEBUG    GPU Devices: ['TITAN X (Pascal)']
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_vram                  DEBUG    GPU VRAM: [12288.0]
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       __init__                  DEBUG    Initialized GPUStats
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       initialize                DEBUG    OS is not macOS. Using pynvml
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_device_count          DEBUG    GPU Device count: 1
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_active_devices        DEBUG    Active GPU Devices: [0]
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_handles               DEBUG    GPU Handles found: 1
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_free                  DEBUG    GPU VRAM free: [11297.69921875]
07/15/2019 13:33:45 Aligner.run     MainThread      gpu_stats       get_card_most_free        DEBUG    Active GPU Card with most free VRAM: {'card_id': 0, 'device': 'TITAN X (Pascal)', 'free': 11297.69921875, 'total': 12288.0}
07/15/2019 13:33:45 Aligner.run     MainThread      _base           get_vram_free             VERBOSE  Using device TITAN X (Pascal) with 11297MB free of 12288MB
07/15/2019 13:33:45 Aligner.run     MainThread      fan             initialize                VERBOSE  Reserving 2240MB for face alignments
07/15/2019 13:33:45 Aligner.run     MainThread      fan             load_graph                VERBOSE  Initializing Face Alignment Network model...
07/15/2019 13:33:49 Aligner.run     MainThread      _base           run                       ERROR    Caught exception in child process: 1724
07/15/2019 13:33:49 Aligner.run     MainThread      _base           run                       ERROR    Traceback:
Traceback (most recent call last):
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\_base.py", line 112, in run
    self.align(*args, **kwargs)
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\_base.py", line 127, in align
    self.initialize(*args, **kwargs)
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\fan.py", line 47, in initialize
    raise err
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\fan.py", line 41, in initialize
    self.model = FAN(self.model_path, ratio=tf_ratio)
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\fan.py", line 199, in __init__
    self.session = self.set_session(ratio)
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\fan.py", line 221, in set_session
    session = self.tf.Session(config=config)
  File "C:\Users\Test Machine\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\client\session.py", line 1551, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "C:\Users\Test Machine\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\client\session.py", line 676, in __init__
    self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
Traceback (most recent call last):
  File "C:\Users\Test Machine\faceswap\lib\cli.py", line 122, in execute_script
    process.process()
  File "C:\Users\Test Machine\faceswap\scripts\extract.py", line 61, in process
    self.run_extraction()
  File "C:\Users\Test Machine\faceswap\scripts\extract.py", line 180, in run_extraction
    self.extractor.launch()
  File "C:\Users\Test Machine\faceswap\plugins\extract\pipeline.py", line 171, in launch
    self.launch_aligner()
  File "C:\Users\Test Machine\faceswap\plugins\extract\pipeline.py", line 206, in launch_aligner
    raise ValueError("Error initializing Aligner")
ValueError: Error initializing Aligner

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         dafde78 Fix Previews for Extract/Convert when window is too small to fit items
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: TITAN X (Pascal)
gpu_devices_active:  GPU_0
gpu_driver:          397.64
gpu_vram:            GPU_0: 12288MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.17763-SP0
os_release:          10
py_command:          C:\Users\Test Machine\faceswap\faceswap.py extract -i C:/Users/Test Machine/faceswap/src/belsky/belsky.mp4 -o C:/Users/Test Machine/faceswap/faces/belsky -l 0.4 --serializer json -D mtcnn -A fan -nm none -bt 0.0 -sz 256 -min 0 -een 1 -si 0 -L INFO -gui
py_conda_version:    conda 4.7.5
py_implementation:   CPython
py_version:          3.6.8
py_virtual_env:      True
sys_cores:           20
sys_processor:       Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
sys_ram:             Total: 130770MB, Available: 122416MB, Used: 8353MB, Free: 122416MB

=============== Pip Packages ===============


============== Conda Packages ==============
# packages in environment at C:\Users\Test Machine\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu  
absl-py 0.7.1 py36_0
astor 0.7.1 py36_0
blas 1.0 mkl
ca-certificates 2019.5.15 0
certifi 2019.6.16 py36_0
cloudpickle 1.2.1 py_0
cudatoolkit 10.0.130 0
cudnn 7.6.0 cuda10.0_0
cycler 0.10.0 py36h009560c_0
cytoolz 0.9.0.1 py36hfa6e2cd_1
dask-core 2.1.0 py_0
decorator 4.4.0 py36_1
fastcluster 1.1.25 py36h830ac7b_1000 conda-forge ffmpeg 4.1.3 h6538335_0 conda-forge ffmpy 0.2.2 pypi_0 pypi freetype 2.9.1 ha9979f8_1
gast 0.2.2 py36_0
grpcio 1.16.1 py36h351948d_1
h5py 2.9.0 py36h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha66f8fd_1
imageio 2.5.0 py36_0
imageio-ffmpeg 0.3.0 py_0 conda-forge intel-openmp 2019.4 245
joblib 0.13.2 py36_0
jpeg 9c hfa6e2cd_1001 conda-forge keras 2.2.4 0
keras-applications 1.0.8 py_0
keras-base 2.2.4 py36_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.1.0 py36ha925a31_0
libblas 3.8.0 8_mkl conda-forge libcblas 3.8.0 8_mkl conda-forge liblapack 3.8.0 8_mkl conda-forge liblapacke 3.8.0 8_mkl conda-forge libmklml 2019.0.3 0
libpng 1.6.37 h7602738_0 conda-forge libprotobuf 3.8.0 h7bd577a_0
libtiff 4.0.10 h6512ee2_1003 conda-forge libwebp 1.0.2 hfa6e2cd_2 conda-forge lz4-c 1.8.3 he025d50_1001 conda-forge markdown 3.1.1 py36_0
matplotlib 2.2.2 py36had4c4a9_2
mkl 2019.4 245
mkl-service 2.0.2 py36he774522_0
mkl_fft 1.0.12 py36h14836fe_0
mkl_random 1.0.2 py36h343c172_0
mock 3.0.5 py36_0
networkx 2.3 py_0
numpy 1.16.2 py36h19fb1c0_0
numpy-base 1.16.2 py36hc3f5095_0
nvidia-ml-py3 7.352.0 pypi_0 pypi olefile 0.46 py36_0
opencv 4.1.0 py36hb4945ee_5 conda-forge openssl 1.1.1c he774522_1
pathlib 1.0.1 py36_1
pillow 6.1.0 py36hdc69c19_0
pip 19.1.1 py36_0
protobuf 3.8.0 py36h33f27b4_0
psutil 5.6.3 py36he774522_0
pyparsing 2.4.0 py_0
pyqt 5.9.2 py36h6538335_2
pyreadline 2.1 py36_1
python 3.6.8 h9f7ef89_7
python-dateutil 2.8.0 py36_0
pytz 2019.1 py_0
pywavelets 1.0.3 py36h8c2d366_1
pywin32 223 py36hfa6e2cd_1
pyyaml 5.1.1 py36he774522_0
qt 5.9.7 hc6833c9_1 conda-forge scikit-image 0.15.0 py36ha925a31_0
scikit-learn 0.21.2 py36h6288b17_0
scipy 1.2.1 py36h29ff71c_0
setuptools 41.0.1 py36_0
sip 4.19.8 py36h6538335_0
six 1.12.0 py36_0
sqlite 3.28.0 he774522_0
tensorboard 1.13.1 py36h33f27b4_0
tensorflow 1.13.1 gpu_py36h9006a92_0
tensorflow-base 1.13.1 gpu_py36h871c8ca_0
tensorflow-estimator 1.13.0 py_0
tensorflow-gpu 1.13.1 h0d30ee6_0
termcolor 1.1.0 py36_1
tk 8.6.8 hfa6e2cd_0
toolz 0.10.0 py_0
toposort 1.5 py_3 conda-forge tornado 6.0.3 py36he774522_0
tqdm 4.32.1 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.15.26706 h3a45250_4
werkzeug 0.15.4 py_0
wheel 0.33.4 py36_0
wincertstore 0.2 py36h7fe50ca_0
xz 5.2.4 h2fa13f4_1001 conda-forge yaml 0.1.7 hc54c509_2
zlib 1.2.11 h2fa13f4_1005 conda-forge zstd 1.4.0 hd8a0e53_0 conda-forge
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Extraction hangs on "Starting, this may take a while..."

Post by torzdf »

You need to update your graphics driver.

Code: Select all

gpu_driver:          397.64

This is insufficient for Cuda 10.

From Nvidia website:

CUDA 10.1 (10.1.105) >= 418.39
CUDA 10.0 (10.0.130) >= 410.48

Get the latest Driver available on Nvidia's website .

Make sure you reboot after installing the drivers

Thanks

My word is final

User avatar
jayekaye
Posts: 7
Joined: Mon Jul 15, 2019 6:27 pm
Been thanked: 1 time

Re: Extraction hangs on "Starting, this may take a while..."

Post by jayekaye »

Thanks! I updated my graphics driver and now I'm hanging at the start again. It's been executing extract.py for over 5 minutes now.

07/15/2019 14:00:13 INFO Loading Detect from Mtcnn plugin...
07/15/2019 14:00:13 INFO Loading Align from Fan plugin...
07/15/2019 14:00:13 INFO Starting, this may take a while...

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Extraction hangs on "Starting, this may take a while..."

Post by torzdf »

First load can take several minutes as Cuda needs to compile a load of stuff. It may time out. If it hasn't after about 10 minutes, kill it and run again.

It will be fine every time after.

My word is final

User avatar
jayekaye
Posts: 7
Joined: Mon Jul 15, 2019 6:27 pm
Been thanked: 1 time

Re: Extraction hangs on "Starting, this may take a while..."

Post by jayekaye »

Terminated and restarted it, now same errors as before. New crash report:

Code: Select all

07/15/2019 14:09:36 MainProcess     MainThread      multithreading  start                     DEBUG    Started all threads 'save_faces': 1
07/15/2019 14:09:36 MainProcess     MainThread      extract         process_item_count        DEBUG    Items already processed: 0
07/15/2019 14:09:36 MainProcess     MainThread      extract         process_item_count        DEBUG    Items to be Processed: 78938
07/15/2019 14:09:36 MainProcess     MainThread      pipeline        launch                    DEBUG    Launching aligner and detector
07/15/2019 14:09:36 MainProcess     MainThread      pipeline        launch_aligner            DEBUG    Launching Aligner
07/15/2019 14:09:36 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initializing SpawnProcess: (target: 'Aligner.run', args: (), kwargs: {})
07/15/2019 14:09:36 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initialized SpawnProcess: 'Aligner.run'
07/15/2019 14:09:36 MainProcess     MainThread      multithreading  start                     DEBUG    Spawning Process: (name: 'Aligner.run', args: (), kwargs: {'event': <multiprocessing.synchronize.Event object at 0x00000297D6656898>, 'error': <multiprocessing.synchronize.Event object at 0x00000297D6688828>, 'log_init': <function set_root_logger at 0x00000297CD482950>, 'log_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x297cd83a4a8>, 'log_level': 10, 'in_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x297d66569e8>, 'out_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x297d66568d0>}, daemon: True)
07/15/2019 14:09:36 MainProcess     MainThread      multithreading  start                     DEBUG    Spawned Process: (name: 'Aligner.run', PID: 4432)
07/15/2019 14:09:38 Aligner.run     MainThread      _base           initialize                DEBUG    _base initialize Align: (PID: 4432, args: (), kwargs: {'event': <multiprocessing.synchronize.Event object at 0x000002349DF2FC88>, 'error': <multiprocessing.synchronize.Event object at 0x00000234A0907588>, 'log_init': <function set_root_logger at 0x000002349E1E71E0>, 'log_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x234a6f880f0>, 'log_level': 10, 'in_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x234a6f88208>, 'out_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x234a6f88278>})
07/15/2019 14:09:38 Aligner.run     MainThread      fan             initialize                INFO     Initializing Face Alignment Network...
07/15/2019 14:09:38 Aligner.run     MainThread      fan             initialize                DEBUG    fan initialize: (args: () kwargs: {'event': <multiprocessing.synchronize.Event object at 0x000002349DF2FC88>, 'error': <multiprocessing.synchronize.Event object at 0x00000234A0907588>, 'log_init': <function set_root_logger at 0x000002349E1E71E0>, 'log_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x234a6f880f0>, 'log_level': 10, 'in_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x234a6f88208>, 'out_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x234a6f88278>})
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       __init__                  DEBUG    Initializing GPUStats
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       initialize                DEBUG    OS is not macOS. Using pynvml
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_device_count          DEBUG    GPU Device count: 1
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_active_devices        DEBUG    Active GPU Devices: [0]
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_handles               DEBUG    GPU Handles found: 1
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_driver                DEBUG    GPU Driver: 399.24
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_devices               DEBUG    GPU Devices: ['TITAN X (Pascal)']
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_vram                  DEBUG    GPU VRAM: [12288.0]
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       __init__                  DEBUG    Initialized GPUStats
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       initialize                DEBUG    OS is not macOS. Using pynvml
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_device_count          DEBUG    GPU Device count: 1
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_active_devices        DEBUG    Active GPU Devices: [0]
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_handles               DEBUG    GPU Handles found: 1
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_free                  DEBUG    GPU VRAM free: [11608.66796875]
07/15/2019 14:09:38 Aligner.run     MainThread      gpu_stats       get_card_most_free        DEBUG    Active GPU Card with most free VRAM: {'card_id': 0, 'device': 'TITAN X (Pascal)', 'free': 11608.66796875, 'total': 12288.0}
07/15/2019 14:09:38 Aligner.run     MainThread      _base           get_vram_free             VERBOSE  Using device TITAN X (Pascal) with 11608MB free of 12288MB
07/15/2019 14:09:38 Aligner.run     MainThread      fan             initialize                VERBOSE  Reserving 2240MB for face alignments
07/15/2019 14:09:38 Aligner.run     MainThread      fan             load_graph                VERBOSE  Initializing Face Alignment Network model...
07/15/2019 14:09:41 Aligner.run     MainThread      _base           run                       ERROR    Caught exception in child process: 4432
07/15/2019 14:09:41 Aligner.run     MainThread      _base           run                       ERROR    Traceback:
Traceback (most recent call last):
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\_base.py", line 112, in run
    self.align(*args, **kwargs)
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\_base.py", line 127, in align
    self.initialize(*args, **kwargs)
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\fan.py", line 47, in initialize
    raise err
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\fan.py", line 41, in initialize
    self.model = FAN(self.model_path, ratio=tf_ratio)
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\fan.py", line 199, in __init__
    self.session = self.set_session(ratio)
  File "C:\Users\Test Machine\faceswap\plugins\extract\align\fan.py", line 221, in set_session
    session = self.tf.Session(config=config)
  File "C:\Users\Test Machine\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\client\session.py", line 1551, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "C:\Users\Test Machine\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\client\session.py", line 676, in __init__
    self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
Traceback (most recent call last):
  File "C:\Users\Test Machine\faceswap\lib\cli.py", line 122, in execute_script
    process.process()
  File "C:\Users\Test Machine\faceswap\scripts\extract.py", line 61, in process
    self.run_extraction()
  File "C:\Users\Test Machine\faceswap\scripts\extract.py", line 180, in run_extraction
    self.extractor.launch()
  File "C:\Users\Test Machine\faceswap\plugins\extract\pipeline.py", line 171, in launch
    self.launch_aligner()
  File "C:\Users\Test Machine\faceswap\plugins\extract\pipeline.py", line 206, in launch_aligner
    raise ValueError("Error initializing Aligner")
ValueError: Error initializing Aligner

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         dafde78 Fix Previews for Extract/Convert when window is too small to fit items
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: TITAN X (Pascal)
gpu_devices_active:  GPU_0
gpu_driver:          399.24
gpu_vram:            GPU_0: 12288MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.17763-SP0
os_release:          10
py_command:          C:\Users\Test Machine\faceswap\faceswap.py extract -i C:/Users/Test Machine/faceswap/src/belsky/belsky.mp4 -o C:/Users/Test Machine/faceswap/faces/belsky -l 0.4 --serializer json -D mtcnn -A fan -nm none -bt 0.0 -sz 256 -min 0 -een 1 -si 0 -L INFO -gui
py_conda_version:    conda 4.7.5
py_implementation:   CPython
py_version:          3.6.8
py_virtual_env:      True
sys_cores:           20
sys_processor:       Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
sys_ram:             Total: 130770MB, Available: 125754MB, Used: 5016MB, Free: 125754MB

=============== Pip Packages ===============


============== Conda Packages ==============
# packages in environment at C:\Users\Test Machine\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu  
absl-py 0.7.1 py36_0
astor 0.7.1 py36_0
blas 1.0 mkl
ca-certificates 2019.5.15 0
certifi 2019.6.16 py36_0
cloudpickle 1.2.1 py_0
cudatoolkit 10.0.130 0
cudnn 7.6.0 cuda10.0_0
cycler 0.10.0 py36h009560c_0
cytoolz 0.9.0.1 py36hfa6e2cd_1
dask-core 2.1.0 py_0
decorator 4.4.0 py36_1
fastcluster 1.1.25 py36h830ac7b_1000 conda-forge ffmpeg 4.1.3 h6538335_0 conda-forge ffmpy 0.2.2 pypi_0 pypi freetype 2.9.1 ha9979f8_1
gast 0.2.2 py36_0
grpcio 1.16.1 py36h351948d_1
h5py 2.9.0 py36h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha66f8fd_1
imageio 2.5.0 py36_0
imageio-ffmpeg 0.3.0 py_0 conda-forge intel-openmp 2019.4 245
joblib 0.13.2 py36_0
jpeg 9c hfa6e2cd_1001 conda-forge keras 2.2.4 0
keras-applications 1.0.8 py_0
keras-base 2.2.4 py36_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.1.0 py36ha925a31_0
libblas 3.8.0 8_mkl conda-forge libcblas 3.8.0 8_mkl conda-forge liblapack 3.8.0 8_mkl conda-forge liblapacke 3.8.0 8_mkl conda-forge libmklml 2019.0.3 0
libpng 1.6.37 h7602738_0 conda-forge libprotobuf 3.8.0 h7bd577a_0
libtiff 4.0.10 h6512ee2_1003 conda-forge libwebp 1.0.2 hfa6e2cd_2 conda-forge lz4-c 1.8.3 he025d50_1001 conda-forge markdown 3.1.1 py36_0
matplotlib 2.2.2 py36had4c4a9_2
mkl 2019.4 245
mkl-service 2.0.2 py36he774522_0
mkl_fft 1.0.12 py36h14836fe_0
mkl_random 1.0.2 py36h343c172_0
mock 3.0.5 py36_0
networkx 2.3 py_0
numpy 1.16.2 py36h19fb1c0_0
numpy-base 1.16.2 py36hc3f5095_0
nvidia-ml-py3 7.352.0 pypi_0 pypi olefile 0.46 py36_0
opencv 4.1.0 py36hb4945ee_5 conda-forge openssl 1.1.1c he774522_1
pathlib 1.0.1 py36_1
pillow 6.1.0 py36hdc69c19_0
pip 19.1.1 py36_0
protobuf 3.8.0 py36h33f27b4_0
psutil 5.6.3 py36he774522_0
pyparsing 2.4.0 py_0
pyqt 5.9.2 py36h6538335_2
pyreadline 2.1 py36_1
python 3.6.8 h9f7ef89_7
python-dateutil 2.8.0 py36_0
pytz 2019.1 py_0
pywavelets 1.0.3 py36h8c2d366_1
pywin32 223 py36hfa6e2cd_1
pyyaml 5.1.1 py36he774522_0
qt 5.9.7 hc6833c9_1 conda-forge scikit-image 0.15.0 py36ha925a31_0
scikit-learn 0.21.2 py36h6288b17_0
scipy 1.2.1 py36h29ff71c_0
setuptools 41.0.1 py36_0
sip 4.19.8 py36h6538335_0
six 1.12.0 py36_0
sqlite 3.28.0 he774522_0
tensorboard 1.13.1 py36h33f27b4_0
tensorflow 1.13.1 gpu_py36h9006a92_0
tensorflow-base 1.13.1 gpu_py36h871c8ca_0
tensorflow-estimator 1.13.0 py_0
tensorflow-gpu 1.13.1 h0d30ee6_0
termcolor 1.1.0 py36_1
tk 8.6.8 hfa6e2cd_0
toolz 0.10.0 py_0
toposort 1.5 py_3 conda-forge tornado 6.0.3 py36he774522_0
tqdm 4.32.1 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.15.26706 h3a45250_4
werkzeug 0.15.4 py_0
wheel 0.33.4 py36_0
wincertstore 0.2 py36h7fe50ca_0
xz 5.2.4 h2fa13f4_1001 conda-forge yaml 0.1.7 hc54c509_2
zlib 1.2.11 h2fa13f4_1005 conda-forge zstd 1.4.0 hd8a0e53_0 conda-forge
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Extraction hangs on "Starting, this may take a while..."

Post by torzdf »

Your driver is still too old.

From Nvidia website:

CUDA 10.1 (10.1.105) >= 418.39
CUDA 10.0 (10.0.130) >= 410.48

You are now on version 399.24... Closer but still not there.

The latest TITAN X (Pascal) Driver available on Nvidia's website is version 431.36

Make sure you reboot after installing the drivers

My word is final

User avatar
jayekaye
Posts: 7
Joined: Mon Jul 15, 2019 6:27 pm
Been thanked: 1 time

Re: Extraction hangs on "Starting, this may take a while..."

Post by jayekaye »

Thanks!!! I just assumed it updated to the latest and I have things working now.

Locked