Got an unexpected crash in training and extracting

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
bribebh
Posts: 3
Joined: Sun Jun 04, 2023 10:00 am

Got an unexpected crash in training and extracting

Post by bribebh »

I installed the faceswap using the linux installer released on github. However, after the installation, the following errors were encountered during the extraction and training phases. Here is the crash report I get during training phase.

Code: Select all

06/04/2023 10:23:25 MainProcess     MainThread                     logger          log_setup                      INFO     Log level set to: INFO
06/04/2023 10:23:28 MainProcess     MainThread                     train           _get_images                    INFO     Model A Directory: '/data/Generation/faceswap/faces/01/001' (116 images)
06/04/2023 10:23:28 MainProcess     MainThread                     train           _get_images                    INFO     Model B Directory: '/data/Generation/faceswap/faces/02/001' (148 images)
06/04/2023 10:23:28 MainProcess     MainThread                     train           _validate_image_counts         WARNING  At least one of your input folders contains fewer than 250 images. Results are likely to be poor.
06/04/2023 10:23:28 MainProcess     MainThread                     train           _validate_image_counts         WARNING  You need to provide a significant number of images to successfully train a Neural Network. Aim for between 500 - 5000 images per side.
06/04/2023 10:23:28 MainProcess     MainThread                     train           process                        INFO     Training data directory: /data/Generation/faceswap/01_02_model
06/04/2023 10:23:28 MainProcess     MainThread                     train           _output_startup_info           INFO     ===================================================
06/04/2023 10:23:28 MainProcess     MainThread                     train           _output_startup_info           INFO       Starting
06/04/2023 10:23:28 MainProcess     MainThread                     train           _output_startup_info           INFO       Press 'ENTER' to save and quit
06/04/2023 10:23:28 MainProcess     MainThread                     train           _output_startup_info           INFO       Press 'S' to save model weights immediately
06/04/2023 10:23:28 MainProcess     MainThread                     train           _output_startup_info           INFO     ===================================================
06/04/2023 10:23:29 MainProcess     _training                      train           _training                      INFO     Loading data, this may take a while...
06/04/2023 10:23:29 MainProcess     _training                      plugin_loader   _import                        INFO     Loading Model from Original plugin...
06/04/2023 10:23:29 MainProcess     _training                      model           _load                          INFO     No existing state file found. Generating.
06/04/2023 10:23:29 MainProcess     _training                      settings        get_mixed_precision_layers     INFO     Storing Mixed Precision compatible layers. Please ignore any following warnings about using mixed precision.
06/04/2023 10:23:29 MainProcess     _training                      device_compatibility_check _log_device_compatibility_check INFO     Mixed precision compatibility check (mixed_float16): OK\nYour GPU will likely run quickly with dtype policy mixed_float16 as it has compute capability of at least 7.0. Your GPU: NVIDIA GeForce RTX 3080 Ti, compute capability 8.6
06/04/2023 10:23:31 MainProcess     _training                      plugin_loader   _import                        INFO     Loading Trainer from Original plugin...
06/04/2023 10:23:47 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
06/04/2023 10:23:47 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training'
06/04/2023 10:23:52 MainProcess     MainThread                     launcher        execute_script                 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/data/Generation/faceswap/lib/cli/launcher.py", line 230, in execute_script
    process.process()
  File "/data/Generation/faceswap/scripts/train.py", line 213, in process
    self._end_thread(thread, err)
  File "/data/Generation/faceswap/scripts/train.py", line 253, in _end_thread
    thread.join()
  File "/data/Generation/faceswap/lib/multithreading.py", line 220, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/data/Generation/faceswap/lib/multithreading.py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "/data/Generation/faceswap/scripts/train.py", line 275, in _training
    raise err
  File "/data/Generation/faceswap/scripts/train.py", line 265, in _training
    self._run_training_cycle(model, trainer)
  File "/data/Generation/faceswap/scripts/train.py", line 353, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/data/Generation/faceswap/plugins/train/trainer/_base.py", line 223, in train_one_step
    loss: List[float] = self._model.model.train_on_batch(model_inputs, y=model_targets)
  File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 2093, in train_on_batch
    logs = self.train_function(iterator)
  File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:

Detected at node 'original/encoder/conv_128_1_conv2d/Conv2D' defined at (most recent call last):
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/threading.py", line 930, in _bootstrap
      self._bootstrap_inner()
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/threading.py", line 973, in _bootstrap_inner
      self.run()
    File "/data/Generation/faceswap/lib/multithreading.py", line 96, in run
      self._target(*self._args, **self._kwargs)
    File "/data/Generation/faceswap/scripts/train.py", line 265, in _training
      self._run_training_cycle(model, trainer)
    File "/data/Generation/faceswap/scripts/train.py", line 353, in _run_training_cycle
      trainer.train_one_step(viewer, timelapse)
    File "/data/Generation/faceswap/plugins/train/trainer/_base.py", line 223, in train_one_step
      loss: List[float] = self._model.model.train_on_batch(model_inputs, y=model_targets)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 2093, in train_on_batch
      logs = self.train_function(iterator)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1021, in train_function
      return step_function(self, iterator)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1010, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1000, in run_step
      outputs = model.train_step(data)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 859, in train_step
      y_pred = self(x, training=True)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 451, in call
      return self._run_internal_graph(
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 589, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 451, in call
      return self._run_internal_graph(
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 589, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/layers/convolutional.py", line 248, in call
      outputs = self.convolution_op(inputs, self.kernel)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/layers/convolutional.py", line 233, in convolution_op
      return tf.nn.convolution(
Node: 'original/encoder/conv_128_1_conv2d/Conv2D'
DNN library is not found.
     [[{{node original/encoder/conv_128_1_conv2d/Conv2D}}]] [Op:__inference_train_function_7661]
06/04/2023 10:23:52 MainProcess     MainThread                     launcher        execute_script                 CRITICAL An unexpected crash has occurred. Crash report written to '/data/Generation/faceswap/crash_report.2023.06.04.102347805853.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

Here is the crash report I get during extract phase.

Code: Select all

06/04/2023 10:25:27 MainProcess     MainThread                     logger          log_setup                      INFO     Log level set to: INFO
06/04/2023 10:25:30 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Detect from S3Fd plugin...
06/04/2023 10:25:30 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Align from Fan plugin...
06/04/2023 10:25:30 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Mask from Components plugin...
06/04/2023 10:25:30 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Mask from Extended plugin...
06/04/2023 10:25:30 MainProcess     MainThread                     extract         process                        INFO     Starting, this may take a while...
06/04/2023 10:25:30 MainProcess     MainThread                     extract         __init__                       INFO     Output Directory: /data/Generation/faceswap/faces/02/002
06/04/2023 10:25:30 MainProcess     MainThread                     _base           initialize                     INFO     Initializing S3FD (Detect)...
06/04/2023 10:25:32 MainProcess     MainThread                     _base           initialize                     INFO     Initialized S3FD (Detect) with batchsize of 1
06/04/2023 10:25:32 MainProcess     MainThread                     _base           initialize                     INFO     Initializing FAN (Align)...
06/04/2023 10:25:54 MainProcess     MainThread                     launcher        execute_script                 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/data/Generation/faceswap/lib/cli/launcher.py", line 230, in execute_script
    process.process()
  File "/data/Generation/faceswap/scripts/extract.py", line 178, in process
    extract.process()
  File "/data/Generation/faceswap/scripts/extract.py", line 702, in process
    self._run_extraction()
  File "/data/Generation/faceswap/scripts/extract.py", line 722, in _run_extraction
    self._extractor.launch()
  File "/data/Generation/faceswap/plugins/extract/pipeline.py", line 271, in launch
    self._launch_plugin(phase)
  File "/data/Generation/faceswap/plugins/extract/pipeline.py", line 700, in _launch_plugin
    plugin.initialize(**kwargs)
  File "/data/Generation/faceswap/plugins/extract/align/_base/aligner.py", line 199, in initialize
    super().initialize(*args, **kwargs)
  File "/data/Generation/faceswap/plugins/extract/_base.py", line 481, in initialize
    self.init_model()
  File "/data/Generation/faceswap/plugins/extract/align/fan.py", line 50, in init_model
    self.model.predict(placeholder)
  File "/data/Generation/faceswap/lib/model/session.py", line 105, in predict
    return self._model.predict(feed, verbose=0, batch_size=batch_size)
  File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:

Detected at node 'model_1/conv1/Conv2D' defined at (most recent call last):
    File "/data/Generation/faceswap/faceswap.py", line 57, in <module>
      _main()
    File "/data/Generation/faceswap/faceswap.py", line 53, in _main
      arguments.func(arguments)
    File "/data/Generation/faceswap/lib/cli/launcher.py", line 230, in execute_script
      process.process()
    File "/data/Generation/faceswap/scripts/extract.py", line 178, in process
      extract.process()
    File "/data/Generation/faceswap/scripts/extract.py", line 702, in process
      self._run_extraction()
    File "/data/Generation/faceswap/scripts/extract.py", line 722, in _run_extraction
      self._extractor.launch()
    File "/data/Generation/faceswap/plugins/extract/pipeline.py", line 271, in launch
      self._launch_plugin(phase)
    File "/data/Generation/faceswap/plugins/extract/pipeline.py", line 700, in _launch_plugin
      plugin.initialize(**kwargs)
    File "/data/Generation/faceswap/plugins/extract/align/_base/aligner.py", line 199, in initialize
      super().initialize(*args, **kwargs)
    File "/data/Generation/faceswap/plugins/extract/_base.py", line 481, in initialize
      self.init_model()
    File "/data/Generation/faceswap/plugins/extract/align/fan.py", line 50, in init_model
      self.model.predict(placeholder)
    File "/data/Generation/faceswap/lib/model/session.py", line 105, in predict
      return self._model.predict(feed, verbose=0, batch_size=batch_size)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1982, in predict
      tmp_batch_outputs = self.predict_function(iterator)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1801, in predict_function
      return step_function(self, iterator)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1790, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1783, in run_step
      outputs = model.predict_step(data)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1751, in predict_step
      return self(x, training=False)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 451, in call
      return self._run_internal_graph(
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 589, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/layers/convolutional.py", line 248, in call
      outputs = self.convolution_op(inputs, self.kernel)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/layers/convolutional.py", line 233, in convolution_op
      return tf.nn.convolution(
Node: 'model_1/conv1/Conv2D'
DNN library is not found.
     [[{{node model_1/conv1/Conv2D}}]] [Op:__inference_predict_function_17968]
06/04/2023 10:25:54 MainProcess     MainThread                     launcher        execute_script                 CRITICAL An unexpected crash has occurred. Crash report written to '/data/Generation/faceswap/crash_report.2023.06.04.102549152473.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

It is worth noting that when I use mtcnn for detection and cv2-dnn for alignment, the extraction process works normally. That is, use the following command

Code: Select all

 python faceswap.py extract -i /***/01.mp4 -o ./faces/02 -A cv2-dnn -D mtcnn

I'm using an Nvidia 3080Ti graphics card and guaranteed to have 11G of video memory at runtime.

I tried some solutions and still can't solve this problem, hope to get some help.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Got an unexpected crash in training and extracting

Post by torzdf »

Most likely a Cuda conflict, but you haven't provided enough information for me to know for sure. Please provide the files:

  • '/data/Generation/faceswap/crash_report.2023.06.04.102347805853.log'
  • '/data/Generation/faceswap/crash_report.2023.06.04.102549152473.log'

as per the error message output

Last edited by torzdf on Sun Jun 04, 2023 11:50 am, edited 1 time in total.

My word is final

User avatar
bribebh
Posts: 3
Joined: Sun Jun 04, 2023 10:00 am

Re: Got an unexpected crash in training and extracting

Post by bribebh »

torzdf wrote: Sun Jun 04, 2023 11:49 am

Most likely a Cuda conflict, but you haven't provided enough information for me to know for sure. Please provide the files:

  • '/data/Generation/faceswap/crash_report.2023.06.04.102347805853.log'
  • '/data/Generation/faceswap/crash_report.2023.06.04.102549152473.log'

as per the error message output

Here is the error log

Code: Select all

06/04/2023 11:30:05 MainProcess     MainThread                     extract         _set_skip_list                 DEBUG    No frames to be skipped
06/04/2023 11:30:05 MainProcess     MainThread                     fsmedia         __init__                       DEBUG    Initializing PostProcess
06/04/2023 11:30:05 MainProcess     MainThread                     fsmedia         _get_items                     DEBUG    Postprocess Items: {}
06/04/2023 11:30:05 MainProcess     MainThread                     fsmedia         __init__                       DEBUG    Initialized PostProcess
06/04/2023 11:30:05 MainProcess     MainThread                     extract         __init__                       DEBUG    Initialized _Extract
06/04/2023 11:30:05 MainProcess     MainThread                     extract         _threaded_redirector           DEBUG    Threading task: (Task: 'load')
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initializing MultiThread: (target: '_load', thread_count: 1)
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initialized MultiThread: '_load'
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread(s): '_load'
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread 1 of 1: '_load'
06/04/2023 11:30:05 MainProcess     _load                          extract         _load                          DEBUG    Load Images: Start
06/04/2023 11:30:05 MainProcess     _load                          image           load                           DEBUG    Initializing Load Generator
06/04/2023 11:30:05 MainProcess     _load                          multithreading  __init__                       DEBUG    Initializing MultiThread: (target: 'ImagesLoader', thread_count: 1)
06/04/2023 11:30:05 MainProcess     _load                          multithreading  __init__                       DEBUG    Initialized MultiThread: 'ImagesLoader'
06/04/2023 11:30:05 MainProcess     _load                          image           _set_thread                    DEBUG    Set thread: <lib.multithreading.MultiThread object at 0x7f43ffd2ceb0>
06/04/2023 11:30:05 MainProcess     _load                          multithreading  start                          DEBUG    Starting thread(s): 'ImagesLoader'
06/04/2023 11:30:05 MainProcess     _load                          multithreading  start                          DEBUG    Starting thread 1 of 1: 'ImagesLoader'
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  start                          DEBUG    Started all threads '_load': 1
06/04/2023 11:30:05 MainProcess     MainThread                     image           __init__                       DEBUG    Initializing ImagesSaver: (path: /data/Generation/faceswap/faces/02/002, queue_size: 8, as_bytes: True)
06/04/2023 11:30:05 MainProcess     MainThread                     image           __init__                       DEBUG    Initializing ImagesSaver: (path: /data/Generation/faceswap/faces/02/002, queue_size: 8, args: None)
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager adding: (name: 'ImagesSaver', maxsize: 8, create_new: True)
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager added: (name: 'ImagesSaver')
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager getting: 'ImagesSaver'
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager got: 'ImagesSaver'
06/04/2023 11:30:05 MainProcess     MainThread                     pipeline        _launch_plugin                 DEBUG    Launching detect plugin
06/04/2023 11:30:05 MainProcess     MainThread                     pipeline        _launch_plugin                 DEBUG    in_qname: extract0_detect_in, out_qname: extract0_align_in
06/04/2023 11:30:05 MainProcess     MainThread                     _base           initialize                     DEBUG    initialize Detect: (args: (), kwargs: {'in_queue': <lib.queue_manager.EventQueue object at 0x7f43ffcb8790>, 'out_queue': <lib.queue_manager.EventQueue object at 0x7f43ffcb8730>})
06/04/2023 11:30:05 MainProcess     ImagesLoader                   image           _process                       DEBUG    Load iterator: <bound method ImagesLoader._from_video of <lib.image.ImagesLoader object at 0x7f43ffcb8040>>
06/04/2023 11:30:05 MainProcess     ImagesLoader                   image           _from_video                    DEBUG    Loading frames from video: '/data/7_daichouzhen/frames/02/1_0.mp4'
06/04/2023 11:30:05 MainProcess     MainThread                     _base           initialize                     INFO     Initializing S3FD (Detect)...
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager getting: 'detect0_predict_s3fd'
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager adding: (name: 'detect0_predict_s3fd', maxsize: 1, create_new: False)
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager added: (name: 'detect0_predict_s3fd')
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager got: 'detect0_predict_s3fd'
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager getting: 'detect0_post_s3fd'
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager adding: (name: 'detect0_post_s3fd', maxsize: 1, create_new: False)
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager added: (name: 'detect0_post_s3fd')
06/04/2023 11:30:05 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager got: 'detect0_post_s3fd'
06/04/2023 11:30:05 MainProcess     MainThread                     _base           _compile_threads               DEBUG    Compiling detect threads
06/04/2023 11:30:05 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Adding thread: (name: detect_s3fd_input, function: <bound method Extractor._process_input of <plugins.extract.detect.s3fd.Detect object at 0x7f440c20eee0>>, in_queue: <lib.queue_manager.EventQueue object at 0x7f43ffcb8790>, out_queue: <lib.queue_manager.EventQueue object at 0x7f43ffd2ca30>)
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initializing MultiThread: (target: 'detect_s3fd_input', thread_count: 1)
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initialized MultiThread: 'detect_s3fd_input'
06/04/2023 11:30:05 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Added thread: detect_s3fd_input
06/04/2023 11:30:05 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Adding thread: (name: detect_s3fd_predict, function: <bound method Detector._predict of <plugins.extract.detect.s3fd.Detect object at 0x7f440c20eee0>>, in_queue: <lib.queue_manager.EventQueue object at 0x7f43ffd2ca30>, out_queue: <lib.queue_manager.EventQueue object at 0x7f43ffd2cf10>)
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initializing MultiThread: (target: 'detect_s3fd_predict', thread_count: 1)
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initialized MultiThread: 'detect_s3fd_predict'
06/04/2023 11:30:05 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Added thread: detect_s3fd_predict
06/04/2023 11:30:05 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Adding thread: (name: detect_s3fd_output, function: <bound method Extractor._process_output of <plugins.extract.detect.s3fd.Detect object at 0x7f440c20eee0>>, in_queue: <lib.queue_manager.EventQueue object at 0x7f43ffd2cf10>, out_queue: <lib.queue_manager.EventQueue object at 0x7f43ffcb8730>)
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initializing MultiThread: (target: 'detect_s3fd_output', thread_count: 1)
06/04/2023 11:30:05 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initialized MultiThread: 'detect_s3fd_output'
06/04/2023 11:30:05 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Added thread: detect_s3fd_output
06/04/2023 11:30:05 MainProcess     MainThread                     _base           _compile_threads               DEBUG    Compiled detect threads: [<lib.multithreading.MultiThread object at 0x7f43ffd2ca60>, <lib.multithreading.MultiThread object at 0x7f43ffd2cd30>, <lib.multithreading.MultiThread object at 0x7f43ffd2ca00>]
06/04/2023 11:30:05 MainProcess     MainThread                     s3fd            __init__                       DEBUG    Initializing: S3fd: (model_path: '/data/Generation/faceswap/.fs_cache/s3fd_keras_v2.h5', model_kwargs: {'custom_objects': {'L2Norm': <class 'plugins.extract.detect.s3fd.L2Norm'>, 'SliceO2K': <class 'plugins.extract.detect.s3fd.SliceO2K'>}}, allow_growth: False, exclude_gpus: None, confidence: 0.7)
06/04/2023 11:30:05 MainProcess     _load                          multithreading  start                          DEBUG    Started all threads 'ImagesLoader': 1
06/04/2023 11:30:07 MainProcess     MainThread                     session         load_model_weights             VERBOSE  Initializing plugin model: S3FD
06/04/2023 11:30:07 MainProcess     MainThread                     attrs           __getitem__                    DEBUG    Creating converter from 3 to 5
06/04/2023 11:30:08 MainProcess     MainThread                     s3fd            __init__                       DEBUG    Initialized: S3fd
06/04/2023 11:30:08 MainProcess     MainThread                     _base           initialize                     INFO     Initialized S3FD (Detect) with batchsize of 1
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread(s): 'detect_s3fd_input'
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread 1 of 1: 'detect_s3fd_input'
06/04/2023 11:30:08 MainProcess     detect_s3fd_input              _base           _thread_process                DEBUG    threading: (function: '_process_input')
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  start                          DEBUG    Started all threads 'detect_s3fd_input': 1
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread(s): 'detect_s3fd_predict'
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread 1 of 1: 'detect_s3fd_predict'
06/04/2023 11:30:08 MainProcess     detect_s3fd_predict            _base           _thread_process                DEBUG    threading: (function: '_predict')
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  start                          DEBUG    Started all threads 'detect_s3fd_predict': 1
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread(s): 'detect_s3fd_output'
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread 1 of 1: 'detect_s3fd_output'
06/04/2023 11:30:08 MainProcess     detect_s3fd_output             _base           _thread_process                DEBUG    threading: (function: '_process_output')
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  start                          DEBUG    Started all threads 'detect_s3fd_output': 1
06/04/2023 11:30:08 MainProcess     MainThread                     pipeline        _launch_plugin                 DEBUG    Launched detect plugin
06/04/2023 11:30:08 MainProcess     MainThread                     pipeline        _launch_plugin                 DEBUG    Launching align plugin
06/04/2023 11:30:08 MainProcess     MainThread                     pipeline        _launch_plugin                 DEBUG    in_qname: extract0_align_in, out_qname: extract0_mask_0_in
06/04/2023 11:30:08 MainProcess     MainThread                     processing      set_input_size_and_centering   DEBUG    input_size: 256, centering: head
06/04/2023 11:30:08 MainProcess     MainThread                     _base           initialize                     DEBUG    initialize Align: (args: (), kwargs: {'in_queue': <lib.queue_manager.EventQueue object at 0x7f43ffcb8730>, 'out_queue': <lib.queue_manager.EventQueue object at 0x7f43ffcb8d90>})
06/04/2023 11:30:08 MainProcess     MainThread                     _base           initialize                     INFO     Initializing FAN (Align)...
06/04/2023 11:30:08 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager getting: 'align0_predict_fan'
06/04/2023 11:30:08 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager adding: (name: 'align0_predict_fan', maxsize: 1, create_new: False)
06/04/2023 11:30:08 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager added: (name: 'align0_predict_fan')
06/04/2023 11:30:08 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager got: 'align0_predict_fan'
06/04/2023 11:30:08 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager getting: 'align0_post_fan'
06/04/2023 11:30:08 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager adding: (name: 'align0_post_fan', maxsize: 1, create_new: False)
06/04/2023 11:30:08 MainProcess     MainThread                     queue_manager   add_queue                      DEBUG    QueueManager added: (name: 'align0_post_fan')
06/04/2023 11:30:08 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager got: 'align0_post_fan'
06/04/2023 11:30:08 MainProcess     MainThread                     _base           _compile_threads               DEBUG    Compiling align threads
06/04/2023 11:30:08 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Adding thread: (name: align_fan_input, function: <bound method Aligner._process_input of <plugins.extract.align.fan.Align object at 0x7f440aa766a0>>, in_queue: <lib.queue_manager.EventQueue object at 0x7f43ffcb8730>, out_queue: <lib.queue_manager.EventQueue object at 0x7f43e86f54f0>)
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initializing MultiThread: (target: 'align_fan_input', thread_count: 1)
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initialized MultiThread: 'align_fan_input'
06/04/2023 11:30:08 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Added thread: align_fan_input
06/04/2023 11:30:08 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Adding thread: (name: align_fan_predict, function: <bound method Aligner._predict of <plugins.extract.align.fan.Align object at 0x7f440aa766a0>>, in_queue: <lib.queue_manager.EventQueue object at 0x7f43e86f54f0>, out_queue: <lib.queue_manager.EventQueue object at 0x7f43e86f5b20>)
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initializing MultiThread: (target: 'align_fan_predict', thread_count: 1)
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initialized MultiThread: 'align_fan_predict'
06/04/2023 11:30:08 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Added thread: align_fan_predict
06/04/2023 11:30:08 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Adding thread: (name: align_fan_output, function: <bound method Aligner._process_output of <plugins.extract.align.fan.Align object at 0x7f440aa766a0>>, in_queue: <lib.queue_manager.EventQueue object at 0x7f43e86f5b20>, out_queue: <lib.queue_manager.EventQueue object at 0x7f43ffcb8d90>)
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initializing MultiThread: (target: 'align_fan_output', thread_count: 1)
06/04/2023 11:30:08 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initialized MultiThread: 'align_fan_output'
06/04/2023 11:30:08 MainProcess     MainThread                     _base           _add_thread                    DEBUG    Added thread: align_fan_output
06/04/2023 11:30:08 MainProcess     MainThread                     _base           _compile_threads               DEBUG    Compiled align threads: [<lib.multithreading.MultiThread object at 0x7f43ffd2c9a0>, <lib.multithreading.MultiThread object at 0x7f43e86f54c0>, <lib.multithreading.MultiThread object at 0x7f43e86f5220>]
06/04/2023 11:30:08 MainProcess     MainThread                     session         load_model                     VERBOSE  Initializing plugin model: FAN
06/04/2023 11:30:15 MainProcess     detect_s3fd_predict            multithreading  run                            DEBUG    Error in thread (detect_s3fd_predict): Graph execution error:\n\nDetected at node 'model/conv1_1/Conv2D' defined at (most recent call last):\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/threading.py", line 930, in _bootstrap\n      self._bootstrap_inner()\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/threading.py", line 973, in _bootstrap_inner\n      self.run()\n    File "/data/Generation/faceswap/lib/multithreading.py", line 96, in run\n      self._target(*self._args, **self._kwargs)\n    File "/data/Generation/faceswap/plugins/extract/_base.py", line 609, in _thread_process\n      batch = function(batch)\n    File "/data/Generation/faceswap/plugins/extract/detect/_base.py", line 279, in _predict\n      batch.prediction = self.predict(batch.feed)\n    File "/data/Generation/faceswap/plugins/extract/detect/s3fd.py", line 66, in predict\n      predictions = self.model.predict(feed)\n    File "/data/Generation/faceswap/lib/model/session.py", line 105, in predict\n      return self._model.predict(feed, verbose=0, batch_size=batch_size)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler\n      return fn(*args, **kwargs)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1982, in predict\n      tmp_batch_outputs = self.predict_function(iterator)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1801, in predict_function\n      return step_function(self, iterator)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1790, in step_function\n      outputs = model.distribute_strategy.run(run_step, args=(data,))\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1783, in run_step\n      outputs = model.predict_step(data)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1751, in predict_step\n      return self(x, training=False)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler\n      return fn(*args, **kwargs)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1096, in __call__\n      outputs = call_fn(inputs, *args, **kwargs)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler\n      return fn(*args, **kwargs)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 451, in call\n      return self._run_internal_graph(\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 589, in _run_internal_graph\n      outputs = node.layer(*args, **kwargs)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler\n      return fn(*args, **kwargs)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1096, in __call__\n      outputs = call_fn(inputs, *args, **kwargs)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler\n      return fn(*args, **kwargs)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/layers/convolutional.py", line 248, in call\n      outputs = self.convolution_op(inputs, self.kernel)\n    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/layers/convolutional.py", line 233, in convolution_op\n      return tf.nn.convolution(\nNode: 'model/conv1_1/Conv2D'\nDNN library is not found.\n	 [[{{node model/conv1_1/Conv2D}}]] [Op:__inference_predict_function_2270]
Traceback (most recent call last):
  File "/data/Generation/faceswap/lib/cli/launcher.py", line 230, in execute_script
    process.process()
  File "/data/Generation/faceswap/scripts/extract.py", line 178, in process
    extract.process()
  File "/data/Generation/faceswap/scripts/extract.py", line 702, in process
    self._run_extraction()
  File "/data/Generation/faceswap/scripts/extract.py", line 722, in _run_extraction
    self._extractor.launch()
  File "/data/Generation/faceswap/plugins/extract/pipeline.py", line 271, in launch
    self._launch_plugin(phase)
  File "/data/Generation/faceswap/plugins/extract/pipeline.py", line 700, in _launch_plugin
    plugin.initialize(**kwargs)
  File "/data/Generation/faceswap/plugins/extract/align/_base/aligner.py", line 199, in initialize
    super().initialize(*args, **kwargs)
  File "/data/Generation/faceswap/plugins/extract/_base.py", line 481, in initialize
    self.init_model()
  File "/data/Generation/faceswap/plugins/extract/align/fan.py", line 50, in init_model
    self.model.predict(placeholder)
  File "/data/Generation/faceswap/lib/model/session.py", line 105, in predict
    return self._model.predict(feed, verbose=0, batch_size=batch_size)
  File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:

Detected at node 'model_1/conv1/Conv2D' defined at (most recent call last):
    File "/data/Generation/faceswap/faceswap.py", line 57, in <module>
      _main()
    File "/data/Generation/faceswap/faceswap.py", line 53, in _main
      arguments.func(arguments)
    File "/data/Generation/faceswap/lib/cli/launcher.py", line 230, in execute_script
      process.process()
    File "/data/Generation/faceswap/scripts/extract.py", line 178, in process
      extract.process()
    File "/data/Generation/faceswap/scripts/extract.py", line 702, in process
      self._run_extraction()
    File "/data/Generation/faceswap/scripts/extract.py", line 722, in _run_extraction
      self._extractor.launch()
    File "/data/Generation/faceswap/plugins/extract/pipeline.py", line 271, in launch
      self._launch_plugin(phase)
    File "/data/Generation/faceswap/plugins/extract/pipeline.py", line 700, in _launch_plugin
      plugin.initialize(**kwargs)
    File "/data/Generation/faceswap/plugins/extract/align/_base/aligner.py", line 199, in initialize
      super().initialize(*args, **kwargs)
    File "/data/Generation/faceswap/plugins/extract/_base.py", line 481, in initialize
      self.init_model()
    File "/data/Generation/faceswap/plugins/extract/align/fan.py", line 50, in init_model
      self.model.predict(placeholder)
    File "/data/Generation/faceswap/lib/model/session.py", line 105, in predict
      return self._model.predict(feed, verbose=0, batch_size=batch_size)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1982, in predict
      tmp_batch_outputs = self.predict_function(iterator)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1801, in predict_function
      return step_function(self, iterator)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1790, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1783, in run_step
      outputs = model.predict_step(data)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/training.py", line 1751, in predict_step
      return self(x, training=False)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 451, in call
      return self._run_internal_graph(
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/functional.py", line 589, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/layers/convolutional.py", line 248, in call
      outputs = self.convolution_op(inputs, self.kernel)
    File "/data/Generation/environment/envs/faceswap/lib/python3.9/site-packages/keras/layers/convolutional.py", line 233, in convolution_op
      return tf.nn.convolution(
Node: 'model_1/conv1/Conv2D'
DNN library is not found.
	 [[{{node model_1/conv1/Conv2D}}]] [Op:__inference_predict_function_17968]

============ System Information ============
backend:             nvidia
encoding:            UTF-8
git_branch:          master
git_commits:         e2ad3e2 add ru locale (#1311)
gpu_cuda:            11.1
gpu_cudnn:           8.0.5
gpu_devices:         GPU_0: NVIDIA GeForce RTX 3080 Ti, GPU_1: NVIDIA GeForce RTX 3080 Ti, GPU_2: NVIDIA GeForce RTX 3080 Ti
gpu_devices_active:  GPU_2
gpu_driver:          470.182.03
gpu_vram:            GPU_0: 12053MB (60MB free), GPU_1: 12053MB (12051MB free), GPU_2: 12053MB (1282MB free)
os_machine:          x86_64
os_platform:         Linux-4.15.0-212-generic-x86_64-with-glibc2.27
os_release:          4.15.0-212-generic
py_command:          faceswap.py extract -i /data/7_daichouzhen/frames/02/1_0.mp4 -o ./faces/02/002
py_conda_version:    conda 4.7.10
py_implementation:   CPython
py_version:          3.9.12
py_virtual_env:      True
sys_cores:           72
sys_processor:       x86_64
sys_ram:             Total: 257603MB, Available: 221414MB, Used: 33304MB, Free: 214910MB

=============== Pip Packages ===============
absl-py @ file:///croot/absl-py_1666362940888/work
astunparse==1.6.3
brotlipy==0.7.0
cachetools==5.3.0
certifi==2023.5.7
cffi @ file:///tmp/abs_98z5h56wf8/croots/recipe/cffi_1659598650955/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1678108872112/work
cloudpickle @ file:///croot/cloudpickle_1683040006038/work
cryptography @ file:///croot/cryptography_1677533068310/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
dm-tree @ file:///croot/dm-tree_1671027432897/work
fastcluster @ file:///home/conda/feedstock_root/build_artifacts/fastcluster_1667858905653/work
ffmpy==0.3.0
flatbuffers==23.5.9
fonttools==4.25.0
gast @ file:///tmp/build/80754af9/gast_1637837472536/work
google-auth==2.18.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.54.0
h5py==3.8.0
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work
imageio @ file:///croot/imageio_1677879560821/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1673483481485/work
importlib-metadata==6.6.0
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1663332044897/work
keras==2.8.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///croot/kiwisolver_1672387140495/work
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.2
matplotlib @ file:///croot/matplotlib-suite_1667356714455/work
mkl-fft==1.3.1
mkl-random @ file:///tmp/build/80754af9/mkl_random_1626186066731/work
mkl-service==2.4.0
munkres==1.1.4
numexpr @ file:///croot/numexpr_1668713893690/work
numpy @ file:///croot/numpy_and_numpy_base_1672336185480/work
nvidia-ml-py==11.525.112
oauthlib==3.2.2
opencv-python==4.7.0.72
opt-einsum==3.3.0
packaging @ file:///croot/packaging_1678965309396/work
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
Pillow==9.4.0
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1683850015520/work
pooch @ file:///home/conda/feedstock_root/build_artifacts/pooch_1679580333621/work
protobuf==3.19.6
psutil @ file:///opt/conda/conda-bld/psutil_1656431268089/work
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pyOpenSSL @ file:///croot/pyopenssl_1677607685877/work
pyparsing @ file:///opt/conda/conda-bld/pyparsing_1661452539315/work
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1648857263093/work
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1684774241324/work
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1685023709438/work
scipy==1.10.1
sip==4.19.13
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.8.0
tensorflow-estimator==2.10.0
tensorflow-gpu==2.10.1
tensorflow-io-gcs-filesystem==0.32.0
tensorflow-probability @ file:///tmp/build/80754af9/tensorflow-probability_1633017132682/work
termcolor==2.3.0
tf-estimator-nightly==2.8.0.dev2021122109
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work
tornado @ file:///opt/conda/conda-bld/tornado_1662061693373/work
tqdm @ file:///croot/tqdm_1679561862951/work
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1685704949284/work
urllib3 @ file:///croot/urllib3_1680254681959/work
Werkzeug==2.3.4
wrapt==1.15.0
zipp==3.15.0

============== Conda Packages ==============
# packages in environment at /data/Generation/environment/envs/faceswap:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   1.3.0            py39h06a4308_0  
astunparse                1.6.3                    pypi_0    pypi
binutils_impl_linux-64    2.38                 h2a08ee3_1  
blas                      1.0                         mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
brotli                    1.0.9                h166bdaf_8    conda-forge
brotli-bin                1.0.9                h166bdaf_8    conda-forge
brotlipy                  0.7.0           py39h27cfd23_1003  
bzip2                     1.0.8                h7b6447c_0  
c-ares                    1.18.1               h7f8727e_0  
ca-certificates           2023.5.7             hbcca054_0    conda-forge
cachetools                5.3.0                    pypi_0    pypi
certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py39h74dc2b5_0  
charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
cloudpickle               2.2.1            py39h06a4308_0  
cryptography              39.0.1           py39h9ce1e76_0  
cudatoolkit               8.0                           3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
cudnn                     6.0.21                cuda8.0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
curl                      7.26.0                        1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
cycler                    0.11.0             pyhd3eb1b0_0  
dbus                      1.13.18              hb2f20db_0  
decorator                 5.1.1              pyhd3eb1b0_0  
dm-tree                   0.1.7            py39h6a678d5_1  
expat                     2.4.4                h295c915_0  
fastcluster               1.2.6            py39h4661b88_2    conda-forge
ffmpeg                    4.2.2                h20bf706_0  
ffmpy                     0.3.0                    pypi_0    pypi
flatbuffers               23.5.9                   pypi_0    pypi
fontconfig                2.14.1               hef1e5e3_0  
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.12.1               h4a9f257_0  
gast                      0.4.0                    pypi_0    pypi
gcc                       12.1.0              h9ea6d83_10    conda-forge
gcc_impl_linux-64         12.1.0              hea43390_17    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h5eee18b_3  
git                       2.34.1          pl5262hc120c5b_0  
glib                      2.69.1               h4ff587b_1  
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
google-auth               2.18.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.54.0                   pypi_0    pypi
gst-plugins-base          1.14.1               h6a678d5_1  
gstreamer                 1.14.1               h5eee18b_1  
h5py                      3.8.0                    pypi_0    pypi
icu                       58.2                 he6710b0_3  
idna                      3.4                pyhd8ed1ab_0    conda-forge
imageio                   2.26.0           py39h06a4308_0  
imageio-ffmpeg            0.4.8              pyhd8ed1ab_0    conda-forge
importlib-metadata        6.6.0                    pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561  
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h5eee18b_1  
keras                     2.8.0                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
kiwisolver                1.4.4            py39h6a678d5_0  
krb5                      1.19.2               hac12032_0  
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
lerc                      3.0                  h295c915_0  
libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
libbrotlidec              1.0.9                h166bdaf_8    conda-forge
libbrotlienc              1.0.9                h166bdaf_8    conda-forge
libclang                  16.0.0                   pypi_0    pypi
libcurl                   7.82.0               h0b77cf5_0  
libdeflate                1.17                 h5eee18b_0  
libedit                   3.1.20210910         h7f8727e_0  
libev                     4.33                 h7f8727e_1  
libffi                    3.3                  he6710b0_2  
libgcc-devel_linux-64     12.1.0              h1ec3361_17    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libnghttp2                1.46.0               hce63b2e_0  
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h5eee18b_0  
libsanitizer              12.1.0              ha89aaad_17    conda-forge
libssh2                   1.10.0               h8f2d780_0  
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libtiff                   4.5.0                h6a678d5_2  
libvpx                    1.7.0                h439df22_0  
libwebp                   1.2.4                h11a3e52_1  
libwebp-base              1.2.4                h5eee18b_1  
libxcb                    1.15                 h7f8727e_0  
libxml2                   2.9.14               h74e7548_0  
lz4-c                     1.9.4                h6a678d5_0  
markdown                  3.4.3                    pypi_0    pypi
markupsafe                2.1.2                    pypi_0    pypi
matplotlib                3.5.3            py39h06a4308_0  
matplotlib-base           3.5.3            py39hf590b9c_0  
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py39h7f8727e_0  
mkl_fft                   1.3.1            py39hd3c417c_0  
mkl_random                1.2.2            py39h51133e4_0  
munkres                   1.1.4                      py_0  
ncurses                   6.3                  h7f8727e_2  
nettle                    3.6                  he412f7d_0    conda-forge
numexpr                   2.8.4            py39he184ba9_0  
numpy                     1.23.5           py39h14f4228_0  
numpy-base                1.23.5           py39h31eccc5_0  
nvidia-ml-py              11.525.112               pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python             4.7.0.72                 pypi_0    pypi
openh264                  2.1.1                h4ff587b_0  
openssl                   1.1.1u               hd590300_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 23.0             py39h06a4308_0  
pcre                      8.45                 h295c915_0  
pcre2                     10.37                h032f7d1_0    conda-forge
perl                      5.26.2               h14c3975_0  
pexpect                   4.8.0              pyhd3eb1b0_3  
pillow                    9.4.0            py39h6a678d5_0  
pip                       23.0.1           py39h06a4308_0  
platformdirs              3.5.1              pyhd8ed1ab_0    conda-forge
pooch                     1.7.0              pyha770c72_3    conda-forge
protobuf                  3.19.6                   pypi_0    pypi
psutil                    5.9.0            py39h5eee18b_0  
ptyprocess                0.7.0              pyhd3eb1b0_2  
pyasn1                    0.5.0                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pycparser                 2.21               pyhd3eb1b0_0  
pyopenssl                 23.0.0           py39h06a4308_0  
pyparsing                 3.0.9            py39h06a4308_0  
pyqt                      5.9.2            py39h2531618_6  
pysocks                   1.7.1            py39hf3d152e_5    conda-forge
python                    3.9.12               h12debd9_1  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python_abi                3.9                      2_cp39    conda-forge
qt                        5.9.7                h5867ecd_1  
readline                  8.1.2                h7f8727e_1  
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-learn              1.2.2            py39hc236052_2    conda-forge
scipy                     1.10.1           py39h14f4228_0  
setuptools                67.8.0           py39h06a4308_0  
sip                       4.19.13          py39h295c915_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.38.5               hc218d9a_0  
sysroot_linux-64          2.12                he073ed8_15    conda-forge
tensorboard               2.8.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.8.0                    pypi_0    pypi
tensorflow-estimator      2.10.0                   pypi_0    pypi
tensorflow-gpu            2.10.1                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.32.0                   pypi_0    pypi
tensorflow-probability    0.14.0             pyhd3eb1b0_0  
termcolor                 2.3.0                    pypi_0    pypi
tf-estimator-nightly      2.8.0.dev2021122109          pypi_0    pypi
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h1ccaba5_0  
tornado                   6.2              py39h5eee18b_0  
tqdm                      4.65.0           py39hb070fc8_0  
typing-extensions         4.6.3                hd8ed1ab_0    conda-forge
typing_extensions         4.6.3              pyha770c72_0    conda-forge
tzdata                    2023c                h04d1e81_0  
urllib3                   1.26.15          py39h06a4308_0  
werkzeug                  2.3.4                    pypi_0    pypi
wheel                     0.38.4           py39h06a4308_0  
wrapt                     1.15.0                   pypi_0    pypi
x264                      1!157.20191217       h7b6447c_0  
xz                        5.4.2                h5eee18b_0  
zipp                      3.15.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0  
zstd                      1.5.5                hc292b87_0  

================= Configs ==================
--------- extract.ini ---------

[global]
allow_growth:             False
aligner_min_scale:        0.07
aligner_max_scale:        2.0
aligner_distance:         22.5
aligner_roll:             45.0
aligner_features:         True
filter_refeed:            True
save_filtered:            False
realign_refeeds:          True
filter_realign:           True

[recognition.vgg_face2]
batch-size:               16
cpu:                      False

[align.fan]
batch-size:               1

[mask.vgg_clear]
batch-size:               6

[mask.bisenet_fp]
batch-size:               8
cpu:                      False
weights:                  faceswap
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.unet_dfl]
batch-size:               8

[mask.vgg_obstructed]
batch-size:               2

[mask.custom]
batch-size:               8
centering:                face
fill:                     False

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
cpu:                      True
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.cv2_dnn]
confidence:               50

[detect.s3fd]
confidence:               70
batch-size:               1

--------- convert.ini ---------

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[color.match_hist]
threshold:                99.0

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.color_transfer]
clip:                     True
preserve_paper:           True

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0
erosion_top:              0.0
erosion_bottom:           0.0
erosion_left:             0.0
erosion_right:            0.0

[writer.pillow]
format:                   png
draw_transparent:         False
separate_mask:            False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.opencv]
format:                   png
draw_transparent:         False
separate_mask:            False
jpg_quality:              75
png_compress_level:       3

--------- .faceswap ---------
backend:                  nvidia

--------- train.ini ---------

[global]
centering:                face
coverage:                 87.5
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
autoclip:                 False
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
loss_function_2:          mse
loss_weight_2:            100
loss_function_3:          none
loss_weight_3:            0
loss_function_4:          none
loss_weight_4:            0
mask_loss_function:       mse
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.phaze_a]
output_size:              128
shared_fc:                none
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              7
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          none
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_upscales_in_fc:       0
dec_norm:                 none
dec_min_filters:          64
dec_max_filters:          512
dec_slope_mode:           full
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
fs_original_use_alt:      False
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.dfaker]
output_size:              128

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.dfl_sae]
input_size:               128
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.original]
lowmem:                   False

[model.dfl_h128]
lowmem:                   False

[model.unbalanced]
input_size:               128
lowmem:                   False
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
mask_opacity:             30
mask_color:               #ff0000
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Got an unexpected crash in training and extracting

Post by torzdf »

Ok, there are 2 potential issues here....

Firstly remove the globally installed version of Cuda and cuDNN, these will conflict with the locally installed versions used for Faceswap.

Secondly, for some reason, the cudatoolkit did not install correctly into the environment, I can't be sure why this is, but the easiest way to attempt to resolve this is to re-run the installer to re-install Faceswap (this will remove and recreate the environment).

Also, unrelated, but won't hurt, you should update your GPU driver, as you are on quite an old driver there. Actually, I also notice you are running a very old Linux kernel too. To be honest, neither of these things should be an issue, but it won't hurt to update if possible.

If you follow these steps and still hit an issue, please provide the latest crash report and we can look to go from there.

My word is final

User avatar
bribebh
Posts: 3
Joined: Sun Jun 04, 2023 10:00 am

Re: Got an unexpected crash in training and extracting

Post by bribebh »

torzdf wrote: Sun Jun 04, 2023 12:02 pm

Ok, there are 2 potential issues here....

Firstly remove the globally installed version of Cuda and cuDNN, these will conflict with the locally installed versions used for Faceswap.

Secondly, for some reason, the cudatoolkit did not install correctly into the environment, I can't be sure why this is, but the easiest way to attempt to resolve this is to re-run the installer to re-install Faceswap (this will remove and recreate the environment).

Also, unrelated, but won't hurt, you should update your GPU driver, as you are on quite an old driver there. Actually, I also notice you are running a very old Linux kernel too. To be honest, neither of these things should be an issue, but it won't hurt to update if possible.

If you follow these steps and still hit an issue, please provide the latest crash report and we can look to go from there.

Thanks for the suggestion, I'll try it out

Locked