Unexpected Crash during training

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Littl3Horn
Posts: 3
Joined: Sat Nov 28, 2020 3:54 am
Has thanked: 1 time

Unexpected Crash during training

Post by Littl3Horn »

Some years ago I had used the program, I reinstalled it but now I can't make it work, my PC is the same as before, same components, same graphics card. Please help.


Code: Select all

Loading...
Setting Faceswap backend to NVIDIA
05/23/2022 23:18:44 INFO     Log level set to: INFO
05/23/2022 23:18:47 INFO     Model A Directory: 'C:\Users\USER-PC\Desktop\Nueva carpeta\wilf\Face Swap Project\FaceA' (306 images)
05/23/2022 23:18:48 INFO     Model B Directory: 'C:\Users\USER-PC\Desktop\Nueva carpeta\wilf\Face Swap Project\FaceB' (36225 images)
05/23/2022 23:18:48 INFO     Training data directory: C:\Users\USER-PC\Desktop\Nueva carpeta\wilf\Face Swap Project\ModeloAB
05/23/2022 23:18:48 INFO     ===================================================
05/23/2022 23:18:48 INFO       Starting
05/23/2022 23:18:48 INFO     ===================================================
05/23/2022 23:18:49 INFO     Loading data, this may take a while...
05/23/2022 23:18:49 INFO     Loading Model from Original plugin...
05/23/2022 23:18:49 INFO     Using configuration saved in state file
05/23/2022 23:18:58 INFO     Loaded model from disk: 'C:\Users\USER-PC\Desktop\Nueva carpeta\wilf\Face Swap Project\ModeloAB\original.h5'
05/23/2022 23:18:58 INFO     Loading Trainer from Original plugin...

05/23/2022 23:19:36 INFO     [Saved models] - Average loss since last save: face_a: 0.10534, face_b: 0.10964

05/23/2022 23:21:40 INFO     [Saved models] - Average loss since last save: face_a: 0.08792, face_b: 0.11906

05/23/2022 23:23:41 INFO     [Saved models] - Average loss since last save: face_a: 0.07993, face_b: 0.10859

05/23/2022 23:25:43 INFO     [Saved models] - Average loss since last save: face_a: 0.07514, face_b: 0.10131

05/23/2022 23:26:45 ERROR    Caught exception in thread: '_training_0'
05/23/2022 23:27:08 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "C:\Users\USER-PC\faceswap\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 190, in process
    self._end_thread(thread, err)
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 230, in _end_thread
    thread.join()
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 252, in _training
    raise err
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "C:\Users\USER-PC\faceswap\plugins\train\trainer\_base.py", line 192, in train_one_step
    model_inputs, model_targets = self._feeder.get_batch()
  File "C:\Users\USER-PC\faceswap\plugins\train\trainer\_base.py", line 426, in get_batch
    batch = next(self._feeds[side])
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 156, in iterator
    self.check_and_raise_error()
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 145, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "C:\Users\USER-PC\faceswap\lib\training\generator.py", line 598, in _minibatch
    yield self._process_batch(img_paths, side)
  File "C:\Users\USER-PC\faceswap\lib\training\generator.py", line 621, in _process_batch
    batch = self._face_cache.cache_metadata(filenames)
  File "C:\Users\USER-PC\faceswap\lib\training\generator.py", line 205, in cache_metadata
    batch, metadata = read_image_batch(filenames, with_metadata=True)
  File "C:\Users\USER-PC\faceswap\lib\image.py", line 380, in read_image_batch
    batch = np.array(batch)
ValueError: could not broadcast input array from shape (512,512,3) into shape (512,512)
05/23/2022 23:27:08 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\USER-PC\faceswap\crash_report.2022.05.23.232645995927.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Process exited.
Attachments
crash_report.2022.05.23.232645995927.log
(59.65 KiB) Downloaded 83 times
User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Unexpected Crash during training

Post by torzdf »

The only way I can see this happening is that there is a greyscale (black and white) image in your training folder. This cannot have been generated by Faceswap, because all images are saved as full color images, even if they are originally black and white.

Double check your training images.

My word is final

User avatar
Littl3Horn
Posts: 3
Joined: Sat Nov 28, 2020 3:54 am
Has thanked: 1 time

Re: Unexpected Crash during training

Post by Littl3Horn »

You are right, there was a black and white image, now it scans more faces, but it gave me the following error.

Code: Select all

Loading...
Setting Faceswap backend to NVIDIA
05/24/2022 21:06:08 INFO     Log level set to: INFO
05/24/2022 21:06:11 INFO     Model A Directory: 'C:\Users\USER-PC\Desktop\Nueva carpeta\wilf\Face Swap Project\FaceA' (306 images)
05/24/2022 21:06:12 INFO     Model B Directory: 'C:\Users\USER-PC\Desktop\Nueva carpeta\wilf\Face Swap Project\FaceB' (35792 images)
05/24/2022 21:06:12 INFO     Training data directory: C:\Users\USER-PC\Desktop\Nueva carpeta\wilf\Face Swap Project\ModeloAB
05/24/2022 21:06:12 INFO     ===================================================
05/24/2022 21:06:12 INFO       Starting
05/24/2022 21:06:12 INFO     ===================================================
05/24/2022 21:06:13 INFO     Loading data, this may take a while...
05/24/2022 21:06:13 INFO     Loading Model from Original plugin...
05/24/2022 21:06:13 INFO     Using configuration saved in state file
05/24/2022 21:06:24 INFO     Loaded model from disk: 'C:\Users\USER-PC\Desktop\Nueva carpeta\wilf\Face Swap Project\ModeloAB\original.h5'
05/24/2022 21:06:24 INFO     Loading Trainer from Original plugin...

05/24/2022 21:07:02 INFO     [Saved models] - Average loss since last save: face_a: 0.09649, face_b: 0.12255

05/24/2022 21:08:57 INFO     [Saved models] - Average loss since last save: face_a: 0.08824, face_b: 0.11918

05/24/2022 21:08:59 ERROR    Caught exception in thread: '_training_0'
05/24/2022 21:09:19 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "C:\Users\USER-PC\faceswap\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 190, in process
    self._end_thread(thread, err)
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 230, in _end_thread
    thread.join()
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 252, in _training
    raise err
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Users\USER-PC\faceswap\scripts\train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "C:\Users\USER-PC\faceswap\plugins\train\trainer\_base.py", line 192, in train_one_step
    model_inputs, model_targets = self._feeder.get_batch()
  File "C:\Users\USER-PC\faceswap\plugins\train\trainer\_base.py", line 426, in get_batch
    batch = next(self._feeds[side])
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 156, in iterator
    self.check_and_raise_error()
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\USER-PC\faceswap\lib\multithreading.py", line 145, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "C:\Users\USER-PC\faceswap\lib\training\generator.py", line 598, in _minibatch
    yield self._process_batch(img_paths, side)
  File "C:\Users\USER-PC\faceswap\lib\training\generator.py", line 621, in _process_batch
    batch = self._face_cache.cache_metadata(filenames)
  File "C:\Users\USER-PC\faceswap\lib\training\generator.py", line 205, in cache_metadata
    batch, metadata = read_image_batch(filenames, with_metadata=True)
  File "C:\Users\USER-PC\faceswap\lib\image.py", line 380, in read_image_batch
    batch = np.array(batch)
ValueError: could not broadcast input array from shape (512,512,3) into shape (512,512)
05/24/2022 21:09:19 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\USER-PC\faceswap\crash_report.2022.05.24.210859802790.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Process exited.
Attachments
crash_report.2022.05.24.210859802790.log
(38.6 KiB) Downloaded 82 times
User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Unexpected Crash during training

Post by torzdf »

This is the same error as before, so the problem persists. Training images must be generated from Faceswap's extract process and must not be manipulated outside of Faceswap.

My word is final

User avatar
Jdkalko
Posts: 1
Joined: Fri May 27, 2022 1:02 pm

Re: Unexpected Crash during training

Post by Jdkalko »

I've got the same probleme, can you help me pls if u have the solution ?

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Unexpected Crash during training

Post by torzdf »

As I said in the previous post, this should not be able to happen if the faces have been extracted by the Faceswap process and have not been edited by third party applications.

If you are sure that your facesets have been created by the Faceswap application, and have not been edited, then the only way I can diagnose this issue is to receive a zipped up copy of your training folder to be able to analyze the images.

My word is final

Locked