Caught exception in thread: '_training_0' , Got Exception on main handler:

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
azerul
Posts: 1
Joined: Wed May 13, 2020 3:02 am

Caught exception in thread: '_training_0' , Got Exception on main handler:

Post by azerul »

Cant seem to make the training part to work , allow growth is ticked , VRAM Savings all ticked , specs i5-9300 , gtx1650 8 gb

message received on the GUI:

Code: Select all

Loading...
Setting Faceswap backend to NVIDIA
05/13/2020 10:59:46 INFO     Log level set to: INFO
Using TensorFlow backend.
05/13/2020 10:59:49 INFO     Model A Directory: C:\Users\user\Desktop\pika\nut
05/13/2020 10:59:49 INFO     Model B Directory: C:\Users\user\Desktop\pika\claree
05/13/2020 10:59:49 INFO     Training data directory: C:\Users\user\Desktop\pika\trainedblend
05/13/2020 10:59:49 INFO     ===================================================
05/13/2020 10:59:49 INFO       Starting
05/13/2020 10:59:49 INFO       Press 'Stop' to save and quit
05/13/2020 10:59:49 INFO     ===================================================
05/13/2020 10:59:50 INFO     Loading data, this may take a while...
05/13/2020 10:59:53 INFO     Loading Model from Original plugin...
05/13/2020 10:59:53 INFO     Using Pingpong Training
05/13/2020 10:59:53 INFO     Using Optimizer Savings
05/13/2020 10:59:53 INFO     Using Memory Saving Gradients
05/13/2020 10:59:54 WARNING  \nThe TensorFlow contrib module will not be included in TensorFlow 2.0.\nFor more information, please see:\n  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n  * https://github.com/tensorflow/addons\n  * https://github.com/tensorflow/io (for I/O related ops)\nIf you depend on functionality not listed there, please file an issue.\n
05/13/2020 10:59:54 INFO     Using configuration saved in state file
05/13/2020 10:59:56 INFO     Loaded model from disk: 'C:\Users\user\Desktop\pika\trainedblend'
05/13/2020 10:59:56 INFO     Loading Trainer from Original plugin...
05/13/2020 10:59:56 WARNING  Currently TensorBoard logging is not supported for Ping-Pong training. Session stats and graphing will not be available for this training session.
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3262).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3023).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (4437).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3684).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (2690).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3659).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3494).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (4359).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (4248).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3630).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (2648).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3871).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3690).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (2754).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (4375).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3933).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (4361).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3285).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (3670).png'
05/13/2020 10:59:56 ERROR    Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (2887).png'
05/13/2020 11:00:15 INFO     Backing up models...
05/13/2020 11:00:18 INFO     [Saved models] - Average since last save: face_loss_A: 0.16964
05/13/2020 11:00:18 INFO     Switching training to side B
05/13/2020 11:00:20 CRITICAL Error caught! Exiting...
05/13/2020 11:00:20 ERROR    Caught exception in thread: '_training_0'
05/13/2020 11:00:22 ERROR    Got Exception on main handler:
Traceback (most recent call last):
File "C:\Users\user\faceswap\lib\image.py", line 271, in read_image
raise ValueError
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\user\faceswap\lib\cli\launcher.py", line 155, in execute_script
process.process()
File "C:\Users\user\faceswap\scripts\train.py", line 161, in process
self._end_thread(thread, err)
File "C:\Users\user\faceswap\scripts\train.py", line 201, in _end_thread
thread.join()
File "C:\Users\user\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\user\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\user\faceswap\scripts\train.py", line 226, in _training
raise err
File "C:\Users\user\faceswap\scripts\train.py", line 216, in _training
self._run_training_cycle(model, trainer)
File "C:\Users\user\faceswap\scripts\train.py", line 305, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "C:\Users\user\faceswap\plugins\train\trainer\_base.py", line 316, in train_one_step
raise err
File "C:\Users\user\faceswap\plugins\train\trainer\_base.py", line 283, in train_one_step
loss[side] = batcher.train_one_batch()
File "C:\Users\user\faceswap\plugins\train\trainer\_base.py", line 422, in train_one_batch
model_inputs, model_targets = self._get_next()
File "C:\Users\user\faceswap\plugins\train\trainer\_base.py", line 452, in _get_next
batch = next(self._feed)
File "C:\Users\user\faceswap\lib\multithreading.py", line 156, in iterator
self.check_and_raise_error()
File "C:\Users\user\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
raise error[1].with_traceback(error[2])
File "C:\Users\user\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\user\faceswap\lib\multithreading.py", line 145, in _run
for item in self.generator(*self._gen_args, **self._gen_kwargs):
File "C:\Users\user\faceswap\lib\training_data.py", line 189, in _minibatch
yield self._process_batch(img_paths, side)
File "C:\Users\user\faceswap\lib\training_data.py", line 197, in _process_batch
batch = read_image_batch(filenames)
File "C:\Users\user\faceswap\lib\image.py", line 334, in read_image_batch
batch[return_indices[images[future]].pop()] = future.result()
File "C:\Users\user\MiniConda3\envs\134756\lib\concurrent\futures\_base.py", line 428, in result
return self.__get_result()
File "C:\Users\user\MiniConda3\envs\134756\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "C:\Users\user\MiniConda3\envs\134756\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\user\faceswap\lib\image.py", line 284, in read_image
raise Exception(msg)
Exception: Error while reading image. This is most likely caused by special characters in the filename: 'C:\Users\user\Desktop\pika\claree\clare (2887).png'
05/13/2020 11:00:22 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\user\faceswap\crash_report.2020.05.13.110020410344.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Process exited.

and these are the crash reports :

Code: Select all

05/13/2020 09:22:15 MainProcess     MainThread      logger          log_setup                 INFO     Log level set to: INFO
05/13/2020 09:22:15 MainProcess     MainThread      launcher        execute_script            DEBUG    Executing: extract. PID: 21544
05/13/2020 09:22:18 MainProcess     MainThread      launcher        _test_for_tf_version      DEBUG    Installed Tensorflow Version: 1.15
05/13/2020 09:22:18 MainProcess     MainThread      queue_manager   __init__                  DEBUG    Initializing QueueManager
05/13/2020 09:22:18 MainProcess     MainThread      queue_manager   __init__                  DEBUG    Initialized QueueManager
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Caught exception in thread: '_training_0' , Got Exception on main handler:

Post by torzdf »

You should not need any of those savings enabled for original model on an 8 GB card. Turn them all back off.

The issue is (at least) one of your training images (although by the output, it looks like all of your images) being corrupted in some way.

The training images are definitely not named correctly, so are you sure that the folder 'C:\Users\user\Desktop\pika\claree just contains extracted face images?

My word is final

Locked