Caught exception in thread '_training_0'

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Swappersoft
Posts: 1
Joined: Tue Jul 28, 2020 6:53 pm

Caught exception in thread '_training_0'

Post by Swappersoft »

Not sure what's wrong here, attached a crash log. Running most recent version of faceswap, all images for both inputs are the same size, no settings were changed during extraction for any of the videos I used.

Thanks!

Code: Select all

[b]CRASH LOG[/b]

07/28/2020 15:06:05 MainProcess     ThreadPoolExecutor-1234_2 _base           save                      DEBUG    Saving model: 'D:\Training Model CB\original_encoder.h5'
07/28/2020 15:06:05 MainProcess     ThreadPoolExecutor-1234_0 _base           save                      DEBUG    Saving model: 'D:\Training Model CB\original_decoder_A.h5'
07/28/2020 15:06:05 MainProcess     ThreadPoolExecutor-1234_3 serializer      marshal                   DEBUG    data type: <class 'dict'>
07/28/2020 15:06:05 MainProcess     ThreadPoolExecutor-1234_3 serializer      marshal                   DEBUG    returned data type: <class 'bytes'>
07/28/2020 15:06:05 MainProcess     ThreadPoolExecutor-1234_3 _base           save                      DEBUG    Saved State
07/28/2020 15:06:07 MainProcess     _training_0     _base           save_models               INFO     [Saved models] - Average since last save: face_loss_A: 0.08470, face_loss_B: 0.08301
07/28/2020 15:07:30 MainProcess     _training_0     _base           generate_preview          DEBUG    Generating preview
07/28/2020 15:07:30 MainProcess     _training_0     _base           largest_face_index        DEBUG    0
07/28/2020 15:07:30 MainProcess     _training_0     _base           compile_sample            DEBUG    Compiling samples: (side: 'a', samples: 14)
07/28/2020 15:07:31 MainProcess     _training_0     _base           generate_preview          DEBUG    Generating preview
07/28/2020 15:07:31 MainProcess     _training_0     _base           largest_face_index        DEBUG    0
07/28/2020 15:07:31 MainProcess     _training_0     _base           compile_sample            DEBUG    Compiling samples: (side: 'b', samples: 14)
07/28/2020 15:07:31 MainProcess     _training_0     _base           show_sample               DEBUG    Showing sample
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_predictions          DEBUG    Getting Predictions
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_predictions          DEBUG    Returning predictions: {'a_a': (14, 64, 64, 3), 'b_a': (14, 64, 64, 3), 'a_b': (14, 64, 64, 3), 'b_b': (14, 64, 64, 3)}
07/28/2020 15:07:31 MainProcess     _training_0     _base           _to_full_frame            DEBUG    side: 'a', number of sample arrays: 3, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
07/28/2020 15:07:31 MainProcess     _training_0     _base           _frame_overlay            DEBUG    full_size: 256, target_size: 176, color: (0, 0, 255)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _frame_overlay            DEBUG    Overlayed background. Shape: (14, 256, 256, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 176, 176, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 176, 176, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 176, 176, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_headers              DEBUG    side: 'a', width: 128
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_headers              DEBUG    height: 32, total_width: 384
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_headers              DEBUG    texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(72, 9), (116, 9), (102, 9)], text_x: [28, 134, 269], text_y: 20
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_headers              DEBUG    header_box.shape: (32, 384, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _to_full_frame            DEBUG    side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
07/28/2020 15:07:31 MainProcess     _training_0     _base           _frame_overlay            DEBUG    full_size: 256, target_size: 176, color: (0, 0, 255)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _frame_overlay            DEBUG    Overlayed background. Shape: (14, 256, 256, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_headers              DEBUG    side: 'b', width: 128
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_headers              DEBUG    height: 32, total_width: 384
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_headers              DEBUG    texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(59, 9), (87, 9), (102, 9)], text_x: [34, 148, 269], text_y: 20
07/28/2020 15:07:31 MainProcess     _training_0     _base           _get_headers              DEBUG    header_box.shape: (32, 384, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _duplicate_headers        DEBUG    side: a header.shape: (32, 384, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _duplicate_headers        DEBUG    side: b header.shape: (32, 384, 3)
07/28/2020 15:07:31 MainProcess     _training_0     _base           _stack_images             DEBUG    Stack images
07/28/2020 15:07:31 MainProcess     _training_0     _base           get_transpose_axes        DEBUG    Even number of images to stack
07/28/2020 15:07:31 MainProcess     _training_0     _base           _stack_images             DEBUG    Stacked images
07/28/2020 15:07:31 MainProcess     _training_0     _base           show_sample               DEBUG    Compiled sample
07/28/2020 15:07:31 MainProcess     _training_0     _base           save_models               DEBUG    Backing up and saving models
07/28/2020 15:07:31 MainProcess     _training_0     _base           get_save_averages         DEBUG    Getting save averages
07/28/2020 15:07:31 MainProcess     _training_0     _base           get_save_averages         DEBUG    Average losses since last save: {'a': 0.08384809833019972, 'b': 0.08275399021804333}
07/28/2020 15:07:31 MainProcess     _training_0     _base           check_loss_drop           DEBUG    Loss for 'a' has dropped
07/28/2020 15:07:31 MainProcess     _training_0     _base           check_loss_drop           DEBUG    Loss for 'b' has dropped
07/28/2020 15:07:31 MainProcess     _training_0     _base           should_backup             DEBUG    Lowest historical save iteration loss average: {'a': 0.08469658568501473, 'b': 0.08300607495009898}
07/28/2020 15:07:31 MainProcess     _training_0     _base           should_backup             DEBUG    Updating lowest save iteration average for 'a': 0.08384809833019972
07/28/2020 15:07:31 MainProcess     _training_0     _base           should_backup             DEBUG    Updating lowest save iteration average for 'b': 0.08275399021804333
07/28/2020 15:07:31 MainProcess     _training_0     _base           should_backup             DEBUG    Backing up: True
07/28/2020 15:07:31 MainProcess     _training_0     _base           save_models               INFO     Backing up models...
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_0 backup_restore  backup_model              VERBOSE  Backing up: 'D:\Training Model CB\original_decoder_A.h5' to 'D:\Training Model CB\original_decoder_A.h5.bk'
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_1 backup_restore  backup_model              VERBOSE  Backing up: 'D:\Training Model CB\original_decoder_B.h5' to 'D:\Training Model CB\original_decoder_B.h5.bk'
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_2 backup_restore  backup_model              VERBOSE  Backing up: 'D:\Training Model CB\original_encoder.h5' to 'D:\Training Model CB\original_encoder.h5.bk'
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_3 _base           save                      DEBUG    Saving State
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_3 backup_restore  backup_model              VERBOSE  Backing up: 'D:\Training Model CB\original_state.json' to 'D:\Training Model CB\original_state.json.bk'
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_1 _base           save                      DEBUG    Saving model: 'D:\Training Model CB\original_decoder_B.h5'
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_0 _base           save                      DEBUG    Saving model: 'D:\Training Model CB\original_decoder_A.h5'
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_2 _base           save                      DEBUG    Saving model: 'D:\Training Model CB\original_encoder.h5'
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_3 serializer      save                      DEBUG    filename: D:\Training Model CB\original_state.json, data type: <class 'dict'>
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_3 serializer      _check_extension          DEBUG    Original filename: 'D:\Training Model CB\original_state.json', final filename: 'D:\Training Model CB\original_state.json'
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_3 serializer      marshal                   DEBUG    data type: <class 'dict'>
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_3 serializer      marshal                   DEBUG    returned data type: <class 'bytes'>
07/28/2020 15:07:31 MainProcess     ThreadPoolExecutor-1437_3 _base           save                      DEBUG    Saved State
07/28/2020 15:07:33 MainProcess     _training_0     _base           save_models               INFO     [Saved models] - Average since last save: face_loss_A: 0.08385, face_loss_B: 0.08275
07/28/2020 15:08:07 MainProcess     _run_0          multithreading  run                       DEBUG    Error in thread (_run_0): could not broadcast input array from shape (256,256,3) into shape (256)
07/28/2020 15:08:08 MainProcess     _training_0     multithreading  check_and_raise_error     DEBUG    Thread error caught: [(<class 'ValueError'>, ValueError('could not broadcast input array from shape (256,256,3) into shape (256)'), <traceback object at 0x000002CEBB7EA8C8>)]
07/28/2020 15:08:08 MainProcess     _training_0     multithreading  run                       DEBUG    Error in thread (_training_0): could not broadcast input array from shape (256,256,3) into shape (256)
07/28/2020 15:08:08 MainProcess     _run_1          multithreading  run                       DEBUG    Error in thread (_run_1): could not broadcast input array from shape (256,256,3) into shape (256)
07/28/2020 15:08:08 MainProcess     MainThread      train           _monitor                  DEBUG    Thread error detected
07/28/2020 15:08:08 MainProcess     MainThread      train           _monitor                  DEBUG    Closed Monitor
07/28/2020 15:08:08 MainProcess     MainThread      train           _end_thread               DEBUG    Ending Training thread
07/28/2020 15:08:08 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
07/28/2020 15:08:08 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Threads: '_training'
07/28/2020 15:08:08 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Thread: '_training_0'
07/28/2020 15:08:08 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "D:\faceswap\lib\cli\launcher.py", line 155, in execute_script
    process.process()
  File "D:\faceswap\scripts\train.py", line 161, in process
    self._end_thread(thread, err)
  File "D:\faceswap\scripts\train.py", line 201, in _end_thread
    thread.join()
  File "D:\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "D:\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "D:\faceswap\scripts\train.py", line 226, in _training
    raise err
  File "D:\faceswap\scripts\train.py", line 216, in _training
    self._run_training_cycle(model, trainer)
  File "D:\faceswap\scripts\train.py", line 305, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "D:\faceswap\plugins\train\trainer\_base.py", line 316, in train_one_step
    raise err
  File "D:\faceswap\plugins\train\trainer\_base.py", line 283, in train_one_step
    loss[side] = batcher.train_one_batch()
  File "D:\faceswap\plugins\train\trainer\_base.py", line 422, in train_one_batch
    model_inputs, model_targets = self._get_next()
  File "D:\faceswap\plugins\train\trainer\_base.py", line 452, in _get_next
    batch = next(self._feed)
  File "D:\faceswap\lib\multithreading.py", line 156, in iterator
    self.check_and_raise_error()
  File "D:\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "D:\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "D:\faceswap\lib\multithreading.py", line 145, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "D:\faceswap\lib\training_data.py", line 189, in _minibatch
    yield self._process_batch(img_paths, side)
  File "D:\faceswap\lib\training_data.py", line 197, in _process_batch
    batch = read_image_batch(filenames)
  File "D:\faceswap\lib\image.py", line 335, in read_image_batch
    batch = np.array(batch)
ValueError: could not broadcast input array from shape (256,256,3) into shape (256)

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         3fd26b5 Manual Tool (#1038)
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1060 3GB
gpu_devices_active:  GPU_0
gpu_driver:          445.75
gpu_vram:            GPU_0: 3072MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.18362-SP0
os_release:          10
py_command:          D:\faceswap\faceswap.py train -A C:/Users/Ryan/Desktop/FS TEST/EXTRACTED FACES (VIDEO)/DH/First Extracted Faces -ala C:/Users/Ryan/Desktop/FS TEST/EXTRACTED FACES (VIDEO)/DH/First Extracted Faces/191208_2234_720P_4000K_54001521_alignments.fsa -B C:/Users/Ryan/Desktop/FS TEST/MODELS/CB/Training -alb C:/Users/Ryan/Desktop/FS TEST/MODELS/CB/Training/alignments_merged_20200728_143055.fsa -m D:/Training Model CB -t original -bs 6 -it 1000000 -g 1 -o -s 100 -ss 25000 -ps 50 -L INFO -gui
py_conda_version:    conda 4.8.3
py_implementation:   CPython
py_version:          3.7.7
py_virtual_env:      True
sys_cores:           6
sys_processor:       Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram:             Total: 16316MB, Available: 7118MB, Used: 9197MB, Free: 7118MB

=============== Pip Packages ===============
absl-py==0.9.0
astor==0.8.0
blinker==1.4
brotlipy==0.7.0
cachetools==4.1.0
certifi==2020.6.20
cffi==1.14.0
chardet==3.0.4
click==7.1.2
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1594141588948/work
cryptography==2.9.2
cycler==0.10.0
cytoolz==0.10.1
dask @ file:///tmp/build/80754af9/dask-core_1594156306305/work
decorator==4.4.2
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.2.2
google-auth @ file:///tmp/build/80754af9/google-auth_1594357566944/work
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.27.2
h5py==2.10.0
idna @ file:///tmp/build/80754af9/idna_1593446292537/work
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1589202782679/work
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
Keras==2.2.4
Keras-Applications @ file:///tmp/build/80754af9/keras-applications_1594366238411/work
Keras-Preprocessing==1.1.0
kiwisolver==1.2.0
Markdown==3.1.1
matplotlib @ file:///C:/ci/matplotlib-base_1592846084747/work
mkl-fft==1.1.0
mkl-random==1.1.1
mkl-service==2.3.0
networkx @ file:///tmp/build/80754af9/networkx_1594377231366/work
numpy==1.18.5
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.3.0.36
opt-einsum==3.1.0
Pillow @ file:///C:/ci/pillow_1594298234712/work
protobuf==3.12.3
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
PyJWT==1.7.1
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1594392929924/work
pyparsing==2.4.7
pyreadline==2.1
PySocks @ file:///C:/ci/pysocks_1594394709107/work
python-dateutil==2.8.1
PyWavelets==1.1.1
pywin32==227
PyYAML==5.3.1
requests @ file:///tmp/build/80754af9/requests_1592841827918/work
requests-oauthlib==1.3.0
rsa==4.0
scikit-image==0.16.2
scikit-learn @ file:///C:/ci/scikit-learn_1592847564598/work
scipy @ file:///C:/ci/scipy_1592916958183/work
six==1.15.0
tensorboard==2.2.1
tensorboard-plugin-wit==1.6.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
toolz==0.10.0
toposort==1.5
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1593446365756/work
urllib3==1.25.9
Werkzeug==0.16.1
win-inet-pton==1.1.0
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\Ryan\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu  
absl-py 0.9.0 py37_0
astor 0.8.0 py37_0
blas 1.0 mkl
blinker 1.4 py37_0
brotlipy 0.7.0 py37he774522_1000
ca-certificates 2020.6.24 0
cachetools 4.1.0 py_1
certifi 2020.6.20 py37_0
cffi 1.14.0 py37h7a1dbc1_0
chardet 3.0.4 py37_1003
click 7.1.2 py_0
cloudpickle 1.5.0 py_0
cryptography 2.9.2 py37h7a1dbc1_0
cudatoolkit 10.0.130 0
cudnn 7.6.5 cuda10.0_0
cycler 0.10.0 py37_0
cytoolz 0.10.1 py37he774522_0
dask-core 2.20.0 py_0
decorator 4.4.2 py_0
fastcluster 1.1.26 py37h9b59f54_1 conda-forge ffmpeg 4.3 ha925a31_0 conda-forge ffmpy 0.2.3 pypi_0 pypi freetype 2.10.2 hd328e21_0
gast 0.2.2 py37_0
git 2.23.0 h6bb4b03_0
google-auth 1.17.2 py_0
google-auth-oauthlib 0.4.1 py_2
google-pasta 0.2.0 py_0
grpcio 1.27.2 py37h351948d_0
h5py 2.10.0 py37h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 py_0
imageio 2.9.0 py_0
imageio-ffmpeg 0.4.2 py_0 conda-forge intel-openmp 2020.1 216
joblib 0.16.0 py_0
jpeg 9b hb83a4c4_2
keras 2.2.4 0
keras-applications 1.0.8 py_1
keras-base 2.2.4 py37_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.2.0 py37h74a9793_0
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.12.3 h7bd577a_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 h62dcd97_0
markdown 3.1.1 py37_0
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py37h64f37c6_0
mkl 2020.1 216
mkl-service 2.3.0 py37hb782905_0
mkl_fft 1.1.0 py37h45dec08_0
mkl_random 1.1.1 py37h47e9c7a_0
networkx 2.4 py_1
numpy 1.18.5 py37h6530119_0
numpy-base 1.18.5 py37hc3f5095_0
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 py_0
olefile 0.46 py37_0
opencv-python 4.3.0.36 pypi_0 pypi openssl 1.1.1g he774522_0
opt_einsum 3.1.0 py_0
pathlib 1.0.1 py37_2
pillow 7.2.0 py37hcc1f983_0
pip 20.1.1 py37_1
protobuf 3.12.3 py37h33f27b4_0
psutil 5.7.0 py37he774522_0
pyasn1 0.4.8 py_0
pyasn1-modules 0.2.7 py_0
pycparser 2.20 py_2
pyjwt 1.7.1 py37_0
pyopenssl 19.1.0 py_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py37h6538335_2
pyreadline 2.1 py37_1
pysocks 1.7.1 py37_1
python 3.7.7 h81c818b_4
python-dateutil 2.8.1 py_0
python_abi 3.7 1_cp37m conda-forge pywavelets 1.1.1 py37he774522_0
pywin32 227 py37he774522_1
pyyaml 5.3.1 py37he774522_1
qt 5.9.7 vc14h73c81de_0
requests 2.24.0 py_0
requests-oauthlib 1.3.0 py_0
rsa 4.0 py_0
scikit-image 0.16.2 py37h47e9c7a_0
scikit-learn 0.23.1 py37h25d0782_0
scipy 1.5.0 py37h9439919_0
setuptools 49.2.0 py37_0
sip 4.19.8 py37h6538335_0
six 1.15.0 py_0
sqlite 3.32.3 h2a8f88b_0
tensorboard 2.2.1 pyh532a8cf_0
tensorboard-plugin-wit 1.6.0 py_0
tensorflow 1.15.0 gpu_py37hc3743a6_0
tensorflow-base 1.15.0 gpu_py37h1afeea4_0
tensorflow-estimator 1.15.1 pyh2649769_0
tensorflow-gpu 1.15.0 h0d30ee6_0
termcolor 1.1.0 py37_1
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
toolz 0.10.0 py_0
toposort 1.5 py_3 conda-forge tornado 6.0.4 py37he774522_1
tqdm 4.47.0 py_0
urllib3 1.25.9 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_3
werkzeug 0.16.1 py_0
wheel 0.34.2 py37_0
win_inet_pton 1.1.0 py37_0
wincertstore 0.2 py37_0
wrapt 1.12.1 py37he774522_1
xz 5.2.5 h62dcd97_0
yaml 0.2.5 he774522_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.5 ha9fde0e_0 =============== State File ================= { "name": "original", "sessions": { "1": { "timestamp": 1595961223.3587685, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "2": { "timestamp": 1595961300.3858752, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 32, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "3": { "timestamp": 1595961405.7269564, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 20, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "4": { "timestamp": 1595961577.6140952, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 8, "iterations": 101, "config": { "learning_rate": 5e-05 } }, "5": { "timestamp": 1595961968.4209135, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 6, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "6": { "timestamp": 1595962090.1445346, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 6, "iterations": 101, "config": { "learning_rate": 5e-05 } }, "7": { "timestamp": 1595962634.649804, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 6, "iterations": 701, "config": { "learning_rate": 5e-05 } } }, "lowest_avg_loss": { "a": 0.08384809833019972, "b": 0.08275399021804333 }, "iterations": 907, "inputs": { "face_in:0": [ 64, 64, 3 ] }, "training_size": 256, "config": { "coverage": 68.75, "mask_type": null, "mask_blur_kernel": 3, "mask_threshold": 4, "learn_mask": false, "icnr_init": false, "conv_aware_init": false, "reflect_padding": false, "penalized_mask_loss": true, "loss_function": "mae", "learning_rate": 5e-05, "lowmem": false } } ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: unsharp_mask amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 70 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] coverage: 68.75 mask_type: none mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False icnr_init: False conv_aware_init: False reflect_padding: False penalized_mask_loss: True loss_function: mae learning_rate: 5e-05 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Caught exception in thread '_training_0'

Post by torzdf »

It looks like you have managed to train successfully in the past, but are now having issues.

The issue is definitely coming from loading training images.

I suggest that you take a copy of your model, then attempt to train with a limited selection of known working images and see if the issue persists. If it this fixes the issue, then you can see if you can identify the problematic image(s)

My word is final

Locked