Critical error

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
ysjen
Posts: 2
Joined: Thu Jun 23, 2022 6:15 pm

Critical error

Post by ysjen »

This used to work on my old pc, but have installed on a multithread server which has more capacity, but running into issues when running the training? is this a compatibility issue?

Code: Select all

06/23/2022 19:02:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
06/23/2022 19:02:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
06/23/2022 19:02:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
06/23/2022 19:02:44 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 14, batchsize: 14, side: 'a', do_shuffle: False, is_preview, False, is_timelapse: True)
06/23/2022 19:02:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
06/23/2022 19:02:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
06/23/2022 19:02:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
06/23/2022 19:02:44 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 14, side: 'a', do_shuffle: False)
06/23/2022 19:02:44 MainProcess     _run_0                         multithreading  run                            DEBUG    Error in thread (_run_0): '20200809_182018_000001_1.jpg'
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
06/23/2022 19:02:44 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 14, side: 'a', do_shuffle: False)
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
06/23/2022 19:02:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
06/23/2022 19:02:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
06/23/2022 19:02:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
06/23/2022 19:02:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
06/23/2022 19:02:44 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 14, batchsize: 14, side: 'b', do_shuffle: False, is_preview, False, is_timelapse: True)
06/23/2022 19:02:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
06/23/2022 19:02:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
06/23/2022 19:02:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
06/23/2022 19:02:44 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 14, side: 'b', do_shuffle: False)
06/23/2022 19:02:44 MainProcess     _run_0                         multithreading  run                            DEBUG    Error in thread (_run_0): '10801st_000001_1.jpg'
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
06/23/2022 19:02:44 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 14, side: 'b', do_shuffle: False)
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
06/23/2022 19:02:44 MainProcess     _training_0                    _base           set_timelapse_feed             DEBUG    Set time-lapse feed: {'a': <generator object BackgroundGenerator.iterator at 0x000001F906352270>, 'b': <generator object BackgroundGenerator.iterator at 0x000001F90ADD8270>}
06/23/2022 19:02:44 MainProcess     _training_0                    _base           _setup                         DEBUG    Set up time-lapse
06/23/2022 19:02:44 MainProcess     _training_0                    _base           output_timelapse               DEBUG    Getting time-lapse samples
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  check_and_raise_error          DEBUG    Thread error caught: [(<class 'KeyError'>, KeyError('20200809_182018_000001_1.jpg'), <traceback object at 0x000001F90D609440>)]
06/23/2022 19:02:44 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): '20200809_182018_000001_1.jpg'
06/23/2022 19:02:44 MainProcess     _run_1                         multithreading  run                            DEBUG    Error in thread (_run_1): '20200809_182018_000001_1.jpg'
06/23/2022 19:02:45 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
06/23/2022 19:02:45 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
06/23/2022 19:02:45 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
06/23/2022 19:02:45 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
06/23/2022 19:02:45 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
06/23/2022 19:02:45 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
06/23/2022 19:02:45 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Critical error

Post by torzdf »

Please post the full crash report from the faceswap folder when posting issues.

However, I'm pretty sure here that you have images in the timelapse folder that does not exist in your training folder.

My word is final

Locked