This used to work on my old pc, but have installed on a multithread server which has more capacity, but running into issues when running the training? is this a compatibility issue?
Code: Select all
06/23/2022 19:02:44 MainProcess _training_0 _base _load_generator DEBUG input_size: 64, output_shapes: [(64, 64, 3)]
06/23/2022 19:02:44 MainProcess _training_0 generator __init__ DEBUG Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
06/23/2022 19:02:44 MainProcess _training_0 generator __init__ DEBUG Initialized TrainingDataGenerator
06/23/2022 19:02:44 MainProcess _training_0 generator minibatch_ab DEBUG Queue batches: (image_count: 14, batchsize: 14, side: 'a', do_shuffle: False, is_preview, False, is_timelapse: True)
06/23/2022 19:02:44 MainProcess _training_0 augmentation __init__ DEBUG Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
06/23/2022 19:02:44 MainProcess _training_0 augmentation __init__ DEBUG Output sizes: [64]
06/23/2022 19:02:44 MainProcess _training_0 augmentation __init__ DEBUG Initialized ImageAugmentation
06/23/2022 19:02:44 MainProcess _training_0 multithreading __init__ DEBUG Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
06/23/2022 19:02:44 MainProcess _training_0 multithreading __init__ DEBUG Initialized BackgroundGenerator: '_run'
06/23/2022 19:02:44 MainProcess _training_0 multithreading start DEBUG Starting thread(s): '_run'
06/23/2022 19:02:44 MainProcess _training_0 multithreading start DEBUG Starting thread 1 of 2: '_run_0'
06/23/2022 19:02:44 MainProcess _run_0 generator _minibatch DEBUG Loading minibatch generator: (image_count: 14, side: 'a', do_shuffle: False)
06/23/2022 19:02:44 MainProcess _run_0 multithreading run DEBUG Error in thread (_run_0): '20200809_182018_000001_1.jpg'
06/23/2022 19:02:44 MainProcess _training_0 multithreading start DEBUG Starting thread 2 of 2: '_run_1'
06/23/2022 19:02:44 MainProcess _run_1 generator _minibatch DEBUG Loading minibatch generator: (image_count: 14, side: 'a', do_shuffle: False)
06/23/2022 19:02:44 MainProcess _training_0 multithreading start DEBUG Started all threads '_run': 2
06/23/2022 19:02:44 MainProcess _training_0 _base _load_generator DEBUG Loading generator
06/23/2022 19:02:44 MainProcess _training_0 _base _load_generator DEBUG input_size: 64, output_shapes: [(64, 64, 3)]
06/23/2022 19:02:44 MainProcess _training_0 generator __init__ DEBUG Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
06/23/2022 19:02:44 MainProcess _training_0 generator __init__ DEBUG Initialized TrainingDataGenerator
06/23/2022 19:02:44 MainProcess _training_0 generator minibatch_ab DEBUG Queue batches: (image_count: 14, batchsize: 14, side: 'b', do_shuffle: False, is_preview, False, is_timelapse: True)
06/23/2022 19:02:44 MainProcess _training_0 augmentation __init__ DEBUG Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
06/23/2022 19:02:44 MainProcess _training_0 augmentation __init__ DEBUG Output sizes: [64]
06/23/2022 19:02:44 MainProcess _training_0 augmentation __init__ DEBUG Initialized ImageAugmentation
06/23/2022 19:02:44 MainProcess _training_0 multithreading __init__ DEBUG Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
06/23/2022 19:02:44 MainProcess _training_0 multithreading __init__ DEBUG Initialized BackgroundGenerator: '_run'
06/23/2022 19:02:44 MainProcess _training_0 multithreading start DEBUG Starting thread(s): '_run'
06/23/2022 19:02:44 MainProcess _training_0 multithreading start DEBUG Starting thread 1 of 2: '_run_0'
06/23/2022 19:02:44 MainProcess _run_0 generator _minibatch DEBUG Loading minibatch generator: (image_count: 14, side: 'b', do_shuffle: False)
06/23/2022 19:02:44 MainProcess _run_0 multithreading run DEBUG Error in thread (_run_0): '10801st_000001_1.jpg'
06/23/2022 19:02:44 MainProcess _training_0 multithreading start DEBUG Starting thread 2 of 2: '_run_1'
06/23/2022 19:02:44 MainProcess _run_1 generator _minibatch DEBUG Loading minibatch generator: (image_count: 14, side: 'b', do_shuffle: False)
06/23/2022 19:02:44 MainProcess _training_0 multithreading start DEBUG Started all threads '_run': 2
06/23/2022 19:02:44 MainProcess _training_0 _base set_timelapse_feed DEBUG Set time-lapse feed: {'a': <generator object BackgroundGenerator.iterator at 0x000001F906352270>, 'b': <generator object BackgroundGenerator.iterator at 0x000001F90ADD8270>}
06/23/2022 19:02:44 MainProcess _training_0 _base _setup DEBUG Set up time-lapse
06/23/2022 19:02:44 MainProcess _training_0 _base output_timelapse DEBUG Getting time-lapse samples
06/23/2022 19:02:44 MainProcess _training_0 multithreading check_and_raise_error DEBUG Thread error caught: [(<class 'KeyError'>, KeyError('20200809_182018_000001_1.jpg'), <traceback object at 0x000001F90D609440>)]
06/23/2022 19:02:44 MainProcess _training_0 multithreading run DEBUG Error in thread (_training_0): '20200809_182018_000001_1.jpg'
06/23/2022 19:02:44 MainProcess _run_1 multithreading run DEBUG Error in thread (_run_1): '20200809_182018_000001_1.jpg'
06/23/2022 19:02:45 MainProcess MainThread train _monitor DEBUG Thread error detected
06/23/2022 19:02:45 MainProcess MainThread train _monitor DEBUG Closed Monitor
06/23/2022 19:02:45 MainProcess MainThread train _end_thread DEBUG Ending Training thread
06/23/2022 19:02:45 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...
06/23/2022 19:02:45 MainProcess MainThread multithreading join DEBUG Joining Threads: '_training'
06/23/2022 19:02:45 MainProcess MainThread multithreading join DEBUG Joining Thread: '_training_0'
06/23/2022 19:02:45 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training_0'
Traceback (most recent call last):