Hello,
I try to create a DFaker model using Warp to Landmarks but after caching the landmarks, it always fails. It can train without WTL. Another model I had previously worked fine with WTL, does anybody know what the issue could be?
Crash report is here:
Code: Select all
04/30/2021 11:37:31 MainProcess _training_0 generator _get_cache DEBUG Creating cache. Side: a
04/30/2021 11:37:31 MainProcess _training_0 augmentation __init__ DEBUG Initializing ImageAugmentation: (batchsize: 14, is_display: False, input_size: 64, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:37:31 MainProcess _training_0 augmentation __init__ DEBUG Output sizes: [128]
04/30/2021 11:37:31 MainProcess _training_0 augmentation __init__ DEBUG Initialized ImageAugmentation
04/30/2021 11:37:31 MainProcess _training_0 image read_image_meta_batch DEBUG Submitting 15522 items to executor
04/30/2021 11:37:32 MainProcess _training_0 image read_image_meta_batch DEBUG Succesfully submitted 15522 items to executor
04/30/2021 11:37:32 MainProcess _training_0 generator _validate_version DEBUG Setting initial extract version: 2.1
04/30/2021 11:38:01 MainProcess _training_0 multithreading __init__ DEBUG Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
04/30/2021 11:38:01 MainProcess _training_0 multithreading __init__ DEBUG Initialized BackgroundGenerator: '_run'
04/30/2021 11:38:01 MainProcess _training_0 multithreading start DEBUG Starting thread(s): '_run'
04/30/2021 11:38:01 MainProcess _training_0 multithreading start DEBUG Starting thread 1 of 2: '_run_0'
04/30/2021 11:38:01 MainProcess _run_0 generator _minibatch DEBUG Loading minibatch generator: (image_count: 15522, side: 'a', do_shuffle: True)
04/30/2021 11:38:01 MainProcess _training_0 multithreading start DEBUG Starting thread 2 of 2: '_run_1'
04/30/2021 11:38:01 MainProcess _run_1 generator _minibatch DEBUG Loading minibatch generator: (image_count: 15522, side: 'a', do_shuffle: True)
04/30/2021 11:38:01 MainProcess _training_0 multithreading start DEBUG Started all threads '_run': 2
04/30/2021 11:38:01 MainProcess _training_0 _base _load_generator DEBUG Loading generator
04/30/2021 11:38:01 MainProcess _training_0 _base _load_generator DEBUG input_size: 64, output_shapes: [(128, 128, 3)]
04/30/2021 11:38:01 MainProcess _training_0 generator __init__ DEBUG Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: True, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:01 MainProcess _training_0 generator __init__ DEBUG Initialized TrainingDataGenerator
04/30/2021 11:38:01 MainProcess _training_0 generator minibatch_ab DEBUG Queue batches: (image_count: 16482, batchsize: 14, side: 'b', do_shuffle: True, is_preview, False, is_timelapse: False)
04/30/2021 11:38:01 MainProcess _training_0 generator _get_cache DEBUG Creating cache. Side: b
04/30/2021 11:38:01 MainProcess _training_0 augmentation __init__ DEBUG Initializing ImageAugmentation: (batchsize: 14, is_display: False, input_size: 64, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:01 MainProcess _training_0 augmentation __init__ DEBUG Output sizes: [128]
04/30/2021 11:38:01 MainProcess _training_0 augmentation __init__ DEBUG Initialized ImageAugmentation
04/30/2021 11:38:01 MainProcess _training_0 image read_image_meta_batch DEBUG Submitting 16482 items to executor
04/30/2021 11:38:01 MainProcess _training_0 image read_image_meta_batch DEBUG Succesfully submitted 16482 items to executor
04/30/2021 11:38:01 MainProcess _training_0 generator _validate_version DEBUG Setting initial extract version: 2.1
04/30/2021 11:38:01 MainProcess _run_0 augmentation initialize DEBUG Initializing constants. training_size: 384
04/30/2021 11:38:01 MainProcess _run_0 augmentation initialize DEBUG Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]]', 'warp_lm_grids': '[[[ 0. 0. 0. ... 0. 0. 0.]\n [ 1. 1. 1. ... 1. 1. 1.]\n [ 2. 2. 2. ... 2. 2. 2.]\n ...\n [381. 381. 381. ... 381. 381. 381.]\n [382. 382. 382. ... 382. 382. 382.]\n [383. 383. 383. ... 383. 383. 383.]]\n\n [[ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]\n ...\n [ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]]]'}
04/30/2021 11:38:01 MainProcess _run_0 multithreading run DEBUG Error in thread (_run_0): 'aligned_face'
04/30/2021 11:38:34 MainProcess _training_0 multithreading __init__ DEBUG Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
04/30/2021 11:38:34 MainProcess _training_0 multithreading __init__ DEBUG Initialized BackgroundGenerator: '_run'
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Starting thread(s): '_run'
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Starting thread 1 of 2: '_run_0'
04/30/2021 11:38:34 MainProcess _run_0 generator _minibatch DEBUG Loading minibatch generator: (image_count: 16482, side: 'b', do_shuffle: True)
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Starting thread 2 of 2: '_run_1'
04/30/2021 11:38:34 MainProcess _run_1 generator _minibatch DEBUG Loading minibatch generator: (image_count: 16482, side: 'b', do_shuffle: True)
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Started all threads '_run': 2
04/30/2021 11:38:34 MainProcess _training_0 _base _set_preview_feed DEBUG Setting preview feed: (side: 'a')
04/30/2021 11:38:34 MainProcess _training_0 _base _load_generator DEBUG Loading generator
04/30/2021 11:38:34 MainProcess _training_0 _base _load_generator DEBUG input_size: 64, output_shapes: [(128, 128, 3)]
04/30/2021 11:38:34 MainProcess _training_0 generator __init__ DEBUG Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: True, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:34 MainProcess _training_0 generator __init__ DEBUG Initialized TrainingDataGenerator
04/30/2021 11:38:34 MainProcess _training_0 generator minibatch_ab DEBUG Queue batches: (image_count: 15522, batchsize: 14, side: 'a', do_shuffle: True, is_preview, True, is_timelapse: False)
04/30/2021 11:38:34 MainProcess _training_0 augmentation __init__ DEBUG Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:34 MainProcess _training_0 augmentation __init__ DEBUG Output sizes: [128]
04/30/2021 11:38:34 MainProcess _training_0 augmentation __init__ DEBUG Initialized ImageAugmentation
04/30/2021 11:38:34 MainProcess _training_0 multithreading __init__ DEBUG Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
04/30/2021 11:38:34 MainProcess _training_0 multithreading __init__ DEBUG Initialized BackgroundGenerator: '_run'
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Starting thread(s): '_run'
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Starting thread 1 of 2: '_run_0'
04/30/2021 11:38:34 MainProcess _run_0 generator _minibatch DEBUG Loading minibatch generator: (image_count: 15522, side: 'a', do_shuffle: True)
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Starting thread 2 of 2: '_run_1'
04/30/2021 11:38:34 MainProcess _run_1 generator _minibatch DEBUG Loading minibatch generator: (image_count: 15522, side: 'a', do_shuffle: True)
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Started all threads '_run': 2
04/30/2021 11:38:34 MainProcess _training_0 _base _set_preview_feed DEBUG Setting preview feed: (side: 'b')
04/30/2021 11:38:34 MainProcess _training_0 _base _load_generator DEBUG Loading generator
04/30/2021 11:38:34 MainProcess _training_0 _base _load_generator DEBUG input_size: 64, output_shapes: [(128, 128, 3)]
04/30/2021 11:38:34 MainProcess _training_0 generator __init__ DEBUG Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: True, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:34 MainProcess _training_0 generator __init__ DEBUG Initialized TrainingDataGenerator
04/30/2021 11:38:34 MainProcess _training_0 generator minibatch_ab DEBUG Queue batches: (image_count: 16482, batchsize: 14, side: 'b', do_shuffle: True, is_preview, True, is_timelapse: False)
04/30/2021 11:38:34 MainProcess _training_0 augmentation __init__ DEBUG Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:34 MainProcess _training_0 augmentation __init__ DEBUG Output sizes: [128]
04/30/2021 11:38:34 MainProcess _training_0 augmentation __init__ DEBUG Initialized ImageAugmentation
04/30/2021 11:38:34 MainProcess _training_0 multithreading __init__ DEBUG Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
04/30/2021 11:38:34 MainProcess _training_0 multithreading __init__ DEBUG Initialized BackgroundGenerator: '_run'
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Starting thread(s): '_run'
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Starting thread 1 of 2: '_run_0'
04/30/2021 11:38:34 MainProcess _run_0 generator _minibatch DEBUG Loading minibatch generator: (image_count: 16482, side: 'b', do_shuffle: True)
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Starting thread 2 of 2: '_run_1'
04/30/2021 11:38:34 MainProcess _run_1 generator _minibatch DEBUG Loading minibatch generator: (image_count: 16482, side: 'b', do_shuffle: True)
04/30/2021 11:38:34 MainProcess _training_0 multithreading start DEBUG Started all threads '_run': 2
04/30/2021 11:38:34 MainProcess _training_0 _base _set_preview_feed DEBUG Set preview feed. Batchsize: 14
04/30/2021 11:38:34 MainProcess _training_0 _base __init__ DEBUG Initialized _Feeder:
04/30/2021 11:38:34 MainProcess _training_0 _base _set_tensorboard DEBUG Enabling TensorBoard Logging
04/30/2021 11:38:34 MainProcess _training_0 _base _set_tensorboard DEBUG Setting up TensorBoard Logging
04/30/2021 11:38:35 MainProcess _training_0 _base _set_tensorboard VERBOSE Enabled TensorBoard Logging
04/30/2021 11:38:35 MainProcess _training_0 _base __init__ DEBUG Initializing _Samples: model: '<plugins.train.model.dfaker.Model object at 0x7f1b165ebfd0>', coverage_ratio: 0.6875)
04/30/2021 11:38:35 MainProcess _training_0 _base __init__ DEBUG Initialized _Samples
04/30/2021 11:38:35 MainProcess _training_0 _base __init__ DEBUG Initializing _Timelapse: model: <plugins.train.model.dfaker.Model object at 0x7f1b165ebfd0>, coverage_ratio: 0.6875, image_count: 14, feeder: '<plugins.train.trainer._base._Feeder object at 0x7f1b1653bac0>', image_paths: 2)
04/30/2021 11:38:35 MainProcess _training_0 _base __init__ DEBUG Initializing _Samples: model: '<plugins.train.model.dfaker.Model object at 0x7f1b165ebfd0>', coverage_ratio: 0.6875)
04/30/2021 11:38:35 MainProcess _training_0 _base __init__ DEBUG Initialized _Samples
04/30/2021 11:38:35 MainProcess _training_0 _base __init__ DEBUG Initialized _Timelapse
04/30/2021 11:38:35 MainProcess _training_0 _base __init__ DEBUG Initialized Trainer
04/30/2021 11:38:35 MainProcess _training_0 train _load_trainer DEBUG Loaded Trainer
04/30/2021 11:38:35 MainProcess _training_0 train _run_training_cycle DEBUG Running Training Cycle
04/30/2021 11:38:35 MainProcess _training_0 multithreading check_and_raise_error DEBUG Thread error caught: [(<class 'KeyError'>, KeyError('aligned_face'), <traceback object at 0x7f18ba491c40>)]
04/30/2021 11:38:35 MainProcess _training_0 multithreading run DEBUG Error in thread (_training_0): 'aligned_face'
04/30/2021 11:38:35 MainProcess _run_1 multithreading run DEBUG Error in thread (_run_1): 'aligned_face'
04/30/2021 11:38:35 MainProcess _run_0 augmentation initialize DEBUG Initializing constants. training_size: 384
04/30/2021 11:38:35 MainProcess _run_0 augmentation initialize DEBUG Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]]', 'warp_lm_grids': '[[[ 0. 0. 0. ... 0. 0. 0.]\n [ 1. 1. 1. ... 1. 1. 1.]\n [ 2. 2. 2. ... 2. 2. 2.]\n ...\n [381. 381. 381. ... 381. 381. 381.]\n [382. 382. 382. ... 382. 382. 382.]\n [383. 383. 383. ... 383. 383. 383.]]\n\n [[ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]\n ...\n [ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]]]'}
04/30/2021 11:38:35 MainProcess _run_1 augmentation initialize DEBUG Initializing constants. training_size: 384
04/30/2021 11:38:35 MainProcess _run_1 augmentation initialize DEBUG Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]\n [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]\n\n [[ 60. 60. 60. 60. 60.]\n [126. 126. 126. 126. 126.]\n [192. 192. 192. 192. 192.]\n [258. 258. 258. 258. 258.]\n [324. 324. 324. 324. 324.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]\n\n [[ 0 0]\n [ 0 383]\n [383 383]\n [383 0]\n [191 0]\n [191 383]\n [383 191]\n [ 0 191]]]', 'warp_lm_grids': '[[[ 0. 0. 0. ... 0. 0. 0.]\n [ 1. 1. 1. ... 1. 1. 1.]\n [ 2. 2. 2. ... 2. 2. 2.]\n ...\n [381. 381. 381. ... 381. 381. 381.]\n [382. 382. 382. ... 382. 382. 382.]\n [383. 383. 383. ... 383. 383. 383.]]\n\n [[ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]\n ...\n [ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]\n [ 0. 1. 2. ... 381. 382. 383.]]]'}
04/30/2021 11:38:36 MainProcess MainThread train _monitor DEBUG Thread error detected
04/30/2021 11:38:36 MainProcess MainThread train _monitor DEBUG Closed Monitor
04/30/2021 11:38:36 MainProcess MainThread train _end_thread DEBUG Ending Training thread
04/30/2021 11:38:36 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...
04/30/2021 11:38:36 MainProcess MainThread multithreading join DEBUG Joining Threads: '_training'
04/30/2021 11:38:36 MainProcess MainThread multithreading join DEBUG Joining Thread: '_training_0'
04/30/2021 11:38:36 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training_0'
Traceback (most recent call last):
File "/home/penguin/FS/lib/cli/launcher.py", line 182, in execute_script
process.process()
File "/home/penguin/FS/scripts/train.py", line 187, in process
self._end_thread(thread, err)
File "/home/penguin/FS/scripts/train.py", line 227, in _end_thread
thread.join()
File "/home/penguin/FS/lib/multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "/home/penguin/FS/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/home/penguin/FS/scripts/train.py", line 249, in _training
raise err
File "/home/penguin/FS/scripts/train.py", line 239, in _training
self._run_training_cycle(model, trainer)
File "/home/penguin/FS/scripts/train.py", line 319, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "/home/penguin/FS/plugins/train/trainer/_base.py", line 187, in train_one_step
model_inputs, model_targets = self._feeder.get_batch()
File "/home/penguin/FS/plugins/train/trainer/_base.py", line 411, in get_batch
batch = next(self._feeds[side])
File "/home/penguin/FS/lib/multithreading.py", line 156, in iterator
self.check_and_raise_error()
File "/home/penguin/FS/lib/multithreading.py", line 84, in check_and_raise_error
raise error[1].with_traceback(error[2])
File "/home/penguin/FS/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/home/penguin/FS/lib/multithreading.py", line 145, in _run
for item in self.generator(*self._gen_args, **self._gen_kwargs):
File "/home/penguin/FS/lib/training/generator.py", line 591, in _minibatch
yield self._process_batch(img_paths, side)
File "/home/penguin/FS/lib/training/generator.py", line 629, in _process_batch
batch_dst_pts = self._get_closest_match(filenames, side, landmarks)
File "/home/penguin/FS/lib/training/generator.py", line 803, in _get_closest_match
landmarks = _FACE_CACHES[lm_side].aligned_landmarks
File "/home/penguin/FS/lib/training/generator.py", line 132, in aligned_landmarks
self._aligned_landmarks = {key: val["aligned_face"].landmarks
File "/home/penguin/FS/lib/training/generator.py", line 132, in <dictcomp>
self._aligned_landmarks = {key: val["aligned_face"].landmarks
KeyError: 'aligned_face'
============ System Information ============
encoding: UTF-8
git_branch: master
git_commits: 91a6a50 Training Bugfix - Unfreeze weights when loading a previously frozen model. e0a98e9 Training bugfixes: - lib.training.generator - Fix duplicate "legacy faceset" warning - Fix missing mask error message - gui: Fix bug in live stats when resuming an old session. a68a9ed Introduce sorting by FFT filtered blur detection. (#1147). c900036 GUI - Analysis Bugfix - Get correct length of loss labels when carrying over raw data. 094ea33 GUI - Bugfixes - Swallow OSErrors when failing to load preview image - Fix event_reader mapping for model output to loss names - stats - Ensure that _tb_logs exists prior to calling stop training
gpu_cuda: 11.3
gpu_cudnn: No global version found. Check Conda packages for Conda cuDNN
gpu_devices: GPU_0: NVIDIA GeForce GTX 1660 Ti
gpu_devices_active: GPU_0
gpu_driver: 465.24.02
gpu_vram: GPU_0: 5941MB
os_machine: x86_64
os_platform: Linux-5.11.15-arch1-2-x86_64-with-glibc2.10
os_release: 5.11.15-arch1-2
py_command: /home/penguin/FS/faceswap.py train -A /media/DATA/inteldrivers/lana1/lana_total -B /media/DATA/inteldrivers/orr1/exte_total -m /media/DATA/inteldrivers/.orr_lana_dfa -t dfaker -bs 14 -it 1000000 -s 250 -ss 25000 -ps 100 -wl -L INFO -gui
py_conda_version: conda 4.9.2
py_implementation: CPython
py_version: 3.8.8
py_virtual_env: True
sys_cores: 16
sys_processor:
sys_ram: Total: 15976MB, Available: 11098MB, Used: 4171MB, Free: 157MB
=============== Pip Packages ===============
absl-py @ file:///tmp/build/80754af9/absl-py_1615411202722/work
aiohttp @ file:///tmp/build/80754af9/aiohttp_1614360992924/work
astunparse==1.6.3
async-timeout==3.0.1
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
blinker==1.4
brotlipy==0.7.0
cachetools @ file:///tmp/build/80754af9/cachetools_1611600262290/work
certifi==2020.12.5
cffi @ file:///tmp/build/80754af9/cffi_1613246945912/work
chardet @ file:///tmp/build/80754af9/chardet_1605303185383/work
click @ file:///home/linux1/recipes/ci/click_1610990599742/work
coverage @ file:///tmp/build/80754af9/coverage_1614613670853/work
cryptography @ file:///tmp/build/80754af9/cryptography_1615532411702/work
cycler==0.10.0
Cython @ file:///tmp/build/80754af9/cython_1614014838717/work
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.3.3
google-auth @ file:///tmp/build/80754af9/google-auth_1616008050444/work
google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1614894617465/work
google-pasta==0.2.0
grpcio @ file:///tmp/build/80754af9/grpcio_1614884175859/work
h5py @ file:///tmp/build/80754af9/h5py_1593454122442/work
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1609799311556/work
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1615900442896/work
joblib @ file:///tmp/build/80754af9/joblib_1613502643832/work
Keras-Preprocessing @ file:///tmp/build/80754af9/keras-preprocessing_1612283640596/work
kiwisolver @ file:///tmp/build/80754af9/kiwisolver_1612282420641/work
Markdown @ file:///tmp/build/80754af9/markdown_1614363528767/work
matplotlib @ file:///tmp/build/80754af9/matplotlib-base_1592846008246/work
mkl-fft==1.3.0
mkl-random==1.1.1
mkl-service==2.3.0
multidict @ file:///tmp/build/80754af9/multidict_1607367757617/work
numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1603570489231/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.5.1.48
opt-einsum==3.1.0
pathlib==1.0.1
Pillow @ file:///tmp/build/80754af9/pillow_1615224089586/work
protobuf==3.14.0
psutil @ file:///tmp/build/80754af9/psutil_1612298023621/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
PyJWT==1.7.1
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work
python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-oauthlib==1.3.0
rsa @ file:///tmp/build/80754af9/rsa_1614366226499/work
scikit-learn @ file:///tmp/build/80754af9/scikit-learn_1614446682169/work
scipy @ file:///tmp/build/80754af9/scipy_1614022789989/work
sip==4.19.13
six @ file:///tmp/build/80754af9/six_1605205327372/work
tensorboard @ file:///home/builder/ktietz/aggregate/tensorflow_recipes/ci_te/tensorboard_1614593728657/work/tmp_pip_dir
tensorboard-plugin-wit==1.6.0
tensorflow==2.2.0
tensorflow-estimator==2.2.0
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado @ file:///tmp/build/80754af9/tornado_1606942300299/work
tqdm @ file:///tmp/build/80754af9/tqdm_1615925068909/work
typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1611751222202/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1615837158687/work
Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/work
wrapt==1.12.1
yarl @ file:///tmp/build/80754af9/yarl_1606939922162/work
zipp @ file:///tmp/build/80754af9/zipp_1615904174917/work
============== Conda Packages ==============
# packages in environment at /home/penguin/miniconda3/envs/faceswap:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_tflow_select 2.1.0 gpu
absl-py 0.12.0 py38h06a4308_0
aiohttp 3.7.4 py38h27cfd23_1
astunparse 1.6.3 py_0
async-timeout 3.0.1 py38h06a4308_0
attrs 20.3.0 pyhd3eb1b0_0
blas 1.0 mkl
blinker 1.4 py38h06a4308_0
brotlipy 0.7.0 py38h27cfd23_1003
bzip2 1.0.8 h516909a_3 conda-forge
c-ares 1.17.1 h27cfd23_0
ca-certificates 2021.1.19 h06a4308_1
cachetools 4.2.1 pyhd3eb1b0_0
certifi 2020.12.5 py38h06a4308_0
cffi 1.14.5 py38h261ae71_0
chardet 3.0.4 py38h06a4308_1003
click 7.1.2 pyhd3eb1b0_0
coverage 5.5 py38h27cfd23_2
cryptography 3.4.6 py38hd23ed53_0
cudatoolkit 10.1.243 h6bb024c_0
cudnn 7.6.5 cuda10.1_0
cupti 10.1.168 0
cycler 0.10.0 py38_0
cython 0.29.22 py38h2531618_0
dbus 1.13.18 hb2f20db_0
expat 2.2.10 he6710b0_2
fastcluster 1.1.26 py38hc5bc63f_2 conda-forge
ffmpeg 4.3.1 h3215721_1 conda-forge
ffmpy 0.2.3 pypi_0 pypi
fontconfig 2.13.1 h6c09931_0
freetype 2.10.4 h5ab3b9f_0
gast 0.3.3 py_0
git 2.23.0 pl526hacde149_0
glib 2.67.4 h36276a3_1
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
google-auth 1.28.0 pyhd3eb1b0_0
google-auth-oauthlib 0.4.3 pyhd3eb1b0_0
google-pasta 0.2.0 py_0
grpcio 1.36.1 py38h2157cd5_1
gst-plugins-base 1.14.0 h8213a91_2
gstreamer 1.14.0 h28cd5cc_2
h5py 2.10.0 py38hd6299e0_1
hdf5 1.10.6 hb1b8bf9_0
icu 58.2 he6710b0_3
idna 2.10 pyhd3eb1b0_0
imageio 2.9.0 py_0
imageio-ffmpeg 0.4.3 pyhd8ed1ab_0 conda-forge
importlib-metadata 3.7.3 py38h06a4308_1
intel-openmp 2020.2 254
joblib 1.0.1 pyhd3eb1b0_0
jpeg 9b h024ee3a_2
keras-preprocessing 1.1.2 pyhd3eb1b0_0
kiwisolver 1.3.1 py38h2531618_0
krb5 1.18.2 h173b8e3_0
lame 3.100 h14c3975_1001 conda-forge
lcms2 2.11 h396b838_0
ld_impl_linux-64 2.33.1 h53a641e_7
libcurl 7.71.1 h20c2e04_1
libedit 3.1.20210216 h27cfd23_1
libffi 3.3 he6710b0_2
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libiconv 1.16 h516909a_0 conda-forge
libpng 1.6.37 hbc83047_0
libprotobuf 3.14.0 h8c45485_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.2.0 h3942068_0
libuuid 1.0.3 h1bed415_2
libwebp-base 1.2.0 h27cfd23_0
libxcb 1.14 h7b6447c_0
libxml2 2.9.10 hb55368b_3
lz4-c 1.9.3 h2531618_0
markdown 3.3.4 py38h06a4308_0
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38hef1b27d_0
mkl 2020.2 256
mkl-service 2.3.0 py38he904b0f_0
mkl_fft 1.3.0 py38h54f3939_0
mkl_random 1.1.1 py38h0573a6f_0
multidict 5.1.0 py38h27cfd23_2
ncurses 6.2 he6710b0_1
nettle 3.6 he412f7d_0 conda-forge
numpy 1.19.2 py38h54aff64_0
numpy-base 1.19.2 py38hfa32c7d_0
nvidia-ml-py3 7.352.1 pypi_0 pypi
oauthlib 3.1.0 py_0
olefile 0.46 py_0
opencv-python 4.5.1.48 pypi_0 pypi
openh264 2.1.1 h8b12597_0 conda-forge
openssl 1.1.1j h27cfd23_0
opt_einsum 3.1.0 py_0
pathlib 1.0.1 py_1
pcre 8.44 he6710b0_0
perl 5.26.2 h14c3975_0
pillow 8.1.2 py38he98fc37_0
pip 21.0.1 py38h06a4308_0
protobuf 3.14.0 py38h2531618_1
psutil 5.8.0 py38h27cfd23_1
pyasn1 0.4.8 py_0
pyasn1-modules 0.2.8 py_0
pycparser 2.20 py_2
pyjwt 1.7.1 py38_0
pyopenssl 20.0.1 pyhd3eb1b0_1
pyparsing 2.4.7 pyhd3eb1b0_0
pyqt 5.9.2 py38h05f1152_4
pysocks 1.7.1 py38h06a4308_0
python 3.8.8 hdb3f193_4
python-dateutil 2.8.1 pyhd3eb1b0_0
python_abi 3.8 1_cp38 conda-forge
qt 5.9.7 h5867ecd_1
readline 8.1 h27cfd23_0
requests 2.25.1 pyhd3eb1b0_0
requests-oauthlib 1.3.0 py_0
rsa 4.7.2 pyhd3eb1b0_1
scikit-learn 0.24.1 py38ha9443f7_0
scipy 1.6.1 py38h91f5cce_0
setuptools 52.0.0 py38h06a4308_0
sip 4.19.13 py38he6710b0_0
six 1.15.0 py38h06a4308_0
sqlite 3.35.2 hdfb4753_0
tensorboard 2.4.0 pyhc547734_0
tensorboard-plugin-wit 1.6.0 py_0
tensorflow 2.2.0 gpu_py38hb782248_0
tensorflow-base 2.2.0 gpu_py38h83e3d50_0
tensorflow-estimator 2.2.0 pyh208ff02_0
tensorflow-gpu 2.2.0 h0d30ee6_0
termcolor 1.1.0 py38h06a4308_1
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 hbc83047_0
tornado 6.1 py38h27cfd23_0
tqdm 4.59.0 pyhd3eb1b0_1
typing-extensions 3.7.4.3 hd3eb1b0_0
typing_extensions 3.7.4.3 pyh06a4308_0
urllib3 1.26.4 pyhd3eb1b0_0
werkzeug 1.0.1 pyhd3eb1b0_0
wheel 0.36.2 pyhd3eb1b0_0
wrapt 1.12.1 py38h7b6447c_1
x264 1!152.20180806 h14c3975_0 conda-forge
xz 5.2.5 h7b6447c_0
yarl 1.6.3 py38h27cfd23_0
zipp 3.4.1 pyhd3eb1b0_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.5 h9ceee32_0
=============== State File =================
{
"name": "dfaker",
"sessions": {
"1": {
"timestamp": 1619779024.1097512,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 14,
"iterations": 12,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": true,
"nan_protection": true,
"convert_batchsize": 10,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
}
},
"lowest_avg_loss": {
"a": 0.24300900508056988,
"b": 0.22170592031695627
},
"iterations": 12,
"config": {
"centering": "face",
"coverage": 68.75,
"optimizer": "adam",
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": true,
"mixed_precision": false,
"nan_protection": true,
"convert_batchsize": 10,
"loss_function": "ssim",
"mask_loss_function": "mse",
"l2_reg_term": 100,
"eye_multiplier": 3,
"mouth_multiplier": 2,
"penalized_mask_loss": true,
"mask_type": "extended",
"mask_blur_kernel": 3,
"mask_threshold": 4,
"learn_mask": false,
"output_size": 128
}
}
================= Configs ==================
--------- train.ini ---------
[global]
centering: face
coverage: 68.75
icnr_init: False
conv_aware_init: False
optimizer: adam
learning_rate: 5e-05
epsilon_exponent: -7
reflect_padding: False
allow_growth: True
mixed_precision: False
nan_protection: True
convert_batchsize: 10
[global.loss]
loss_function: ssim
mask_loss_function: mse
l2_reg_term: 100
eye_multiplier: 3
mouth_multiplier: 2
penalized_mask_loss: True
mask_type: extended
mask_blur_kernel: 3
mask_threshold: 4
learn_mask: False
[trainer.original]
preview_images: 14
zoom_amount: 5
rotation_range: 10
shift_range: 5
flip_chance: 50
color_lightness: 30
color_ab: 8
color_clahe_chance: 50
color_clahe_max_size: 4
[model.realface]
input_size: 64
output_size: 128
dense_nodes: 1536
complexity_encoder: 128
complexity_decoder: 512
[model.original]
lowmem: False
[model.dfl_sae]
input_size: 128
clipnorm: True
architecture: df
autoencoder_dims: 0
encoder_dims: 42
decoder_dims: 21
multiscale_decoder: False
[model.unbalanced]
input_size: 128
lowmem: False
clipnorm: True
nodes: 1024
complexity_encoder: 128
complexity_decoder_a: 384
complexity_decoder_b: 512
[model.villain]
lowmem: False
[model.dlight]
features: best
details: good
output_size: 256
[model.dfaker]
output_size: 128
[model.dfl_h128]
lowmem: False
--------- extract.ini ---------
[global]
allow_growth: True
[align.fan]
batch-size: 12
[detect.s3fd]
confidence: 70
batch-size: 4
[detect.mtcnn]
minsize: 20
scalefactor: 0.709
batch-size: 8
threshold_1: 0.6
threshold_2: 0.7
threshold_3: 0.7
[detect.cv2_dnn]
confidence: 50
[mask.unet_dfl]
batch-size: 8
[mask.vgg_clear]
batch-size: 6
[mask.vgg_obstructed]
batch-size: 2
--------- gui.ini ---------
[global]
fullscreen: False
tab: extract
options_panel_width: 30
console_panel_height: 20
icon_size: 14
font: default
font_size: 9
autosave_last_session: prompt
timeout: 120
auto_load_model_stats: True
--------- convert.ini ---------
[scaling.sharpen]
method: gaussian
amount: 134
radius: 1.4
threshold: 5.0
[color.match_hist]
threshold: 99.0
[color.color_transfer]
clip: True
preserve_paper: True
[color.manual_balance]
colorspace: HSV
balance_1: 0.0
balance_2: 0.0
balance_3: 0.0
contrast: 0.0
brightness: 0.0
[writer.ffmpeg]
container: mp4
codec: libx264
crf: 23
preset: medium
tune: None
profile: auto
level: auto
skip_mux: False
[writer.gif]
fps: 25
loop: 0
palettesize: 256
subrectangles: False
[writer.pillow]
format: png
draw_transparent: False
optimize: False
gif_interlace: True
jpg_quality: 75
png_compress_level: 3
tif_compression: tiff_deflate
[writer.opencv]
format: png
draw_transparent: False
jpg_quality: 75
png_compress_level: 3
[mask.mask_blend]
type: normalized
kernel_size: 3
passes: 4
threshold: 4
erosion: 0.0
[mask.box_blend]
type: gaussian
distance: 11.0
radius: 5.0
passes: 1
--------- .faceswap ---------
backend: nvidia
Many thanks