Critical Crash when training with Warp to Landmarks (dfaker)

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Post Reply
User avatar
infinitygorkem
Posts: 2
Joined: Fri Apr 30, 2021 10:41 am

Critical Crash when training with Warp to Landmarks (dfaker)

Post by infinitygorkem »

Hello,

I try to create a DFaker model using Warp to Landmarks but after caching the landmarks, it always fails. It can train without WTL. Another model I had previously worked fine with WTL, does anybody know what the issue could be?

Crash report is here:

Code: Select all

04/30/2021 11:37:31 MainProcess     _training_0                    generator       _get_cache                     DEBUG    Creating cache. Side: a
04/30/2021 11:37:31 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: False, input_size: 64, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:37:31 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [128]
04/30/2021 11:37:31 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
04/30/2021 11:37:31 MainProcess     _training_0                    image           read_image_meta_batch          DEBUG    Submitting 15522 items to executor
04/30/2021 11:37:32 MainProcess     _training_0                    image           read_image_meta_batch          DEBUG    Succesfully submitted 15522 items to executor
04/30/2021 11:37:32 MainProcess     _training_0                    generator       _validate_version              DEBUG    Setting initial extract version: 2.1
04/30/2021 11:38:01 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
04/30/2021 11:38:01 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
04/30/2021 11:38:01 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
04/30/2021 11:38:01 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
04/30/2021 11:38:01 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 15522, side: 'a', do_shuffle: True)
04/30/2021 11:38:01 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
04/30/2021 11:38:01 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 15522, side: 'a', do_shuffle: True)
04/30/2021 11:38:01 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
04/30/2021 11:38:01 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
04/30/2021 11:38:01 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(128, 128, 3)]
04/30/2021 11:38:01 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: True, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:01 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
04/30/2021 11:38:01 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 16482, batchsize: 14, side: 'b', do_shuffle: True, is_preview, False, is_timelapse: False)
04/30/2021 11:38:01 MainProcess     _training_0                    generator       _get_cache                     DEBUG    Creating cache. Side: b
04/30/2021 11:38:01 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: False, input_size: 64, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:01 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [128]
04/30/2021 11:38:01 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
04/30/2021 11:38:01 MainProcess     _training_0                    image           read_image_meta_batch          DEBUG    Submitting 16482 items to executor
04/30/2021 11:38:01 MainProcess     _training_0                    image           read_image_meta_batch          DEBUG    Succesfully submitted 16482 items to executor
04/30/2021 11:38:01 MainProcess     _training_0                    generator       _validate_version              DEBUG    Setting initial extract version: 2.1
04/30/2021 11:38:01 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initializing constants. training_size: 384
04/30/2021 11:38:01 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
04/30/2021 11:38:01 MainProcess     _run_0                         multithreading  run                            DEBUG    Error in thread (_run_0): 'aligned_face'
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
04/30/2021 11:38:34 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 16482, side: 'b', do_shuffle: True)
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
04/30/2021 11:38:34 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 16482, side: 'b', do_shuffle: True)
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
04/30/2021 11:38:34 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Setting preview feed: (side: 'a')
04/30/2021 11:38:34 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
04/30/2021 11:38:34 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(128, 128, 3)]
04/30/2021 11:38:34 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: True, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:34 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
04/30/2021 11:38:34 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 15522, batchsize: 14, side: 'a', do_shuffle: True, is_preview, True, is_timelapse: False)
04/30/2021 11:38:34 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:34 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [128]
04/30/2021 11:38:34 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
04/30/2021 11:38:34 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 15522, side: 'a', do_shuffle: True)
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
04/30/2021 11:38:34 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 15522, side: 'a', do_shuffle: True)
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
04/30/2021 11:38:34 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Setting preview feed: (side: 'b')
04/30/2021 11:38:34 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
04/30/2021 11:38:34 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(128, 128, 3)]
04/30/2021 11:38:34 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: True, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:34 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
04/30/2021 11:38:34 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 16482, batchsize: 14, side: 'b', do_shuffle: True, is_preview, True, is_timelapse: False)
04/30/2021 11:38:34 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': True, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 10, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
04/30/2021 11:38:34 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [128]
04/30/2021 11:38:34 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
04/30/2021 11:38:34 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 16482, side: 'b', do_shuffle: True)
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
04/30/2021 11:38:34 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 16482, side: 'b', do_shuffle: True)
04/30/2021 11:38:34 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
04/30/2021 11:38:34 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Set preview feed. Batchsize: 14
04/30/2021 11:38:34 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Feeder:
04/30/2021 11:38:34 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Enabling TensorBoard Logging
04/30/2021 11:38:34 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Setting up TensorBoard Logging
04/30/2021 11:38:35 MainProcess     _training_0                    _base           _set_tensorboard               VERBOSE  Enabled TensorBoard Logging
04/30/2021 11:38:35 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Samples: model: '<plugins.train.model.dfaker.Model object at 0x7f1b165ebfd0>', coverage_ratio: 0.6875)
04/30/2021 11:38:35 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Samples
04/30/2021 11:38:35 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Timelapse: model: <plugins.train.model.dfaker.Model object at 0x7f1b165ebfd0>, coverage_ratio: 0.6875, image_count: 14, feeder: '<plugins.train.trainer._base._Feeder object at 0x7f1b1653bac0>', image_paths: 2)
04/30/2021 11:38:35 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Samples: model: '<plugins.train.model.dfaker.Model object at 0x7f1b165ebfd0>', coverage_ratio: 0.6875)
04/30/2021 11:38:35 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Samples
04/30/2021 11:38:35 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Timelapse
04/30/2021 11:38:35 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized Trainer
04/30/2021 11:38:35 MainProcess     _training_0                    train           _load_trainer                  DEBUG    Loaded Trainer
04/30/2021 11:38:35 MainProcess     _training_0                    train           _run_training_cycle            DEBUG    Running Training Cycle
04/30/2021 11:38:35 MainProcess     _training_0                    multithreading  check_and_raise_error          DEBUG    Thread error caught: [(<class 'KeyError'>, KeyError('aligned_face'), <traceback object at 0x7f18ba491c40>)]
04/30/2021 11:38:35 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): 'aligned_face'
04/30/2021 11:38:35 MainProcess     _run_1                         multithreading  run                            DEBUG    Error in thread (_run_1): 'aligned_face'
04/30/2021 11:38:35 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initializing constants. training_size: 384
04/30/2021 11:38:35 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
04/30/2021 11:38:35 MainProcess     _run_1                         augmentation    initialize                     DEBUG    Initializing constants. training_size: 384
04/30/2021 11:38:35 MainProcess     _run_1                         augmentation    initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
04/30/2021 11:38:36 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
04/30/2021 11:38:36 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
04/30/2021 11:38:36 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
04/30/2021 11:38:36 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
04/30/2021 11:38:36 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
04/30/2021 11:38:36 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
04/30/2021 11:38:36 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "/home/penguin/FS/lib/cli/launcher.py", line 182, in execute_script
    process.process()
  File "/home/penguin/FS/scripts/train.py", line 187, in process
    self._end_thread(thread, err)
  File "/home/penguin/FS/scripts/train.py", line 227, in _end_thread
    thread.join()
  File "/home/penguin/FS/lib/multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/home/penguin/FS/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/home/penguin/FS/scripts/train.py", line 249, in _training
    raise err
  File "/home/penguin/FS/scripts/train.py", line 239, in _training
    self._run_training_cycle(model, trainer)
  File "/home/penguin/FS/scripts/train.py", line 319, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/home/penguin/FS/plugins/train/trainer/_base.py", line 187, in train_one_step
    model_inputs, model_targets = self._feeder.get_batch()
  File "/home/penguin/FS/plugins/train/trainer/_base.py", line 411, in get_batch
    batch = next(self._feeds[side])
  File "/home/penguin/FS/lib/multithreading.py", line 156, in iterator
    self.check_and_raise_error()
  File "/home/penguin/FS/lib/multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "/home/penguin/FS/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/home/penguin/FS/lib/multithreading.py", line 145, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "/home/penguin/FS/lib/training/generator.py", line 591, in _minibatch
    yield self._process_batch(img_paths, side)
  File "/home/penguin/FS/lib/training/generator.py", line 629, in _process_batch
    batch_dst_pts = self._get_closest_match(filenames, side, landmarks)
  File "/home/penguin/FS/lib/training/generator.py", line 803, in _get_closest_match
    landmarks = _FACE_CACHES[lm_side].aligned_landmarks
  File "/home/penguin/FS/lib/training/generator.py", line 132, in aligned_landmarks
    self._aligned_landmarks = {key: val["aligned_face"].landmarks
  File "/home/penguin/FS/lib/training/generator.py", line 132, in <dictcomp>
    self._aligned_landmarks = {key: val["aligned_face"].landmarks
KeyError: 'aligned_face'

============ System Information ============
encoding:            UTF-8
git_branch:          master
git_commits:         91a6a50 Training Bugfix - Unfreeze weights when loading a previously frozen model. e0a98e9 Training bugfixes:   - lib.training.generator       - Fix duplicate "legacy faceset" warning       - Fix missing mask error message   - gui: Fix bug in live stats when resuming an old session. a68a9ed Introduce sorting by FFT filtered blur detection. (#1147). c900036 GUI - Analysis Bugfix    - Get correct length of loss labels when carrying over raw data. 094ea33 GUI - Bugfixes   - Swallow OSErrors when failing to load preview image   - Fix event_reader mapping for model output to loss names   - stats - Ensure that _tb_logs exists prior to calling stop training
gpu_cuda:            11.3
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce GTX 1660 Ti
gpu_devices_active:  GPU_0
gpu_driver:          465.24.02
gpu_vram:            GPU_0: 5941MB
os_machine:          x86_64
os_platform:         Linux-5.11.15-arch1-2-x86_64-with-glibc2.10
os_release:          5.11.15-arch1-2
py_command:          /home/penguin/FS/faceswap.py train -A /media/DATA/inteldrivers/lana1/lana_total -B /media/DATA/inteldrivers/orr1/exte_total -m /media/DATA/inteldrivers/.orr_lana_dfa -t dfaker -bs 14 -it 1000000 -s 250 -ss 25000 -ps 100 -wl -L INFO -gui
py_conda_version:    conda 4.9.2
py_implementation:   CPython
py_version:          3.8.8
py_virtual_env:      True
sys_cores:           16
sys_processor:       
sys_ram: Total: 15976MB, Available: 11098MB, Used: 4171MB, Free: 157MB =============== Pip Packages =============== absl-py @ file:///tmp/build/80754af9/absl-py_1615411202722/work aiohttp @ file:///tmp/build/80754af9/aiohttp_1614360992924/work astunparse==1.6.3 async-timeout==3.0.1 attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work blinker==1.4 brotlipy==0.7.0 cachetools @ file:///tmp/build/80754af9/cachetools_1611600262290/work certifi==2020.12.5 cffi @ file:///tmp/build/80754af9/cffi_1613246945912/work chardet @ file:///tmp/build/80754af9/chardet_1605303185383/work click @ file:///home/linux1/recipes/ci/click_1610990599742/work coverage @ file:///tmp/build/80754af9/coverage_1614613670853/work cryptography @ file:///tmp/build/80754af9/cryptography_1615532411702/work cycler==0.10.0 Cython @ file:///tmp/build/80754af9/cython_1614014838717/work fastcluster==1.1.26 ffmpy==0.2.3 gast==0.3.3 google-auth @ file:///tmp/build/80754af9/google-auth_1616008050444/work google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1614894617465/work google-pasta==0.2.0 grpcio @ file:///tmp/build/80754af9/grpcio_1614884175859/work h5py @ file:///tmp/build/80754af9/h5py_1593454122442/work idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1609799311556/work importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1615900442896/work joblib @ file:///tmp/build/80754af9/joblib_1613502643832/work Keras-Preprocessing @ file:///tmp/build/80754af9/keras-preprocessing_1612283640596/work kiwisolver @ file:///tmp/build/80754af9/kiwisolver_1612282420641/work Markdown @ file:///tmp/build/80754af9/markdown_1614363528767/work matplotlib @ file:///tmp/build/80754af9/matplotlib-base_1592846008246/work mkl-fft==1.3.0 mkl-random==1.1.1 mkl-service==2.3.0 multidict @ file:///tmp/build/80754af9/multidict_1607367757617/work numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1603570489231/work nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6 oauthlib==3.1.0 olefile==0.46 opencv-python==4.5.1.48 opt-einsum==3.1.0 pathlib==1.0.1 Pillow @ file:///tmp/build/80754af9/pillow_1615224089586/work protobuf==3.14.0 psutil @ file:///tmp/build/80754af9/psutil_1612298023621/work pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work PyJWT==1.7.1 pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work requests @ file:///tmp/build/80754af9/requests_1608241421344/work requests-oauthlib==1.3.0 rsa @ file:///tmp/build/80754af9/rsa_1614366226499/work scikit-learn @ file:///tmp/build/80754af9/scikit-learn_1614446682169/work scipy @ file:///tmp/build/80754af9/scipy_1614022789989/work sip==4.19.13 six @ file:///tmp/build/80754af9/six_1605205327372/work tensorboard @ file:///home/builder/ktietz/aggregate/tensorflow_recipes/ci_te/tensorboard_1614593728657/work/tmp_pip_dir tensorboard-plugin-wit==1.6.0 tensorflow==2.2.0 tensorflow-estimator==2.2.0 termcolor==1.1.0 threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl tornado @ file:///tmp/build/80754af9/tornado_1606942300299/work tqdm @ file:///tmp/build/80754af9/tqdm_1615925068909/work typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1611751222202/work urllib3 @ file:///tmp/build/80754af9/urllib3_1615837158687/work Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/work wrapt==1.12.1 yarl @ file:///tmp/build/80754af9/yarl_1606939922162/work zipp @ file:///tmp/build/80754af9/zipp_1615904174917/work ============== Conda Packages ============== # packages in environment at /home/penguin/miniconda3/envs/faceswap: # # Name Version Build Channel _libgcc_mutex 0.1 main
_tflow_select 2.1.0 gpu
absl-py 0.12.0 py38h06a4308_0
aiohttp 3.7.4 py38h27cfd23_1
astunparse 1.6.3 py_0
async-timeout 3.0.1 py38h06a4308_0
attrs 20.3.0 pyhd3eb1b0_0
blas 1.0 mkl
blinker 1.4 py38h06a4308_0
brotlipy 0.7.0 py38h27cfd23_1003
bzip2 1.0.8 h516909a_3 conda-forge c-ares 1.17.1 h27cfd23_0
ca-certificates 2021.1.19 h06a4308_1
cachetools 4.2.1 pyhd3eb1b0_0
certifi 2020.12.5 py38h06a4308_0
cffi 1.14.5 py38h261ae71_0
chardet 3.0.4 py38h06a4308_1003
click 7.1.2 pyhd3eb1b0_0
coverage 5.5 py38h27cfd23_2
cryptography 3.4.6 py38hd23ed53_0
cudatoolkit 10.1.243 h6bb024c_0
cudnn 7.6.5 cuda10.1_0
cupti 10.1.168 0
cycler 0.10.0 py38_0
cython 0.29.22 py38h2531618_0
dbus 1.13.18 hb2f20db_0
expat 2.2.10 he6710b0_2
fastcluster 1.1.26 py38hc5bc63f_2 conda-forge ffmpeg 4.3.1 h3215721_1 conda-forge ffmpy 0.2.3 pypi_0 pypi fontconfig 2.13.1 h6c09931_0
freetype 2.10.4 h5ab3b9f_0
gast 0.3.3 py_0
git 2.23.0 pl526hacde149_0
glib 2.67.4 h36276a3_1
gmp 6.2.1 h58526e2_0 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge google-auth 1.28.0 pyhd3eb1b0_0
google-auth-oauthlib 0.4.3 pyhd3eb1b0_0
google-pasta 0.2.0 py_0
grpcio 1.36.1 py38h2157cd5_1
gst-plugins-base 1.14.0 h8213a91_2
gstreamer 1.14.0 h28cd5cc_2
h5py 2.10.0 py38hd6299e0_1
hdf5 1.10.6 hb1b8bf9_0
icu 58.2 he6710b0_3
idna 2.10 pyhd3eb1b0_0
imageio 2.9.0 py_0
imageio-ffmpeg 0.4.3 pyhd8ed1ab_0 conda-forge importlib-metadata 3.7.3 py38h06a4308_1
intel-openmp 2020.2 254
joblib 1.0.1 pyhd3eb1b0_0
jpeg 9b h024ee3a_2
keras-preprocessing 1.1.2 pyhd3eb1b0_0
kiwisolver 1.3.1 py38h2531618_0
krb5 1.18.2 h173b8e3_0
lame 3.100 h14c3975_1001 conda-forge lcms2 2.11 h396b838_0
ld_impl_linux-64 2.33.1 h53a641e_7
libcurl 7.71.1 h20c2e04_1
libedit 3.1.20210216 h27cfd23_1
libffi 3.3 he6710b0_2
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libiconv 1.16 h516909a_0 conda-forge libpng 1.6.37 hbc83047_0
libprotobuf 3.14.0 h8c45485_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.2.0 h3942068_0
libuuid 1.0.3 h1bed415_2
libwebp-base 1.2.0 h27cfd23_0
libxcb 1.14 h7b6447c_0
libxml2 2.9.10 hb55368b_3
lz4-c 1.9.3 h2531618_0
markdown 3.3.4 py38h06a4308_0
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38hef1b27d_0
mkl 2020.2 256
mkl-service 2.3.0 py38he904b0f_0
mkl_fft 1.3.0 py38h54f3939_0
mkl_random 1.1.1 py38h0573a6f_0
multidict 5.1.0 py38h27cfd23_2
ncurses 6.2 he6710b0_1
nettle 3.6 he412f7d_0 conda-forge numpy 1.19.2 py38h54aff64_0
numpy-base 1.19.2 py38hfa32c7d_0
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 py_0
olefile 0.46 py_0
opencv-python 4.5.1.48 pypi_0 pypi openh264 2.1.1 h8b12597_0 conda-forge openssl 1.1.1j h27cfd23_0
opt_einsum 3.1.0 py_0
pathlib 1.0.1 py_1
pcre 8.44 he6710b0_0
perl 5.26.2 h14c3975_0
pillow 8.1.2 py38he98fc37_0
pip 21.0.1 py38h06a4308_0
protobuf 3.14.0 py38h2531618_1
psutil 5.8.0 py38h27cfd23_1
pyasn1 0.4.8 py_0
pyasn1-modules 0.2.8 py_0
pycparser 2.20 py_2
pyjwt 1.7.1 py38_0
pyopenssl 20.0.1 pyhd3eb1b0_1
pyparsing 2.4.7 pyhd3eb1b0_0
pyqt 5.9.2 py38h05f1152_4
pysocks 1.7.1 py38h06a4308_0
python 3.8.8 hdb3f193_4
python-dateutil 2.8.1 pyhd3eb1b0_0
python_abi 3.8 1_cp38 conda-forge qt 5.9.7 h5867ecd_1
readline 8.1 h27cfd23_0
requests 2.25.1 pyhd3eb1b0_0
requests-oauthlib 1.3.0 py_0
rsa 4.7.2 pyhd3eb1b0_1
scikit-learn 0.24.1 py38ha9443f7_0
scipy 1.6.1 py38h91f5cce_0
setuptools 52.0.0 py38h06a4308_0
sip 4.19.13 py38he6710b0_0
six 1.15.0 py38h06a4308_0
sqlite 3.35.2 hdfb4753_0
tensorboard 2.4.0 pyhc547734_0
tensorboard-plugin-wit 1.6.0 py_0
tensorflow 2.2.0 gpu_py38hb782248_0
tensorflow-base 2.2.0 gpu_py38h83e3d50_0
tensorflow-estimator 2.2.0 pyh208ff02_0
tensorflow-gpu 2.2.0 h0d30ee6_0
termcolor 1.1.0 py38h06a4308_1
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 hbc83047_0
tornado 6.1 py38h27cfd23_0
tqdm 4.59.0 pyhd3eb1b0_1
typing-extensions 3.7.4.3 hd3eb1b0_0
typing_extensions 3.7.4.3 pyh06a4308_0
urllib3 1.26.4 pyhd3eb1b0_0
werkzeug 1.0.1 pyhd3eb1b0_0
wheel 0.36.2 pyhd3eb1b0_0
wrapt 1.12.1 py38h7b6447c_1
x264 1!152.20180806 h14c3975_0 conda-forge xz 5.2.5 h7b6447c_0
yarl 1.6.3 py38h27cfd23_0
zipp 3.4.1 pyhd3eb1b0_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.5 h9ceee32_0 =============== State File ================= { "name": "dfaker", "sessions": { "1": { "timestamp": 1619779024.1097512, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 14, "iterations": 12, "config": { "learning_rate": 5e-05, "epsilon_exponent": -7, "allow_growth": true, "nan_protection": true, "convert_batchsize": 10, "eye_multiplier": 3, "mouth_multiplier": 2 } } }, "lowest_avg_loss": { "a": 0.24300900508056988, "b": 0.22170592031695627 }, "iterations": 12, "config": { "centering": "face", "coverage": 68.75, "optimizer": "adam", "learning_rate": 5e-05, "epsilon_exponent": -7, "allow_growth": true, "mixed_precision": false, "nan_protection": true, "convert_batchsize": 10, "loss_function": "ssim", "mask_loss_function": "mse", "l2_reg_term": 100, "eye_multiplier": 3, "mouth_multiplier": 2, "penalized_mask_loss": true, "mask_type": "extended", "mask_blur_kernel": 3, "mask_threshold": 4, "learn_mask": false, "output_size": 128 } } ================= Configs ================== --------- train.ini --------- [global] centering: face coverage: 68.75 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 epsilon_exponent: -7 reflect_padding: False allow_growth: True mixed_precision: False nan_protection: True convert_batchsize: 10 [global.loss] loss_function: ssim mask_loss_function: mse l2_reg_term: 100 eye_multiplier: 3 mouth_multiplier: 2 penalized_mask_loss: True mask_type: extended mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4 [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.original] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [model.dlight] features: best details: good output_size: 256 [model.dfaker] output_size: 128 [model.dfl_h128] lowmem: False --------- extract.ini --------- [global] allow_growth: True [align.fan] batch-size: 12 [detect.s3fd] confidence: 70 batch-size: 4 [detect.mtcnn] minsize: 20 scalefactor: 0.709 batch-size: 8 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 [detect.cv2_dnn] confidence: 50 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- convert.ini --------- [scaling.sharpen] method: gaussian amount: 134 radius: 1.4 threshold: 5.0 [color.match_hist] threshold: 99.0 [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: None profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 --------- .faceswap --------- backend: nvidia

Many thanks


User avatar
torzdf
Posts: 1317
Joined: Fri Jul 12, 2019 12:53 am
Answers: 122
Has thanked: 42 times
Been thanked: 262 times

Re: Critical Crash when training with Warp to Landmarks (dfaker)

Post by torzdf »

I will tag this as bug and look at it when I have a second.

In the meantime, I know that WTL is bugged, so doesn't work correctly.

To be honest, I don't feel it brings any benefit, so my advice is to leave it off.

My word is final


User avatar
infinitygorkem
Posts: 2
Joined: Fri Apr 30, 2021 10:41 am

Re: Critical Crash when training with Warp to Landmarks (dfaker)

Post by infinitygorkem »

Thanks torzdf.

The last time I tried it (which was the first time I used it), I turned it on midway through training the dfaker model as I saw it was the "dfaker" way, thought it might make better results, but it did work. Maybe it was always bugged and just doesn't work with new models or something.

I'll leave it off until it is fixed I guess. Thanks for looking at this.


User avatar
torzdf
Posts: 1317
Joined: Fri Jul 12, 2019 12:53 am
Answers: 122
Has thanked: 42 times
Been thanked: 262 times

Re: Critical Crash when training with Warp to Landmarks (dfaker)

Post by torzdf »

This bug should now be squashed.

My word is final


Post Reply