Faceswap Forum

Posted: **Wed Feb 03, 2021 3:09 am**

I don't see this one in the knowledge base. Any help would be appreciated.

InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run LogicalNot: Dst tensor is not initialized. [Op:LogicalNot]

02/02/2021 21:49:33 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\master\faceswap\crash_report.2021.02.02.214930539552.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

Process exited.
No summary data loaded. Nothing to save

I have the crash log and system info, but I don't see a way to attach them. Should I just put them in the text?

Posted: **Wed Feb 03, 2021 12:17 pm**

Yes, you can copy and paste the crash report (ideally inside [code] tags).

This looks like an Out Of Memory issue, so you can try reducing batch size. I will need you log to provide any more insight though.

Posted: **Wed Feb 03, 2021 2:14 pm**

OK, thanks. Here is the first half of the crash report. System specs will also follow.

Code: Select all

02/02/2021 21:54:26 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 128, output_shapes: [(128, 128, 3)]
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 128, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, augment_color: True, no_flip: False, warp_to_landmarks: False, alignments: ['aligned_faces', 'versions'], config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': False, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'disable_warp': False, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Initialized TrainingDataGenerator
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   minibatch_ab                   DEBUG    Queue batches: (image_count: 517, batchsize: 14, side: 'a', do_shuffle: True, is_preview, True, is_timelapse: False)
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 128, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': False, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'disable_warp': False, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Output sizes: [128]
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Initialized ImageAugmentation
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   _minibatch                     DEBUG    Loading minibatch generator: (image_count: 517, side: 'a', do_shuffle: True)
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
02/02/2021 21:54:26 MainProcess     _run_1                         training_data   _minibatch                     DEBUG    Loading minibatch generator: (image_count: 517, side: 'a', do_shuffle: True)
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
02/02/2021 21:54:26 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Setting preview feed: (side: 'b')
02/02/2021 21:54:26 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
02/02/2021 21:54:26 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 128, output_shapes: [(128, 128, 3)]
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 128, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, augment_color: True, no_flip: False, warp_to_landmarks: False, alignments: ['aligned_faces', 'versions'], config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': False, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'disable_warp': False, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Initialized TrainingDataGenerator
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   minibatch_ab                   DEBUG    Queue batches: (image_count: 723, batchsize: 14, side: 'b', do_shuffle: True, is_preview, True, is_timelapse: False)
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 128, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': False, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'disable_warp': False, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Output sizes: [128]
02/02/2021 21:54:26 MainProcess     _training_0                    training_data   __init__                       DEBUG    Initialized ImageAugmentation
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   _minibatch                     DEBUG    Loading minibatch generator: (image_count: 723, side: 'b', do_shuffle: True)
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
02/02/2021 21:54:26 MainProcess     _run_1                         training_data   _minibatch                     DEBUG    Loading minibatch generator: (image_count: 723, side: 'b', do_shuffle: True)
02/02/2021 21:54:26 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
02/02/2021 21:54:26 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Set preview feed. Batchsize: 14
02/02/2021 21:54:26 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Feeder:
02/02/2021 21:54:26 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Enabling TensorBoard Logging
02/02/2021 21:54:26 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Setting up TensorBoard Logging
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   _crop_to_center                DEBUG    caching crop size: (centering: 'face', full size: 512, crop size: 384)
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   _crop_to_center                DEBUG    caching crop size: (centering: 'face', full size: 512, crop size: 384)
02/02/2021 21:54:26 MainProcess     _run_1                         training_data   _crop_to_center                DEBUG    caching crop size: (centering: 'face', full size: 512, crop size: 384)
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   _crop_to_center                DEBUG    caching crop size: (centering: 'face', full size: 512, crop size: 384)
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   initialize                     DEBUG    Initializing constants. training_size: 384
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]]', 'warp_pad': 160, 'warp_slices': slice(16, -16, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   initialize                     DEBUG    Initializing constants. training_size: 384
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]]', 'warp_pad': 160, 'warp_slices': slice(16, -16, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
02/02/2021 21:54:26 MainProcess     _run_1                         training_data   initialize                     DEBUG    Initializing constants. training_size: 384
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   initialize                     DEBUG    Initializing constants. training_size: 384
02/02/2021 21:54:26 MainProcess     _run_0                         training_data   initialize                     DEBUG    Initializing constants. training_size: 384
02/02/2021 21:54:26 MainProcess     _run_1                         training_data   initialize                     DEBUG    Initializing constants. training_size: 384
02/02/2021 21:54:27 MainProcess     _run_1                         training_data   initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]]', 'warp_pad': 160, 'warp_slices': slice(16, -16, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
02/02/2021 21:54:27 MainProcess     _run_1                         training_data   initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]]', 'warp_pad': 160, 'warp_slices': slice(16, -16, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
02/02/2021 21:54:27 MainProcess     _run_0                         training_data   initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]]', 'warp_pad': 160, 'warp_slices': slice(16, -16, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}

Posted: **Wed Feb 03, 2021 2:14 pm**

This is the rest of the crash report.

Code: Select all

02/02/2021 21:54:27 MainProcess     _run_0                         training_data   initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(60, 324, None), 'warp_mapx': '[[[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]\n\n [[ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]\n  [ 60. 126. 192. 258. 324.]]]', 'warp_mapy': '[[[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]\n\n [[ 60.  60.  60.  60.  60.]\n  [126. 126. 126. 126. 126.]\n  [192. 192. 192. 192. 192.]\n  [258. 258. 258. 258. 258.]\n  [324. 324. 324. 324. 324.]]]', 'warp_pad': 160, 'warp_slices': slice(16, -16, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
02/02/2021 21:54:27 MainProcess     _training_0                    _base           _set_tensorboard               VERBOSE  Enabled TensorBoard Logging
02/02/2021 21:54:27 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Samples: model: '<plugins.train.model.villain.Model object at 0x00000203ED256A00>', coverage_ratio: 0.6875)
02/02/2021 21:54:27 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Samples
02/02/2021 21:54:27 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Timelapse: model: <plugins.train.model.villain.Model object at 0x00000203ED256A00>, coverage_ratio: 0.6875, image_count: 14, feeder: '<plugins.train.trainer._base._Feeder object at 0x0000020403AC2B50>', image_paths: 2)
02/02/2021 21:54:27 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Samples: model: '<plugins.train.model.villain.Model object at 0x00000203ED256A00>', coverage_ratio: 0.6875)
02/02/2021 21:54:27 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Samples
02/02/2021 21:54:27 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Timelapse
02/02/2021 21:54:27 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized Trainer
02/02/2021 21:54:27 MainProcess     _training_0                    train           _load_trainer                  DEBUG    Loaded Trainer
02/02/2021 21:54:27 MainProcess     _training_0                    train           _run_training_cycle            DEBUG    Running Training Cycle
02/02/2021 21:54:28 MainProcess     _training_0                    ag_logging      warn                           DEBUG    AutoGraph could not transform <bound method Logger.isEnabledFor of <FaceswapLogger lib.model.losses_tf (DEBUG)>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
02/02/2021 21:54:28 MainProcess     _training_0                    ag_logging      warn                           DEBUG    AutoGraph could not transform <bound method Logger.findCaller of <FaceswapLogger lib.model.losses_tf (DEBUG)>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
02/02/2021 21:54:28 MainProcess     _training_0                    ag_logging      warn                           DEBUG    AutoGraph could not transform <bound method Logger.makeRecord of <FaceswapLogger lib.model.losses_tf (DEBUG)>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
02/02/2021 21:54:29 MainProcess     _training_0                    ag_logging      warn                           DEBUG    AutoGraph could not transform <bound method FaceswapFormatter.format of <lib.logger.FaceswapFormatter object at 0x00000203E25B7F70>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
02/02/2021 21:54:28 MainProcess     _training_0                    api             converted_call                 DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A2F85B0>, weight: 1.0, mask_channel: -1)
02/02/2021 21:54:29 MainProcess     _training_0                    ag_logging      warn                           DEBUG    AutoGraph could not transform <bound method LossWrapper._apply_mask of <class 'lib.model.losses_tf.LossWrapper'>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    No mask to apply
02/02/2021 21:54:29 MainProcess     _training_0                    ag_logging      warn                           DEBUG    AutoGraph could not transform <bound method DSSIMObjective.call of <lib.model.losses_tf.DSSIMObjective object at 0x00000203ED2E6D00>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A30DE50>, weight: 1.0, mask_channel: -1)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    No mask to apply
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A3150D0>, weight: 3.0, mask_channel: -1)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    No mask to apply
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A324340>, weight: 1.0, mask_channel: 1)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    Applying mask from channel 1
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A324B50>, weight: 2.0, mask_channel: -1)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    No mask to apply
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A328370>, weight: 1.0, mask_channel: 2)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    Applying mask from channel 2
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A328B50>, weight: 1.0, mask_channel: -1)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    No mask to apply
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A32E370>, weight: 1.0, mask_channel: -1)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    No mask to apply
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A32E9D0>, weight: 3.0, mask_channel: -1)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    No mask to apply
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A3343D0>, weight: 1.0, mask_channel: 1)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    Applying mask from channel 1
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A334B80>, weight: 2.0, mask_channel: -1)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    No mask to apply
02/02/2021 21:54:29 MainProcess     _training_0                    tmpeqpyer9_     if_body                        DEBUG    Processing loss function: (func: <tensorflow.python.keras.engine.compile_utils.LossesContainer object at 0x000002038A33C3A0>, weight: 1.0, mask_channel: 2)
02/02/2021 21:54:29 MainProcess     _training_0                    losses_tf       _apply_mask                    DEBUG    Applying mask from channel 2
02/02/2021 21:54:41 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): in user code:\n\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:806 train_function  *\n        return step_function(self, iterator)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:796 step_function  **\n        outputs = model.distribute_strategy.run(run_step, args=(data,))\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:1211 run\n        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2585 call_for_each_replica\n        return self._call_for_each_replica(fn, args, kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2945 _call_for_each_replica\n        return fn(*args, **kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:789 run_step  **\n        outputs = model.train_step(data)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:756 train_step\n        _minimize(self.distribute_strategy, tape, self.optimizer, loss,\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:2743 _minimize\n        optimizer.apply_gradients(\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py:519 apply_gradients\n        self._create_all_weights(var_list)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py:704 _create_all_weights\n        self._create_slots(var_list)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\optimizer_v2\adam.py:129 _create_slots\n        self.add_slot(var, 'v')\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py:760 add_slot\n        weight = tf_variables.Variable(\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:262 __call__\n        return cls._variable_v2_call(*args, **kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:244 _variable_v2_call\n        return previous_getter(\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:67 getter\n        return captured_getter(captured_previous, **kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2857 creator\n        return next_creator(**kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:67 getter\n        return captured_getter(captured_previous, **kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2857 creator\n        return next_creator(**kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:67 getter\n        return captured_getter(captured_previous, **kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2857 creator\n        return next_creator(**kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:67 getter\n        return captured_getter(captured_previous, **kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py:683 variable_capturing_scope\n        v = UnliftedInitializerVariable(\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:264 __call__\n        return super(VariableMetaclass, cls).__call__(*args, **kwargs)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py:237 __init__\n        super(UnliftedInitializerVariable, self).__init__(\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py:1879 __init__\n        handle = _variable_handle_from_shape_and_dtype(\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py:175 _variable_handle_from_shape_and_dtype\n        math_ops.logical_not(exists), [exists], name="EagerVariableNameReuse")\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_math_ops.py:5463 logical_not\n        _ops.raise_from_not_ok_status(e, name)\n    C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:6843 raise_from_not_ok_status\n        six.raise_from(core._status_to_exception(e.code, message), None)\n    <string>:3 raise_from\n        \n\n    InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run LogicalNot: Dst tensor is not initialized. [Op:LogicalNot]\n
02/02/2021 21:54:42 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
02/02/2021 21:54:42 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
02/02/2021 21:54:42 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
02/02/2021 21:54:42 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
02/02/2021 21:54:42 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
02/02/2021 21:54:42 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
02/02/2021 21:54:42 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "C:\Users\master\faceswap\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "C:\Users\master\faceswap\scripts\train.py", line 170, in process
    self._end_thread(thread, err)
  File "C:\Users\master\faceswap\scripts\train.py", line 210, in _end_thread
    thread.join()
  File "C:\Users\master\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\master\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\master\faceswap\scripts\train.py", line 232, in _training
    raise err
  File "C:\Users\master\faceswap\scripts\train.py", line 222, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Users\master\faceswap\scripts\train.py", line 302, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "C:\Users\master\faceswap\plugins\train\trainer\_base.py", line 238, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1695, in train_on_batch
    logs = train_function(iterator)
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 823, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 696, in _initialize
    self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2855, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 3213, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 3065, in _create_graph_function
    func_graph_module.func_graph_from_py_func(
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 600, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\func_graph.py", line 973, in wrapper
    raise e.ag_error_metadata.to_exception(e)
tensorflow.python.framework.errors_impl.InternalError: in user code:

C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:806 train_function  *
    return step_function(self, iterator)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:796 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:1211 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2585 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2945 _call_for_each_replica
    return fn(*args, **kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:789 run_step  **
    outputs = model.train_step(data)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:756 train_step
    _minimize(self.distribute_strategy, tape, self.optimizer, loss,
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py:2743 _minimize
    optimizer.apply_gradients(
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py:519 apply_gradients
    self._create_all_weights(var_list)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py:704 _create_all_weights
    self._create_slots(var_list)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\optimizer_v2\adam.py:129 _create_slots
    self.add_slot(var, 'v')
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py:760 add_slot
    weight = tf_variables.Variable(
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:262 __call__
    return cls._variable_v2_call(*args, **kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:244 _variable_v2_call
    return previous_getter(
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:67 getter
    return captured_getter(captured_previous, **kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2857 creator
    return next_creator(**kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:67 getter
    return captured_getter(captured_previous, **kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2857 creator
    return next_creator(**kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:67 getter
    return captured_getter(captured_previous, **kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2857 creator
    return next_creator(**kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:67 getter
    return captured_getter(captured_previous, **kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py:683 variable_capturing_scope
    v = UnliftedInitializerVariable(
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\variables.py:264 __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py:237 __init__
    super(UnliftedInitializerVariable, self).__init__(
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py:1879 __init__
    handle = _variable_handle_from_shape_and_dtype(
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py:175 _variable_handle_from_shape_and_dtype
    math_ops.logical_not(exists), [exists], name="EagerVariableNameReuse")
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_math_ops.py:5463 logical_not
    _ops.raise_from_not_ok_status(e, name)
C:\Users\master\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py:6843 raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
<string>:3 raise_from
    

InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run LogicalNot: Dst tensor is not initialized. [Op:LogicalNot]


============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         27a7adb Update README.md. 0941d6d realface model - Bugfix resblock
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 965M
gpu_devices_active:  GPU_0
gpu_driver:          461.40
gpu_vram:            GPU_0: 2048MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.19041-SP0
os_release:          10
py_command:          C:\Users\master\faceswap\faceswap.py train -A F:/Common/Brian/National Treasure Trailer/Deepfake training 1 James Roday 2014 -ala F:/Common/Brian/National Treasure Trailer/Deepfake training 1 James Roday 2014/alignments.fsa -B F:/Common/Brian/National Treasure Trailer/Deepfake training 2 Nicolas Cage 2004 -alb F:/Common/Brian/National Treasure Trailer/Deepfake training 2 Nicolas Cage 2004/alignments.fsa -m F:/Common/Brian/National Treasure Trailer/Training -t villain -bs 16 -it 1000000 -s 250 -ss 25000 -ps 100 -L INFO -gui
py_conda_version:    conda 4.9.2
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           8
sys_processor:       Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
sys_ram:             Total: 16333MB, Available: 8402MB, Used: 7930MB, Free: 8402MB

=============== Pip Packages ===============
absl-py @ file:///tmp/build/80754af9/absl-py_1607439979954/work
aiohttp @ file:///C:/ci/aiohttp_1607109697839/work
astunparse==1.6.3
async-timeout==3.0.1
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
blinker==1.4
brotlipy==0.7.0
cachetools @ file:///tmp/build/80754af9/cachetools_1611600262290/work
certifi==2020.12.5
cffi @ file:///C:/ci/cffi_1606255208697/work
chardet @ file:///C:/ci/chardet_1605303225733/work
click @ file:///home/linux1/recipes/ci/click_1610990599742/work
cryptography==2.9.2
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast @ file:///tmp/build/80754af9/gast_1597433534803/work
google-auth @ file:///tmp/build/80754af9/google-auth_1607969906642/work
google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1603929124518/work
google-pasta==0.2.0
grpcio @ file:///C:/ci/grpcio_1597406462198/work
h5py==2.10.0
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1609799311556/work
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1602276842396/work
joblib @ file:///tmp/build/80754af9/joblib_1607970656719/work
Keras-Applications @ file:///tmp/build/80754af9/keras-applications_1594366238411/work
Keras-Preprocessing==1.1.0
kiwisolver @ file:///C:/ci/kiwisolver_1604014703538/work
Markdown @ file:///C:/ci/markdown_1605111189761/work
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
multidict @ file:///C:/ci/multidict_1600456481656/work
numpy @ file:///C:/ci/numpy_and_numpy_base_1603466732592/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.5.1.48
opt-einsum==3.1.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1609786840597/work
protobuf==3.13.0
psutil @ file:///C:/ci/psutil_1598370330503/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
PyJWT @ file:///C:/ci/pyjwt_1610893382614/work
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
pyreadline==2.1
PySocks @ file:///C:/ci/pysocks_1605287845585/work
python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work
pywin32==227
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-oauthlib==1.3.0
rsa @ file:///tmp/build/80754af9/rsa_1610483308194/work
scikit-learn @ file:///C:/ci/scikit-learn_1598377018496/work
scipy @ file:///C:/ci/scipy_1604596260408/work
sip==4.19.13
six @ file:///C:/ci/six_1605187374963/work
tensorboard @ file:///home/builder/ktietz/conda/conda-bld/tensorboard_1604313476433/work/tmp_pip_dir
tensorboard-plugin-wit==1.6.0
tensorflow==2.3.0
tensorflow-estimator @ file:///tmp/build/80754af9/tensorflow-estimator_1599136169057/work/whl_temp/tensorflow_estimator-2.3.0-py2.py3-none-any.whl
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado @ file:///C:/ci/tornado_1606942392901/work
tqdm @ file:///tmp/build/80754af9/tqdm_1611857934208/work
typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1611751222202/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1611694770489/work
Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/work
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
wincertstore==0.2
wrapt==1.12.1
yarl @ file:///C:/ci/yarl_1598045274898/work
zipp @ file:///tmp/build/80754af9/zipp_1604001098328/work

============== Conda Packages ==============
# packages in environment at C:\Users\master\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.3.0                       gpu  

absl-py                   0.11.0             pyhd3eb1b0_1  

aiohttp                   3.7.3            py38h2bbff1b_1  

astunparse                1.6.3                      py_0  

async-timeout             3.0.1            py38haa95532_0  

attrs                     20.3.0             pyhd3eb1b0_0  

blas                      1.0                         mkl  

blinker                   1.4              py38haa95532_0  

brotlipy                  0.7.0           py38h2bbff1b_1003  

ca-certificates           2021.1.19            haa95532_0  

cachetools                4.2.1              pyhd3eb1b0_0  

certifi                   2020.12.5        py38haa95532_0  

cffi                      1.14.4           py38hcd4344a_0  

chardet                   3.0.4           py38haa95532_1003  

click                     7.1.2              pyhd3eb1b0_0  

cryptography              2.9.2            py38h7a1dbc1_0  

cudatoolkit               10.1.243             h74a9793_0  

cudnn                     7.6.5                cuda10.1_0  

cycler                    0.10.0                   py38_0  

fastcluster               1.1.26           py38h251f6bf_2    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
freetype                  2.10.4               hd328e21_0  

gast                      0.4.0                      py_0  

git                       2.23.0               h6bb4b03_0  

google-auth               1.24.0             pyhd3eb1b0_0  

google-auth-oauthlib      0.4.2              pyhd3eb1b0_2  

google-pasta              0.2.0                      py_0  

grpcio                    1.31.0           py38he7da953_0  

h5py                      2.10.0           py38h5e291fa_0  

hdf5                      1.10.4               h7ebc959_0  

icc_rt                    2019.0.0             h0cc432a_1  

icu                       58.2                 ha925a31_3  

idna                      2.10               pyhd3eb1b0_0  

imageio                   2.9.0                      py_0  

imageio-ffmpeg            0.4.3              pyhd8ed1ab_0    conda-forge
importlib-metadata        2.0.0                      py_1  

intel-openmp              2020.2                      254  

joblib                    1.0.0              pyhd3eb1b0_0  

jpeg                      9b                   hb83a4c4_2  

keras-applications        1.0.8                      py_1  

keras-preprocessing       1.1.0                      py_1  

kiwisolver                1.3.0            py38hd77b12b_0  

libpng                    1.6.37               h2a8f88b_0  

libprotobuf               3.13.0.1             h200bbdf_0  

libtiff                   4.1.0                h56a325e_1  

lz4-c                     1.9.3                h2bbff1b_0  

markdown                  3.3.3            py38haa95532_0  

matplotlib                3.2.2                         0  

matplotlib-base           3.2.2            py38h64f37c6_0  

mkl                       2020.2                      256  

mkl-service               2.3.0            py38h196d8e1_0  

mkl_fft                   1.2.0            py38h45dec08_0  

mkl_random                1.1.1            py38h47e9c7a_0  

multidict                 4.7.6            py38he774522_1  

numpy                     1.19.2           py38hadc3359_0  

numpy-base                1.19.2           py38ha3acd2a_0  

nvidia-ml-py3             7.352.1                  pypi_0    pypi
oauthlib                  3.1.0                      py_0  

olefile                   0.46                       py_0  

opencv-python             4.5.1.48                 pypi_0    pypi
openssl                   1.1.1i               h2bbff1b_0  

opt_einsum                3.1.0                      py_0  

pathlib                   1.0.1                      py_1  

pillow                    8.1.0            py38h4fa10fc_0  

pip                       20.3.3           py38haa95532_0  

protobuf                  3.13.0.1         py38ha925a31_1  

psutil                    5.7.2            py38he774522_0  

pyasn1                    0.4.8                      py_0  

pyasn1-modules            0.2.8                      py_0  

pycparser                 2.20                       py_2  

pyjwt                     2.0.1            py38haa95532_0  

pyopenssl                 20.0.1             pyhd3eb1b0_1  

pyparsing                 2.4.7              pyhd3eb1b0_0  

pyqt                      5.9.2            py38ha925a31_4  

pyreadline                2.1                      py38_1  

pysocks                   1.7.1            py38haa95532_0  

python                    3.8.5                h5fd99cc_1  

python-dateutil           2.8.1              pyhd3eb1b0_0  

python_abi                3.8                      1_cp38    conda-forge
pywin32                   227              py38he774522_1  

qt                        5.9.7            vc14h73c81de_0  

requests                  2.25.1             pyhd3eb1b0_0  

requests-oauthlib         1.3.0                      py_0  

rsa                       4.7                pyhd3eb1b0_1  

scikit-learn              0.23.2           py38h47e9c7a_0  

scipy                     1.5.2            py38h14eb087_0  

setuptools                52.0.0           py38haa95532_0  

sip                       4.19.13          py38ha925a31_0  

six                       1.15.0           py38haa95532_0  

sqlite                    3.33.0               h2a8f88b_0  

tensorboard               2.3.0              pyh4dce500_0  

tensorboard-plugin-wit    1.6.0                      py_0  

tensorflow                2.3.0           mkl_py38h1fcfbd6_0  

tensorflow-base           2.3.0           gpu_py38h7339f5a_0  

tensorflow-estimator      2.3.0              pyheb71bc4_0  

tensorflow-gpu            2.3.0                he13fc11_0  

termcolor                 1.1.0                    py38_1  

threadpoolctl             2.1.0              pyh5ca1d4c_0  

tk                        8.6.10               he774522_0  

tornado                   6.1              py38h2bbff1b_0  

tqdm                      4.56.0             pyhd3eb1b0_0  

typing-extensions         3.7.4.3              hd3eb1b0_0  

typing_extensions         3.7.4.3            pyh06a4308_0  

urllib3                   1.26.3             pyhd3eb1b0_0  

vc                        14.2                 h21ff451_1  

vs2015_runtime            14.27.29016          h5e58377_2  

werkzeug                  1.0.1              pyhd3eb1b0_0  

wheel                     0.36.2             pyhd3eb1b0_0  

win_inet_pton             1.1.0            py38haa95532_0  

wincertstore              0.2                      py38_0  

wrapt                     1.12.1           py38he774522_1  

xz                        5.2.5                h62dcd97_0  

yarl                      1.5.1            py38he774522_0  

zipp                      3.4.0              pyhd3eb1b0_0  

zlib                      1.2.11               h62dcd97_4  

zstd                      1.4.5                h04227a9_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7
scalefactor:              0.709
batch-size:               8

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                face
coverage:                 68.75
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
convert_batchsize:        16

[global.loss]
loss_function:            ssim
mask_loss_function:       mse
l2_reg_term:              100
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      False
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
disable_warp:             False
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

Posted: **Wed Feb 03, 2021 2:16 pm**

And here is the system info.

Code: Select all


============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         27a7adb Update README.md. 0941d6d realface model - Bugfix resblock
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 965M
gpu_devices_active:  GPU_0
gpu_driver:          461.40
gpu_vram:            GPU_0: 2048MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.19041-SP0
os_release:          10
py_command:          C:\Users\master\faceswap/faceswap.py gui
py_conda_version:    conda 4.9.2
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           8
sys_processor:       Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
sys_ram:             Total: 16333MB, Available: 9339MB, Used: 6994MB, Free: 9339MB

=============== Pip Packages ===============
absl-py @ file:///tmp/build/80754af9/absl-py_1607439979954/work
aiohttp @ file:///C:/ci/aiohttp_1607109697839/work
astunparse==1.6.3
async-timeout==3.0.1
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
blinker==1.4
brotlipy==0.7.0
cachetools @ file:///tmp/build/80754af9/cachetools_1611600262290/work
certifi==2020.12.5
cffi @ file:///C:/ci/cffi_1606255208697/work
chardet @ file:///C:/ci/chardet_1605303225733/work
click @ file:///home/linux1/recipes/ci/click_1610990599742/work
cryptography==2.9.2
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast @ file:///tmp/build/80754af9/gast_1597433534803/work
google-auth @ file:///tmp/build/80754af9/google-auth_1607969906642/work
google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1603929124518/work
google-pasta==0.2.0
grpcio @ file:///C:/ci/grpcio_1597406462198/work
h5py==2.10.0
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1609799311556/work
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1602276842396/work
joblib @ file:///tmp/build/80754af9/joblib_1607970656719/work
Keras-Applications @ file:///tmp/build/80754af9/keras-applications_1594366238411/work
Keras-Preprocessing==1.1.0
kiwisolver @ file:///C:/ci/kiwisolver_1604014703538/work
Markdown @ file:///C:/ci/markdown_1605111189761/work
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
multidict @ file:///C:/ci/multidict_1600456481656/work
numpy @ file:///C:/ci/numpy_and_numpy_base_1603466732592/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.5.1.48
opt-einsum==3.1.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1609786840597/work
protobuf==3.13.0
psutil @ file:///C:/ci/psutil_1598370330503/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
PyJWT @ file:///C:/ci/pyjwt_1610893382614/work
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
pyreadline==2.1
PySocks @ file:///C:/ci/pysocks_1605287845585/work
python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work
pywin32==227
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-oauthlib==1.3.0
rsa @ file:///tmp/build/80754af9/rsa_1610483308194/work
scikit-learn @ file:///C:/ci/scikit-learn_1598377018496/work
scipy @ file:///C:/ci/scipy_1604596260408/work
sip==4.19.13
six @ file:///C:/ci/six_1605187374963/work
tensorboard @ file:///home/builder/ktietz/conda/conda-bld/tensorboard_1604313476433/work/tmp_pip_dir
tensorboard-plugin-wit==1.6.0
tensorflow==2.3.0
tensorflow-estimator @ file:///tmp/build/80754af9/tensorflow-estimator_1599136169057/work/whl_temp/tensorflow_estimator-2.3.0-py2.py3-none-any.whl
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado @ file:///C:/ci/tornado_1606942392901/work
tqdm @ file:///tmp/build/80754af9/tqdm_1611857934208/work
typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1611751222202/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1611694770489/work
Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/work
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
wincertstore==0.2
wrapt==1.12.1
yarl @ file:///C:/ci/yarl_1598045274898/work
zipp @ file:///tmp/build/80754af9/zipp_1604001098328/work

============== Conda Packages ==============
# packages in environment at C:\Users\master\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.3.0                       gpu  

absl-py                   0.11.0             pyhd3eb1b0_1  

aiohttp                   3.7.3            py38h2bbff1b_1  

astunparse                1.6.3                      py_0  

async-timeout             3.0.1            py38haa95532_0  

attrs                     20.3.0             pyhd3eb1b0_0  

blas                      1.0                         mkl  

blinker                   1.4              py38haa95532_0  

brotlipy                  0.7.0           py38h2bbff1b_1003  

ca-certificates           2021.1.19            haa95532_0  

cachetools                4.2.1              pyhd3eb1b0_0  

certifi                   2020.12.5        py38haa95532_0  

cffi                      1.14.4           py38hcd4344a_0  

chardet                   3.0.4           py38haa95532_1003  

click                     7.1.2              pyhd3eb1b0_0  

cryptography              2.9.2            py38h7a1dbc1_0  

cudatoolkit               10.1.243             h74a9793_0  

cudnn                     7.6.5                cuda10.1_0  

cycler                    0.10.0                   py38_0  

fastcluster               1.1.26           py38h251f6bf_2    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
freetype                  2.10.4               hd328e21_0  

gast                      0.4.0                      py_0  

git                       2.23.0               h6bb4b03_0  

google-auth               1.24.0             pyhd3eb1b0_0  

google-auth-oauthlib      0.4.2              pyhd3eb1b0_2  

google-pasta              0.2.0                      py_0  

grpcio                    1.31.0           py38he7da953_0  

h5py                      2.10.0           py38h5e291fa_0  

hdf5                      1.10.4               h7ebc959_0  

icc_rt                    2019.0.0             h0cc432a_1  

icu                       58.2                 ha925a31_3  

idna                      2.10               pyhd3eb1b0_0  

imageio                   2.9.0                      py_0  

imageio-ffmpeg            0.4.3              pyhd8ed1ab_0    conda-forge
importlib-metadata        2.0.0                      py_1  

intel-openmp              2020.2                      254  

joblib                    1.0.0              pyhd3eb1b0_0  

jpeg                      9b                   hb83a4c4_2  

keras-applications        1.0.8                      py_1  

keras-preprocessing       1.1.0                      py_1  

kiwisolver                1.3.0            py38hd77b12b_0  

libpng                    1.6.37               h2a8f88b_0  

libprotobuf               3.13.0.1             h200bbdf_0  

libtiff                   4.1.0                h56a325e_1  

lz4-c                     1.9.3                h2bbff1b_0  

markdown                  3.3.3            py38haa95532_0  

matplotlib                3.2.2                         0  

matplotlib-base           3.2.2            py38h64f37c6_0  

mkl                       2020.2                      256  

mkl-service               2.3.0            py38h196d8e1_0  

mkl_fft                   1.2.0            py38h45dec08_0  

mkl_random                1.1.1            py38h47e9c7a_0  

multidict                 4.7.6            py38he774522_1  

numpy                     1.19.2           py38hadc3359_0  

numpy-base                1.19.2           py38ha3acd2a_0  

nvidia-ml-py3             7.352.1                  pypi_0    pypi
oauthlib                  3.1.0                      py_0  

olefile                   0.46                       py_0  

opencv-python             4.5.1.48                 pypi_0    pypi
openssl                   1.1.1i               h2bbff1b_0  

opt_einsum                3.1.0                      py_0  

pathlib                   1.0.1                      py_1  

pillow                    8.1.0            py38h4fa10fc_0  

pip                       20.3.3           py38haa95532_0  

protobuf                  3.13.0.1         py38ha925a31_1  

psutil                    5.7.2            py38he774522_0  

pyasn1                    0.4.8                      py_0  

pyasn1-modules            0.2.8                      py_0  

pycparser                 2.20                       py_2  

pyjwt                     2.0.1            py38haa95532_0  

pyopenssl                 20.0.1             pyhd3eb1b0_1  

pyparsing                 2.4.7              pyhd3eb1b0_0  

pyqt                      5.9.2            py38ha925a31_4  

pyreadline                2.1                      py38_1  

pysocks                   1.7.1            py38haa95532_0  

python                    3.8.5                h5fd99cc_1  

python-dateutil           2.8.1              pyhd3eb1b0_0  

python_abi                3.8                      1_cp38    conda-forge
pywin32                   227              py38he774522_1  

qt                        5.9.7            vc14h73c81de_0  

requests                  2.25.1             pyhd3eb1b0_0  

requests-oauthlib         1.3.0                      py_0  

rsa                       4.7                pyhd3eb1b0_1  

scikit-learn              0.23.2           py38h47e9c7a_0  

scipy                     1.5.2            py38h14eb087_0  

setuptools                52.0.0           py38haa95532_0  

sip                       4.19.13          py38ha925a31_0  

six                       1.15.0           py38haa95532_0  

sqlite                    3.33.0               h2a8f88b_0  

tensorboard               2.3.0              pyh4dce500_0  

tensorboard-plugin-wit    1.6.0                      py_0  

tensorflow                2.3.0           mkl_py38h1fcfbd6_0  

tensorflow-base           2.3.0           gpu_py38h7339f5a_0  

tensorflow-estimator      2.3.0              pyheb71bc4_0  

tensorflow-gpu            2.3.0                he13fc11_0  

termcolor                 1.1.0                    py38_1  

threadpoolctl             2.1.0              pyh5ca1d4c_0  

tk                        8.6.10               he774522_0  

tornado                   6.1              py38h2bbff1b_0  

tqdm                      4.56.0             pyhd3eb1b0_0  

typing-extensions         3.7.4.3              hd3eb1b0_0  

typing_extensions         3.7.4.3            pyh06a4308_0  

urllib3                   1.26.3             pyhd3eb1b0_0  

vc                        14.2                 h21ff451_1  

vs2015_runtime            14.27.29016          h5e58377_2  

werkzeug                  1.0.1              pyhd3eb1b0_0  

wheel                     0.36.2             pyhd3eb1b0_0  

win_inet_pton             1.1.0            py38haa95532_0  

wincertstore              0.2                      py38_0  

wrapt                     1.12.1           py38he774522_1  

xz                        5.2.5                h62dcd97_0  

yarl                      1.5.1            py38he774522_0  

zipp                      3.4.0              pyhd3eb1b0_0  

zlib                      1.2.11               h62dcd97_4  

zstd                      1.4.5                h04227a9_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7
scalefactor:              0.709
batch-size:               8

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                face
coverage:                 68.75
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
convert_batchsize:        16

[global.loss]
loss_function:            ssim
mask_loss_function:       mse
l2_reg_term:              100
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      False
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
disable_warp:             False
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

Posted: **Wed Feb 03, 2021 3:08 pm**

Reducing the batch size from 18 to 8 to 4 to 1 to 1 did not help.

Posted: **Wed Feb 03, 2021 9:24 pm**

You have a 2GB GPU. There is literally no way you are running Villain model on that card.

At best you may be able to run the "lightweight" model at a very low batchsize (with mixed precision enabled)... but you may also not be able to run Faceswap at all on that card, I'm afraid.

Posted: **Wed Feb 03, 2021 9:50 pm**

Aha! Well, that explains it anyway. Thank you.

Faceswap Forum

Failed copying input tensor

Failed copying input tensor

Re: Failed copying input tensor

Re: Failed copying input tensor

Re: Failed copying input tensor

Re: Failed copying input tensor

Re: Failed copying input tensor

Re: Failed copying input tensor

Re: Failed copying input tensor