ValueError: setting an array element with a sequence

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Groundseeker
Posts: 1
Joined: Sat Jan 09, 2021 4:54 pm

ValueError: setting an array element with a sequence

Post by Groundseeker »

Hello everyone,
after a long time i wanted to start some new tests.
I recently got a Problem in a new Project where i could not start my training.
I think i narrowed it down to the Mask Type.
I couldn't start with Unet-Dfl getting an Error. (attached log file)
As for Vgg-Obstructed it was possible to start training.
I wanted to find out how the "learn mask" Option Impacts and in my Opinion the the unet-dfl showed better relsults than the vgg. So im not sure if that makes a difference for the learning process.
Maybe some can find the Problem, i already checked for newer updates.

Code: Select all

09/18/2021 12:28:44 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'color_clahe_max_size')
09/18/2021 12:28:44 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 4)
09/18/2021 12:28:44 MainProcess     _training_0                    _base           add_session_batchsize          DEBUG    Adding session batch size: 8
09/18/2021 12:28:44 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Feeder: num_images: 2, batch_size: 8, config: {'centering': 'face', 'coverage': 85.0, 'icnr_init': False, 'conv_aware_init': True, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 4, 'mouth_multiplier': 4, 'penalized_mask_loss': True, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': True, 'preview_images': 12, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(384, 384, 3), (384, 384, 1)]
09/18/2021 12:28:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(384, 384, 3), (384, 384, 1)], coverage_ratio: 0.85, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 85.0, 'icnr_init': False, 'conv_aware_init': True, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 4, 'mouth_multiplier': 4, 'penalized_mask_loss': True, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': True, 'preview_images': 12, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
09/18/2021 12:28:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
09/18/2021 12:28:44 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 3812, batchsize: 8, side: 'a', do_shuffle: True, is_preview, False, is_timelapse: False)
09/18/2021 12:28:44 MainProcess     _training_0                    generator       _get_cache                     DEBUG    Creating cache. Side: a
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 8, is_display: False, input_size: 64, output_shapes: [(384, 384, 3), (384, 384, 1)], coverage_ratio: 0.85, config: {'centering': 'face', 'coverage': 85.0, 'icnr_init': False, 'conv_aware_init': True, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 4, 'mouth_multiplier': 4, 'penalized_mask_loss': True, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': True, 'preview_images': 12, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [384]
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
09/18/2021 12:28:44 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 3812, side: 'a', do_shuffle: True)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
09/18/2021 12:28:44 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 3812, side: 'a', do_shuffle: True)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(384, 384, 3), (384, 384, 1)]
09/18/2021 12:28:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(384, 384, 3), (384, 384, 1)], coverage_ratio: 0.85, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 85.0, 'icnr_init': False, 'conv_aware_init': True, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 4, 'mouth_multiplier': 4, 'penalized_mask_loss': True, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': True, 'preview_images': 12, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
09/18/2021 12:28:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
09/18/2021 12:28:44 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 106, batchsize: 8, side: 'b', do_shuffle: True, is_preview, False, is_timelapse: False)
09/18/2021 12:28:44 MainProcess     _training_0                    generator       _get_cache                     DEBUG    Creating cache. Side: b
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 8, is_display: False, input_size: 64, output_shapes: [(384, 384, 3), (384, 384, 1)], coverage_ratio: 0.85, config: {'centering': 'face', 'coverage': 85.0, 'icnr_init': False, 'conv_aware_init': True, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 4, 'mouth_multiplier': 4, 'penalized_mask_loss': True, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': True, 'preview_images': 12, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [384]
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
09/18/2021 12:28:44 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 106, side: 'b', do_shuffle: True)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
09/18/2021 12:28:44 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 106, side: 'b', do_shuffle: True)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Setting preview feed: (side: 'a')
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(384, 384, 3), (384, 384, 1)]
09/18/2021 12:28:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(384, 384, 3), (384, 384, 1)], coverage_ratio: 0.85, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 85.0, 'icnr_init': False, 'conv_aware_init': True, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 4, 'mouth_multiplier': 4, 'penalized_mask_loss': True, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': True, 'preview_images': 12, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
09/18/2021 12:28:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
09/18/2021 12:28:44 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 3812, batchsize: 12, side: 'a', do_shuffle: True, is_preview, True, is_timelapse: False)
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 12, is_display: True, input_size: 64, output_shapes: [(384, 384, 3), (384, 384, 1)], coverage_ratio: 0.85, config: {'centering': 'face', 'coverage': 85.0, 'icnr_init': False, 'conv_aware_init': True, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 4, 'mouth_multiplier': 4, 'penalized_mask_loss': True, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': True, 'preview_images': 12, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [384]
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
09/18/2021 12:28:44 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 3812, side: 'a', do_shuffle: True)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
09/18/2021 12:28:44 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 3812, side: 'a', do_shuffle: True)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Setting preview feed: (side: 'b')
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(384, 384, 3), (384, 384, 1)]
09/18/2021 12:28:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(384, 384, 3), (384, 384, 1)], coverage_ratio: 0.85, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 85.0, 'icnr_init': False, 'conv_aware_init': True, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 4, 'mouth_multiplier': 4, 'penalized_mask_loss': True, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': True, 'preview_images': 12, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
09/18/2021 12:28:44 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
09/18/2021 12:28:44 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 106, batchsize: 12, side: 'b', do_shuffle: True, is_preview, True, is_timelapse: False)
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 12, is_display: True, input_size: 64, output_shapes: [(384, 384, 3), (384, 384, 1)], coverage_ratio: 0.85, config: {'centering': 'face', 'coverage': 85.0, 'icnr_init': False, 'conv_aware_init': True, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 4, 'mouth_multiplier': 4, 'penalized_mask_loss': True, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': True, 'preview_images': 12, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [384]
09/18/2021 12:28:44 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
09/18/2021 12:28:44 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 106, side: 'b', do_shuffle: True)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
09/18/2021 12:28:44 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 106, side: 'b', do_shuffle: True)
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Set preview feed. Batchsize: 12
09/18/2021 12:28:44 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Feeder:
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Enabling TensorBoard Logging
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Setting up TensorBoard Logging
09/18/2021 12:28:44 MainProcess     _training_0                    _base           _set_tensorboard               VERBOSE  Enabled TensorBoard Logging
09/18/2021 12:28:44 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Samples: model: '<plugins.train.model.phaze_a.Model object at 0x0000018246032670>', coverage_ratio: 0.85)
09/18/2021 12:28:44 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Samples
09/18/2021 12:28:44 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Timelapse: model: <plugins.train.model.phaze_a.Model object at 0x0000018246032670>, coverage_ratio: 0.85, image_count: 12, feeder: '<plugins.train.trainer._base._Feeder object at 0x0000018204818700>', image_paths: 2)
09/18/2021 12:28:44 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Samples: model: '<plugins.train.model.phaze_a.Model object at 0x0000018246032670>', coverage_ratio: 0.85)
09/18/2021 12:28:44 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Samples
09/18/2021 12:28:44 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Timelapse
09/18/2021 12:28:44 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized Trainer
09/18/2021 12:28:44 MainProcess     _training_0                    train           _load_trainer                  DEBUG    Loaded Trainer
09/18/2021 12:28:44 MainProcess     _training_0                    train           _run_training_cycle            DEBUG    Running Training Cycle
09/18/2021 12:28:44 MainProcess     _run_0                         generator       _validate_version              DEBUG    Setting initial extract version: 2.2
09/18/2021 12:28:44 MainProcess     _run_0                         generator       _validate_version              DEBUG    Setting initial extract version: 2.2
09/18/2021 12:28:44 MainProcess     _run_0                         multithreading  run                            DEBUG    Error in thread (_run_0): setting an array element with a sequence.
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  check_and_raise_error          DEBUG    Thread error caught: [(<class 'ValueError'>, ValueError('setting an array element with a sequence.'), <traceback object at 0x0000018204660E00>)]
09/18/2021 12:28:44 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): setting an array element with a sequence.
09/18/2021 12:28:44 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initializing constants. training_size: 768
09/18/2021 12:28:44 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 6, 'tgt_slices': slice(58, 710, None), 'warp_mapx': '[[[ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]]\n\n [[ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]]\n\n [[ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]]\n\n [[ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]]\n\n [[ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]]\n\n [[ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]]\n\n [[ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]]\n\n [[ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]\n  [ 58. 221. 384. 547. 710.]]]', 'warp_mapy': '[[[ 58.  58.  58.  58.  58.]\n  [221. 221. 221. 221. 221.]\n  [384. 384. 384. 384. 384.]\n  [547. 547. 547. 547. 547.]\n  [710. 710. 710. 710. 710.]]\n\n [[ 58.  58.  58.  58.  58.]\n  [221. 221. 221. 221. 221.]\n  [384. 384. 384. 384. 384.]\n  [547. 547. 547. 547. 547.]\n  [710. 710. 710. 710. 710.]]\n\n [[ 58.  58.  58.  58.  58.]\n  [221. 221. 221. 221. 221.]\n  [384. 384. 384. 384. 384.]\n  [547. 547. 547. 547. 547.]\n  [710. 710. 710. 710. 710.]]\n\n [[ 58.  58.  58.  58.  58.]\n  [221. 221. 221. 221. 221.]\n  [384. 384. 384. 384. 384.]\n  [547. 547. 547. 547. 547.]\n  [710. 710. 710. 710. 710.]]\n\n [[ 58.  58.  58.  58.  58.]\n  [221. 221. 221. 221. 221.]\n  [384. 384. 384. 384. 384.]\n  [547. 547. 547. 547. 547.]\n  [710. 710. 710. 710. 710.]]\n\n [[ 58.  58.  58.  58.  58.]\n  [221. 221. 221. 221. 221.]\n  [384. 384. 384. 384. 384.]\n  [547. 547. 547. 547. 547.]\n  [710. 710. 710. 710. 710.]]\n\n [[ 58.  58.  58.  58.  58.]\n  [221. 221. 221. 221. 221.]\n  [384. 384. 384. 384. 384.]\n  [547. 547. 547. 547. 547.]\n  [710. 710. 710. 710. 710.]]\n\n [[ 58.  58.  58.  58.  58.]\n  [221. 221. 221. 221. 221.]\n  [384. 384. 384. 384. 384.]\n  [547. 547. 547. 547. 547.]\n  [710. 710. 710. 710. 710.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 767]\n  [767 767]\n  [767   0]\n  [383   0]\n  [383 767]\n  [767 383]\n  [  0 383]]\n\n [[  0   0]\n  [  0 767]\n  [767 767]\n  [767   0]\n  [383   0]\n  [383 767]\n  [767 383]\n  [  0 383]]\n\n [[  0   0]\n  [  0 767]\n  [767 767]\n  [767   0]\n  [383   0]\n  [383 767]\n  [767 383]\n  [  0 383]]\n\n [[  0   0]\n  [  0 767]\n  [767 767]\n  [767   0]\n  [383   0]\n  [383 767]\n  [767 383]\n  [  0 383]]\n\n [[  0   0]\n  [  0 767]\n  [767 767]\n  [767   0]\n  [383   0]\n  [383 767]\n  [767 383]\n  [  0 383]]\n\n [[  0   0]\n  [  0 767]\n  [767 767]\n  [767   0]\n  [383   0]\n  [383 767]\n  [767 383]\n  [  0 383]]\n\n [[  0   0]\n  [  0 767]\n  [767 767]\n  [767   0]\n  [383   0]\n  [383 767]\n  [767 383]\n  [  0 383]]\n\n [[  0   0]\n  [  0 767]\n  [767 767]\n  [767   0]\n  [383   0]\n  [383 767]\n  [767 383]\n  [  0 383]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [765. 765. 765. ... 765. 765. 765.]\n  [766. 766. 766. ... 766. 766. 766.]\n  [767. 767. 767. ... 767. 767. 767.]]\n\n [[  0.   1.   2. ... 765. 766. 767.]\n  [  0.   1.   2. ... 765. 766. 767.]\n  [  0.   1.   2. ... 765. 766. 767.]\n  ...\n  [  0.   1.   2. ... 765. 766. 767.]\n  [  0.   1.   2. ... 765. 766. 767.]\n  [  0.   1.   2. ... 765. 766. 767.]]]'}
09/18/2021 12:28:44 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
09/18/2021 12:28:44 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
09/18/2021 12:28:44 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
09/18/2021 12:28:44 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
09/18/2021 12:28:44 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
09/18/2021 12:28:44 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
09/18/2021 12:28:44 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
TypeError: only size-1 arrays can be converted to Python scalars

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\Admin\FS\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "C:\Users\Admin\FS\scripts\train.py", line 190, in process
    self._end_thread(thread, err)
  File "C:\Users\Admin\FS\scripts\train.py", line 230, in _end_thread
    thread.join()
  File "C:\Users\Admin\FS\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\Admin\FS\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Admin\FS\scripts\train.py", line 252, in _training
    raise err
  File "C:\Users\Admin\FS\scripts\train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Users\Admin\FS\scripts\train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "C:\Users\Admin\FS\plugins\train\trainer\_base.py", line 191, in train_one_step
    model_inputs, model_targets = self._feeder.get_batch()
  File "C:\Users\Admin\FS\plugins\train\trainer\_base.py", line 416, in get_batch
    batch = next(self._feeds[side])
  File "C:\Users\Admin\FS\lib\multithreading.py", line 156, in iterator
    self.check_and_raise_error()
  File "C:\Users\Admin\FS\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\Admin\FS\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Admin\FS\lib\multithreading.py", line 145, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "C:\Users\Admin\FS\lib\training\generator.py", line 598, in _minibatch
    yield self._process_batch(img_paths, side)
  File "C:\Users\Admin\FS\lib\training\generator.py", line 627, in _process_batch
    batch = self._apply_mask(filenames, cache, batch, side)
  File "C:\Users\Admin\FS\lib\training\generator.py", line 758, in _apply_mask
    masks = np.array([self._get_mask(item[key], size)
ValueError: setting an array element with a sequence.

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         860ccb9 Suppress Tensorflow stderr custom error messages. 24888f6 Suppress ptxas error in GUI on Windows. 15501b1 typofix. 7798a19 bugfix: _requirements_base.txt - Exclude badly versioned nvml. cf4b567 Tensorflow 2.6 Support (#1182)
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: Advanced Micro Devices, Inc. - gfx1030 (experimental)
gpu_devices_active:  GPU_0
gpu_driver:          ['3276.6 (PAL,LC)']
gpu_vram:            GPU_0: 16368MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.19043-SP0
os_release:          10
py_command:          C:\Users\Admin\FS\faceswap.py train -A W:/Temp2/TrainsetA -B W:/Temp2/TrainsetB -m W:/Temp2/Model -t phaze-a -bs 8 -it 1000000 -s 250 -ss 25000 -ps 100 -L INFO -gui
py_conda_version:    conda 4.10.3
py_implementation:   CPython
py_version:          3.8.11
py_virtual_env:      True
sys_cores:           16
sys_processor:       AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD
sys_ram:             Total: 32711MB, Available: 23690MB, Used: 9020MB, Free: 23690MB

=============== Pip Packages ===============
absl-py==0.13.0
astunparse==1.6.3
cachetools==4.2.2
certifi==2021.5.30
cffi==1.14.6
charset-normalizer==2.0.4
cycler==0.10.0
enum34==1.1.10
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.3.3
google-auth==1.35.0
google-auth-oauthlib==0.4.5
google-pasta==0.2.0
grpcio==1.39.0
h5py==2.10.0
idna==3.2
imageio @ file:///tmp/build/80754af9/imageio_1617700267927/work
imageio-ffmpeg==0.4.5
joblib @ file:///tmp/build/80754af9/joblib_1613502643832/work
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/ci/kiwisolver_1612282606037/work
Markdown==3.3.4
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.3.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy==1.18.5
nvidia-ml-py==11.470.66
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.1
olefile==0.46
opencv-python==4.5.3.56
opt-einsum==3.3.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1625663293593/work
plaidml==0.7.0
plaidml-keras==0.7.0
protobuf==3.17.3
psutil @ file:///C:/ci/psutil_1612298324802/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==228
PyYAML==5.4.1
requests==2.26.0
requests-oauthlib==1.3.0
rsa==4.7.2
scikit-learn @ file:///C:/ci/scikit-learn_1622739500535/work
scipy @ file:///C:/ci/scipy_1616703433439/work
sip==4.19.13
six @ file:///tmp/build/80754af9/six_1623709665295/work
tensorboard==2.2.2
tensorboard-plugin-wit==1.8.0
tensorflow==2.2.3
tensorflow-estimator==2.2.0
termcolor==1.1.0
threadpoolctl @ file:///Users/ktietz/demo/mc3/conda-bld/threadpoolctl_1628668762525/work
tornado @ file:///C:/ci/tornado_1606942392901/work
tqdm @ file:///tmp/build/80754af9/tqdm_1629302309755/work
urllib3==1.26.6
Werkzeug==2.0.1
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\Admin\MiniConda3\envs\FS-AMD:
#
# Name                    Version                   Build  Channel
absl-py                   0.13.0                   pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
ca-certificates 2021.7.5 haa95532_1
cachetools 4.2.2 pypi_0 pypi certifi 2021.5.30 py38haa95532_0
cffi 1.14.6 pypi_0 pypi charset-normalizer 2.0.4 pypi_0 pypi cycler 0.10.0 py38_0
enum34 1.1.10 pypi_0 pypi fastcluster 1.1.26 py38h251f6bf_2 conda-forge ffmpy 0.2.3 pypi_0 pypi freetype 2.10.4 hd328e21_0
gast 0.3.3 pypi_0 pypi git 2.32.0 haa95532_1
google-auth 1.35.0 pypi_0 pypi google-auth-oauthlib 0.4.5 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.39.0 pypi_0 pypi h5py 2.10.0 pypi_0 pypi icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 3.2 pypi_0 pypi imageio 2.9.0 pyhd3eb1b0_0
imageio-ffmpeg 0.4.5 pypi_0 pypi intel-openmp 2021.3.0 haa95532_3372
joblib 1.0.1 pyhd3eb1b0_0
jpeg 9b hb83a4c4_2
keras 2.2.4 pypi_0 pypi keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.3.1 py38hd77b12b_0
libpng 1.6.37 h2a8f88b_0
libtiff 4.2.0 hd0e1b90_0
lz4-c 1.9.3 h2bbff1b_1
markdown 3.3.4 pypi_0 pypi matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38h64f37c6_0
mkl 2020.2 256
mkl-service 2.3.0 py38h196d8e1_0
mkl_fft 1.3.0 py38h46781fe_0
mkl_random 1.1.1 py38h47e9c7a_0
numpy 1.18.5 pypi_0 pypi nvidia-ml-py 11.470.66 pypi_0 pypi nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.1 pypi_0 pypi olefile 0.46 py_0
opencv-python 4.5.3.56 pypi_0 pypi openssl 1.1.1k h2bbff1b_0
opt-einsum 3.3.0 pypi_0 pypi pathlib 1.0.1 py_1
pillow 8.3.1 py38h4fa10fc_0
pip 21.0.1 py38haa95532_0
plaidml 0.7.0 pypi_0 pypi plaidml-keras 0.7.0 pypi_0 pypi protobuf 3.17.3 pypi_0 pypi psutil 5.8.0 py38h2bbff1b_1
pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycparser 2.20 pypi_0 pypi pyparsing 2.4.7 pyhd3eb1b0_0
pyqt 5.9.2 py38ha925a31_4
python 3.8.11 h6244533_1
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.8 2_cp38 conda-forge pywin32 228 py38hbaba5e8_1
pyyaml 5.4.1 pypi_0 pypi qt 5.9.7 vc14h73c81de_0
requests 2.26.0 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rsa 4.7.2 pypi_0 pypi scikit-learn 0.24.2 py38hf11a4ad_1
scipy 1.6.2 py38h14eb087_0
setuptools 52.0.0 py38haa95532_0
sip 4.19.13 py38ha925a31_0
six 1.16.0 pyhd3eb1b0_0
sqlite 3.36.0 h2bbff1b_0
tensorboard 2.2.2 pypi_0 pypi tensorboard-plugin-wit 1.8.0 pypi_0 pypi tensorflow 2.2.3 pypi_0 pypi tensorflow-estimator 2.2.0 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi threadpoolctl 2.2.0 pyhbf3da8f_0
tk 8.6.10 he774522_0
tornado 6.1 py38h2bbff1b_0
tqdm 4.62.1 pyhd3eb1b0_1
urllib3 1.26.6 pypi_0 pypi vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 2.0.1 pypi_0 pypi wheel 0.37.0 pyhd3eb1b0_0
wincertstore 0.2 py38_0
wrapt 1.12.1 pypi_0 pypi xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.9 h19a0ad4_0 ================= Configs ================== --------- .faceswap --------- backend: amd --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 6 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: unsharp_mask amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mov codec: libx264 crf: 20 preset: medium tune: None profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False [align.fan] batch-size: 8 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 scalefactor: 0.709 batch-size: 8 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 [detect.s3fd] confidence: 75 batch-size: 4 [mask.bisenet_fp] batch-size: 8 include_ears: False include_hair: False include_glasses: False [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] centering: face coverage: 85.0 icnr_init: False conv_aware_init: True optimizer: adam learning_rate: 5e-05 epsilon_exponent: -7 reflect_padding: False allow_growth: False mixed_precision: False nan_protection: True convert_batchsize: 16 [global.loss] loss_function: ssim mask_loss_function: mse l2_reg_term: 100 eye_multiplier: 4 mouth_multiplier: 4 penalized_mask_loss: True mask_type: unet-dfl mask_blur_kernel: 3 mask_threshold: 4 learn_mask: True [model.dfaker] output_size: 256 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 192 clipnorm: False architecture: liae autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 128 [model.original] lowmem: False [model.phaze_a] output_size: 384 shared_fc: None enable_gblock: True split_fc: True split_gblock: False split_decoders: False enc_architecture: fs_original enc_scaling: 40 enc_load_weights: True bottleneck_type: dense bottleneck_norm: None bottleneck_size: 1024 bottleneck_in_encoder: True fc_depth: 1 fc_min_filters: 1024 fc_max_filters: 1024 fc_dimensions: 4 fc_filter_slope: -0.5 fc_dropout: 0.0 fc_upsampler: upsample2d fc_upsamples: 1 fc_upsample_filters: 512 fc_gblock_depth: 3 fc_gblock_min_nodes: 512 fc_gblock_max_nodes: 512 fc_gblock_filter_slope: -0.5 fc_gblock_dropout: 0.0 dec_upscale_method: subpixel dec_norm: None dec_min_filters: 64 dec_max_filters: 512 dec_filter_slope: -0.45 dec_res_blocks: 1 dec_output_kernel: 5 dec_gaussian: True dec_skip_last_residual: True freeze_layers: keras_encoder load_layers: encoder fs_original_depth: 4 fs_original_min_filters: 128 fs_original_max_filters: 1024 mobilenet_width: 1.0 mobilenet_depth: 1 mobilenet_dropout: 0.001 [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 256 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 12 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
Locked