suddently error when training

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
adam_macchiato
Posts: 16
Joined: Tue Jul 26, 2022 5:26 am
Has thanked: 4 times

suddently error when training

Post by adam_macchiato »

i just chane the training source folder , cannot train and show these error ,

Code: Select all

05/23/2023 13:54:05 MainProcess     _training                      config          _check_exists                  DEBUG    Config file exists: 'C:\Users\adama\faceswap\config\train.ini'
05/23/2023 13:54:05 MainProcess     _training                      config          _load_config                   VERBOSE  Loading config: 'C:\Users\adama\faceswap\config\train.ini'
05/23/2023 13:54:05 MainProcess     _training                      config          _validate_config               DEBUG    Validating config
05/23/2023 13:54:05 MainProcess     _training                      config          _check_config_change           DEBUG    Default config has not changed
05/23/2023 13:54:05 MainProcess     _training                      config          _check_config_choices          DEBUG    Checking config choices
05/23/2023 13:54:05 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'keras_encoder' to list ['keras_encoder'] for section 'model.phaze_a', option 'freeze_layers'
05/23/2023 13:54:05 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'encoder' to list ['encoder'] for section 'model.phaze_a', option 'load_layers'
05/23/2023 13:54:05 MainProcess     _training                      config          _check_config_choices          DEBUG    Checked config choices
05/23/2023 13:54:05 MainProcess     _training                      config          _validate_config               DEBUG    Validated config
05/23/2023 13:54:05 MainProcess     _training                      config          _handle_config                 DEBUG    Handled config
05/23/2023 13:54:05 MainProcess     _training                      config          __init__                       DEBUG    Initialized: Config
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'learning_rate')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 3e-05)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'epsilon_exponent')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: -3)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'autoclip')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'allow_growth')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'mixed_precision')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: True)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'nan_protection')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: True)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'convert_batchsize')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 16)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: ms_ssim)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_2')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mae)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_2')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 25)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_3')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: lpips_alex)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_3')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 5)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_4')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: ffl)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_4')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 100)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mask_loss_function')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mse)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'eye_multiplier')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 3)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mouth_multiplier')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 2)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'model.phaze_a', option: 'fc_dropout')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 0.0)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'model.phaze_a', option: 'fc_gblock_dropout')
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 0.0)
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'model.phaze_a', option: 'freeze_layers')
05/23/2023 13:54:05 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'keras_encoder' to list ['keras_encoder'] for section 'model.phaze_a', option 'freeze_layers'
05/23/2023 13:54:05 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'list'>, value: ['keras_encoder'])
05/23/2023 13:54:05 MainProcess     _training                      config          changeable_items               DEBUG    Alterable for existing models: {'learning_rate': 3e-05, 'epsilon_exponent': -3, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ms_ssim', 'loss_function_2': 'mae', 'loss_weight_2': 25, 'loss_function_3': 'lpips_alex', 'loss_weight_3': 5, 'loss_function_4': 'ffl', 'loss_weight_4': 100, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'fc_dropout': 0.0, 'fc_gblock_dropout': 0.0, 'freeze_layers': ['keras_encoder']}
05/23/2023 13:54:05 MainProcess     _training                      model           __init__                       DEBUG    Initializing State: (model_dir: 'D:\facewarp\Model Alice & Li V2_L\Model Alice & Li V2_L', model_name: 'phaze_a', config_changeable_items: '{'learning_rate': 3e-05, 'epsilon_exponent': -3, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ms_ssim', 'loss_function_2': 'mae', 'loss_weight_2': 25, 'loss_function_3': 'lpips_alex', 'loss_weight_3': 5, 'loss_function_4': 'ffl', 'loss_weight_4': 100, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'fc_dropout': 0.0, 'fc_gblock_dropout': 0.0, 'freeze_layers': ['keras_encoder']}', no_logs: False
05/23/2023 13:54:05 MainProcess     _training                      serializer      get_serializer                 DEBUG    <lib.serializer._JSONSerializer object at 0x0000021AEA344460>
05/23/2023 13:54:05 MainProcess     _training                      model           _load                          DEBUG    Loading State
05/23/2023 13:54:05 MainProcess     _training                      serializer      load                           DEBUG    filename: D:\facewarp\Model Alice & Li V2_L\Model Alice & Li V2_L\phaze_a_state.json
05/23/2023 13:54:05 MainProcess     _training                      serializer      load                           DEBUG    stored data type: <class 'bytes'>
05/23/2023 13:54:05 MainProcess     _training                      serializer      unmarshal                      DEBUG    data type: <class 'bytes'>
05/23/2023 13:54:05 MainProcess     _training                      serializer      unmarshal                      DEBUG    returned data type: <class 'dict'>
05/23/2023 13:54:05 MainProcess     _training                      serializer      load                           DEBUG    data type: <class 'dict'>
05/23/2023 13:54:05 MainProcess     _training                      model           _load                          DEBUG    Loaded state: {'name': 'phaze_a', 'sessions': {'1': {'timestamp': 1683966772.4561841, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 8, 'iterations': 2055, 'config': {'learning_rate': 3e-05, 'epsilon_exponent': -3, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mae', 'loss_weight_2': 100, 'loss_function_3': 'ffl', 'loss_weight_3': 50, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mae', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'fc_dropout': 0.0, 'fc_gblock_dropout': 0.0, 'freeze_layers': ['keras_encoder']}}, '2': {'timestamp': 1683969323.2565155, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 8, 'iterations': 206801, 'config': {'learning_rate': 3e-05, 'epsilon_exponent': -3, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ms_ssim', 'loss_function_2': 'mae', 'loss_weight_2': 100, 'loss_function_3': 'ffl', 'loss_weight_3': 50, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mae', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'fc_dropout': 0.0, 'fc_gblock_dropout': 0.0, 'freeze_layers': ['keras_encoder']}}, '3': {'timestamp': 1684218367.0934753, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 8, 'iterations': 771, 'config': {'learning_rate': 3e-05, 'epsilon_exponent': -3, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ms_ssim', 'loss_function_2': 'mae', 'loss_weight_2': 100, 'loss_function_3': 'ffl', 'loss_weight_3': 50, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mae', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'fc_dropout': 0.0, 'fc_gblock_dropout': 0.0, 'freeze_layers': ['keras_encoder']}}, '4': {'timestamp': 1684220514.234757, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 6, 'iterations': 56765, 'config': {'learning_rate': 3e-05, 'epsilon_exponent': -4, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ms_ssim', 'loss_function_2': 'mae', 'loss_weight_2': 25, 'loss_function_3': 'lpips_vgg16', 'loss_weight_3': 5, 'loss_function_4': 'ffl', 'loss_weight_4': 100, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'fc_dropout': 0.0, 'fc_gblock_dropout': 0.0, 'freeze_layers': ['keras_encoder']}}, '5': {'timestamp': 1684301890.4861472, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 6, 'iterations': 369538, 'config': {'learning_rate': 3e-05, 'epsilon_exponent': -4, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ms_ssim', 'loss_function_2': 'mae', 'loss_weight_2': 25, 'loss_function_3': 'lpips_vgg16', 'loss_weight_3': 5, 'loss_function_4': 'ffl', 'loss_weight_4': 100, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'fc_dropout': 0.0, 'fc_gblock_dropout': 0.0, 'freeze_layers': ['keras_encoder']}}}, 'lowest_avg_loss': {'a': 0.02910966745018959, 'b': 0.02204980169609189}, 'iterations': 635930, 'mixed_precision_layers': [], 'config': {'centering': 'face', 'coverage': 87.5, 'optimizer': 'adam', 'learning_rate': 3e-05, 'epsilon_exponent': -4, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ms_ssim', 'loss_function_2': 'mae', 'loss_weight_2': 25, 'loss_function_3': 'lpips_vgg16', 'loss_weight_3': 5, 'loss_function_4': 'ffl', 'loss_weight_4': 100, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'output_size': 256, 'shared_fc': None, 'enable_gblock': True, 'split_fc': True, 'split_gblock': False, 'split_decoders': False, 'enc_architecture': 'efficientnet_v2_l', 'enc_scaling': 60, 'enc_load_weights': True, 'bottleneck_type': 'dense', 'bottleneck_norm': None, 'bottleneck_size': 512, 'bottleneck_in_encoder': True, 'fc_depth': 1, 'fc_min_filters': 1280, 'fc_max_filters': 1280, 'fc_dimensions': 8, 'fc_filter_slope': -0.5, 'fc_dropout': 0.0, 'fc_upsampler': 'upsample2d', 'fc_upsamples': 1, 'fc_upsample_filters': 1280, 'fc_gblock_depth': 3, 'fc_gblock_min_nodes': 512, 'fc_gblock_max_nodes': 512, 'fc_gblock_filter_slope': -0.5, 'fc_gblock_dropout': 0.0, 'dec_upscale_method': 'resize_images', 'dec_upscales_in_fc': 0, 'dec_norm': None, 'dec_min_filters': 160, 'dec_max_filters': 640, 'dec_slope_mode': 'full', 'dec_filter_slope': -0.33, 'dec_res_blocks': 1, 'dec_output_kernel': 3, 'dec_gaussian': True, 'dec_skip_last_residual': False, 'freeze_layers': ['efficientnetv2-l'], 'load_layers': ['encoder'], 'fs_original_depth': 4, 'fs_original_min_filters': 128, 'fs_original_max_filters': 1024, 'fs_original_use_alt': False, 'mobilenet_width': 1.0, 'mobilenet_depth': 1, 'mobilenet_dropout': 0.001, 'mobilenet_minimalistic': False}}
05/23/2023 13:54:05 MainProcess     _training                      model           _update_legacy_config          DEBUG    Checking for legacy state file update
05/23/2023 13:54:05 MainProcess     _training                      model           _update_legacy_config          DEBUG    Legacy item 'dssim_loss' not in config. Skipping update
05/23/2023 13:54:05 MainProcess     _training                      model           _update_legacy_config          DEBUG    Legacy item 'l2_reg_term' not in config. Skipping update
05/23/2023 13:54:05 MainProcess     _training                      model           _update_legacy_config          DEBUG    Legacy item 'clipnorm' not in config. Skipping update
05/23/2023 13:54:05 MainProcess     _training                      model           _update_legacy_config          DEBUG    State file updated for legacy config: False
05/23/2023 13:54:05 MainProcess     _training                      model           _update_changed_config_items   INFO     Config item: 'epsilon_exponent' has been updated from '-4' to '-3'
05/23/2023 13:54:05 MainProcess     _training                      model           _update_changed_config_items   INFO     Config item: 'loss_function_3' has been updated from 'lpips_vgg16' to 'lpips_alex'
05/23/2023 13:54:05 MainProcess     _training                      model           _update_changed_config_items   INFO     Config item: 'freeze_layers' has been updated from '['efficientnetv2-l']' to '['keras_encoder']'
05/23/2023 13:54:05 MainProcess     _training                      model           _replace_config                DEBUG    Replacing config. Old config: {'centering': 'face', 'coverage': 87.5, 'optimizer': 'adam', 'learning_rate': 3e-05, 'epsilon_exponent': -3, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ms_ssim', 'loss_function_2': 'mae', 'loss_weight_2': 25, 'loss_function_3': 'lpips_alex', 'loss_weight_3': 5, 'loss_function_4': 'ffl', 'loss_weight_4': 100, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'output_size': 256, 'shared_fc': None, 'enable_gblock': True, 'split_fc': True, 'split_gblock': False, 'split_decoders': False, 'enc_architecture': 'efficientnet_v2_l', 'enc_scaling': 60, 'enc_load_weights': True, 'bottleneck_type': 'dense', 'bottleneck_norm': None, 'bottleneck_size': 512, 'bottleneck_in_encoder': True, 'fc_depth': 1, 'fc_min_filters': 1280, 'fc_max_filters': 1280, 'fc_dimensions': 8, 'fc_filter_slope': -0.5, 'fc_dropout': 0.0, 'fc_upsampler': 'upsample2d', 'fc_upsamples': 1, 'fc_upsample_filters': 1280, 'fc_gblock_depth': 3, 'fc_gblock_min_nodes': 512, 'fc_gblock_max_nodes': 512, 'fc_gblock_filter_slope': -0.5, 'fc_gblock_dropout': 0.0, 'dec_upscale_method': 'resize_images', 'dec_upscales_in_fc': 0, 'dec_norm': None, 'dec_min_filters': 160, 'dec_max_filters': 640, 'dec_slope_mode': 'full', 'dec_filter_slope': -0.33, 'dec_res_blocks': 1, 'dec_output_kernel': 3, 'dec_gaussian': True, 'dec_skip_last_residual': False, 'freeze_layers': ['keras_encoder'], 'load_layers': ['encoder'], 'fs_original_depth': 4, 'fs_original_min_filters': 128, 'fs_original_max_filters': 1024, 'fs_original_use_alt': False, 'mobilenet_width': 1.0, 'mobilenet_depth': 1, 'mobilenet_dropout': 0.001, 'mobilenet_minimalistic': False}
05/23/2023 13:54:05 MainProcess     _training                      model           _replace_config                DEBUG    Replaced config. New config: {'centering': 'face', 'coverage': 87.5, 'optimizer': 'adam', 'learning_rate': 3e-05, 'epsilon_exponent': -3, 'autoclip': False, 'allow_growth': False, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ms_ssim', 'loss_function_2': 'mae', 'loss_weight_2': 25, 'loss_function_3': 'lpips_alex', 'loss_weight_3': 5, 'loss_function_4': 'ffl', 'loss_weight_4': 100, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'output_size': 256, 'shared_fc': None, 'enable_gblock': True, 'split_fc': True, 'split_gblock': False, 'split_decoders': False, 'enc_architecture': 'efficientnet_v2_l', 'enc_scaling': 60, 'enc_load_weights': True, 'bottleneck_type': 'dense', 'bottleneck_norm': None, 'bottleneck_size': 512, 'bottleneck_in_encoder': True, 'fc_depth': 1, 'fc_min_filters': 1280, 'fc_max_filters': 1280, 'fc_dimensions': 8, 'fc_filter_slope': -0.5, 'fc_dropout': 0.0, 'fc_upsampler': 'upsample2d', 'fc_upsamples': 1, 'fc_upsample_filters': 1280, 'fc_gblock_depth': 3, 'fc_gblock_min_nodes': 512, 'fc_gblock_max_nodes': 512, 'fc_gblock_filter_slope': -0.5, 'fc_gblock_dropout': 0.0, 'dec_upscale_method': 'resize_images', 'dec_upscales_in_fc': 0, 'dec_norm': None, 'dec_min_filters': 160, 'dec_max_filters': 640, 'dec_slope_mode': 'full', 'dec_filter_slope': -0.33, 'dec_res_blocks': 1, 'dec_output_kernel': 3, 'dec_gaussian': True, 'dec_skip_last_residual': False, 'freeze_layers': ['keras_encoder'], 'load_layers': ['encoder'], 'fs_original_depth': 4, 'fs_original_min_filters': 128, 'fs_original_max_filters': 1024, 'fs_original_use_alt': False, 'mobilenet_width': 1.0, 'mobilenet_depth': 1, 'mobilenet_dropout': 0.001, 'mobilenet_minimalistic': False}
05/23/2023 13:54:05 MainProcess     _training                      model           _replace_config                INFO     Using configuration saved in state file
05/23/2023 13:54:05 MainProcess     _training                      model           _new_session_id                DEBUG    6
05/23/2023 13:54:05 MainProcess     _training                      model           _create_new_session            DEBUG    Creating new session. id: 6
05/23/2023 13:54:05 MainProcess     _training                      model           __init__                       DEBUG    Initialized State:
05/23/2023 13:54:05 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Settings: (arguments: Namespace(func=<bound method ScriptExecutor.execute_script of <lib.cli.launcher.ScriptExecutor object at 0x0000021ADE1B00D0>>, exclude_gpus=None, configfile=None, loglevel='INFO', logfile=None, redirect_gui=True, colab=False, input_a='D:\\facewarp\\Face Alice Finall', input_b='D:\\facewarp\\Face Li', model_dir='D:\\facewarp\\Model Alice & Li V2_L\\Model Alice & Li V2_L', load_weights=None, trainer='phaze-a', summary=False, freeze_weights=False, batch_size=8, iterations=2000000, distribution_strategy='default', save_interval=250, snapshot_interval=25000, timelapse_input_a='D:\\facewarp\\Face Alice Finall', timelapse_input_b='D:\\facewarp\\Face Li', timelapse_output='D:\\facewarp\\Timeline', preview=False, write_image=False, no_logs=False, warp_to_landmarks=False, no_flip=False, no_augment_color=False, no_warp=True), mixed_precision: True, allow_growth: False, is_predict: False)
05/23/2023 13:54:05 MainProcess     _training                      settings        _set_tf_settings               DEBUG    Not setting any specific Tensorflow settings
05/23/2023 13:54:05 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    use_mixed_precision: True
05/23/2023 13:54:05 MainProcess     _training                      device_compatibility_check _log_device_compatibility_check INFO     Mixed precision compatibility check (mixed_float16): OK\nYour GPU will likely run quickly with dtype policy mixed_float16 as it has compute capability of at least 7.0. Your GPU: NVIDIA GeForce RTX 3090, compute capability 8.6
05/23/2023 13:54:05 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    Enabled mixed precision. (Compute dtype: float16, variable_dtype: float32)
05/23/2023 13:54:05 MainProcess     _training                      settings        __init__                       INFO     Enabling Mixed Precision Training.
05/23/2023 13:54:05 MainProcess     _training                      settings        _get_strategy                  DEBUG    Using strategy: <tensorflow.python.distribute.distribute_lib._DefaultDistributionStrategy object at 0x0000021AEB378CD0>
05/23/2023 13:54:05 MainProcess     _training                      settings        __init__                       DEBUG    Initialized Settings
05/23/2023 13:54:05 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Loss: (color_order: bgr)
05/23/2023 13:54:05 MainProcess     _training                      settings        _get_mask_channels             DEBUG    uses_masks: (True, True, True), mask_channels: [3, 4, 5]
05/23/2023 13:54:05 MainProcess     _training                      settings        __init__                       DEBUG    Initialized: Loss
05/23/2023 13:54:05 MainProcess     _training                      model           __init__                       DEBUG    Initialized ModelBase (Model)
05/23/2023 13:54:05 MainProcess     _training                      phaze_a         _select_freeze_layers          DEBUG    Substituting 'keras_encoder' for 'efficientnet_v2_l'
05/23/2023 13:54:05 MainProcess     _training                      phaze_a         _get_input_shape               DEBUG    Encoder input set to: (288, 288, 3)
05/23/2023 13:54:05 MainProcess     _training                      settings        strategy_scope                 DEBUG    Using strategy scope: <tensorflow.python.distribute.distribute_lib._DefaultDistributionContext object at 0x0000021AEA166080>
05/23/2023 13:54:05 MainProcess     _training                      io              _load                          DEBUG    Loading model: D:\facewarp\Model Alice & Li V2_L\Model Alice & Li V2_L\phaze_a.h5
05/23/2023 13:54:05 MainProcess     _training                      multithreading  run                            DEBUG    Error in thread (_training): Unable to open file (bad object header version number)
05/23/2023 13:54:05 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
05/23/2023 13:54:05 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
05/23/2023 13:54:05 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
05/23/2023 13:54:05 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
05/23/2023 13:54:05 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
05/23/2023 13:54:05 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training'
05/23/2023 13:54:05 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training'
Traceback (most recent call last):
  File "C:\Users\adama\faceswap\lib\cli\launcher.py", line 230, in execute_script
    process.process()
  File "C:\Users\adama\faceswap\scripts\train.py", line 213, in process
    self._end_thread(thread, err)
  File "C:\Users\adama\faceswap\scripts\train.py", line 253, in _end_thread
    thread.join()
  File "C:\Users\adama\faceswap\lib\multithreading.py", line 220, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\adama\faceswap\lib\multithreading.py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\adama\faceswap\scripts\train.py", line 275, in _training
    raise err
  File "C:\Users\adama\faceswap\scripts\train.py", line 263, in _training
    model = self._load_model()
  File "C:\Users\adama\faceswap\scripts\train.py", line 291, in _load_model
    model.build()
  File "C:\Users\adama\faceswap\plugins\train\model\phaze_a.py", line 220, in build
    model = self._io._load()  # pylint:disable=protected-access
  File "C:\Users\adama\faceswap\plugins\train\model\_base\io.py", line 152, in _load
    model = load_model(self._filename, compile=False)
  File "C:\Users\adama\MiniConda3\envs\faceswap\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\adama\MiniConda3\envs\faceswap\lib\site-packages\h5py\_hl\files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "C:\Users\adama\MiniConda3\envs\faceswap\lib\site-packages\h5py\_hl\files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5f.pyx", line 106, in h5py.h5f.open
OSError: Unable to open file (bad object header version number)

============ System Information ============
backend:             nvidia
encoding:            cp950
git_branch:          master
git_commits:         e2ad3e2 add ru locale (#1311)
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce RTX 3090
gpu_devices_active:  GPU_0
gpu_driver:          531.79
gpu_vram:            GPU_0: 24576MB (23701MB free)
os_machine:          AMD64
os_platform:         Windows-10-10.0.19045-SP0
os_release:          10
py_command:          C:\Users\adama\faceswap\faceswap.py train -A D:/facewarp/Face Alice Finall -B D:/facewarp/Face Li -m D:/facewarp/Model Alice & Li V2_L/Model Alice & Li V2_L -t phaze-a -bs 8 -it 2000000 -D default -s 250 -ss 25000 -tia D:/facewarp/Face Alice Finall -tib D:/facewarp/Face Li -to D:/facewarp/Timeline -nw -L INFO -gui
py_conda_version:    conda 23.3.1
py_implementation:   CPython
py_version:          3.9.16
py_virtual_env:      True
sys_cores:           24
sys_processor:       AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
sys_ram:             Total: 32677MB, Available: 27160MB, Used: 5516MB, Free: 27160MB

=============== Pip Packages ===============
absl-py @ file:///C:/b/abs_5babsu7y5x/croot/absl-py_1666362945682/work
astunparse==1.6.3
cachetools==5.3.0
certifi==2023.5.7
charset-normalizer==3.1.0
cloudpickle @ file:///C:/b/abs_3796yxesic/croot/cloudpickle_1683040098851/work
colorama @ file:///C:/b/abs_a9ozq0l032/croot/colorama_1672387194846/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
dm-tree @ file:///C:/b/abs_10z0iy5knj/croot/dm-tree_1671027465819/work
fastcluster @ file:///D:/bld/fastcluster_1649783471014/work
ffmpy==0.3.0
flatbuffers==23.5.8
fonttools==4.25.0
gast==0.4.0
google-auth==2.17.3
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.54.0
h5py==3.8.0
idna==3.4
imageio @ file:///C:/b/abs_27kq2gy1us/croot/imageio_1677879918708/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1673483481485/work
importlib-metadata==6.6.0
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1663332044897/work
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/b/abs_88mdhvtahm/croot/kiwisolver_1672387921783/work
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.2
matplotlib @ file:///C:/b/abs_ae02atcfur/croot/matplotlib-suite_1667356722968/work
mkl-fft==1.3.6
mkl-random @ file:///C:/Users/dev-admin/mkl/mkl_random_1682977971003/work
mkl-service==2.4.0
munkres==1.1.4
numexpr @ file:///C:/b/abs_afm0oewmmt/croot/numexpr_1683221839116/work
numpy @ file:///C:/Users/dev-admin/mkl/numpy_and_numpy_base_1682981337988/work
nvidia-ml-py==11.525.112
oauthlib==3.2.2
opencv-python==4.7.0.72
opt-einsum==3.3.0
packaging @ file:///C:/b/abs_ed_kb9w6g4/croot/packaging_1678965418855/work
Pillow==9.4.0
ply==3.11
protobuf==3.19.6
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
pyasn1==0.5.0
pyasn1-modules==0.3.0
pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work
PyQt5==5.15.7
PyQt5-sip @ file:///C:/Windows/Temp/abs_d7gmd2jg8i/croots/recipe/pyqt-split_1659273064801/work/pyqt_sip
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==305.1
pywinpty @ file:///C:/ci_310/pywinpty_1644230983541/work/target/wheels/pywinpty-2.0.2-cp39-none-win_amd64.whl
requests==2.30.0
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn @ file:///C:/b/abs_a0emltxu06/croot/scikit-learn_1680198772090/work
scipy==1.9.3
sip @ file:///C:/Windows/Temp/abs_b8fxd17m2u/croots/recipe/sip_1659012372737/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.10.0
tensorflow-gpu==2.10.1
tensorflow-io-gcs-filesystem==0.31.0
tensorflow-probability @ file:///tmp/build/80754af9/tensorflow-probability_1633017132682/work
termcolor==2.3.0
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
tornado @ file:///C:/ci/tornado_1662458743919/work
tqdm @ file:///C:/b/abs_f76j9hg7pv/croot/tqdm_1679561871187/work
typing_extensions @ file:///C:/b/abs_a1bb332wcs/croot/typing_extensions_1681939523095/work
urllib3==2.0.2
Werkzeug==2.3.4
wrapt==1.15.0
zipp==3.15.0

============== Conda Packages ==============
# packages in environment at C:\Users\adama\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   1.3.0            py39haa95532_0  
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
brotli                    1.0.9                h2bbff1b_7  
brotli-bin                1.0.9                h2bbff1b_7  
ca-certificates           2023.5.7             h56e8100_0    conda-forge
cachetools                5.3.0                    pypi_0    pypi
certifi                   2023.5.7                 pypi_0    pypi
charset-normalizer        3.1.0                    pypi_0    pypi
cloudpickle               2.2.1            py39haa95532_0  
colorama                  0.4.6            py39haa95532_0  
cudatoolkit               11.2.2              h933977f_10    conda-forge
cudnn                     8.1.0.77             h3e0f4f4_0    conda-forge
cycler                    0.11.0             pyhd3eb1b0_0  
decorator                 5.1.1              pyhd3eb1b0_0  
dm-tree                   0.1.7            py39hd77b12b_1  
fastcluster               1.2.6            py39h2e25243_1    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.3.0                    pypi_0    pypi
flatbuffers               23.5.8                   pypi_0    pypi
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.12.1               ha860e81_0  
gast                      0.4.0                    pypi_0    pypi
giflib                    5.2.1                h8cc25b3_3  
git                       2.34.1               haa95532_0  
glib                      2.69.1               h5dc1a3c_2  
google-auth               2.17.3                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.54.0                   pypi_0    pypi
gst-plugins-base          1.18.5               h9e645db_0  
gstreamer                 1.18.5               hd78058f_0  
h5py                      3.8.0                    pypi_0    pypi
icc_rt                    2022.1.0             h6049295_2  
icu                       58.2                 ha925a31_3  
idna                      3.4                      pypi_0    pypi
imageio                   2.26.0           py39haa95532_0  
imageio-ffmpeg            0.4.8              pyhd8ed1ab_0    conda-forge
importlib-metadata        6.6.0                    pypi_0    pypi
intel-openmp              2023.1.0         h59b6b97_46319  
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h2bbff1b_1  
keras                     2.10.0                   pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.4            py39hd77b12b_0  
krb5                      1.19.4               h5b6d351_0  
lerc                      3.0                  hd77b12b_0  
libbrotlicommon           1.0.9                h2bbff1b_7  
libbrotlidec              1.0.9                h2bbff1b_7  
libbrotlienc              1.0.9                h2bbff1b_7  
libclang                  16.0.0                   pypi_0    pypi
libclang13                14.0.6          default_h8e68704_1  
libdeflate                1.17                 h2bbff1b_0  
libffi                    3.4.4                hd77b12b_0  
libiconv                  1.16                 h2bbff1b_2  
libogg                    1.3.5                h2bbff1b_1  
libpng                    1.6.39               h8cc25b3_0  
libtiff                   4.5.0                h6c2663c_2  
libvorbis                 1.3.7                he774522_0  
libwebp                   1.2.4                hbc33d0d_1  
libwebp-base              1.2.4                h2bbff1b_1  
libxml2                   2.10.3               h0ad7f3c_0  
libxslt                   1.1.37               h2bbff1b_0  
lz4-c                     1.9.4                h2bbff1b_0  
markdown                  3.4.3                    pypi_0    pypi
markupsafe                2.1.2                    pypi_0    pypi
matplotlib                3.5.3            py39haa95532_0  
matplotlib-base           3.5.3            py39hd77b12b_0  
mkl                       2023.1.0         h8bd8f75_46356  
mkl-service               2.4.0            py39h2bbff1b_1  
mkl_fft                   1.3.6            py39hf11a4ad_1  
mkl_random                1.2.2            py39hf11a4ad_1  
munkres                   1.1.4                      py_0  
numexpr                   2.8.4            py39h7b80656_1  
numpy                     1.23.5           py39h6917f2d_1  
numpy-base                1.23.5           py39h46c4fa8_1  
nvidia-ml-py              11.525.112               pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python             4.7.0.72                 pypi_0    pypi
openssl                   1.1.1t               h2bbff1b_0  
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 23.0             py39haa95532_0  
pcre                      8.45                 hd77b12b_0  
pillow                    9.4.0            py39hd77b12b_0  
pip                       23.0.1           py39haa95532_0  
ply                       3.11             py39haa95532_0  
protobuf                  3.19.6                   pypi_0    pypi
psutil                    5.9.0            py39h2bbff1b_0  
pyasn1                    0.5.0                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pyparsing                 3.0.9            py39haa95532_0  
pyqt                      5.15.7           py39hd77b12b_0  
pyqt5-sip                 12.11.0          py39hd77b12b_0  
python                    3.9.16               h6244533_2  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python_abi                3.9                      2_cp39    conda-forge
pywin32                   305              py39h2bbff1b_0  
pywinpty                  2.0.2            py39h5da7b33_0  
qt-main                   5.15.2               he8e5bd7_8  
qt-webengine              5.15.9               hb9a9bb5_5  
qtwebkit                  5.212                h2bbfb41_5  
requests                  2.30.0                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-learn              1.2.2            py39hd77b12b_0  
scipy                     1.9.3            py39hdcfc7df_2  
setuptools                66.0.0           py39haa95532_0  
sip                       6.6.2            py39hd77b12b_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.41.2               h2bbff1b_0  
tbb                       2021.8.0             h59b6b97_0  
tensorboard               2.10.1                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow-estimator      2.10.0                   pypi_0    pypi
tensorflow-gpu            2.10.1                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.31.0                   pypi_0    pypi
tensorflow-probability    0.14.0             pyhd3eb1b0_0  
termcolor                 2.3.0                    pypi_0    pypi
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
tornado                   6.2              py39h2bbff1b_0  
tqdm                      4.65.0           py39hd4e2768_0  
typing-extensions         4.5.0            py39haa95532_0  
typing_extensions         4.5.0            py39haa95532_0  
tzdata                    2023c                h04d1e81_0  
urllib3                   2.0.2                    pypi_0    pypi
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
werkzeug                  2.3.4                    pypi_0    pypi
wheel                     0.38.4           py39haa95532_0  
winpty                    0.4.3                         4  
wrapt                     1.15.0                   pypi_0    pypi
xz                        5.4.2                h8cc25b3_0  
zipp                      3.15.0                   pypi_0    pypi
zlib                      1.2.13               h8cc25b3_0  
zstd                      1.5.5                hd43e919_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0
erosion_top:              0.0
erosion_bottom:           0.0
erosion_left:             0.0
erosion_right:            0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
separate_mask:            False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
separate_mask:            False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False
aligner_min_scale:        0.07
aligner_max_scale:        2.0
aligner_distance:         22.5
aligner_roll:             45.0
aligner_features:         True
filter_refeed:            True
save_filtered:            False
realign_refeeds:          True
filter_realign:           True

[align.fan]
batch-size:               64

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
cpu:                      True
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               70
batch-size:               20

[mask.bisenet_fp]
batch-size:               8
cpu:                      False
weights:                  faceswap
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.custom]
batch-size:               8
centering:                face
fill:                     False

[mask.unet_dfl]
batch-size:               32

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

[recognition.vgg_face2]
batch-size:               32
cpu:                      False

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    always
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                face
coverage:                 87.5
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            3e-05
epsilon_exponent:         -3
autoclip:                 False
reflect_padding:          False
allow_growth:             False
mixed_precision:          True
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ms_ssim
loss_function_2:          mae
loss_weight_2:            25
loss_function_3:          lpips_alex
loss_weight_3:            5
loss_function_4:          ffl
loss_weight_4:            100
mask_loss_function:       mse
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.phaze_a]
output_size:              256
shared_fc:                None
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         efficientnet_v2_l
enc_scaling:              60
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          None
bottleneck_size:          512
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1280
fc_max_filters:           1280
fc_dimensions:            8
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      1280
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       resize_images
dec_upscales_in_fc:       0
dec_norm:                 None
dec_min_filters:          160
dec_max_filters:          640
dec_slope_mode:           full
dec_filter_slope:         -0.33
dec_res_blocks:           1
dec_output_kernel:        3
dec_gaussian:             True
dec_skip_last_residual:   False
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
fs_original_use_alt:      False
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
mask_opacity:             30
mask_color:               #ff0000
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4
Last edited by torzdf on Thu May 25, 2023 11:34 am, edited 2 times in total.
User avatar
adam_macchiato
Posts: 16
Joined: Tue Jul 26, 2022 5:26 am
Has thanked: 4 times

Re: suddently error when training

Post by adam_macchiato »

i fix it by model restore already , thnak you :)

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: suddently error when training

Post by torzdf »

Yes, this is a corrupt model file. Restore will fix it

My word is final

Locked