Crash: Unable to synchronously open object (object 'face_in_a' doesn't exist)"

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Post Reply
User avatar
KrkiKhan
Posts: 1
Joined: Tue Dec 12, 2023 2:27 pm

Crash: Unable to synchronously open object (object 'face_in_a' doesn't exist)"

Post by KrkiKhan »

This is what is in the log file. Thank you!

Code: Select all

12/14/2023 03:36:06 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Adding defaults: (filename: original_defaults.py, module_path: plugins.train.trainer, plugin_type: trainer
12/14/2023 03:36:06 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Importing defaults module: plugins.train.trainer.original_defaults
12/14/2023 03:36:06 MainProcess     _training                      config          add_section                    DEBUG    Add section: (title: 'trainer.original', info: 'Original Trainer Options.\nWARNING: The defaults for augmentation will be fine for 99.9% of use cases. Only change them if you absolutely know what you are doing!')
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'preview_images', datatype: '<class 'int'>', default: '14', info: 'Number of sample faces to display for each side in the preview when training.', rounding: '2', min_max: (2, 16), choices: None, gui_radio: False, fixed: True, group: evaluation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'mask_opacity', datatype: '<class 'int'>', default: '30', info: 'The opacity of the mask overlay in the training preview. Lower values are more transparent.', rounding: '2', min_max: (0, 100), choices: None, gui_radio: False, fixed: True, group: evaluation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'mask_color', datatype: '<class 'str'>', default: '#ff0000', info: 'The RGB hex color to use for the mask overlay in the training preview.', rounding: 'None', min_max: None, choices: colorchooser, gui_radio: False, fixed: True, group: evaluation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'zoom_amount', datatype: '<class 'int'>', default: '5', info: 'Percentage amount to randomly zoom each training image in and out.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'rotation_range', datatype: '<class 'int'>', default: '10', info: 'Percentage amount to randomly rotate each training image.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'shift_range', datatype: '<class 'int'>', default: '5', info: 'Percentage amount to randomly shift each training image horizontally and vertically.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'flip_chance', datatype: '<class 'int'>', default: '50', info: 'Percentage chance to randomly flip each training image horizontally.\nNB: This is ignored if the 'no-flip' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: True, group: image augmentation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_lightness', datatype: '<class 'int'>', default: '30', info: 'Percentage amount to randomly alter the lightness of each training image.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: True, group: color augmentation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_ab', datatype: '<class 'int'>', default: '8', info: 'Percentage amount to randomly alter the 'a' and 'b' colors of the L*a*b* color space of each training image.\nNB: This is ignored if the 'no-augment-color' optionis enabled', rounding: '1', min_max: (0, 50), choices: None, gui_radio: False, fixed: True, group: color augmentation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_clahe_chance', datatype: '<class 'int'>', default: '50', info: 'Percentage chance to perform Contrast Limited Adaptive Histogram Equalization on each training image.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: False, group: color augmentation)
12/14/2023 03:36:06 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_clahe_max_size', datatype: '<class 'int'>', default: '4', info: 'The grid size dictates how much Contrast Limited Adaptive Histogram Equalization is performed on any training image selected for clahe. Contrast will be applied randomly with a gridsize of 0 up to the maximum. This value is a multiplier calculated from the training image size.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (1, 8), choices: None, gui_radio: False, fixed: True, group: color augmentation)
12/14/2023 03:36:06 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Added defaults: trainer.original
12/14/2023 03:36:06 MainProcess     _training                      config          _handle_config                 DEBUG    Handling config: (section: model.original, configfile: 'C:\Users\Krstan\faceswap\config\train.ini')
12/14/2023 03:36:06 MainProcess     _training                      config          _check_exists                  DEBUG    Config file exists: 'C:\Users\Krstan\faceswap\config\train.ini'
12/14/2023 03:36:06 MainProcess     _training                      config          _load_config                   VERBOSE  Loading config: 'C:\Users\Krstan\faceswap\config\train.ini'
12/14/2023 03:36:06 MainProcess     _training                      config          _validate_config               DEBUG    Validating config
12/14/2023 03:36:06 MainProcess     _training                      config          _check_config_change           DEBUG    Default config has not changed
12/14/2023 03:36:06 MainProcess     _training                      config          _check_config_choices          DEBUG    Checking config choices
12/14/2023 03:36:06 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'keras_encoder' to list ['keras_encoder'] for section 'model.phaze_a', option 'freeze_layers'
12/14/2023 03:36:06 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'encoder' to list ['encoder'] for section 'model.phaze_a', option 'load_layers'
12/14/2023 03:36:06 MainProcess     _training                      config          _check_config_choices          DEBUG    Checked config choices
12/14/2023 03:36:06 MainProcess     _training                      config          _validate_config               DEBUG    Validated config
12/14/2023 03:36:06 MainProcess     _training                      config          _handle_config                 DEBUG    Handled config
12/14/2023 03:36:06 MainProcess     _training                      config          __init__                       DEBUG    Initialized: Config
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'learning_rate')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 5e-05)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'epsilon_exponent')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: -7)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'save_optimizer')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: exit)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'autoclip')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'allow_growth')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'mixed_precision')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'nan_protection')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: True)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'convert_batchsize')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 16)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: ssim)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_2')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mse)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_2')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 100)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_3')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: None)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_3')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 0)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_4')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: None)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_4')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 0)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mask_loss_function')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mse)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'eye_multiplier')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 3)
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mouth_multiplier')
12/14/2023 03:36:06 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 2)
12/14/2023 03:36:06 MainProcess     _training                      config          changeable_items               DEBUG    Alterable for existing models: {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}
12/14/2023 03:36:06 MainProcess     _training                      model           __init__                       DEBUG    Initializing State: (model_dir: 'D:\Krstan\Media Files\Kino\Face\Katheryn Winnick\ModelKS', model_name: 'original', config_changeable_items: '{'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}', no_logs: False
12/14/2023 03:36:06 MainProcess     _training                      serializer      get_serializer                 DEBUG    <lib.serializer._JSONSerializer object at 0x000002B2AAC06E30>
12/14/2023 03:36:06 MainProcess     _training                      model           _load                          DEBUG    Loading State
12/14/2023 03:36:06 MainProcess     _training                      serializer      load                           DEBUG    filename: D:\Krstan\Media Files\Kino\Face\Katheryn Winnick\ModelKS\original_state.json
12/14/2023 03:36:06 MainProcess     _training                      serializer      load                           DEBUG    stored data type: <class 'bytes'>
12/14/2023 03:36:06 MainProcess     _training                      serializer      unmarshal                      DEBUG    data type: <class 'bytes'>
12/14/2023 03:36:06 MainProcess     _training                      serializer      unmarshal                      DEBUG    returned data type: <class 'dict'>
12/14/2023 03:36:06 MainProcess     _training                      serializer      load                           DEBUG    data type: <class 'dict'>
12/14/2023 03:36:06 MainProcess     _training                      model           _load                          DEBUG    Loaded state: {'name': 'original', 'sessions': {'1': {'timestamp': 1701781839.726914, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 10000, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '2': {'timestamp': 1701809681.0274732, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 27000, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '3': {'timestamp': 1701817739.0616052, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 16500, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '4': {'timestamp': 1701822397.273921, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 26750, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '5': {'timestamp': 1701870494.6018848, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 32250, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '6': {'timestamp': 1701902365.8886678, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 39250, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '7': {'timestamp': 1701912969.6696975, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 20250, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '8': {'timestamp': 1701919121.3341663, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 54000, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '9': {'timestamp': 1701956835.932841, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 96000, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '10': {'timestamp': 1701990948.3299236, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 1, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '11': {'timestamp': 1701991109.323849, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 47250, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '12': {'timestamp': 1702005754.57795, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 12250, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '13': {'timestamp': 1702043353.591677, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 16250, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '14': {'timestamp': 1702067942.4673264, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 51000, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '15': {'timestamp': 1702096221.8046615, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 40500, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '16': {'timestamp': 1702108030.8833964, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 10000, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '17': {'timestamp': 1702142844.502077, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 1750, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '18': {'timestamp': 1702229308.6918175, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 7500, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '19': {'timestamp': 1702237227.172034, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 5500, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '20': {'timestamp': 1702239172.8819082, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 22500, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '21': {'timestamp': 1702296169.5863414, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 107250, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}, '22': {'timestamp': 1702329464.912981, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 64, 'iterations': 12250, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}}, 'lowest_avg_loss': {'a': 0.008340940507128835, 'b': 0.007072083059698343}, 'iterations': 656001, 'mixed_precision_layers': ['conv_128_0_conv2d', 'conv_128_0_leakyrelu', 'conv_256_0_conv2d', 'conv_256_0_leakyrelu', 'conv_512_0_conv2d', 'conv_512_0_leakyrelu', 'conv_1024_0_conv2d', 'conv_1024_0_leakyrelu', 'flatten', 'dense', 'dense_1', 'reshape', 'upscale_512_0_conv2d_conv2d', 'upscale_512_0_conv2d_leakyrelu', 'upscale_512_0_pixelshuffler', 'upscale_256_0_conv2d_conv2d', 'upscale_256_0_conv2d_leakyrelu', 'upscale_256_0_pixelshuffler', 'upscale_128_0_conv2d_conv2d', 'upscale_128_0_conv2d_leakyrelu', 'upscale_128_0_pixelshuffler', 'upscale_64_0_conv2d_conv2d', 'upscale_64_0_conv2d_leakyrelu', 'upscale_64_0_pixelshuffler', 'face_out_a_0_conv2d', 'upscale_256_1_conv2d_conv2d', 'upscale_256_1_conv2d_leakyrelu', 'upscale_256_1_pixelshuffler', 'upscale_128_1_conv2d_conv2d', 'upscale_128_1_conv2d_leakyrelu', 'upscale_128_1_pixelshuffler', 'upscale_64_1_conv2d_conv2d', 'upscale_64_1_conv2d_leakyrelu', 'upscale_64_1_pixelshuffler', 'face_out_b_0_conv2d'], 'config': {'centering': 'face', 'coverage': 87.5, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'lr_finder_iterations': 1000, 'lr_finder_mode': 'set', 'lr_finder_strength': 'default', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'lowmem': False}}
12/14/2023 03:36:06 MainProcess     _training                      model           _update_legacy_config          DEBUG    Checking for legacy state file update
12/14/2023 03:36:06 MainProcess     _training                      model           _update_legacy_config          DEBUG    Legacy item 'dssim_loss' not in config. Skipping update
12/14/2023 03:36:06 MainProcess     _training                      model           _update_legacy_config          DEBUG    Legacy item 'l2_reg_term' not in config. Skipping update
12/14/2023 03:36:06 MainProcess     _training                      model           _update_legacy_config          DEBUG    Legacy item 'clipnorm' not in config. Skipping update
12/14/2023 03:36:06 MainProcess     _training                      model           _update_legacy_config          DEBUG    State file updated for legacy config: False
12/14/2023 03:36:06 MainProcess     _training                      model           _replace_config                DEBUG    Replacing config. Old config: {'centering': 'face', 'coverage': 87.5, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'lr_finder_iterations': 1000, 'lr_finder_mode': 'set', 'lr_finder_strength': 'default', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'lowmem': False}
12/14/2023 03:36:06 MainProcess     _training                      model           _replace_config                DEBUG    Replaced config. New config: {'centering': 'face', 'coverage': 87.5, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'lr_finder_iterations': 1000, 'lr_finder_mode': 'set', 'lr_finder_strength': 'default', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'lowmem': False}
12/14/2023 03:36:06 MainProcess     _training                      model           _replace_config                INFO     Using configuration saved in state file
12/14/2023 03:36:06 MainProcess     _training                      model           _new_session_id                DEBUG    23
12/14/2023 03:36:06 MainProcess     _training                      model           _create_new_session            DEBUG    Creating new session. id: 23
12/14/2023 03:36:06 MainProcess     _training                      model           __init__                       DEBUG    Initialized State:
12/14/2023 03:36:06 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Settings: (arguments: Namespace(func=<bound method ScriptExecutor.execute_script of <lib.cli.launcher.ScriptExecutor object at 0x000002B29EE5BA90>>, exclude_gpus=None, configfile=None, loglevel='INFO', logfile=None, redirect_gui=True, colab=False, input_a='D:\\Krstan\\Media Files\\Kino\\Face\\Katheryn Winnick\\Star', input_b='D:\\Krstan\\Media Files\\Kino\\Face\\Katheryn Winnick\\Katheryn', model_dir='D:\\Krstan\\Media Files\\Kino\\Face\\Katheryn Winnick\\ModelKS', load_weights=None, trainer='original', summary=False, freeze_weights=False, batch_size=64, iterations=1000000, distribution_strategy='default', no_logs=False, use_lr_finder=False, save_interval=250, snapshot_interval=25000, timelapse_input_a='D:\\Krstan\\Media Files\\Kino\\Face\\Katheryn Winnick\\Star', timelapse_input_b='D:\\Krstan\\Media Files\\Kino\\Face\\Katheryn Winnick\\Katheryn', timelapse_output='D:\\Krstan\\Media Files\\Kino\\Face\\Katheryn Winnick\\Timelapse', preview=True, write_image=False, warp_to_landmarks=False, no_flip=False, no_augment_color=False, no_warp=False), mixed_precision: False, allow_growth: False, is_predict: False)
12/14/2023 03:36:06 MainProcess     _training                      settings        _set_tf_settings               DEBUG    Not setting any specific Tensorflow settings
12/14/2023 03:36:06 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    use_mixed_precision: False
12/14/2023 03:36:06 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    Disabling mixed precision. (Compute dtype: float32, variable_dtype: float32)
12/14/2023 03:36:06 MainProcess     _training                      settings        _get_strategy                  DEBUG    Using strategy: <tensorflow.python.distribute.distribute_lib._DefaultDistributionStrategy object at 0x000002B2AAD39DB0>
12/14/2023 03:36:06 MainProcess     _training                      settings        __init__                       DEBUG    Initialized Settings
12/14/2023 03:36:06 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Loss: (color_order: bgr)
12/14/2023 03:36:06 MainProcess     _training                      settings        _get_mask_channels             DEBUG    uses_masks: (True, True, True), mask_channels: [3, 4, 5]
12/14/2023 03:36:06 MainProcess     _training                      settings        __init__                       DEBUG    Initialized: Loss
12/14/2023 03:36:06 MainProcess     _training                      model           __init__                       DEBUG    Initialized ModelBase (Model)
12/14/2023 03:36:06 MainProcess     _training                      settings        strategy_scope                 DEBUG    Using strategy scope: <tensorflow.python.distribute.distribute_lib._DefaultDistributionContext object at 0x000002B2AAA131C0>
12/14/2023 03:36:06 MainProcess     _training                      io              load                           DEBUG    Loading model: D:\Krstan\Media Files\Kino\Face\Katheryn Winnick\ModelKS\original.h5
12/14/2023 03:36:06 MainProcess     _training                      attrs           __getitem__                    DEBUG    Creating converter from 3 to 5
12/14/2023 03:36:06 MainProcess     preview                        preview_cv      _launch                        DEBUG    Waiting for preview image
12/14/2023 03:36:07 MainProcess     _training                      multithreading  run                            DEBUG    Error in thread (_training): "Unable to synchronously open object (object 'face_in_a' doesn't exist)"
12/14/2023 03:36:07 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training'
Traceback (most recent call last):
  File "C:\Users\Krstan\faceswap\lib\cli\launcher.py", line 225, in execute_script
    process.process()
  File "C:\Users\Krstan\faceswap\scripts\train.py", line 209, in process
    self._end_thread(thread, err)
  File "C:\Users\Krstan\faceswap\scripts\train.py", line 249, in _end_thread
    thread.join()
  File "C:\Users\Krstan\faceswap\lib\multithreading.py", line 224, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\Krstan\faceswap\lib\multithreading.py", line 100, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Krstan\faceswap\scripts\train.py", line 274, in _training
    raise err
  File "C:\Users\Krstan\faceswap\scripts\train.py", line 259, in _training
    model = self._load_model()
  File "C:\Users\Krstan\faceswap\scripts\train.py", line 290, in _load_model
    model.build()
  File "C:\Users\Krstan\faceswap\plugins\train\model\_base\model.py", line 255, in build
    model = self.io.load()
  File "C:\Users\Krstan\faceswap\plugins\train\model\_base\io.py", line 165, in load
    raise err
  File "C:\Users\Krstan\faceswap\plugins\train\model\_base\io.py", line 147, in load
    model = kmodels.load_model(self.filename, compile=False)
  File "C:\Users\Krstan\MiniConda3\envs\faceswap\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "C:\Users\Krstan\MiniConda3\envs\faceswap\lib\site-packages\h5py\_hl\group.py", line 357, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5o.pyx", line 189, in h5py.h5o.open
KeyError: "Unable to synchronously open object (object 'face_in_a' doesn't exist)"

============ System Information ============
backend:             nvidia
encoding:            cp1252
git_branch:          master
git_commits:         a62a85c bugfix: setup.py - Force pip for imagio-ffmpeg
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce RTX 4060 Laptop GPU
gpu_devices_active:  GPU_0
gpu_driver:          546.17
gpu_vram:            GPU_0: 8188MB (2299MB free)
os_machine:          AMD64
os_platform:         Windows-10-10.0.22621-SP0
os_release:          10
py_command:          C:\Users\Krstan\faceswap\faceswap.py train -A D:/Krstan/Media Files/Kino/Face/Katheryn Winnick/Star -B D:/Krstan/Media Files/Kino/Face/Katheryn Winnick/Katheryn -m D:/Krstan/Media Files/Kino/Face/Katheryn Winnick/ModelKS -t original -bs 64 -it 1000000 -D default -s 250 -ss 25000 -tia D:/Krstan/Media Files/Kino/Face/Katheryn Winnick/Star -tib D:/Krstan/Media Files/Kino/Face/Katheryn Winnick/Katheryn -to D:/Krstan/Media Files/Kino/Face/Katheryn Winnick/Timelapse -p -L INFO -gui
py_conda_version:    conda 23.10.0
py_implementation:   CPython
py_version:          3.10.13
py_virtual_env:      True
sys_cores:           16
sys_processor:       Intel64 Family 6 Model 186 Stepping 2, GenuineIntel
sys_ram:             Total: 32471MB, Available: 20438MB, Used: 12032MB, Free: 20438MB

=============== Pip Packages ===============
absl-py==2.0.0
astunparse==1.6.3
cachetools==5.3.2
certifi==2023.11.17
charset-normalizer==3.3.2
colorama @ file:///C:/b/abs_a9ozq0l032/croot/colorama_1672387194846/work
contourpy @ file:///C:/b/abs_853rfy8zse/croot/contourpy_1700583617587/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
fastcluster @ file:///D:/bld/fastcluster_1695650232190/work
ffmpy @ file:///home/conda/feedstock_root/build_artifacts/ffmpy_1659474992694/work
flatbuffers==23.5.26
fonttools==4.25.0
gast==0.4.0
google-auth==2.23.4
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.59.3
h5py==3.10.0
idna==3.4
imageio @ file:///C:/b/abs_3eijmwdodc/croot/imageio_1695996500830/work
imageio-ffmpeg==0.4.9
joblib @ file:///C:/b/abs_1anqjntpan/croot/joblib_1685113317150/work
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/b/abs_88mdhvtahm/croot/kiwisolver_1672387921783/work
libclang==16.0.6
Markdown==3.5.1
MarkupSafe==2.1.3
matplotlib @ file:///C:/b/abs_e26vnvd5s1/croot/matplotlib-suite_1698692153288/work
mkl-fft @ file:///C:/b/abs_19i1y8ykas/croot/mkl_fft_1695058226480/work
mkl-random @ file:///C:/b/abs_edwkj1_o69/croot/mkl_random_1695059866750/work
mkl-service==2.4.0
munkres==1.1.4
numexpr @ file:///C:/b/abs_5fucrty5dc/croot/numexpr_1696515448831/work
numpy @ file:///C:/b/abs_9fu2cs2527/croot/numpy_and_numpy_base_1695830496596/work/dist/numpy-1.26.0-cp310-cp310-win_amd64.whl#sha256=11367989d61b64039738e0c68c95c6b797a41c4c75ec2147c0541b21163786eb
nvidia-ml-py @ file:///home/conda/feedstock_root/build_artifacts/nvidia-ml-py_1698947663801/work
oauthlib==3.2.2
opencv-python==4.8.1.78
opt-einsum==3.3.0
packaging @ file:///C:/b/abs_28t5mcoltc/croot/packaging_1693575224052/work
Pillow @ file:///C:/b/abs_153xikw91n/croot/pillow_1695134603563/work
ply==3.11
protobuf==3.19.6
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
pyasn1==0.5.1
pyasn1-modules==0.3.0
pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work
PyQt5==5.15.10
PyQt5-sip @ file:///C:/b/abs_c0pi2mimq3/croot/pyqt-split_1698769125270/work/pyqt_sip
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==305.1
pywinpty @ file:///C:/ci_310/pywinpty_1644230983541/work/target/wheels/pywinpty-2.0.2-cp310-none-win_amd64.whl
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn @ file:///C:/b/abs_55olq_4gzc/croot/scikit-learn_1690978955123/work
scipy==1.11.3
sip @ file:///C:/b/abs_edevan3fce/croot/sip_1698675983372/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.1
tensorflow-estimator==2.10.0
tensorflow-io-gcs-filesystem==0.31.0
termcolor==2.3.0
threadpoolctl @ file:///Users/ktietz/demo/mc3/conda-bld/threadpoolctl_1629802263681/work
tomli @ file:///C:/Windows/TEMP/abs_ac109f85-a7b3-4b4d-bcfd-52622eceddf0hy332ojo/croots/recipe/tomli_1657175513137/work
tornado @ file:///C:/b/abs_0cbrstidzg/croot/tornado_1696937003724/work
tqdm @ file:///C:/b/abs_f76j9hg7pv/croot/tqdm_1679561871187/work
typing_extensions==4.8.0
urllib3==2.1.0
Werkzeug==3.0.1
wrapt==1.16.0

============== Conda Packages ==============
# packages in environment at C:\Users\Krstan\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   2.0.0                    pypi_0    pypi
aom                       3.7.1                h63175ca_0    conda-forge
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
brotli                    1.0.9                h2bbff1b_7  
brotli-bin                1.0.9                h2bbff1b_7  
bzip2                     1.0.8                he774522_0  
ca-certificates           2023.11.17           h56e8100_0    conda-forge
cachetools                5.3.2                    pypi_0    pypi
certifi                   2023.11.17               pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
colorama                  0.4.6           py310haa95532_0  
contourpy                 1.2.0           py310h59b6b97_0  
cudatoolkit               11.8.0               hd77b12b_0  
cudnn                     8.9.2.26               cuda11_0  
cycler                    0.11.0             pyhd3eb1b0_0  
dav1d                     1.2.1                hcfcfb64_0    conda-forge
expat                     2.5.0                h63175ca_1    conda-forge
fastcluster               1.2.6           py310hecd3228_3    conda-forge
ffmpeg                    6.1.0           gpl_h2616e31_101    conda-forge
ffmpy                     0.3.0              pyhb6f538c_0    conda-forge
flatbuffers               23.5.26                  pypi_0    pypi
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               hbde0cde_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.12.1               ha860e81_0  
gast                      0.4.0                    pypi_0    pypi
giflib                    5.2.1                h8cc25b3_3  
git                       2.40.1               haa95532_1  
google-auth               2.23.4                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.59.3                   pypi_0    pypi
h5py                      3.10.0                   pypi_0    pypi
icc_rt                    2022.1.0             h6049295_2  
icu                       73.1                 h6c2663c_0  
idna                      3.4                      pypi_0    pypi
imageio                   2.31.4          py310haa95532_0  
imageio-ffmpeg            0.4.9                    pypi_0    pypi
intel-openmp              2023.1.0         h59b6b97_46320  
joblib                    1.2.0           py310haa95532_0  
jpeg                      9e                   h2bbff1b_1  
keras                     2.10.0                   pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.4           py310hd77b12b_0  
krb5                      1.20.1               h5b6d351_0  
lerc                      3.0                  hd77b12b_0  
libbrotlicommon           1.0.9                h2bbff1b_7  
libbrotlidec              1.0.9                h2bbff1b_7  
libbrotlienc              1.0.9                h2bbff1b_7  
libclang                  16.0.6                   pypi_0    pypi
libclang13                14.0.6          default_h8e68704_1  
libdeflate                1.17                 h2bbff1b_1  
libexpat                  2.5.0                h63175ca_1    conda-forge
libffi                    3.4.4                hd77b12b_0  
libiconv                  1.17                 h8ffe710_0    conda-forge
libopus                   1.3.1                h8ffe710_1    conda-forge
libpng                    1.6.39               h8cc25b3_0  
libpq                     12.15                h906ac69_1  
libtiff                   4.5.1                hd77b12b_0  
libwebp                   1.3.2                hbc33d0d_0  
libwebp-base              1.3.2                h2bbff1b_0  
libxml2                   2.11.6               hc3477c8_0    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
libzlib-wapi              1.2.13               hcfcfb64_5    conda-forge
lz4-c                     1.9.4                h2bbff1b_0  
markdown                  3.5.1                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.8.0           py310haa95532_0  
matplotlib-base           3.8.0           py310h4ed8f06_0  
mkl                       2023.1.0         h6b88ed4_46358  
mkl-service               2.4.0           py310h2bbff1b_1  
mkl_fft                   1.3.8           py310h2bbff1b_0  
mkl_random                1.2.4           py310h59b6b97_0  
munkres                   1.1.4                      py_0  
numexpr                   2.8.7           py310h2cd9be0_0  
numpy                     1.26.0          py310h055cbcc_0  
numpy-base                1.26.0          py310h65a83cf_0  
nvidia-ml-py              12.535.133         pyhd8ed1ab_0    conda-forge
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python             4.8.1.78                 pypi_0    pypi
openh264                  2.3.1                h63175ca_2    conda-forge
openssl                   3.1.4                hcfcfb64_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 23.1            py310haa95532_0  
pillow                    9.4.0           py310hd77b12b_1  
pip                       23.3.1          py310haa95532_0  
ply                       3.11            py310haa95532_0  
protobuf                  3.19.6                   pypi_0    pypi
psutil                    5.9.0           py310h2bbff1b_0  
pyasn1                    0.5.1                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pyparsing                 3.0.9           py310haa95532_0  
pyqt                      5.15.10         py310hd77b12b_0  
pyqt5-sip                 12.13.0         py310h2bbff1b_0  
python                    3.10.13              he1021f5_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python_abi                3.10                    2_cp310    conda-forge
pywin32                   305             py310h2bbff1b_0  
pywinpty                  2.0.2           py310h5da7b33_0  
qt-main                   5.15.2              h19c9488_10  
requests                  2.31.0                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-learn              1.3.0           py310h4ed8f06_0  
scipy                     1.11.3          py310h309d312_0  
setuptools                68.0.0          py310haa95532_0  
sip                       6.7.12          py310hd77b12b_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.41.2               h2bbff1b_0  
svt-av1                   1.7.0                h63175ca_0    conda-forge
tbb                       2021.8.0             h59b6b97_0  
tensorboard               2.10.1                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.10.1                   pypi_0    pypi
tensorflow-estimator      2.10.0                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.31.0                   pypi_0    pypi
termcolor                 2.3.0                    pypi_0    pypi
threadpoolctl             2.2.0              pyh0d69192_0  
tk                        8.6.12               h2bbff1b_0  
tomli                     2.0.1           py310haa95532_0  
tornado                   6.3.3           py310h2bbff1b_0  
tqdm                      4.65.0          py310h9909e9c_0  
typing-extensions         4.8.0                    pypi_0    pypi
tzdata                    2023c                h04d1e81_0  
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
urllib3                   2.1.0                    pypi_0    pypi
vc                        14.2                 h21ff451_1  
vc14_runtime              14.36.32532         hdcecf7f_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
werkzeug                  3.0.1                    pypi_0    pypi
wheel                     0.41.2          py310haa95532_0  
winpty                    0.4.3                         4  
wrapt                     1.16.0                   pypi_0    pypi
x264                      1!164.3095           h8ffe710_2    conda-forge
x265                      3.5                  h2d74725_3    conda-forge
xz                        5.4.2                h8cc25b3_0  
zlib                      1.2.13               hcfcfb64_5    conda-forge
zlib-wapi                 1.2.13               hcfcfb64_5    conda-forge
zstd                      1.5.5                hd43e919_0  

=============== State File =================
{
  "name": "original",
  "sessions": {
    "1": {
      "timestamp": 1701781839.726914,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 10000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "2": {
      "timestamp": 1701809681.0274732,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 27000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "3": {
      "timestamp": 1701817739.0616052,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 16500,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "4": {
      "timestamp": 1701822397.273921,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 26750,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "5": {
      "timestamp": 1701870494.6018848,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 32250,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "6": {
      "timestamp": 1701902365.8886678,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 39250,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "7": {
      "timestamp": 1701912969.6696975,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 20250,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "8": {
      "timestamp": 1701919121.3341663,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 54000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "9": {
      "timestamp": 1701956835.932841,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 96000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "10": {
      "timestamp": 1701990948.3299236,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 1,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "11": {
      "timestamp": 1701991109.323849,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 47250,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "12": {
      "timestamp": 1702005754.57795,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 12250,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "13": {
      "timestamp": 1702043353.591677,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 16250,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "14": {
      "timestamp": 1702067942.4673264,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 51000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "15": {
      "timestamp": 1702096221.8046615,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 40500,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "16": {
      "timestamp": 1702108030.8833964,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 10000,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "17": {
      "timestamp": 1702142844.502077,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 1750,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "18": {
      "timestamp": 1702229308.6918175,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 7500,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "19": {
      "timestamp": 1702237227.172034,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 5500,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "20": {
      "timestamp": 1702239172.8819082,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 22500,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "21": {
      "timestamp": 1702296169.5863414,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 107250,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "22": {
      "timestamp": 1702329464.912981,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 64,
      "iterations": 12250,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "save_optimizer": "exit",
        "autoclip": false,
        "allow_growth": false,
        "mixed_precision": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    }
  },
  "lowest_avg_loss": {
    "a": 0.008340940507128835,
    "b": 0.007072083059698343
  },
  "iterations": 656001,
  "mixed_precision_layers": [
    "conv_128_0_conv2d",
    "conv_128_0_leakyrelu",
    "conv_256_0_conv2d",
    "conv_256_0_leakyrelu",
    "conv_512_0_conv2d",
    "conv_512_0_leakyrelu",
    "conv_1024_0_conv2d",
    "conv_1024_0_leakyrelu",
    "flatten",
    "dense",
    "dense_1",
    "reshape",
    "upscale_512_0_conv2d_conv2d",
    "upscale_512_0_conv2d_leakyrelu",
    "upscale_512_0_pixelshuffler",
    "upscale_256_0_conv2d_conv2d",
    "upscale_256_0_conv2d_leakyrelu",
    "upscale_256_0_pixelshuffler",
    "upscale_128_0_conv2d_conv2d",
    "upscale_128_0_conv2d_leakyrelu",
    "upscale_128_0_pixelshuffler",
    "upscale_64_0_conv2d_conv2d",
    "upscale_64_0_conv2d_leakyrelu",
    "upscale_64_0_pixelshuffler",
    "face_out_a_0_conv2d",
    "upscale_256_1_conv2d_conv2d",
    "upscale_256_1_conv2d_leakyrelu",
    "upscale_256_1_pixelshuffler",
    "upscale_128_1_conv2d_conv2d",
    "upscale_128_1_conv2d_leakyrelu",
    "upscale_128_1_pixelshuffler",
    "upscale_64_1_conv2d_conv2d",
    "upscale_64_1_conv2d_leakyrelu",
    "upscale_64_1_pixelshuffler",
    "face_out_b_0_conv2d"
  ],
  "config": {
    "centering": "face",
    "coverage": 87.5,
    "optimizer": "adam",
    "learning_rate": 5e-05,
    "epsilon_exponent": -7,
    "save_optimizer": "exit",
    "lr_finder_iterations": 1000,
    "lr_finder_mode": "set",
    "lr_finder_strength": "default",
    "autoclip": false,
    "allow_growth": false,
    "mixed_precision": false,
    "nan_protection": true,
    "convert_batchsize": 16,
    "loss_function": "ssim",
    "loss_function_2": "mse",
    "loss_weight_2": 100,
    "loss_function_3": null,
    "loss_weight_3": 0,
    "loss_function_4": null,
    "loss_weight_4": 0,
    "mask_loss_function": "mse",
    "eye_multiplier": 3,
    "mouth_multiplier": 2,
    "penalized_mask_loss": true,
    "mask_type": "extended",
    "mask_blur_kernel": 3,
    "mask_threshold": 4,
    "learn_mask": false,
    "lowmem": false
  }
}

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0
erosion_top:              0.0
erosion_bottom:           0.0
erosion_left:             0.0
erosion_right:            0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
separate_mask:            False
jpg_quality:              75
png_compress_level:       3

[writer.patch]
start_index:              0
index_offset:             0
number_padding:           6
include_filename:         True
face_index_location:      before
origin:                   bottom-left
empty_frames:             blank
json_output:              False
separate_mask:            False
bit_depth:                16
format:                   png
png_compress_level:       3
tiff_compression_method:  lzw

[writer.pillow]
format:                   png
draw_transparent:         False
separate_mask:            False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False
aligner_min_scale:        0.07
aligner_max_scale:        2.0
aligner_distance:         22.5
aligner_roll:             45.0
aligner_features:         True
filter_refeed:            True
save_filtered:            False
realign_refeeds:          True
filter_realign:           True

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
cpu:                      True
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.bisenet_fp]
batch-size:               8
cpu:                      False
weights:                  faceswap
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.custom]
batch-size:               8
centering:                face
fill:                     False

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

[recognition.vgg_face2]
batch-size:               16
cpu:                      False

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                face
coverage:                 87.5
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
save_optimizer:           exit
lr_finder_iterations:     1000
lr_finder_mode:           set
lr_finder_strength:       default
autoclip:                 False
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
loss_function_2:          mse
loss_weight_2:            100
loss_function_3:          none
loss_weight_3:            0
loss_function_4:          none
loss_weight_4:            0
mask_loss_function:       mse
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.phaze_a]
output_size:              128
shared_fc:                none
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              7
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          none
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_upscales_in_fc:       0
dec_norm:                 none
dec_min_filters:          64
dec_max_filters:          512
dec_slope_mode:           full
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
fs_original_use_alt:      False
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
mask_opacity:             30
mask_color:               #ff0000
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4
Last edited by torzdf on Thu Dec 14, 2023 12:46 pm, edited 1 time in total.
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Crash: Unable to synchronously open object (object 'face_in_a' doesn't exist)"

Post by torzdf »

This looks like it might be model corruption.

See here for restoring:
viewtopic.php?p=8361#p8361

My word is final

Post Reply