I stopped training to do some other stuff on my computer, finished my things, started to train again and it wouldn't work. I didnt mess with any settings.
08/20/2020 20:19:14 CRITICAL Error caught! Exiting...
08/20/2020 20:19:14 ERROR Caught exception in thread: '_training_0'
08/20/2020 20:19:20 ERROR Got Exception on main handler:
OSError: Unable to open file (bad object header version number)
In the process of trying to fix it, I uninstalled an reinstalled, Also Im pretty sure I successfully installed tensorflow 2.3 manually. but neither of those fixed my problems
Code: Select all
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Importing defaults module: plugins.train.model.original_defaults
08/20/2020 20:19:12 MainProcess _training_0 config add_section DEBUG Add section: (title: 'model.original', info: 'Original Faceswap Model.\nNB: Unless specifically stated, values changed here will only take effect when creating a new model.')
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.original', title: 'lowmem', datatype: '<class 'bool'>', default: 'False', info: 'Lower memory mode. Set to 'True' if having issues with VRAM useage.\nNB: Models with a changed lowmem mode are not compatible with each other.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Added defaults: model.original
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Adding defaults: (filename: realface_defaults.py, module_path: plugins.train.model, plugin_type: model
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Importing defaults module: plugins.train.model.realface_defaults
08/20/2020 20:19:12 MainProcess _training_0 config add_section DEBUG Add section: (title: 'model.realface', info: 'An extra detailed variant of Original model.\nIncorporates ideas from Bryanlyon and inspiration from the Villain model.\nRequires about 6GB-8GB of VRAM (batchsize 8-16).\n\nNB: Unless specifically stated, values changed here will only take effect when creating a new model.')
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.realface', title: 'input_size', datatype: '<class 'int'>', default: '64', info: 'Resolution (in pixels) of the input image to train on.\nBE AWARE Larger resolution will dramatically increase VRAM requirements.\nHigher resolutions may increase prediction accuracy, but does not effect the resulting output size.\nMust be between 64 and 128 and be divisible by 16.', rounding: '16', min_max: (64, 128), choices: [], gui_radio: False, fixed: True, group: size)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.realface', title: 'output_size', datatype: '<class 'int'>', default: '128', info: 'Output image resolution (in pixels).\nBe aware that larger resolution will increase VRAM requirements.\nNB: Must be between 64 and 256 and be divisible by 16.', rounding: '16', min_max: (64, 256), choices: [], gui_radio: False, fixed: True, group: size)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.realface', title: 'dense_nodes', datatype: '<class 'int'>', default: '1536', info: 'Number of nodes for decoder. Might affect your model's ability to learn in general.\nNote that: Lower values will affect the ability to predict details.', rounding: '64', min_max: (768, 2048), choices: [], gui_radio: False, fixed: True, group: network)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.realface', title: 'complexity_encoder', datatype: '<class 'int'>', default: '128', info: 'Encoder Convolution Layer Complexity. sensible ranges: 128 to 150.', rounding: '4', min_max: (96, 160), choices: [], gui_radio: False, fixed: True, group: network)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.realface', title: 'complexity_decoder', datatype: '<class 'int'>', default: '512', info: 'Decoder Complexity.', rounding: '4', min_max: (512, 544), choices: [], gui_radio: False, fixed: True, group: network)
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Added defaults: model.realface
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Adding defaults: (filename: unbalanced_defaults.py, module_path: plugins.train.model, plugin_type: model
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Importing defaults module: plugins.train.model.unbalanced_defaults
08/20/2020 20:19:12 MainProcess _training_0 config add_section DEBUG Add section: (title: 'model.unbalanced', info: 'An unbalanced model with adjustable input size options.\nThis is an unbalanced model so b>a swaps may not work well\n\nNB: Unless specifically stated, values changed here will only take effect when creating a new model.')
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.unbalanced', title: 'input_size', datatype: '<class 'int'>', default: '128', info: 'Resolution (in pixels) of the image to train on.\nBE AWARE Larger resolution will dramatically increaseVRAM requirements.\nMake sure your resolution is divisible by 64 (e.g. 64, 128, 256 etc.).\nNB: Your faceset must be at least 1.6x larger than your required input size.\n(e.g. 160 is the maximum input size for a 256x256 faceset).', rounding: '64', min_max: (64, 512), choices: [], gui_radio: False, fixed: True, group: size)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.unbalanced', title: 'lowmem', datatype: '<class 'bool'>', default: 'False', info: 'Lower memory mode. Set to 'True' if having issues with VRAM useage.\nNB: Models with a changed lowmem mode are not compatible with each other.\nNB: lowmem will override cutom nodes and complexity settings.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.unbalanced', title: 'clipnorm', datatype: '<class 'bool'>', default: 'True', info: 'Controls gradient clipping of the optimizer. Can prevent model corruption at the expense of VRAM.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.unbalanced', title: 'nodes', datatype: '<class 'int'>', default: '1024', info: 'Number of nodes for decoder. Don't change this unless you know what you are doing!', rounding: '64', min_max: (512, 4096), choices: [], gui_radio: False, fixed: True, group: network)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.unbalanced', title: 'complexity_encoder', datatype: '<class 'int'>', default: '128', info: 'Encoder Convolution Layer Complexity. sensible ranges: 128 to 160.', rounding: '16', min_max: (64, 1024), choices: [], gui_radio: False, fixed: True, group: network)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.unbalanced', title: 'complexity_decoder_a', datatype: '<class 'int'>', default: '384', info: 'Decoder A Complexity.', rounding: '16', min_max: (64, 1024), choices: [], gui_radio: False, fixed: True, group: network)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.unbalanced', title: 'complexity_decoder_b', datatype: '<class 'int'>', default: '512', info: 'Decoder B Complexity.', rounding: '16', min_max: (64, 1024), choices: [], gui_radio: False, fixed: True, group: network)
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Added defaults: model.unbalanced
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Adding defaults: (filename: villain_defaults.py, module_path: plugins.train.model, plugin_type: model
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Importing defaults module: plugins.train.model.villain_defaults
08/20/2020 20:19:12 MainProcess _training_0 config add_section DEBUG Add section: (title: 'model.villain', info: 'A Higher resolution version of the Original Model by VillainGuy.\nExtremely VRAM heavy. Don't try to run this if you have a small GPU.\n\nNB: Unless specifically stated, values changed here will only take effect when creating a new model.')
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'model.villain', title: 'lowmem', datatype: '<class 'bool'>', default: 'False', info: 'Lower memory mode. Set to 'True' if having issues with VRAM useage.\nNB: Models with a changed lowmem mode are not compatible with each other.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Added defaults: model.villain
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Adding defaults: (filename: original_defaults.py, module_path: plugins.train.trainer, plugin_type: trainer
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Importing defaults module: plugins.train.trainer.original_defaults
08/20/2020 20:19:12 MainProcess _training_0 config add_section DEBUG Add section: (title: 'trainer.original', info: 'Original Trainer Options.\nWARNING: The defaults for augmentation will be fine for 99.9% of use cases. Only change them if you absolutely know what you are doing!')
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'trainer.original', title: 'preview_images', datatype: '<class 'int'>', default: '14', info: 'Number of sample faces to display for each side in the preview when training.', rounding: '2', min_max: (2, 16), choices: None, gui_radio: False, fixed: True, group: evaluation)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'trainer.original', title: 'zoom_amount', datatype: '<class 'int'>', default: '5', info: 'Percentage amount to randomly zoom each training image in and out.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'trainer.original', title: 'rotation_range', datatype: '<class 'int'>', default: '10', info: 'Percentage amount to randomly rotate each training image.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'trainer.original', title: 'shift_range', datatype: '<class 'int'>', default: '5', info: 'Percentage amount to randomly shift each training image horizontally and vertically.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'trainer.original', title: 'flip_chance', datatype: '<class 'int'>', default: '50', info: 'Percentage chance to randomly flip each training image horizontally.\nNB: This is ignored if the 'no-flip' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: True, group: image augmentation)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'trainer.original', title: 'color_lightness', datatype: '<class 'int'>', default: '30', info: 'Percentage amount to randomly alter the lightness of each training image.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: True, group: color augmentation)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'trainer.original', title: 'color_ab', datatype: '<class 'int'>', default: '8', info: 'Percentage amount to randomly alter the 'a' and 'b' colors of the L*a*b* color space of each training image.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (0, 50), choices: None, gui_radio: False, fixed: True, group: color augmentation)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'trainer.original', title: 'color_clahe_chance', datatype: '<class 'int'>', default: '50', info: 'Percentage chance to perform Contrast Limited Adaptive Histogram Equalization on each training image.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: False, group: color augmentation)
08/20/2020 20:19:12 MainProcess _training_0 config add_item DEBUG Add item: (section: 'trainer.original', title: 'color_clahe_max_size', datatype: '<class 'int'>', default: '4', info: 'The grid size dictates how much Contrast Limited Adaptive Histogram Equalization is performed on any training image selected for clahe. Contrast will be applied randomly with a gridsize of 0 up to the maximum. This value is a multiplier calculated from the training image size.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (1, 8), choices: None, gui_radio: False, fixed: True, group: color augmentation)
08/20/2020 20:19:12 MainProcess _training_0 _config load_module DEBUG Added defaults: trainer.original
08/20/2020 20:19:12 MainProcess _training_0 config handle_config DEBUG Handling config
08/20/2020 20:19:12 MainProcess _training_0 config check_exists DEBUG Config file exists: 'C:\Users\user\faceswap\config\train.ini'
08/20/2020 20:19:12 MainProcess _training_0 config load_config VERBOSE Loading config: 'C:\Users\user\faceswap\config\train.ini'
08/20/2020 20:19:12 MainProcess _training_0 config validate_config DEBUG Validating config
08/20/2020 20:19:12 MainProcess _training_0 config check_config_change DEBUG Default config has not changed
08/20/2020 20:19:12 MainProcess _training_0 config check_config_choices DEBUG Checking config choices
08/20/2020 20:19:12 MainProcess _training_0 config check_config_choices DEBUG Checked config choices
08/20/2020 20:19:12 MainProcess _training_0 config validate_config DEBUG Validated config
08/20/2020 20:19:12 MainProcess _training_0 config handle_config DEBUG Handled config
08/20/2020 20:19:12 MainProcess _training_0 config __init__ DEBUG Initialized: Config
08/20/2020 20:19:12 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'learning_rate')
08/20/2020 20:19:12 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'float'>, value: 5e-05)
08/20/2020 20:19:12 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'allow_growth')
08/20/2020 20:19:12 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'bool'>, value: False)
08/20/2020 20:19:12 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'convert_batchsize')
08/20/2020 20:19:12 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 16)
08/20/2020 20:19:12 MainProcess _training_0 config get DEBUG Getting config item: (section: 'model.dfl_sae', option: 'clipnorm')
08/20/2020 20:19:12 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'bool'>, value: True)
08/20/2020 20:19:12 MainProcess _training_0 config changeable_items DEBUG Alterable for existing models: {'learning_rate': 5e-05, 'allow_growth': False, 'convert_batchsize': 16, 'clipnorm': True}
08/20/2020 20:19:12 MainProcess _training_0 _base __init__ DEBUG Initializing State: (model_dir: 'D:\Nueral Network programs\secndary\dflsae model', model_name: 'dfl_sae', config_changeable_items: '{'learning_rate': 5e-05, 'allow_growth': False, 'convert_batchsize': 16, 'clipnorm': True}', no_logs: False, training_image_size: '208'
08/20/2020 20:19:12 MainProcess _training_0 serializer get_serializer DEBUG <lib.serializer._JSONSerializer object at 0x000001E7A54DDCA0>
08/20/2020 20:19:12 MainProcess _training_0 _base _load DEBUG Loading State
08/20/2020 20:19:12 MainProcess _training_0 serializer load DEBUG filename: D:\Nueral Network programs\secndary\dflsae model\dfl_sae_state.json
08/20/2020 20:19:12 MainProcess _training_0 serializer load DEBUG stored data type: <class 'bytes'>
08/20/2020 20:19:12 MainProcess _training_0 serializer unmarshal DEBUG data type: <class 'bytes'>
08/20/2020 20:19:12 MainProcess _training_0 serializer unmarshal DEBUG returned data type: <class 'dict'>
08/20/2020 20:19:12 MainProcess _training_0 serializer load DEBUG data type: <class 'dict'>
08/20/2020 20:19:12 MainProcess _training_0 _base _load DEBUG Loaded state: {'name': 'dfl_sae', 'sessions': {'1': {'timestamp': 1597856042.5730655, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 1, 'iterations': 319, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': False}}, '2': {'timestamp': 1597856175.0856314, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 2, 'iterations': 323, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': False}}, '3': {'timestamp': 1597856339.583101, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 321, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': False}}, '4': {'timestamp': 1597856681.881946, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 8254, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': False}}, '5': {'timestamp': 1597861556.4950275, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 5, 'iterations': 1, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': False}}, '6': {'timestamp': 1597861645.3876421, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 33900, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': False}}, '7': {'timestamp': 1597882403.9973261, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 21086, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': False}}, '8': {'timestamp': 1597895406.2477026, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 4831, 'config': {'learning_rate': 5e-05, 'allow_growth': True, 'clipnorm': False}}, '9': {'timestamp': 1597898351.2048326, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 3, 'iterations': 400, 'config': {'learning_rate': 5e-05, 'allow_growth': True, 'clipnorm': False}}, '10': {'timestamp': 1597898708.903881, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 2198, 'config': {'learning_rate': 4.9e-05, 'allow_growth': True, 'clipnorm': False}}, '11': {'timestamp': 1597900040.0676365, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 6757, 'config': {'learning_rate': 4.8e-05, 'allow_growth': True, 'clipnorm': False}}, '12': {'timestamp': 1597903865.230849, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 52696, 'config': {'learning_rate': 4.8e-05, 'allow_growth': True, 'clipnorm': True}}, '13': {'timestamp': 1597933328.6587334, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 19323, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': True}}, '14': {'timestamp': 1597944398.3239489, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 4600, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': True}}, '15': {'timestamp': 1597948123.137797, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 19764, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': True}}, '16': {'timestamp': 1597962125.0118682, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 200, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': True}}, '17': {'timestamp': 1597963495.7711692, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 7404, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': True}}, '18': {'timestamp': 1597968266.6555011, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 3615, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': True}}, '19': {'timestamp': 1597970693.0466576, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 4, 'iterations': 7800, 'config': {'learning_rate': 5.1e-05, 'allow_growth': True, 'clipnorm': True}}}, 'lowest_avg_loss': {'a': 0.016461588442325592, 'b': 0.017985491082072258}, 'iterations': 193792, 'training_size': 208, 'config': {'coverage': 82.52, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': False, 'loss_function': 'ssim', 'optimizer': 'adam', 'learning_rate': 5.1e-05, 'allow_growth': True, 'mixed_precision': True, 'input_size': 208, 'clipnorm': True, 'architecture': 'df', 'autoencoder_dims': 0, 'encoder_dims': 30, 'decoder_dims': 18, 'multiscale_decoder': False}}
08/20/2020 20:19:12 MainProcess _training_0 _base _update_legacy_config DEBUG Checking for legacy state file update
08/20/2020 20:19:12 MainProcess _training_0 _base _update_legacy_config DEBUG Legacy item 'dssim_loss' not in config. Skipping update
08/20/2020 20:19:12 MainProcess _training_0 _base _update_legacy_config DEBUG State file updated for legacy config: False
08/20/2020 20:19:12 MainProcess _training_0 _base _replace_config INFO Adding new config item to state file: 'convert_batchsize': '16'
08/20/2020 20:19:12 MainProcess _training_0 _base _update_changed_config_items INFO Config item: 'learning_rate' has been updated from '5.1e-05' to '5e-05'
08/20/2020 20:19:12 MainProcess _training_0 _base _update_changed_config_items INFO Config item: 'allow_growth' has been updated from 'True' to 'False'
08/20/2020 20:19:12 MainProcess _training_0 _base _replace_config DEBUG Replacing config. Old config: {'coverage': 68.75, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True, 'loss_function': 'mae', 'optimizer': 'adam', 'learning_rate': 5e-05, 'allow_growth': False, 'mixed_precision': False, 'convert_batchsize': 16, 'input_size': 128, 'clipnorm': True, 'architecture': 'df', 'autoencoder_dims': 0, 'encoder_dims': 42, 'decoder_dims': 21, 'multiscale_decoder': False}
08/20/2020 20:19:12 MainProcess _training_0 _base _replace_config DEBUG Replaced config. New config: {'coverage': 82.52, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': False, 'loss_function': 'ssim', 'optimizer': 'adam', 'learning_rate': 5e-05, 'allow_growth': False, 'mixed_precision': True, 'input_size': 208, 'clipnorm': True, 'architecture': 'df', 'autoencoder_dims': 0, 'encoder_dims': 30, 'decoder_dims': 18, 'multiscale_decoder': False, 'convert_batchsize': 16}
08/20/2020 20:19:12 MainProcess _training_0 _base _replace_config INFO Using configuration saved in state file
08/20/2020 20:19:12 MainProcess _training_0 _base _new_session_id DEBUG 20
08/20/2020 20:19:12 MainProcess _training_0 _base _create_new_session DEBUG Creating new session. id: 20
08/20/2020 20:19:12 MainProcess _training_0 _base __init__ DEBUG Initialized State:
08/20/2020 20:19:12 MainProcess _training_0 _base __init__ DEBUG Initializing _Settings: (arguments: Namespace(alignments_path_a=None, alignments_path_b=None, batch_size=4, colab=False, configfile=None, distributed=False, exclude_gpus=None, func=<bound method ScriptExecutor.execute_script of <lib.cli.launcher.ScriptExecutor object at 0x000001E78EE64DC0>>, input_a='D:\\Nueral Network programs\\secndary\\third frames 208', input_b='D:\\Nueral Network programs\\secndary\\5400 frames sized 208', iterations=1000000, logfile=None, loglevel='INFO', model_dir='D:\\Nueral Network programs\\secndary\\dflsae model', no_augment_color=False, no_flip=False, no_logs=False, preview=False, preview_scale=50, redirect_gui=True, save_interval=200, snapshot_interval=25000, timelapse_input_a='D:\\Nueral Network programs\\secndary\\third frames 208', timelapse_input_b='D:\\Nueral Network programs\\secndary\\5400 frames sized 208', timelapse_output='D:\\Nueral Network programs\\secndary\\Project VR\\third fourth\\time lapse for dflsae 208', trainer='dfl-sae', warp_to_landmarks=False, write_image=False), mixed_precision: True, allow_growth: False, is_predict: False)
08/20/2020 20:19:12 MainProcess _training_0 _base _set_tf_settings DEBUG Not setting any specific Tensorflow settings
08/20/2020 20:19:12 MainProcess _training_0 _base _set_keras_mixed_precision DEBUG use_mixed_precision: True, skip_check: False
08/20/2020 20:19:12 MainProcess _training_0 _base _set_keras_mixed_precision INFO Enabling Mixed Precision Training.
08/20/2020 20:19:13 MainProcess _training_0 _base _set_keras_mixed_precision DEBUG Enabled mixed precision. (Compute dtype: float16, variable_dtype: float32)
08/20/2020 20:19:13 MainProcess _training_0 _base _get_strategy DEBUG Using strategy: <tensorflow.python.distribute.distribute_lib._DefaultDistributionStrategy object at 0x000001E7A54DDB80>
08/20/2020 20:19:13 MainProcess _training_0 _base __init__ DEBUG Initialized _Settings
08/20/2020 20:19:13 MainProcess _training_0 _base __init__ DEBUG Initialized ModelBase (Model)
08/20/2020 20:19:13 MainProcess _training_0 _base strategy_scope DEBUG Using strategy scope: <tensorflow.python.distribute.distribute_lib._DefaultDistributionContext object at 0x000001E7A5350E80>
08/20/2020 20:19:13 MainProcess _training_0 _base _load DEBUG Loading model: D:\Nueral Network programs\secndary\dflsae model\dfl_sae.h5
08/20/2020 20:19:13 MainProcess _training_0 multithreading run DEBUG Error in thread (_training_0): Unable to open file (bad object header version number)
08/20/2020 20:19:14 MainProcess MainThread train _monitor DEBUG Thread error detected
08/20/2020 20:19:14 MainProcess MainThread train _monitor DEBUG Closed Monitor
08/20/2020 20:19:14 MainProcess MainThread train _end_thread DEBUG Ending Training thread
08/20/2020 20:19:14 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...
08/20/2020 20:19:14 MainProcess MainThread multithreading join DEBUG Joining Threads: '_training'
08/20/2020 20:19:14 MainProcess MainThread multithreading join DEBUG Joining Thread: '_training_0'
08/20/2020 20:19:14 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training_0'
Traceback (most recent call last):
File "C:\Users\user\faceswap\lib\cli\launcher.py", line 156, in execute_script
process.process()
File "C:\Users\user\faceswap\scripts\train.py", line 165, in process
self._end_thread(thread, err)
File "C:\Users\user\faceswap\scripts\train.py", line 205, in _end_thread
thread.join()
File "C:\Users\user\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\user\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\user\faceswap\scripts\train.py", line 227, in _training
raise err
File "C:\Users\user\faceswap\scripts\train.py", line 215, in _training
model = self._load_model()
File "C:\Users\user\faceswap\scripts\train.py", line 244, in _load_model
model.build()
File "C:\Users\user\faceswap\plugins\train\model\_base.py", line 259, in build
model = self._io._load() # pylint:disable=protected-access
File "C:\Users\user\faceswap\plugins\train\model\_base.py", line 495, in _load
model = load_model(self._filename, compile=False)
File "C:\Users\user\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\saving\save.py", line 184, in load_model
return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
File "C:\Users\user\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py", line 166, in load_model_from_hdf5
f = h5py.File(filepath, mode='r')
File "C:\Users\user\MiniConda3\envs\faceswap\lib\site-packages\h5py\_hl\files.py", line 406, in __init__
fid = make_fid(name, mode, userblock_size,
File "C:\Users\user\MiniConda3\envs\faceswap\lib\site-packages\h5py\_hl\files.py", line 173, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5f.pyx", line 88, in h5py.h5f.open
OSError: Unable to open file (bad object header version number)
============ System Information ============
encoding: cp1252
git_branch: master
git_commits: 619bd41 Catch further AMD OOM errors
gpu_cuda: No global version found. Check Conda packages for Conda Cuda
gpu_cudnn: No global version found. Check Conda packages for Conda cuDNN
gpu_devices: GPU_0: GeForce RTX 2060
gpu_devices_active: GPU_0
gpu_driver: 432.00
gpu_vram: GPU_0: 6144MB
os_machine: AMD64
os_platform: Windows-10-10.0.19041-SP0
os_release: 10
py_command: C:\Users\user\faceswap\faceswap.py train -A D:/Nueral Network programs/secndary/third frames 208 -B D:/Nueral Network programs/secndary/5400 frames sized 208 -m D:/Nueral Network programs/secndary/dflsae model -t dfl-sae -bs 4 -it 1000000 -s 200 -ss 25000 -tia D:/Nueral Network programs/secndary/third frames 208 -tib D:/Nueral Network programs/secndary/5400 frames sized 208 -to D:/Nueral Network programs/secndary/Project fourth/third fourth/time lapse for dflsae 208 -ps 50 -L INFO -gui
py_conda_version: conda 4.8.4
py_implementation: CPython
py_version: 3.8.5
py_virtual_env: True
sys_cores: 8
sys_processor: AMD64 Family 23 Model 17 Stepping 0, AuthenticAMD
sys_ram: Total: 15270MB, Available: 9093MB, Used: 6177MB, Free: 9093MB
=============== Pip Packages ===============
absl-py==0.10.0
astunparse==1.6.3
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.3.3
google-auth==1.20.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.31.0
h5py==2.10.0
idna==2.10
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1589202782679/work
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
Markdown==3.2.2
matplotlib @ file:///C:/ci/matplotlib-base_1597876438601/work
mkl-fft==1.1.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.4.0.42
opt-einsum==3.3.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1594298230227/work
protobuf==3.13.0
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
pywin32==227
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-learn @ file:///C:/ci/scikit-learn_1592853510272/work
scipy==1.4.1
sip==4.19.13
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tensorflow-gpu==2.2.0
tensorflow-gpu-estimator==2.2.0
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1596810128862/work
urllib3==1.25.10
Werkzeug==1.0.1
wincertstore==0.2
wrapt==1.12.1
============== Conda Packages ==============
# packages in environment at C:\Users\user\MiniConda3\envs\faceswap:
#
# Name Version Build Channel
absl-py 0.10.0 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
blas 1.0 mkl
ca-certificates 2020.6.24 0
cachetools 4.1.1 pypi_0 pypi
certifi 2020.6.20 py38_0
chardet 3.0.4 pypi_0 pypi
cudatoolkit 10.1.243 h74a9793_0
cudnn 7.6.5 cuda10.1_0
cycler 0.10.0 py38_0
fastcluster 1.1.26 py38hbe40bda_1 conda-forge
ffmpeg 4.3.1 ha925a31_0 conda-forge
ffmpy 0.2.3 pypi_0 pypi
freetype 2.10.2 hd328e21_0
gast 0.3.3 pypi_0 pypi
git 2.23.0 h6bb4b03_0
google-auth 1.20.1 pypi_0 pypi
google-auth-oauthlib 0.4.1 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.31.0 pypi_0 pypi
h5py 2.10.0 pypi_0 pypi
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 pypi_0 pypi
imageio 2.9.0 py_0
imageio-ffmpeg 0.4.2 py_0 conda-forge
intel-openmp 2020.1 216
joblib 0.16.0 py_0
jpeg 9b hb83a4c4_2
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.2.0 py38h74a9793_0
libpng 1.6.37 h2a8f88b_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 h62dcd97_1
markdown 3.2.2 pypi_0 pypi
matplotlib 3.3.1 0
matplotlib-base 3.3.1 py38hba9282a_0
mkl 2020.1 216
mkl-service 2.3.0 py38hb782905_0
mkl_fft 1.1.0 py38h45dec08_0
mkl_random 1.1.1 py38h47e9c7a_0
numpy 1.19.1 py38h5510c5b_0
numpy-base 1.19.1 py38ha3acd2a_0
nvidia-ml-py3 7.352.1 pypi_0 pypi
oauthlib 3.1.0 pypi_0 pypi
olefile 0.46 py_0
opencv-python 4.4.0.42 pypi_0 pypi
openssl 1.1.1g he774522_1
opt-einsum 3.3.0 pypi_0 pypi
pathlib 1.0.1 py_1
pillow 7.2.0 py38hcc1f983_0
pip 20.2.2 py38_0
protobuf 3.13.0 pypi_0 pypi
psutil 5.7.0 py38he774522_0
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pyparsing 2.4.7 py_0
pyqt 5.9.2 py38ha925a31_4
python 3.8.5 he1778fa_0
python-dateutil 2.8.1 py_0
python_abi 3.8 1_cp38 conda-forge
pywin32 227 py38he774522_1
qt 5.9.7 vc14h73c81de_0
requests 2.24.0 pypi_0 pypi
requests-oauthlib 1.3.0 pypi_0 pypi
rsa 4.6 pypi_0 pypi
scikit-learn 0.23.1 py38h25d0782_0
scipy 1.4.1 pypi_0 pypi
setuptools 49.6.0 py38_0
sip 4.19.13 py38ha925a31_0
six 1.15.0 py_0
sqlite 3.33.0 h2a8f88b_0
tensorboard 2.2.2 pypi_0 pypi
tensorboard-plugin-wit 1.7.0 pypi_0 pypi
tensorflow-gpu 2.2.0 pypi_0 pypi
tensorflow-gpu-estimator 2.2.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
tornado 6.0.4 py38he774522_1
tqdm 4.48.2 py_0
urllib3 1.25.10 pypi_0 pypi
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_3
werkzeug 1.0.1 pypi_0 pypi
wheel 0.34.2 py38_0
wincertstore 0.2 py38_0
wrapt 1.12.1 pypi_0 pypi
xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.5 h04227a9_0
================= Configs ==================
--------- .faceswap ---------
backend: nvidia
--------- convert.ini ---------
[color.color_transfer]
clip: True
preserve_paper: True
[color.manual_balance]
colorspace: HSV
balance_1: 0.0
balance_2: 0.0
balance_3: 0.0
contrast: 0.0
brightness: 0.0
[color.match_hist]
threshold: 99.0
[mask.box_blend]
type: gaussian
distance: 11.0
radius: 5.0
passes: 1
[mask.mask_blend]
type: normalized
kernel_size: 3
passes: 4
threshold: 4
erosion: 0.0
[scaling.sharpen]
method: unsharp_mask
amount: 150
radius: 0.3
threshold: 5.0
[writer.ffmpeg]
container: mp4
codec: libx264
crf: 23
preset: medium
tune: none
profile: auto
level: auto
skip_mux: False
[writer.gif]
fps: 25
loop: 0
palettesize: 256
subrectangles: False
[writer.opencv]
format: png
draw_transparent: False
jpg_quality: 75
png_compress_level: 3
[writer.pillow]
format: png
draw_transparent: False
optimize: False
gif_interlace: True
jpg_quality: 75
png_compress_level: 3
tif_compression: tiff_deflate
--------- extract.ini ---------
[global]
allow_growth: False
[align.fan]
batch-size: 12
[detect.cv2_dnn]
confidence: 50
[detect.mtcnn]
minsize: 20
threshold_1: 0.6
threshold_2: 0.7
threshold_3: 0.7
scalefactor: 0.709
batch-size: 8
[detect.s3fd]
confidence: 70
batch-size: 4
[mask.unet_dfl]
batch-size: 8
[mask.vgg_clear]
batch-size: 6
[mask.vgg_obstructed]
batch-size: 2
--------- gui.ini ---------
[global]
fullscreen: False
tab: extract
options_panel_width: 30
console_panel_height: 20
icon_size: 14
font: default
font_size: 9
autosave_last_session: prompt
timeout: 120
auto_load_model_stats: True
--------- train.ini ---------
[global]
coverage: 68.75
mask_type: extended
mask_blur_kernel: 3
mask_threshold: 4
learn_mask: False
penalized_mask_loss: True
loss_function: mae
icnr_init: False
conv_aware_init: False
optimizer: adam
learning_rate: 5e-05
reflect_padding: False
allow_growth: False
mixed_precision: False
convert_batchsize: 16
[model.dfl_h128]
lowmem: False
[model.dfl_sae]
input_size: 128
clipnorm: True
architecture: df
autoencoder_dims: 0
encoder_dims: 42
decoder_dims: 21
multiscale_decoder: False
[model.dlight]
features: best
details: good
output_size: 256
[model.original]
lowmem: False
[model.realface]
input_size: 64
output_size: 128
dense_nodes: 1536
complexity_encoder: 128
complexity_decoder: 512
[model.unbalanced]
input_size: 128
lowmem: False
clipnorm: True
nodes: 1024
complexity_encoder: 128
complexity_decoder_a: 384
complexity_decoder_b: 512
[model.villain]
lowmem: False
[trainer.original]
preview_images: 14
zoom_amount: 5
rotation_range: 10
shift_range: 5
flip_chance: 50
color_lightness: 30
color_ab: 8
color_clahe_chance: 50
color_clahe_max_size: 4
