Training Continuation Error: Unknown config_item: RandomNormal

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
XxArTiuSxX
Posts: 10
Joined: Tue May 31, 2022 3:20 am
Has thanked: 34 times

Training Continuation Error: Unknown config_item: RandomNormal

Post by XxArTiuSxX »

Hello great community!

I tried searching for threads with issues that mention "Unknown config_item: RandomNormal" but was unable to locate any

Please forgive the miss if there is already a thread about this issue :oops:

Some background:

  1. Running in Colab Pro
  2. Training started normally and completed over 25000 iterations (Model: Villain)
  3. When I resume, it throws up the attached error.
    3.1 Started a new model (Villain) which works till I stop the model. When I try to continue it, it gives the same error again.

Any and all help is much appreciated!

Code: Select all

06/29/2022 18:19:31 MainProcess     _training_0                    config          _load_defaults_from_module     DEBUG    Importing defaults module: plugins.train.model.original_defaults
06/29/2022 18:19:31 MainProcess     _training_0                    config          add_section                    DEBUG    Add section: (title: 'model.original', info: 'Original Faceswap Model.')
06/29/2022 18:19:31 MainProcess     _training_0                    config          add_item                       DEBUG    Add item: (section: 'model.original', title: 'lowmem', datatype: '<class 'bool'>', default: 'False', info: 'Lower memory mode. Set to 'True' if having issues with VRAM useage.\nNB: Models with a changed lowmem mode are not compatible with each other.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
06/29/2022 18:19:31 MainProcess     _training_0                    config          _load_defaults_from_module     DEBUG    Added defaults: model.original
06/29/2022 18:19:31 MainProcess     _training_0                    config          handle_config                  DEBUG    Handling config: (section: model.villain, configfile: '/content/faceswap/config/train.ini')
06/29/2022 18:19:31 MainProcess     _training_0                    config          check_exists                   DEBUG    Config file exists: '/content/faceswap/config/train.ini'
06/29/2022 18:19:31 MainProcess     _training_0                    config          load_config                    VERBOSE  Loading config: '/content/faceswap/config/train.ini'
06/29/2022 18:19:31 MainProcess     _training_0                    config          validate_config                DEBUG    Validating config
06/29/2022 18:19:31 MainProcess     _training_0                    config          check_config_change            DEBUG    Default config has not changed
06/29/2022 18:19:31 MainProcess     _training_0                    config          check_config_choices           DEBUG    Checking config choices
06/29/2022 18:19:31 MainProcess     _training_0                    config          _parse_list                    DEBUG    Processed raw option 'keras_encoder' to list ['keras_encoder'] for section 'model.phaze_a', option 'freeze_layers'
06/29/2022 18:19:31 MainProcess     _training_0                    config          _parse_list                    DEBUG    Processed raw option 'encoder' to list ['encoder'] for section 'model.phaze_a', option 'load_layers'
06/29/2022 18:19:31 MainProcess     _training_0                    config          check_config_choices           DEBUG    Checked config choices
06/29/2022 18:19:31 MainProcess     _training_0                    config          validate_config                DEBUG    Validated config
06/29/2022 18:19:31 MainProcess     _training_0                    config          handle_config                  DEBUG    Handled config
06/29/2022 18:19:31 MainProcess     _training_0                    config          __init__                       DEBUG    Initialized: Config
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global', option: 'learning_rate')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 5e-05)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global', option: 'epsilon_exponent')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: -7)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global', option: 'autoclip')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global', option: 'allow_growth')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: True)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global', option: 'mixed_precision')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: True)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global', option: 'nan_protection')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: True)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global', option: 'convert_batchsize')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 16)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'str'>, value: ssim)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_2')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mse)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_2')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 100)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_3')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'str'>, value: None)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_3')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 0)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_4')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'str'>, value: None)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_4')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 0)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mask_loss_function')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mse)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'eye_multiplier')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 3)
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mouth_multiplier')
06/29/2022 18:19:31 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 2)
06/29/2022 18:19:31 MainProcess     _training_0                    config          changeable_items               DEBUG    Alterable for existing models: {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': True, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}
06/29/2022 18:19:31 MainProcess     _training_0                    model           __init__                       DEBUG    Initializing State: (model_dir: '/content/drive/MyDrive/Colab/faceswap/Podel', model_name: 'villain', config_changeable_items: '{'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': True, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}', no_logs: False
06/29/2022 18:19:31 MainProcess     _training_0                    serializer      get_serializer                 DEBUG    <lib.serializer._JSONSerializer object at 0x7fb13adab590>
06/29/2022 18:19:31 MainProcess     _training_0                    model           _load                          DEBUG    Loading State
06/29/2022 18:19:31 MainProcess     _training_0                    serializer      load                           DEBUG    filename: /content/drive/MyDrive/Colab/faceswap/Podel/villain_state.json
06/29/2022 18:19:31 MainProcess     _training_0                    serializer      load                           DEBUG    stored data type: <class 'bytes'>
06/29/2022 18:19:31 MainProcess     _training_0                    serializer      unmarshal                      DEBUG    data type: <class 'bytes'>
06/29/2022 18:19:31 MainProcess     _training_0                    serializer      unmarshal                      DEBUG    returned data type: <class 'dict'>
06/29/2022 18:19:31 MainProcess     _training_0                    serializer      load                           DEBUG    data type: <class 'dict'>
06/29/2022 18:19:31 MainProcess     _training_0                    model           _load                          DEBUG    Loaded state: {'name': 'villain', 'sessions': {'1': {'timestamp': 1656477580.8066294, 'no_logs': False, 'loss_names': ['total', 'face_a', 'face_b'], 'batchsize': 16, 'iterations': 30240, 'config': {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': True, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2}}}, 'lowest_avg_loss': {'a': 0.04317956857590212, 'b': 0.03314127560911907}, 'iterations': 30240, 'mixed_precision_layers': [], 'config': {'centering': 'face', 'coverage': 100.0, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': True, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'bisenet-fp_face', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'lowmem': False}}
06/29/2022 18:19:31 MainProcess     _training_0                    model           _update_legacy_config          DEBUG    Checking for legacy state file update
06/29/2022 18:19:31 MainProcess     _training_0                    model           _update_legacy_config          DEBUG    Legacy item 'dssim_loss' not in config. Skipping update
06/29/2022 18:19:31 MainProcess     _training_0                    model           _update_legacy_config          DEBUG    Legacy item 'l2_reg_term' not in config. Skipping update
06/29/2022 18:19:31 MainProcess     _training_0                    model           _update_legacy_config          DEBUG    Legacy item 'clipnorm' not in config. Skipping update
06/29/2022 18:19:31 MainProcess     _training_0                    model           _update_legacy_config          DEBUG    State file updated for legacy config: False
06/29/2022 18:19:31 MainProcess     _training_0                    model           _replace_config                DEBUG    Replacing config. Old config: {'centering': 'face', 'coverage': 100.0, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': True, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'bisenet-fp_face', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'lowmem': False}
06/29/2022 18:19:31 MainProcess     _training_0                    model           _replace_config                DEBUG    Replaced config. New config: {'centering': 'face', 'coverage': 100.0, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': True, 'mixed_precision': True, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'bisenet-fp_face', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'lowmem': False}
06/29/2022 18:19:31 MainProcess     _training_0                    model           _replace_config                INFO     Using configuration saved in state file
06/29/2022 18:19:31 MainProcess     _training_0                    model           _new_session_id                DEBUG    2
06/29/2022 18:19:31 MainProcess     _training_0                    model           _create_new_session            DEBUG    Creating new session. id: 2
06/29/2022 18:19:31 MainProcess     _training_0                    model           __init__                       DEBUG    Initialized State:
06/29/2022 18:19:31 MainProcess     _training_0                    settings        __init__                       DEBUG    Initializing Settings: (arguments: Namespace(batch_size=16, colab=False, configfile=None, distributed=False, exclude_gpus=None, freeze_weights=False, func=<bound method ScriptExecutor.execute_script of <lib.cli.launcher.ScriptExecutor object at 0x7fb1cf3a2550>>, input_a='/content/drive/MyDrive/Colab/faceswap/faces/A', input_b='/content/drive/MyDrive/Colab/faceswap/faces/Pooja', iterations=100000, load_weights=None, logfile=None, loglevel='INFO', model_dir='/content/drive/MyDrive/Colab/faceswap/Podel', no_augment_color=False, no_flip=False, no_logs=False, no_warp=False, preview=False, preview_scale=100, redirect_gui=False, save_interval=360, snapshot_interval=20000, summary=False, timelapse_input_a='/content/drive/MyDrive/Colab/faceswap/faces/A', timelapse_input_b='/content/drive/MyDrive/Colab/faceswap/faces/Pooja', timelapse_output='/content/drive/MyDrive/Colab/faceswap/timelapse', trainer='villain', warp_to_landmarks=False, write_image=True), mixed_precision: True, allow_growth: True, is_predict: False)
06/29/2022 18:19:32 MainProcess     _training_0                    settings        _set_tf_settings               DEBUG    Setting Tensorflow 'allow_growth' option
06/29/2022 18:19:32 MainProcess     _training_0                    settings        _set_tf_settings               INFO     Setting allow growth for GPU: PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
06/29/2022 18:19:32 MainProcess     _training_0                    settings        _set_tf_settings               DEBUG    Set Tensorflow 'allow_growth' option
06/29/2022 18:19:32 MainProcess     _training_0                    settings        _set_keras_mixed_precision     DEBUG    use_mixed_precision: True
06/29/2022 18:19:32 MainProcess     _training_0                    device_compatibility_check _log_device_compatibility_check WARNING  Mixed precision compatibility check (mixed_float16): WARNING
Your GPU may run slowly with dtype policy mixed_float16 because it does not have compute capability of at least 7.0. Your GPU:
  Tesla P100-PCIE-16GB, compute capability 6.0
See https://developer.nvidia.com/cuda-gpus for a list of GPUs and their compute capabilities.
If you will use compatible GPU(s) not attached to this host, e.g. by running a multi-worker model, you can ignore this warning. This message will only be logged once
06/29/2022 18:19:32 MainProcess     _training_0                    settings        _set_keras_mixed_precision     DEBUG    Enabled mixed precision. (Compute dtype: float16, variable_dtype: float32)
06/29/2022 18:19:32 MainProcess     _training_0                    settings        __init__                       INFO     Enabling Mixed Precision Training.
06/29/2022 18:19:32 MainProcess     _training_0                    settings        _get_strategy                  DEBUG    Using strategy: <tensorflow.python.distribute.distribute_lib._DefaultDistributionStrategy object at 0x7fb139d2bd90>
06/29/2022 18:19:32 MainProcess     _training_0                    settings        __init__                       DEBUG    Initialized Settings
06/29/2022 18:19:32 MainProcess     _training_0                    settings        __init__                       DEBUG    Initializing Loss
06/29/2022 18:19:32 MainProcess     _training_0                    settings        _get_mask_channels             DEBUG    uses_masks: (True, True, True), mask_channels: [3, 4, 5]
06/29/2022 18:19:32 MainProcess     _training_0                    settings        __init__                       DEBUG    Initialized: Loss
06/29/2022 18:19:32 MainProcess     _training_0                    model           __init__                       DEBUG    Initialized ModelBase (Model)
06/29/2022 18:19:32 MainProcess     _training_0                    settings        strategy_scope                 DEBUG    Using strategy scope: <tensorflow.python.distribute.distribute_lib._DefaultDistributionContext object at 0x7fb139df1c30>
06/29/2022 18:19:32 MainProcess     _training_0                    io              _load                          DEBUG    Loading model: /content/drive/MyDrive/Colab/faceswap/Podel/villain.h5
06/29/2022 18:19:32 MainProcess     _training_0                    attrs           __getitem__                    DEBUG    Creating converter from 3 to 5
06/29/2022 18:19:32 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): Unknown config_item: RandomNormal. Please ensure this object is passed to the `custom_objects` argument. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.
06/29/2022 18:19:33 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
06/29/2022 18:19:33 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
06/29/2022 18:19:33 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
06/29/2022 18:19:33 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
06/29/2022 18:19:33 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
06/29/2022 18:19:33 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
06/29/2022 18:19:33 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 188, in execute_script
    process.process()
  File "/content/faceswap/scripts/train.py", line 190, in process
    self._end_thread(thread, err)
  File "/content/faceswap/scripts/train.py", line 230, in _end_thread
    thread.join()
  File "/content/faceswap/lib/multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/content/faceswap/scripts/train.py", line 252, in _training
    raise err
  File "/content/faceswap/scripts/train.py", line 240, in _training
    model = self._load_model()
  File "/content/faceswap/scripts/train.py", line 268, in _load_model
    model.build()
  File "/content/faceswap/plugins/train/model/_base/model.py", line 287, in build
    model = self._io._load()  # pylint:disable=protected-access
  File "/content/faceswap/plugins/train/model/_base/io.py", line 152, in _load
    model = load_model(self._filename, compile=False)
  File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.7/dist-packages/keras/utils/generic_utils.py", line 563, in class_and_config_for_serialized_keras_object
    f'Unknown {printable_module_name}: {class_name}. Please ensure this '
ValueError: Unknown config_item: RandomNormal. Please ensure this object is passed to the `custom_objects` argument. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.

============ System Information ============
encoding:            UTF-8
git_branch:          Not Found
git_commits:         Not Found
gpu_cuda:            11.1
gpu_cudnn:           No global version found
gpu_devices:         GPU_0: Tesla P100-PCIE-16GB
gpu_devices_active:  GPU_0
gpu_driver:          460.32.03
gpu_vram:            GPU_0: 16280MB
os_machine:          x86_64
os_platform:         Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
os_release:          5.4.188+
py_command:          faceswap/faceswap.py train -A /content/drive/MyDrive/Colab/faceswap/faces/A -B /content/drive/MyDrive/Colab/faceswap/faces/Pooja -m /content/drive/MyDrive/Colab/faceswap/Podel -t villain -bs 16 -it 100000 -s 360 -ss 20000 -tia /content/drive/MyDrive/Colab/faceswap/faces/A -tib /content/drive/MyDrive/Colab/faceswap/faces/Pooja -to /content/drive/MyDrive/Colab/faceswap/timelapse -w
py_conda_version:    N/A
py_implementation:   CPython
py_version:          3.7.13
py_virtual_env:      False
sys_cores:           2
sys_processor:       x86_64
sys_ram:             Total: 12986MB, Available: 11651MB, Used: 1112MB, Free: 7492MB

=============== Pip Packages ===============
absl-py==1.1.0
alabaster==0.7.12
albumentations==0.1.12
altair==4.2.0
appdirs==1.4.4
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arviz==0.12.1
astor==0.8.1
astropy==4.3.1
astunparse==1.6.3
atari-py==0.2.9
atomicwrites==1.4.0
attrs==21.4.0
audioread==2.1.9
autograd==1.4
Babel==2.10.2
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==5.0.0
blis==0.7.7
bokeh==2.3.3
branca==0.5.0
bs4==0.0.1
CacheControl==0.12.11
cached-property==1.5.2
cachetools==4.2.4
catalogue==2.0.7
certifi==2022.6.15
cffi==1.15.0
cftime==1.6.0
chardet==3.0.4
charset-normalizer==2.0.12
click==7.1.2
cloudpickle==1.3.0
cmake==3.22.5
cmdstanpy==0.9.5
colorcet==3.0.0
colorlover==0.3.0
community==1.0.0b1
contextlib2==0.5.5
convertdate==2.4.0
coverage==3.7.1
coveralls==0.5
crcmod==1.7
cufflinks==0.17.3
cupy-cuda111==9.4.0
cvxopt==1.2.7
cvxpy==1.0.31
cycler==0.11.0
cymem==2.0.6
Cython==0.29.30
daft==0.0.4
dask==2.12.0
datascience==0.10.6
debugpy==1.0.0
decorator==4.4.2
defusedxml==0.7.1
descartes==1.1.0
dill==0.3.5.1
distributed==1.25.3
dlib==19.18.0+zzzcolab20220513001918
dm-tree==0.1.7
docopt==0.6.2
docutils==0.17.1
dopamine-rl==1.0.5
earthengine-api==0.1.315
easydict==1.9
ecos==2.0.10
editdistance==0.5.3
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl
entrypoints==0.4
ephem==4.1.3
et-xmlfile==1.1.0
fa2==0.3.5
fastai==2.6.3
fastcluster==1.2.6
fastcore==1.4.4
fastdownload==0.0.6
fastdtw==0.3.4
fastjsonschema==2.15.3
fastprogress==1.0.2
fastrlock==0.8
fbprophet==0.7.1
feather-format==0.4.1
ffmpy==0.2.3
filelock==3.7.1
firebase-admin==4.4.0
fix-yahoo-finance==0.0.22
Flask==1.1.4
flatbuffers==2.0
folium==0.8.3
fonttools==4.33.3
future==0.16.0
gast==0.5.3
GDAL==2.2.2
gdown==4.4.0
gensim==3.6.0
geographiclib==1.52
geopy==1.17.0
gin-config==0.5.0
glob2==0.7
google==2.0.3
google-api-core==1.31.6
google-api-python-client==1.12.11
google-auth==1.35.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.6
google-cloud-bigquery==1.21.0
google-cloud-bigquery-storage==1.1.2
google-cloud-core==1.0.3
google-cloud-datastore==1.8.0
google-cloud-firestore==1.7.0
google-cloud-language==1.2.0
google-cloud-storage==1.18.1
google-cloud-translate==1.5.0
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz
google-pasta==0.2.0
google-resumable-media==0.4.1
googleapis-common-protos==1.56.2
googledrivedownloader==0.4
graphviz==0.10.1
greenlet==1.1.2
grpcio==1.46.3
gspread==3.4.2
gspread-dataframe==3.0.8
gym==0.17.3
h5py==3.1.0
HeapDict==1.0.1
hijri-converter==2.2.4
holidays==0.10.5.2
holoviews==1.14.9
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.17.4
httplib2shim==0.0.3
humanize==0.5.1
hyperopt==0.1.2
ideep4py==2.0.0.post3
idna==2.10
imageio==2.19.3
imageio-ffmpeg==0.4.7
imagesize==1.3.0
imbalanced-learn==0.8.1
imblearn==0.0
imgaug==0.2.9
importlib-metadata==4.11.4
importlib-resources==5.7.1
imutils==0.5.4
inflect==2.1.0
iniconfig==1.1.1
intel-openmp==2022.1.0
intervaltree==2.1.0
ipykernel==4.10.1
ipython==5.5.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.7.0
itsdangerous==1.1.0
jax==0.3.8
jaxlib @ https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.3.7+cuda11.cudnn805-cp37-none-manylinux2014_x86_64.whl
jedi==0.18.1
jieba==0.42.1
Jinja2==2.11.3
joblib==1.1.0
jpeg4py==0.1.4
jsonschema==4.3.3
jupyter==1.0.0
jupyter-client==5.3.5
jupyter-console==5.2.0
jupyter-core==4.10.0
jupyterlab-pygments==0.2.2
jupyterlab-widgets==1.1.0
kaggle==1.5.12
kapre==0.3.7
keras==2.8.0
Keras-Preprocessing==1.1.2
keras-vis==0.4.1
kiwisolver==1.4.3
korean-lunar-calendar==0.2.1
langcodes==3.3.0
libclang==14.0.1
librosa==0.8.1
lightgbm==2.2.3
llvmlite==0.34.0
lmdb==0.99
LunarCalendar==0.0.9
lxml==4.2.6
Markdown==3.3.7
MarkupSafe==2.0.1
matplotlib==3.5.2
matplotlib-inline==0.1.3
matplotlib-venn==0.11.7
missingno==0.5.1
mistune==0.8.4
mizani==0.6.0
mkl==2019.0
mlxtend==0.14.0
more-itertools==8.13.0
moviepy==0.2.3.5
mpmath==1.2.1
msgpack==1.0.4
multiprocess==0.70.13
multitasking==0.0.10
murmurhash==1.0.7
music21==5.5.0
natsort==5.5.0
nbclient==0.6.4
nbconvert==5.6.1
nbformat==5.4.0
nest-asyncio==1.5.5
netCDF4==1.5.8
networkx==2.6.3
nibabel==3.0.2
nltk==3.7
notebook==5.3.1
numba==0.51.2
numexpr==2.8.1
numpy==1.21.6
nvidia-ml-py==11.510.69
oauth2client==4.1.3
oauthlib==3.2.0
okgrade==0.4.3
opencv-contrib-python==4.1.2.30
opencv-python==4.6.0.66
openpyxl==3.0.10
opt-einsum==3.3.0
osqp==0.6.2.post0
packaging==21.3
palettable==3.3.0
pandas==1.3.5
pandas-datareader==0.9.0
pandas-gbq==0.13.3
pandas-profiling==1.4.1
pandocfilters==1.5.0
panel==0.12.1
param==1.12.1
parso==0.8.3
pathlib==1.0.1
pathy==0.6.1
patsy==0.5.2
pep517==0.12.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.1.1
pip-tools==6.2.0
plotly==5.5.0
plotnine==0.6.0
pluggy==0.7.1
pooch==1.6.0
portpicker==1.3.9
prefetch-generator==1.0.1
preshed==3.0.6
prettytable==3.3.0
progressbar2==3.38.0
prometheus-client==0.14.1
promise==2.3
prompt-toolkit==1.0.18
protobuf==3.17.3
psutil==5.9.1
psycopg2==2.7.6.1
ptyprocess==0.7.0
py==1.11.0
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.4
pycparser==2.21
pyct==0.4.8
pydantic==1.8.2
pydata-google-auth==1.4.0
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyerfa==2.0.0.1
pyglet==1.5.0
Pygments==2.6.1
pygobject==3.26.1
pymc3==3.11.4
PyMeeus==0.5.11
pymongo==4.1.1
pymystem3==0.2.0
PyOpenGL==3.1.6
pyparsing==3.0.9
pyrsistent==0.18.1
pysndfile==1.3.8
PySocks==1.7.1
pystan==2.19.1.1
pytest==3.6.4
python-apt==0.0.0
python-chess==0.23.11
python-dateutil==2.8.2
python-louvain==0.16
python-slugify==6.1.2
python-utils==3.3.3
pytz==2022.1
pyviz-comms==2.2.0
PyWavelets==1.3.0
PyYAML==3.13
pyzmq==23.1.0
qdldl==0.1.5.post2
qtconsole==5.3.1
QtPy==2.1.0
regex==2022.6.2
requests==2.23.0
requests-oauthlib==1.3.1
resampy==0.2.2
rpy2==3.4.5
rsa==4.8
scikit-image==0.18.3
scikit-learn==1.0.2
scipy==1.4.1
screen-resolution-extra==0.0.0
scs==3.2.0
seaborn==0.11.2
semver==2.13.0
Send2Trash==1.8.0
setuptools-git==1.2
Shapely==1.8.2
simplegeneric==0.8.1
six==1.15.0
sklearn==0.0
sklearn-pandas==1.8.0
smart-open==5.2.1
snowballstemmer==2.2.0
sortedcontainers==2.4.0
SoundFile==0.10.3.post1
soupsieve==2.3.2.post1
spacy==3.3.1
spacy-legacy==3.0.9
spacy-loggers==1.0.2
Sphinx==1.8.6
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib-websupport==1.2.4
SQLAlchemy==1.4.37
sqlparse==0.4.2
srsly==2.4.3
statsmodels==0.10.2
sympy==1.7.1
tables==3.7.0
tabulate==0.8.9
tblib==1.7.0
tenacity==8.0.1
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.8.2+zzzcolab20220527125636
tensorflow-datasets==4.0.1
tensorflow-estimator==2.8.0
tensorflow-gcs-config==2.8.0
tensorflow-hub==0.12.0
tensorflow-io-gcs-filesystem==0.26.0
tensorflow-metadata==1.8.0
tensorflow-probability==0.16.0
termcolor==1.1.0
terminado==0.13.3
testpath==0.6.0
text-unidecode==1.3
textblob==0.15.3
Theano-PyMC==1.1.2
thinc==8.0.17
threadpoolctl==3.1.0
tifffile==2021.11.2
tinycss2==1.1.1
tomli==2.0.1
toolz==0.11.2
torch @ https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp37-cp37m-linux_x86_64.whl
torchaudio @ https://download.pytorch.org/whl/cu113/torchaudio-0.11.0%2Bcu113-cp37-cp37m-linux_x86_64.whl
torchsummary==1.5.1
torchtext==0.12.0
torchvision @ https://download.pytorch.org/whl/cu113/torchvision-0.12.0%2Bcu113-cp37-cp37m-linux_x86_64.whl
tornado==5.1.1
tqdm==4.64.0
traitlets==5.1.1
tweepy==3.10.0
typeguard==2.7.1
typer==0.4.1
typing-extensions==4.1.1
tzlocal==1.5.1
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.9.0
wasabi==0.9.1
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.6.0
wordcloud==1.5.0
wrapt==1.14.1
xarray==0.20.2
xarray-einstats==0.2.2
xgboost==0.90
xkit==0.0.0
xlrd==1.1.0
xlwt==1.3.0
yellowbrick==1.4
zict==2.2.0
zipp==3.8.0

=============== State File =================
{
  "name": "villain",
  "sessions": {
    "1": {
      "timestamp": 1656477580.8066294,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 30240,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "autoclip": false,
        "allow_growth": true,
        "mixed_precision": true,
        "nan_protection": true,
        "convert_batchsize": 16,
        "loss_function": "ssim",
        "loss_function_2": "mse",
        "loss_weight_2": 100,
        "loss_function_3": null,
        "loss_weight_3": 0,
        "loss_function_4": null,
        "loss_weight_4": 0,
        "mask_loss_function": "mse",
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    }
  },
  "lowest_avg_loss": {
    "a": 0.04317956857590212,
    "b": 0.03314127560911907
  },
  "iterations": 30240,
  "mixed_precision_layers": [],
  "config": {
    "centering": "face",
    "coverage": 100.0,
    "optimizer": "adam",
    "learning_rate": 5e-05,
    "epsilon_exponent": -7,
    "autoclip": false,
    "allow_growth": true,
    "mixed_precision": true,
    "nan_protection": true,
    "convert_batchsize": 16,
    "loss_function": "ssim",
    "loss_function_2": "mse",
    "loss_weight_2": 100,
    "loss_function_3": null,
    "loss_weight_3": 0,
    "loss_function_4": null,
    "loss_weight_4": 0,
    "mask_loss_function": "mse",
    "eye_multiplier": 3,
    "mouth_multiplier": 2,
    "penalized_mask_loss": true,
    "mask_type": "bisenet-fp_face",
    "mask_blur_kernel": 3,
    "mask_threshold": 4,
    "learn_mask": false,
    "lowmem": false
  }
}

================= Configs ==================
--------- train.ini ---------

[global]
centering:                face
coverage:                 100.0
icnr_init:                True
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
autoclip:                 False
reflect_padding:          True
allow_growth:             True
mixed_precision:          True
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
loss_function_2:          mse
loss_weight_2:            100
loss_function_3:          none
loss_weight_3:            0
loss_function_4:          none
loss_weight_4:            0
mask_loss_function:       mse
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                bisenet-fp_face
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

[model.dfl_h128]
lowmem:                   False

[model.dfaker]
output_size:              128

[model.unbalanced]
input_size:               128
lowmem:                   False
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.dfl_sae]
input_size:               128
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.phaze_a]
output_size:              128
shared_fc:                None
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              40
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          None
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_upscales_in_fc:       0
dec_norm:                 None
dec_min_filters:          64
dec_max_filters:          512
dec_slope_mode:           full
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
fs_original_use_alt:      False
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.original]
lowmem:                   False

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               75
batch-size:               64

[mask.bisenet_fp]
batch-size:               8
weights:                  faceswap
include_ears:             True
include_hair:             False
include_glasses:          True

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     gaussian
kernel_size:              3
passes:                   5
threshold:                0
erosion:                  -12.2

[scaling.sharpen]
method:                   None
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     None
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   jpg
draw_transparent:         False
optimize:                 True
gif_interlace:            True
jpg_quality:              95
png_compress_level:       0
tif_compression:          tiff_deflate
Last edited by XxArTiuSxX on Wed Aug 17, 2022 9:35 pm, edited 2 times in total.
User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Training Continuation Error: Unknown config_item: RandomNormal

Post by torzdf »

Hmmm. Not seen this before, and really don't know what the issue could be. RandomNormal is a standard initializer which definitely exists in Keras:
https://github.com/keras-team/keras/blo ... v2.py#L302

A quick bit of googling suggests this may be a version mismatch issue in Colab:
https://stackoverflow.com/questions/531 ... eras-model

My word is final

Locked