Training Continuation Error: Unknown config_item: RandomNormal

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
chedger
Posts: 4
Joined: Thu Feb 09, 2023 12:06 am

Training Continuation Error: Unknown config_item: RandomNormal

Post by chedger »

I tried searching for threads with issues that mention "Unknown config_item: RandomNormal" but was unable to locate any

Please forgive the miss if there is already a thread about this issue.

Some background:

  1. Running in Windows 10 64x with DirectML backend
  2. Training started normally and completed over 50000 iterations (Model: Villain)
  3. When I resume, it throws up the attached error.
    3.1 Started a new model (Villain) which works till I stop the model. When I try to continue it, it gives the same error again.
    Any and all help is much appreciated!

Code: Select all

02/08/2023 17:07:30 MainProcess     _training                      model           _update_legacy_config          DEBUG    Checking for legacy state file update
02/08/2023 17:07:30 MainProcess     _training                      model           _update_legacy_config          DEBUG    Legacy item 'dssim_loss' not in config. Skipping update
02/08/2023 17:07:30 MainProcess     _training                      model           _update_legacy_config          DEBUG    Legacy item 'l2_reg_term' not in config. Skipping update
02/08/2023 17:07:30 MainProcess     _training                      model           _update_legacy_config          DEBUG    Legacy item 'clipnorm' not in config. Skipping update
02/08/2023 17:07:30 MainProcess     _training                      model           _update_legacy_config          DEBUG    State file updated for legacy config: False
02/08/2023 17:07:30 MainProcess     _training                      model           _replace_config                DEBUG    Replacing config. Old config: {'centering': 'face', 'coverage': 87.5, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': 'ssim', 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 5, 'mouth_multiplier': 15, 'penalized_mask_loss': True, 'mask_type': 'vgg-obstructed', 'mask_blur_kernel': 5, 'mask_threshold': 4, 'learn_mask': False, 'lowmem': False}
02/08/2023 17:07:30 MainProcess     _training                      model           _replace_config                DEBUG    Replaced config. New config: {'centering': 'face', 'coverage': 87.5, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': 'ssim', 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 5, 'mouth_multiplier': 15, 'penalized_mask_loss': True, 'mask_type': 'vgg-obstructed', 'mask_blur_kernel': 5, 'mask_threshold': 4, 'learn_mask': False, 'lowmem': False}
02/08/2023 17:07:30 MainProcess     _training                      model           _replace_config                INFO     Using configuration saved in state file
02/08/2023 17:07:30 MainProcess     _training                      model           _new_session_id                DEBUG    2
02/08/2023 17:07:30 MainProcess     _training                      model           _create_new_session            DEBUG    Creating new session. id: 2
02/08/2023 17:07:30 MainProcess     _training                      model           __init__                       DEBUG    Initialized State:
02/08/2023 17:07:30 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Settings: (arguments: Namespace(func=<bound method ScriptExecutor.execute_script of <lib.cli.launcher.ScriptExecutor object at 0x000001C689E68700>>, exclude_gpus=None, configfile=None, loglevel='DEBUG', logfile=None, redirect_gui=True, colab=False, input_a='F:\\faceswap\\Source\\Emma Brown Faces', input_b='F:\\faceswap\\Source\\Hailee Stienfeld Faces', model_dir='F:\\faceswap\\Projects\\Models\\HaiPee\\Villain', load_weights=None, trainer='villain', summary=False, freeze_weights=False, batch_size=10, iterations=1000000, distribution_strategy='default', save_interval=250, snapshot_interval=2500, timelapse_input_a=None, timelapse_input_b=None, timelapse_output=None, preview=False, write_image=False, no_logs=True, warp_to_landmarks=False, no_flip=False, no_augment_color=False, no_warp=False), mixed_precision: False, allow_growth: False, is_predict: False)
02/08/2023 17:07:30 MainProcess     _training                      settings        _set_tf_settings               DEBUG    Not setting any specific Tensorflow settings
02/08/2023 17:07:30 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    use_mixed_precision: False
02/08/2023 17:07:30 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    Disabling mixed precision. (Compute dtype: float32, variable_dtype: float32)
02/08/2023 17:07:30 MainProcess     _training                      settings        _get_strategy                  DEBUG    Using strategy: <tensorflow python.distribute.distribute_lib._DefaultDistributionStrategy object at 0x000001C693AC1430>
02/08/2023 17:07:30 MainProcess     _training                      settings        __init__                       DEBUG    Initialized Settings
02/08/2023 17:07:30 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Loss: (color_order: bgr)
02/08/2023 17:07:30 MainProcess     _training                      settings        _get_mask_channels             DEBUG    uses_masks: (True, True, True), mask_channels: [3, 4, 5]
02/08/2023 17:07:30 MainProcess     _training                      settings        __init__                       DEBUG    Initialized: Loss
02/08/2023 17:07:30 MainProcess     _training                      model           __init__                       DEBUG    Initialized ModelBase (Model)
02/08/2023 17:07:30 MainProcess     _training                      settings        strategy_scope                 DEBUG    Using strategy scope: <tensorflow python.distribute.distribute_lib._DefaultDistributionContext object at 0x000001C6938CA180>
02/08/2023 17:07:30 MainProcess     _training                      io              _load                          DEBUG    Loading model: F:\faceswap\Projects\Models\HaiPee\Villain\villain.h5
02/08/2023 17:07:30 MainProcess     _training                      attrs           __getitem__                    DEBUG    Creating converter from 3 to 5
02/08/2023 17:07:31 MainProcess     _training                      multithreading  run                            DEBUG    Error in thread (_training): Unknown config_item: RandomNormal. Please ensure this object is passed to the `custom_objects` argument. 02/08/2023 17:07:32 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
02/08/2023 17:07:32 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
02/08/2023 17:07:32 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
02/08/2023 17:07:32 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
02/08/2023 17:07:32 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
02/08/2023 17:07:32 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training'
02/08/2023 17:07:32 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training'
Traceback (most recent call last):
  File "F:\faceswap\lib\cli\launcher py", line 230, in execute_script
    process.process()
  File "F:\faceswap\scripts\train py", line 213, in process
    self._end_thread(thread, err)
  File "F:\faceswap\scripts\train py", line 253, in _end_thread
    thread.join()
  File "F:\faceswap\lib\multithreading py", line 217, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "F:\faceswap\lib\multithreading py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "F:\faceswap\scripts\train py", line 275, in _training
    raise err
  File "F:\faceswap\scripts\train py", line 263, in _training
    model = self._load_model()
  File "F:\faceswap\scripts\train py", line 291, in _load_model
    model.build()
  File "F:\faceswap\plugins\train\model\_base\model py", line 303, in build
    model = self._io._load()  # pylint:disable=protected-access
  File "F:\faceswap\plugins\train\model\_base\io py", line 152, in _load
    model = load_model(self._filename, compile=False)
  File "C:\Users\Unybo\miniconda3\envs\265938\lib\site-packages\keras\utils\traceback_utils py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\Unybo\miniconda3\envs\265938\lib\site-packages\keras\utils\generic_utils py", line 605, in class_and_config_for_serialized_keras_object
    raise ValueError(
ValueError: Unknown config_item: RandomNormal. Please ensure this object is passed to the `custom_objects` argument.

============ System Information ============
backend:             directml
encoding:            cp1252
git_branch:          master
git_commits:         b4212de bugfix: pin numpy
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: Radeon RX Vega, GPU_1: Radeon RX Vega, GPU_2: NVIDIA GeForce GT 1030, GPU_3: Intel(R) HD Graphics 630
gpu_devices_active:  GPU_0, GPU_1, GPU_2, GPU_3
gpu_driver:          31.0.12029.10015|31.0.12029.10015|31.0.15.2756|31.0.101.2111
gpu_vram:            GPU_0: 8120MB (7352MB free), GPU_1: 8120MB (7352MB free), GPU_2: 1981MB (1684MB free), GPU_3: 128MB (9408MB free)
os_machine:          AMD64
os_platform:         Windows-10-10.0.19045-SP0
os_release:          10
py_command:          F:\faceswap\faceswap py train -A F:\faceswap\Source\Emma Brown Faces -B F:\faceswap\Source\Hailee Stienfeld Faces -m F:/faceswap/Projects/Models/HaiPee/Villain -t villain -bs 10 -it 1000000 -D default -s 250 -ss 2500 -nl -L DEBUG -gui
py_conda_version:    conda 23.1.0
py_implementation:   CPython
py_version:          3.9.16
py_virtual_env:      True
sys_cores:           4
sys_processor:       Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
sys_ram:             Total: 20346MB, Available: 12942MB, Used: 7403MB, Free: 12942MB

=============== Pip Packages ===============
absl-py @ file:///C:/b/abs_5babsu7y5x/croot/absl-py_1666362945682/work
astunparse==1.6.3
cachetools==5.3.0
certifi @ file:///C:/b/abs_85o_6fm0se/croot/certifi_1671487778835/work/certifi
charset-normalizer==3.0.1
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1632508026186/work
colorama @ file:///C:/b/abs_a9ozq0l032/croot/colorama_1672387194846/work
comtypes==1.1.10
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
dm-tree @ file:///C:/b/abs_10z0iy5knj/croot/dm-tree_1671027465819/work
fastcluster @ file:///D:/bld/fastcluster_1649783471014/work
ffmpy==0.3.0
flatbuffers==23.1.21
flit_core @ file:///opt/conda/conda-bld/flit-core_1644941570762/work/source/flit_core
fonttools==4.25.0
gast==0.4.0
google-auth==2.16.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.1
h5py==3.8.0
idna==3.4
imageio @ file:///C:/Windows/TEMP/abs_24c1b783-7540-4ca9-a1b1-0e8aa8e6ae64hb79ssux/croots/recipe/imageio_1658785038775/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1673483481485/work
importlib-metadata==6.0.0
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1663332044897/work
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/b/abs_88mdhvtahm/croot/kiwisolver_1672387921783/work
libclang==15.0.6.1
Markdown==3.4.1
MarkupSafe==2.1.2
matplotlib @ file:///C:/ci/matplotlib-suite_1660169687702/work
mkl-fft==1.3.1
mkl-random @ file:///C:/ci/mkl_random_1626186184308/work
mkl-service==2.4.0
munkres==1.1.4
numexpr @ file:///C:/b/abs_a7kbak88hk/croot/numexpr_1668713882979/work
numpy @ file:///C:/b/abs_datssh7cer/croot/numpy_and_numpy_base_1672336199388/work
nvidia-ml-py==11.525.84
oauthlib==3.2.2
opencv-python==4.7.0.68
opt-einsum==3.3.0
packaging @ file:///C:/b/abs_cfsup8ur87/croot/packaging_1671697442297/work
Pillow==9.3.0
ply==3.11
protobuf==3.19.6
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work
PyQt5==5.15.7
PyQt5-sip @ file:///C:/Windows/Temp/abs_d7gmd2jg8i/croots/recipe/pyqt-split_1659273064801/work/pyqt_sip
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==305.1
pywinpty @ file:///C:/ci_310/pywinpty_1644230983541/work/target/wheels/pywinpty-2.0.2-cp39-none-win_amd64.whl
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn @ file:///D:/bld/scikit-learn_1659726281030/work
scipy @ file:///C:/bld/scipy_1658811088396/work
sip @ file:///C:/Windows/Temp/abs_b8fxd17m2u/croots/recipe/sip_1659012372737/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-cpu==2.10.0
tensorflow-directml-plugin==0.4.0.dev230202
tensorflow-estimator==2.10.0
tensorflow-io-gcs-filesystem==0.30.0
tensorflow-probability @ file:///tmp/build/80754af9/tensorflow-probability_1633017132682/work
tensorflow_intel==2.10.0
termcolor==2.2.0
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
tornado @ file:///C:/ci/tornado_1662458743919/work
tqdm @ file:///C:/b/abs_0axbz66qik/croots/recipe/tqdm_1664392691071/work
typing_extensions @ file:///C:/b/abs_89eui86zuq/croot/typing_extensions_1669923792806/work
urllib3==1.26.14
Werkzeug==2.2.2
wincertstore==0.2
wrapt==1.14.1
zipp==3.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\Unybo\miniconda3\envs\265938:
#
# Name                    Version                   Build  Channel
absl-py                   1.3.0            py39haa95532_0  
astunparse 1.6.3 pypi_0 pypi blas 1.0 mkl
brotli 1.0.9 h2bbff1b_7
brotli-bin 1.0.9 h2bbff1b_7
ca-certificates 2023.01.10 haa95532_0
cachetools 5.3.0 pypi_0 pypi certifi 2022.12.7 py39haa95532_0
charset-normalizer 3.0.1 pypi_0 pypi cloudpickle 2.0.0 pyhd3eb1b0_0
colorama 0.4.6 py39haa95532_0
comtypes 1.1.10 py39haa95532_1002
cycler 0.11.0 pyhd3eb1b0_0
decorator 5.1.1 pyhd3eb1b0_0
dm-tree 0.1.7 py39hd77b12b_1
fastcluster 1.2.6 py39h2e25243_1 conda-forge ffmpeg 4.3.1 ha925a31_0 conda-forge ffmpy 0.3.0 pypi_0 pypi flatbuffers 23.1.21 pypi_0 pypi flit-core 3.6.0 pyhd3eb1b0_0
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.12.1 ha860e81_0
gast 0.4.0 pypi_0 pypi git 2.34.1 haa95532_0
glib 2.69.1 h5dc1a3c_2
google-auth 2.16.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.51.1 pypi_0 pypi gst-plugins-base 1.18.5 h9e645db_0
gstreamer 1.18.5 hd78058f_0
h5py 3.8.0 pypi_0 pypi icu 58.2 ha925a31_3
idna 3.4 pypi_0 pypi imageio 2.19.3 py39haa95532_0
imageio-ffmpeg 0.4.8 pyhd8ed1ab_0 conda-forge importlib-metadata 6.0.0 pypi_0 pypi intel-openmp 2021.4.0 haa95532_3556
joblib 1.2.0 pyhd8ed1ab_0 conda-forge jpeg 9e h2bbff1b_0
keras 2.10.0 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.4.4 py39hd77b12b_0
lerc 3.0 hd77b12b_0
libblas 3.9.0 1_h8933c1f_netlib conda-forge libbrotlicommon 1.0.9 h2bbff1b_7
libbrotlidec 1.0.9 h2bbff1b_7
libbrotlienc 1.0.9 h2bbff1b_7
libcblas 3.9.0 5_hd5c7e75_netlib conda-forge libclang 15.0.6.1 pypi_0 pypi libdeflate 1.8 h2bbff1b_5
libffi 3.4.2 hd77b12b_6
libiconv 1.16 h2bbff1b_2
liblapack 3.9.0 5_hd5c7e75_netlib conda-forge libogg 1.3.5 h2bbff1b_1
libpng 1.6.37 h2a8f88b_0
libtiff 4.5.0 h6c2663c_1
libvorbis 1.3.7 he774522_0
libwebp 1.2.4 h2bbff1b_0
libwebp-base 1.2.4 h2bbff1b_0
libxml2 2.9.14 h0ad7f3c_0
libxslt 1.1.35 h2bbff1b_0
lz4-c 1.9.4 h2bbff1b_0
m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown 3.4.1 pypi_0 pypi markupsafe 2.1.2 pypi_0 pypi matplotlib 3.5.2 py39haa95532_0
matplotlib-base 3.5.2 py39hd77b12b_0
mkl 2021.4.0 haa95532_640
mkl-service 2.4.0 py39h2bbff1b_0
mkl_fft 1.3.1 py39h277e83a_0
mkl_random 1.2.2 py39hf11a4ad_0
msys2-conda-epoch 20160418 1 conda-forge munkres 1.1.4 py_0
numexpr 2.8.4 py39h5b0cc5e_0
numpy 1.23.5 py39h3b20f71_0
numpy-base 1.23.5 py39h4da318b_0
nvidia-ml-py 11.525.84 pypi_0 pypi oauthlib 3.2.2 pypi_0 pypi opencv-python 4.7.0.68 pypi_0 pypi openssl 1.1.1s h2bbff1b_0
opt-einsum 3.3.0 pypi_0 pypi packaging 22.0 py39haa95532_0
pcre 8.45 hd77b12b_0
pillow 9.3.0 py39hd77b12b_2
pip 22.3.1 py39haa95532_0
ply 3.11 py39haa95532_0
protobuf 3.19.6 pypi_0 pypi psutil 5.9.0 py39h2bbff1b_0
pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pyparsing 3.0.9 py39haa95532_0
pyqt 5.15.7 py39hd77b12b_0
pyqt5-sip 12.11.0 py39hd77b12b_0
python 3.9.16 h6244533_0
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.9 2_cp39 conda-forge pywin32 305 py39h2bbff1b_0
pywinpty 2.0.2 py39h5da7b33_0
qt-main 5.15.2 he8e5bd7_7
qt-webengine 5.15.9 hb9a9bb5_5
qtwebkit 5.212 h3ad3cdb_4
requests 2.28.2 pypi_0 pypi requests-oauthlib 1.3.1 pypi_0 pypi rsa 4.9 pypi_0 pypi scikit-learn 1.1.2 py39hfd4428b_0 conda-forge scipy 1.8.1 py39h5567194_2 conda-forge setuptools 65.6.3 py39haa95532_0
sip 6.6.2 py39hd77b12b_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.40.1 h2bbff1b_0
tensorboard 2.10.1 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow-cpu 2.10.0 pypi_0 pypi tensorflow-directml-plugin 0.4.0.dev230202 pypi_0 pypi tensorflow-estimator 2.10.0 pypi_0 pypi tensorflow-intel 2.10.0 pypi_0 pypi tensorflow-io-gcs-filesystem 0.30.0 pypi_0 pypi tensorflow-probability 0.14.0 pyhd3eb1b0_0
termcolor 2.2.0 pypi_0 pypi threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tk 8.6.12 h2bbff1b_0
toml 0.10.2 pyhd3eb1b0_0
tornado 6.2 py39h2bbff1b_0
tqdm 4.64.1 py39haa95532_0
typing-extensions 4.4.0 py39haa95532_0
typing_extensions 4.4.0 py39haa95532_0
tzdata 2022g h04d1e81_0
urllib3 1.26.14 pypi_0 pypi vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 2.2.2 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0
wincertstore 0.2 py39haa95532_2
winpty 0.4.3 4
wrapt 1.14.1 pypi_0 pypi xz 5.2.10 h8cc25b3_1
zipp 3.12.1 pypi_0 pypi zlib 1.2.13 h8cc25b3_0
zstd 1.5.2 h19a0ad4_0 =============== State File ================= { "name": "villain", "sessions": { "1": { "timestamp": 1675806813.8550851, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 10, "iterations": 50000, "config": { "learning_rate": 5e-05, "epsilon_exponent": -7, "autoclip": false, "allow_growth": false, "mixed_precision": false, "nan_protection": true, "convert_batchsize": 16, "loss_function": "ssim", "loss_function_2": "mse", "loss_weight_2": 100, "loss_function_3": null, "loss_weight_3": 0, "loss_function_4": "ssim", "loss_weight_4": 0, "mask_loss_function": "mse", "eye_multiplier": 5, "mouth_multiplier": 15 } } }, "lowest_avg_loss": { "a": 0.10296544040739536, "b": 0.07080858370661736 }, "iterations": 50000, "mixed_precision_layers": [ "conv_128_0_conv2d", "leaky_re_lu", "residual_128_0_conv2d_0", "residual_128_0_leakyrelu_1", "residual_128_0_conv2d_1", "add", "residual_128_0_leakyrelu_3", "residual_128_1_conv2d_0", "residual_128_1_leakyrelu_1", "residual_128_1_conv2d_1", "add_1", "residual_128_1_leakyrelu_3", "residual_128_2_conv2d_0", "residual_128_2_leakyrelu_1", "residual_128_2_conv2d_1", "add_2", "residual_128_2_leakyrelu_3", "residual_128_3_conv2d_0", "residual_128_3_leakyrelu_1", "residual_128_3_conv2d_1", "add_3", "residual_128_3_leakyrelu_3", "residual_128_4_conv2d_0", "residual_128_4_leakyrelu_1", "residual_128_4_conv2d_1", "add_4", "residual_128_4_leakyrelu_3", "residual_128_5_conv2d_0", "residual_128_5_leakyrelu_1", "residual_128_5_conv2d_1", "add_5", "residual_128_5_leakyrelu_3", "residual_128_6_conv2d_0", "residual_128_6_leakyrelu_1", "residual_128_6_conv2d_1", "add_6", "residual_128_6_leakyrelu_3", "residual_128_7_conv2d_0", "residual_128_7_leakyrelu_1", "residual_128_7_conv2d_1", "add_7", "residual_128_7_leakyrelu_3", "residual_128_8_conv2d_0", "residual_128_8_leakyrelu_1", "residual_128_8_conv2d_1", "add_8", "residual_128_8_leakyrelu_3", "residual_128_9_conv2d_0", "residual_128_9_leakyrelu_1", "residual_128_9_conv2d_1", "add_9", "residual_128_9_leakyrelu_3", "residual_128_10_conv2d_0", "residual_128_10_leakyrelu_1", "residual_128_10_conv2d_1", "add_10", "residual_128_10_leakyrelu_3", "residual_128_11_conv2d_0", "residual_128_11_leakyrelu_1", "residual_128_11_conv2d_1", "add_11", "residual_128_11_leakyrelu_3", "residual_128_12_conv2d_0", "residual_128_12_leakyrelu_1", "residual_128_12_conv2d_1", "add_12", "residual_128_12_leakyrelu_3", "residual_128_13_conv2d_0", "residual_128_13_leakyrelu_1", "residual_128_13_conv2d_1", "add_13", "residual_128_13_leakyrelu_3", "residual_128_14_conv2d_0", "residual_128_14_leakyrelu_1", "residual_128_14_conv2d_1", "add_14", "residual_128_14_leakyrelu_3", "residual_128_15_conv2d_0", "residual_128_15_leakyrelu_1", "residual_128_15_conv2d_1", "add_15", "residual_128_15_leakyrelu_3", "leaky_re_lu_1", "add_16", "conv_128_1_conv2d", "conv_128_1_leakyrelu", "pixel_shuffler", "conv_128_2_conv2d", "conv_128_2_leakyrelu", "pixel_shuffler_1", "conv_128_3_conv2d", "conv_128_3_leakyrelu", "separableconv2d_256_0_seperableconv2d", "separableconv2d_256_0_relu", "conv_512_0_conv2d", "conv_512_0_leakyrelu", "separableconv2d_1024_0_seperableconv2d", "separableconv2d_1024_0_relu", "flatten", "dense", "dense_1", "reshape", "upscale_512_0_conv2d_conv2d", "upscale_512_0_conv2d_leakyrelu", "upscale_512_0_pixelshuffler", "upscale_512_1_conv2d_conv2d", "upscale_512_1_pixelshuffler", "leaky_re_lu_2", "residual_512_0_conv2d_0", "residual_512_0_leakyrelu_1", "residual_512_0_conv2d_1", "add_17", "residual_512_0_leakyrelu_3", "upscale_256_0_conv2d_conv2d", "upscale_256_0_pixelshuffler", "leaky_re_lu_3", "residual_256_0_conv2d_0", "residual_256_0_leakyrelu_1", "residual_256_0_conv2d_1", "add_18", "residual_256_0_leakyrelu_3", "upscale_128_0_conv2d_conv2d", "upscale_128_0_pixelshuffler", "leaky_re_lu_4", "residual_128_16_conv2d_0", "residual_128_16_leakyrelu_1", "residual_128_16_conv2d_1", "add_19", "residual_128_16_leakyrelu_3", "face_out_a_conv2d", "upscale_512_2_conv2d_conv2d", "upscale_512_2_pixelshuffler", "leaky_re_lu_5", "residual_512_1_conv2d_0", "residual_512_1_leakyrelu_1", "residual_512_1_conv2d_1", "add_20", "residual_512_1_leakyrelu_3", "upscale_256_1_conv2d_conv2d", "upscale_256_1_pixelshuffler", "leaky_re_lu_6", "residual_256_1_conv2d_0", "residual_256_1_leakyrelu_1", "residual_256_1_conv2d_1", "add_21", "residual_256_1_leakyrelu_3", "upscale_128_1_conv2d_conv2d", "upscale_128_1_pixelshuffler", "leaky_re_lu_7", "residual_128_17_conv2d_0", "residual_128_17_leakyrelu_1", "residual_128_17_conv2d_1", "add_22", "residual_128_17_leakyrelu_3", "face_out_b_conv2d" ], "config": { "centering": "face", "coverage": 87.5, "optimizer": "adam", "learning_rate": 5e-05, "epsilon_exponent": -7, "autoclip": false, "allow_growth": false, "mixed_precision": false, "nan_protection": true, "convert_batchsize": 16, "loss_function": "ssim", "loss_function_2": "mse", "loss_weight_2": 100, "loss_function_3": null, "loss_weight_3": 0, "loss_function_4": "ssim", "loss_weight_4": 0, "mask_loss_function": "mse", "eye_multiplier": 5, "mouth_multiplier": 15, "penalized_mask_loss": true, "mask_type": "vgg-obstructed", "mask_blur_kernel": 5, "mask_threshold": 4, "learn_mask": false, "lowmem": false } } ================= Configs ================== --------- .faceswap --------- backend: directml --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 erosion_top: 0.0 erosion_bottom: 0.0 erosion_left: 0.0 erosion_right: 0.0 [scaling.sharpen] method: none amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False separate_mask: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False separate_mask: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False aligner_min_scale: 0.07 aligner_max_scale: 2.0 aligner_distance: 22.5 aligner_roll: 45.0 aligner_features: True filter_refeed: True save_filtered: False realign_refeeds: True filter_realign: True [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 scalefactor: 0.709 batch-size: 8 cpu: True threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 [detect.s3fd] confidence: 75 batch-size: 6 [mask.bisenet_fp] batch-size: 8 cpu: False weights: faceswap include_ears: False include_hair: False include_glasses: True [mask.custom] batch-size: 8 centering: face fill: False [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 4 [recognition.vgg_face2] batch-size: 16 cpu: False --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] centering: face coverage: 87.5 icnr_init: True conv_aware_init: False optimizer: adam learning_rate: 5e-05 epsilon_exponent: -7 autoclip: False reflect_padding: False allow_growth: False mixed_precision: False nan_protection: True convert_batchsize: 16 [global.loss] loss_function: ssim loss_function_2: mse loss_weight_2: 100 loss_function_3: none loss_weight_3: 0 loss_function_4: ssim loss_weight_4: 0 mask_loss_function: mse eye_multiplier: 5 mouth_multiplier: 15 penalized_mask_loss: True mask_type: vgg-obstructed mask_blur_kernel: 5 mask_threshold: 4 learn_mask: False [model.dfaker] output_size: 128 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.phaze_a] output_size: 128 shared_fc: None enable_gblock: True split_fc: True split_gblock: False split_decoders: False enc_architecture: fs_original enc_scaling: 7 enc_load_weights: True bottleneck_type: dense bottleneck_norm: None bottleneck_size: 1024 bottleneck_in_encoder: True fc_depth: 1 fc_min_filters: 1024 fc_max_filters: 1024 fc_dimensions: 4 fc_filter_slope: -0.5 fc_dropout: 0.0 fc_upsampler: upsample2d fc_upsamples: 1 fc_upsample_filters: 512 fc_gblock_depth: 3 fc_gblock_min_nodes: 512 fc_gblock_max_nodes: 512 fc_gblock_filter_slope: -0.5 fc_gblock_dropout: 0.0 dec_upscale_method: subpixel dec_upscales_in_fc: 0 dec_norm: None dec_min_filters: 64 dec_max_filters: 512 dec_slope_mode: full dec_filter_slope: -0.45 dec_res_blocks: 1 dec_output_kernel: 5 dec_gaussian: True dec_skip_last_residual: True freeze_layers: keras_encoder load_layers: encoder fs_original_depth: 4 fs_original_min_filters: 128 fs_original_max_filters: 1024 fs_original_use_alt: False mobilenet_width: 1.0 mobilenet_depth: 1 mobilenet_dropout: 0.001 mobilenet_minimalistic: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Training Continuation Error: Unknown config_item: RandomNormal

Post by torzdf »

See here:
viewtopic.php?t=2055

Most likely a version conflict.

I find your post suspicious as it is almost a direct c+p from the above post, with a different crash report and some text changed.

Either way, assuming you aren't a spammer, and I won't be shortly having to remove you and your message from this board, you should try this:
app.php/faqpage#f1r1

My word is final

User avatar
chedger
Posts: 4
Joined: Thu Feb 09, 2023 12:06 am

Re: Training Continuation Error: Unknown config_item: RandomNormal

Post by chedger »

I did base my post off that one, sorry. It was Identical to my issue in every way, except I was not using Collab. Regardless, I tried completely reinstalling everything, and the problem persisted. Any other suggestions?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Training Continuation Error: Unknown config_item: RandomNormal

Post by torzdf »

You definitely fully removed everything (including the python folders) as suggested in the link? Ultimately "RandomNormal" is a standard keras initialization method, so the error message does not make sense if everything is up to date and the same.

The model file you are using is from the installed Faceswap you are currently using? It doesn't come from a different system/install?

Can you try starting new models with other trainers, and resuming and seeing if the issue persists over each version?

Please provide the model .h5 file (it will be large, so you will need to host it somewhere and link it) so I can have a look at the versioning.

Sorry for thinking you may be a spammer.... it is their MO to copy and paste existing messages in the hope of flying under the radar!

My word is final

User avatar
chedger
Posts: 4
Joined: Thu Feb 09, 2023 12:06 am

Re: Training Continuation Error: Unknown config_item: RandomNormal

Post by chedger »

I followed all the steps twice, and removed anything on my computer that said conda, python, or faceswap. Does that mean I am sure I followed the instructions correctly, no not really but I think I did. Here is the folder for the new model I just replicated it with, along with the crash report: https://drive.google.com/drive/folders/ ... sp=sharing

Last edited by chedger on Fri Feb 17, 2023 4:11 pm, edited 1 time in total.
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Training Continuation Error: Unknown config_item: RandomNormal

Post by torzdf »

Ok, I have tested your file and it won't load for me either (same error). When I create my own model and save + reload, it works fine (same settings as you).

The only time I have seen this before is because of a version conflict, where another version of Keras exists somewhere on your system, and is being loaded instead of the correct version.

All I can suggest is you double check the steps listed earlier and make sure you got rid of absolutely everything python related. Like I said before, RandomNormal is a standard Keras initializer. It is not a custom object. As this issue is coming from within Keras and I cannot recreate it, there is not much I can do to fix the issue.

Similar issues online suggest that the model file was created with an older version of Keras. As this model was not created with an older version of Keras, the only logical conclusion is that you have an older version somewhere on your system which is conflicting.

If you still can't get this working, then search you entire hard disk for files/folders containing "keras" and post the output.

Last edited by torzdf on Sat Feb 18, 2023 12:58 pm, edited 1 time in total.

My word is final

Locked