I'm unable to start training as I get NaN error every time.
Is it normal that "Loading Trainer from Original plugin" takes about 15 minutes to load?
Code: Select all
06/14/2021 08:00:19 MainProcess MainThread logger log_setup INFO Log level set to: INFO
06/14/2021 08:00:21 MainProcess MainThread train _get_images INFO Model A Directory: 'C:\Users\Henri\Desktop\Deepfake\FaceA' (2718 images)
06/14/2021 08:00:21 MainProcess MainThread train _get_images INFO Model B Directory: 'C:\Users\Henri\Desktop\Deepfake\FaceB' (2628 images)
06/14/2021 08:00:21 MainProcess MainThread train process INFO Training data directory: C:\Users\Henri\Desktop\Deepfake\Model
06/14/2021 08:00:21 MainProcess MainThread train _monitor INFO ===================================================
06/14/2021 08:00:21 MainProcess MainThread train _monitor INFO Starting
06/14/2021 08:00:21 MainProcess MainThread train _monitor INFO Press 'Stop' to save and quit
06/14/2021 08:00:21 MainProcess MainThread train _monitor INFO ===================================================
06/14/2021 08:00:22 MainProcess _training_0 train _training INFO Loading data, this may take a while...
06/14/2021 08:00:22 MainProcess _training_0 plugin_loader _import INFO Loading Model from Original plugin...
06/14/2021 08:00:22 MainProcess _training_0 _base _update_changed_config_items INFO Config item: 'allow_growth' has been updated from 'True' to 'False'
06/14/2021 08:00:22 MainProcess _training_0 _base _replace_config INFO Using configuration saved in state file
06/14/2021 08:05:14 MainProcess _training_0 _base _load INFO Loaded model from disk: 'C:\Users\Henri\Desktop\Deepfake\Model\original.h5'
06/14/2021 08:05:14 MainProcess _training_0 plugin_loader _import INFO Loading Trainer from Original plugin...
06/14/2021 08:05:16 MainProcess _training_0 ag_logging warn DEBUG AutoGraph could not transform <bound method Logger.isEnabledFor of <FaceswapLogger lib.model.losses_tf (DEBUG)>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
06/14/2021 08:05:16 MainProcess _training_0 ag_logging warn DEBUG AutoGraph could not transform <bound method Logger.findCaller of <FaceswapLogger lib.model.losses_tf (DEBUG)>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
06/14/2021 08:05:16 MainProcess _training_0 ag_logging warn DEBUG AutoGraph could not transform <bound method Logger.makeRecord of <FaceswapLogger lib.model.losses_tf (DEBUG)>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
06/14/2021 08:05:17 MainProcess _training_0 ag_logging warn DEBUG AutoGraph could not transform <bound method FaceswapFormatter.format of <lib.logger.FaceswapFormatter object at 0x00000166958A7F70>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
06/14/2021 08:05:17 MainProcess _training_0 ag_logging warn DEBUG AutoGraph could not transform <bound method LossWrapper._apply_mask of <class 'lib.model.losses_tf.LossWrapper'>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
06/14/2021 08:05:17 MainProcess _training_0 ag_logging warn DEBUG AutoGraph could not transform <bound method DSSIMObjective.call of <lib.model.losses_tf.DSSIMObjective object at 0x000001669EC32A60>> and will run it as-is.\nPlease report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\nCause: module 'gast' has no attribute 'Index'\nTo silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
06/14/2021 08:22:08 MainProcess _training_0 _base _collate_and_store_loss CRITICAL NaN Detected. Loss: [nan, nan]
06/14/2021 08:22:08 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...
06/14/2021 08:22:08 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training_0'
06/14/2021 08:22:08 MainProcess MainThread launcher execute_script ERROR A NaN was detected and you have NaN protection enabled. Training has been terminated.