Training Error, need assistance.

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Shiokkou
Posts: 1
Joined: Sun Jan 23, 2022 4:24 pm

Training Error, need assistance.

Post by Shiokkou »

Hi, never had this problem before and it just suddenly occurred recently, any help is appreciated! :D

GUI

Code: Select all

Loading...
Setting Faceswap backend to NVIDIA
01/23/2022 16:30:53 INFO     Log level set to: INFO
01/23/2022 16:30:56 INFO     Model A Directory: 'D:\Face\Faceswap Proj\FaceA' (9593 images)
01/23/2022 16:30:56 INFO     Model B Directory: 'D:\Face\Faceswap Proj\FaceB' (889 images)
01/23/2022 16:30:56 INFO     Training data directory: D:\Face\Faceswap Proj\ModelAB
01/23/2022 16:30:56 INFO     ===================================================
01/23/2022 16:30:56 INFO       Starting
01/23/2022 16:30:56 INFO       Press 'Stop' to save and quit
01/23/2022 16:30:56 INFO     ===================================================
01/23/2022 16:30:57 INFO     Loading data, this may take a while...
01/23/2022 16:30:57 INFO     Loading Model from Original plugin...
01/23/2022 16:30:57 INFO     No existing state file found. Generating.
01/23/2022 16:30:58 INFO     Loading Trainer from Original plugin...
01/23/2022 16:30:59 CRITICAL Error caught! Exiting...
01/23/2022 16:30:59 ERROR    Caught exception in thread: '_training_0'
01/23/2022 16:31:01 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "D:\Face\faceswap\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "D:\Face\faceswap\scripts\train.py", line 190, in process
    self._end_thread(thread, err)
  File "D:\Face\faceswap\scripts\train.py", line 230, in _end_thread
    thread.join()
  File "D:\Face\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "D:\Face\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Face\faceswap\scripts\train.py", line 252, in _training
    raise err
  File "D:\Face\faceswap\scripts\train.py", line 241, in _training
    trainer = self._load_trainer(model)
  File "D:\Face\faceswap\scripts\train.py", line 287, in _load_trainer
    trainer = trainer(model,
  File "D:\Face\faceswap\plugins\train\trainer\original.py", line 10, in __init__
    super().__init__(*args, **kwargs)
  File "D:\Face\faceswap\plugins\train\trainer\_base.py", line 79, in __init__
    self._tensorboard = self._set_tensorboard()
  File "D:\Face\faceswap\plugins\train\trainer\_base.py", line 142, in _set_tensorboard
    tensorboard.set_model(self._model.model)
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\callbacks.py", line 2237, in set_model
    self._write_keras_model_summary()
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\callbacks.py", line 2276, in _write_keras_model_summary
    with self._train_writer.as_default():
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\callbacks.py", line 2245, in _train_writer
    self._writers['train'] = summary_ops_v2.create_file_writer_v2(
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py", line 553, in create_file_writer_v2
    return _ResourceSummaryWriter(
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py", line 313, in __init__
    self._init_op = init_op_fn(self._resource)
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_summary_ops.py", line 145, in create_summary_file_writer
    _ops.raise_from_not_ok_status(e, name)
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py", line 6941, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Failed to create a NewWriteableFile: D:\Face\Faceswap Proj\ModelAB\original_logs\session_1\train/events.out.tfevents.1642955458.123???.22172.0.v2 : The filename, directory name, or volume label syntax is incorrect.

; no protocol option
	Creating writable file D:\Face\Faceswap Proj\ModelAB\original_logs\session_1\train/events.out.tfevents.1642955458.123???.22172.0.v2
	Could not initialize events writer. [Op:CreateSummaryFileWriter]
01/23/2022 16:31:01 CRITICAL An unexpected crash has occurred. Crash report written to 'D:\Face\faceswap\crash_report.2022.01.23.163059142488.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Process exited.

Crash Report

Code: Select all

01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'flip_chance')
01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 50)
01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'color_lightness')
01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 30)
01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'color_ab')
01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 8)
01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'color_clahe_chance')
01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 50)
01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'color_clahe_max_size')
01/23/2022 16:30:58 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 4)
01/23/2022 16:30:58 MainProcess     _training_0                    _base           add_session_batchsize          DEBUG    Adding session batch size: 16
01/23/2022 16:30:58 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Feeder: num_images: 2, batch_size: 16, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
01/23/2022 16:30:58 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
01/23/2022 16:30:58 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
01/23/2022 16:30:58 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 9593, batchsize: 16, side: 'a', do_shuffle: True, is_preview, False, is_timelapse: False)
01/23/2022 16:30:58 MainProcess     _training_0                    generator       _get_cache                     DEBUG    Creating cache. Side: a
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 16, is_display: False, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
01/23/2022 16:30:58 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 9593, side: 'a', do_shuffle: True)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
01/23/2022 16:30:58 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 9593, side: 'a', do_shuffle: True)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
01/23/2022 16:30:58 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
01/23/2022 16:30:58 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
01/23/2022 16:30:58 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 889, batchsize: 16, side: 'b', do_shuffle: True, is_preview, False, is_timelapse: False)
01/23/2022 16:30:58 MainProcess     _training_0                    generator       _get_cache                     DEBUG    Creating cache. Side: b
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 16, is_display: False, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
01/23/2022 16:30:58 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 889, side: 'b', do_shuffle: True)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
01/23/2022 16:30:58 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 889, side: 'b', do_shuffle: True)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Setting preview feed: (side: 'a')
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
01/23/2022 16:30:58 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
01/23/2022 16:30:58 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
01/23/2022 16:30:58 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 9593, batchsize: 14, side: 'a', do_shuffle: True, is_preview, True, is_timelapse: False)
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
01/23/2022 16:30:58 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 9593, side: 'a', do_shuffle: True)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
01/23/2022 16:30:58 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 9593, side: 'a', do_shuffle: True)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Setting preview feed: (side: 'b')
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
01/23/2022 16:30:58 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
01/23/2022 16:30:58 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
01/23/2022 16:30:58 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 889, batchsize: 14, side: 'b', do_shuffle: True, is_preview, True, is_timelapse: False)
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.875, config: {'centering': 'face', 'coverage': 87.5, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
01/23/2022 16:30:58 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
01/23/2022 16:30:58 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 889, side: 'b', do_shuffle: True)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
01/23/2022 16:30:58 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 889, side: 'b', do_shuffle: True)
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Set preview feed. Batchsize: 14
01/23/2022 16:30:58 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Feeder:
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Enabling TensorBoard Logging
01/23/2022 16:30:58 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Setting up TensorBoard Logging
01/23/2022 16:30:58 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): Failed to create a NewWriteableFile: D:\Face\Faceswap Proj\ModelAB\original_logs\session_1\train/events.out.tfevents.1642955458.123???.22172.0.v2 : The filename, directory name, or volume label syntax is incorrect.\r\n; no protocol option\n	Creating writable file D:\Face\Faceswap Proj\ModelAB\original_logs\session_1\train/events.out.tfevents.1642955458.123???.22172.0.v2\n	Could not initialize events writer. [Op:CreateSummaryFileWriter]
01/23/2022 16:30:58 MainProcess     _run_0                         generator       _validate_version              DEBUG    Setting initial extract version: 2.2
01/23/2022 16:30:58 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initializing constants. training_size: 384
01/23/2022 16:30:58 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(24, 360, None), 'warp_mapx': '[[[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]]', 'warp_mapy': '[[[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
01/23/2022 16:30:58 MainProcess     _run_0                         generator       _validate_version              DEBUG    Setting initial extract version: 2.2
01/23/2022 16:30:59 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initializing constants. training_size: 384
01/23/2022 16:30:59 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(24, 360, None), 'warp_mapx': '[[[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]]', 'warp_mapy': '[[[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
01/23/2022 16:30:59 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initializing constants. training_size: 384
01/23/2022 16:30:59 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
01/23/2022 16:30:59 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
01/23/2022 16:30:59 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
01/23/2022 16:30:59 MainProcess     _run_0                         augmentation    initialize                     DEBUG    Initialized constants: {'clahe_base_contrast': 3, 'tgt_slices': slice(24, 360, None), 'warp_mapx': '[[[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]\n\n [[ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]\n  [ 24. 108. 192. 276. 360.]]]', 'warp_mapy': '[[[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]\n\n [[ 24.  24.  24.  24.  24.]\n  [108. 108. 108. 108. 108.]\n  [192. 192. 192. 192. 192.]\n  [276. 276. 276. 276. 276.]\n  [360. 360. 360. 360. 360.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]\n\n [[  0   0]\n  [  0 383]\n  [383 383]\n  [383   0]\n  [191   0]\n  [191 383]\n  [383 191]\n  [  0 191]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [381. 381. 381. ... 381. 381. 381.]\n  [382. 382. 382. ... 382. 382. 382.]\n  [383. 383. 383. ... 383. 383. 383.]]\n\n [[  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  ...\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]\n  [  0.   1.   2. ... 381. 382. 383.]]]'}
01/23/2022 16:30:59 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
01/23/2022 16:30:59 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
01/23/2022 16:30:59 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
01/23/2022 16:30:59 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "D:\Face\faceswap\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "D:\Face\faceswap\scripts\train.py", line 190, in process
    self._end_thread(thread, err)
  File "D:\Face\faceswap\scripts\train.py", line 230, in _end_thread
    thread.join()
  File "D:\Face\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "D:\Face\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Face\faceswap\scripts\train.py", line 252, in _training
    raise err
  File "D:\Face\faceswap\scripts\train.py", line 241, in _training
    trainer = self._load_trainer(model)
  File "D:\Face\faceswap\scripts\train.py", line 287, in _load_trainer
    trainer = trainer(model,
  File "D:\Face\faceswap\plugins\train\trainer\original.py", line 10, in __init__
    super().__init__(*args, **kwargs)
  File "D:\Face\faceswap\plugins\train\trainer\_base.py", line 79, in __init__
    self._tensorboard = self._set_tensorboard()
  File "D:\Face\faceswap\plugins\train\trainer\_base.py", line 142, in _set_tensorboard
    tensorboard.set_model(self._model.model)
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\callbacks.py", line 2237, in set_model
    self._write_keras_model_summary()
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\callbacks.py", line 2276, in _write_keras_model_summary
    with self._train_writer.as_default():
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\callbacks.py", line 2245, in _train_writer
    self._writers['train'] = summary_ops_v2.create_file_writer_v2(
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py", line 553, in create_file_writer_v2
    return _ResourceSummaryWriter(
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py", line 313, in __init__
    self._init_op = init_op_fn(self._resource)
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_summary_ops.py", line 145, in create_summary_file_writer
    _ops.raise_from_not_ok_status(e, name)
  File "C:\Users\jianj\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py", line 6941, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Failed to create a NewWriteableFile: D:\Face\Faceswap Proj\ModelAB\original_logs\session_1\train/events.out.tfevents.1642955458.123???.22172.0.v2 : The filename, directory name, or volume label syntax is incorrect.
; no protocol option
	Creating writable file D:\Face\Faceswap Proj\ModelAB\original_logs\session_1\train/events.out.tfevents.1642955458.123???.22172.0.v2
	Could not initialize events writer. [Op:CreateSummaryFileWriter]

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         183aee3 bugfix: Pin pynvml to <11.515
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce RTX 3060 Ti
gpu_devices_active:  GPU_0
gpu_driver:          497.09
gpu_vram:            GPU_0: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.22000-SP0
os_release:          10
py_command:          D:\Face\faceswap\faceswap.py train -A D:/Face/Faceswap Proj/FaceA -B D:/Face/Faceswap Proj/FaceB -m D:/Face/Faceswap Proj/ModelAB -t original -bs 16 -it 1000000 -s 250 -ss 25000 -tia D:/Face/Faceswap Proj/FaceA -tib D:/Face/Faceswap Proj/FaceB -to D:/Face/Faceswap Proj/TimelapseAB -ps 100 -L INFO -gui
py_conda_version:    conda 4.11.0
py_implementation:   CPython
py_version:          3.8.12
py_virtual_env:      True
sys_cores:           16
sys_processor:       AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
sys_ram:             Total: 16299MB, Available: 7414MB, Used: 8884MB, Free: 7414MB

=============== Pip Packages ===============
absl-py==0.15.0
astunparse==1.6.3
cachetools==4.2.4
certifi==2021.10.8
charset-normalizer==2.0.10
clang==5.0
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
fastcluster==1.1.26
ffmpy==0.2.3
flatbuffers==1.12
gast==0.4.0
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.43.0
h5py==3.1.0
idna==3.3
imageio @ file:///tmp/build/80754af9/imageio_1617700267927/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1629987409325/work
importlib-metadata==4.10.1
joblib @ file:///tmp/build/80754af9/joblib_1635411271373/work
keras==2.6.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/ci/kiwisolver_1612282606037/work
Markdown==3.3.6
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.3.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1603466732592/work
nvidia-ml-py==11.495.46
oauthlib==3.1.1
olefile @ file:///Users/ktietz/demo/mc3/conda-bld/olefile_1629805411829/work
opencv-python==4.5.5.62
opt-einsum==3.3.0
Pillow==8.4.0
protobuf==3.19.3
psutil @ file:///C:/ci/psutil_1612298324802/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing @ file:///tmp/build/80754af9/pyparsing_1635766073266/work
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==302
requests==2.27.1
requests-oauthlib==1.3.0
rsa==4.8
scikit-learn @ file:///C:/ci/scikit-learn_1642599125707/work
scipy @ file:///C:/ci/scipy_1616703433439/work
sip==4.19.13
six==1.15.0
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.6.0
tensorflow-gpu==2.6.2
termcolor==1.1.0
threadpoolctl @ file:///Users/ktietz/demo/mc3/conda-bld/threadpoolctl_1629802263681/work
tornado @ file:///C:/ci/tornado_1606942392901/work
tqdm @ file:///tmp/build/80754af9/tqdm_1635330843403/work
typing-extensions==3.7.4.3
urllib3==1.26.8
Werkzeug==2.0.2
wincertstore==0.2
wrapt==1.12.1
zipp==3.7.0

============== Conda Packages ==============
# packages in environment at C:\Users\jianj\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   0.15.0                   pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
ca-certificates 2021.10.8 h5b45459_0 conda-forge cachetools 4.2.4 pypi_0 pypi certifi 2021.10.8 py38haa244fe_1 conda-forge charset-normalizer 2.0.10 pypi_0 pypi clang 5.0 pypi_0 pypi cudatoolkit 11.2.2 h933977f_9 conda-forge cudnn 8.1.0.77 h3e0f4f4_0 conda-forge cycler 0.11.0 pyhd3eb1b0_0
fastcluster 1.1.26 py38h5d928e2_3 conda-forge ffmpeg 4.3.1 ha925a31_0 conda-forge ffmpy 0.2.3 pypi_0 pypi flatbuffers 1.12 pypi_0 pypi freetype 2.10.4 hd328e21_0
gast 0.4.0 pypi_0 pypi git 2.34.1 haa95532_0
google-auth 1.35.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.43.0 pypi_0 pypi h5py 3.1.0 pypi_0 pypi icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 3.3 pypi_0 pypi imageio 2.9.0 pyhd3eb1b0_0
imageio-ffmpeg 0.4.5 pyhd8ed1ab_0 conda-forge importlib-metadata 4.10.1 pypi_0 pypi intel-openmp 2022.0.0 haa95532_3663
joblib 1.1.0 pyhd3eb1b0_0
jpeg 9d h2bbff1b_0
keras 2.6.0 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.3.1 py38hd77b12b_0
libpng 1.6.37 h2a8f88b_0
libtiff 4.2.0 hd0e1b90_0
libwebp 1.2.0 h2bbff1b_0
lz4-c 1.9.3 h2bbff1b_1
markdown 3.3.6 pypi_0 pypi matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38h64f37c6_0
mkl 2020.2 256
mkl-service 2.3.0 py38h196d8e1_0
mkl_fft 1.3.0 py38h46781fe_0
mkl_random 1.1.1 py38h47e9c7a_0
numpy 1.19.2 py38hadc3359_0
numpy-base 1.19.2 py38ha3acd2a_0
nvidia-ml-py 11.495.46 pypi_0 pypi oauthlib 3.1.1 pypi_0 pypi olefile 0.46 pyhd3eb1b0_0
opencv-python 4.5.5.62 pypi_0 pypi openssl 1.1.1l h8ffe710_0 conda-forge opt-einsum 3.3.0 pypi_0 pypi pillow 8.4.0 py38hd45dc43_0
pip 21.2.2 py38haa95532_0
protobuf 3.19.3 pypi_0 pypi psutil 5.8.0 py38h2bbff1b_1
pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pyparsing 3.0.4 pyhd3eb1b0_0
pyqt 5.9.2 py38ha925a31_4
python 3.8.12 h6244533_0
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.8 2_cp38 conda-forge pywin32 302 py38h827c3e9_1
qt 5.9.7 vc14h73c81de_0
requests 2.27.1 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rsa 4.8 pypi_0 pypi scikit-learn 1.0.2 py38hf11a4ad_1
scipy 1.6.2 py38h14eb087_0
setuptools 58.0.4 py38haa95532_0
sip 4.19.13 py38ha925a31_0
six 1.15.0 pypi_0 pypi sqlite 3.37.0 h2bbff1b_0
tensorboard 2.6.0 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow-estimator 2.6.0 pypi_0 pypi tensorflow-gpu 2.6.2 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi threadpoolctl 2.2.0 pyh0d69192_0
tk 8.6.11 h2bbff1b_0
tornado 6.1 py38h2bbff1b_0
tqdm 4.62.3 pyhd3eb1b0_1
typing-extensions 3.7.4.3 pypi_0 pypi urllib3 1.26.8 pypi_0 pypi vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 2.0.2 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0
wincertstore 0.2 py38haa95532_2
wrapt 1.12.1 pypi_0 pypi xz 5.2.5 h62dcd97_0
zipp 3.7.0 pypi_0 pypi zlib 1.2.11 h8cc25b3_4
zstd 1.4.9 h19a0ad4_0 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: none amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 scalefactor: 0.709 batch-size: 8 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 [detect.s3fd] confidence: 70 batch-size: 4 [mask.bisenet_fp] batch-size: 8 include_ears: False include_hair: False include_glasses: True [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] centering: face coverage: 87.5 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 epsilon_exponent: -7 reflect_padding: False allow_growth: False mixed_precision: False nan_protection: True convert_batchsize: 16 [global.loss] loss_function: ssim mask_loss_function: mse l2_reg_term: 100 eye_multiplier: 3 mouth_multiplier: 2 penalized_mask_loss: True mask_type: extended mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False [model.dfaker] output_size: 128 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.phaze_a] output_size: 128 shared_fc: None enable_gblock: True split_fc: True split_gblock: False split_decoders: False enc_architecture: fs_original enc_scaling: 40 enc_load_weights: True bottleneck_type: dense bottleneck_norm: None bottleneck_size: 1024 bottleneck_in_encoder: True fc_depth: 1 fc_min_filters: 1024 fc_max_filters: 1024 fc_dimensions: 4 fc_filter_slope: -0.5 fc_dropout: 0.0 fc_upsampler: upsample2d fc_upsamples: 1 fc_upsample_filters: 512 fc_gblock_depth: 3 fc_gblock_min_nodes: 512 fc_gblock_max_nodes: 512 fc_gblock_filter_slope: -0.5 fc_gblock_dropout: 0.0 dec_upscale_method: subpixel dec_norm: None dec_min_filters: 64 dec_max_filters: 512 dec_filter_slope: -0.45 dec_res_blocks: 1 dec_output_kernel: 5 dec_gaussian: True dec_skip_last_residual: True freeze_layers: keras_encoder load_layers: encoder fs_original_depth: 4 fs_original_min_filters: 128 fs_original_max_filters: 1024 mobilenet_width: 1.0 mobilenet_depth: 1 mobilenet_dropout: 0.001 [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
User avatar
torzdf
Posts: 2672
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Training Error, need assistance.

Post by torzdf »

I know what the issue is, I don't know why the issue is (have not seen before).

Ultimately TF is trying to create logfiles with question marks (?) in the filename. This is invalid, thus the error. You will need to look into why TF is trying to insert question marks into the logfile names. Feel free to report back here with your findings.

Specifically:

Code: Select all

1642955458.123???.22172.0.v2 : The filename, directory name, or volume label syntax is incorrect.

My word is final

Locked