Failure to create Tensorboard logging folder

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
visionaryfire
Posts: 1
Joined: Thu Jul 22, 2021 12:42 am

Failure to create Tensorboard logging folder

Post by visionaryfire »

Code: Select all

CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\Visionary2\faceswap\crash_report.2021.07.21.204451304634.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Process exited.

Code: Select all

07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'preview_images')
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 14)
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'zoom_amount')
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 5)
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'rotation_range')
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 10)
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'shift_range')
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 5)
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'flip_chance')
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 50)
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'color_lightness')
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 30)
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'color_ab')
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 8)
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'color_clahe_chance')
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 50)
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Getting config item: (section: 'trainer.original', option: 'color_clahe_max_size')
07/21/2021 20:40:51 MainProcess     _training_0                    config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 4)
07/21/2021 20:40:51 MainProcess     _training_0                    _base           add_session_batchsize          DEBUG    Adding session batch size: 16
07/21/2021 20:40:51 MainProcess     _training_0                    _base           __init__                       DEBUG    Initializing _Feeder: num_images: 2, batch_size: 16, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
07/21/2021 20:40:51 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
07/21/2021 20:40:51 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
07/21/2021 20:40:51 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 54, batchsize: 16, side: 'a', do_shuffle: True, is_preview, False, is_timelapse: False)
07/21/2021 20:40:51 MainProcess     _training_0                    generator       _get_cache                     DEBUG    Creating cache. Side: a
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 16, is_display: False, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
07/21/2021 20:40:51 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 54, side: 'a', do_shuffle: True)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
07/21/2021 20:40:51 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 54, side: 'a', do_shuffle: True)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
07/21/2021 20:40:51 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
07/21/2021 20:40:51 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
07/21/2021 20:40:51 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 54, batchsize: 16, side: 'b', do_shuffle: True, is_preview, False, is_timelapse: False)
07/21/2021 20:40:51 MainProcess     _training_0                    generator       _get_cache                     DEBUG    Creating cache. Side: b
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 16, is_display: False, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
07/21/2021 20:40:51 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 54, side: 'b', do_shuffle: True)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
07/21/2021 20:40:51 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 54, side: 'b', do_shuffle: True)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Setting preview feed: (side: 'a')
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
07/21/2021 20:40:51 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
07/21/2021 20:40:51 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
07/21/2021 20:40:51 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 54, batchsize: 14, side: 'a', do_shuffle: True, is_preview, True, is_timelapse: False)
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
07/21/2021 20:40:51 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 54, side: 'a', do_shuffle: True)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
07/21/2021 20:40:51 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 54, side: 'a', do_shuffle: True)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Setting preview feed: (side: 'b')
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _load_generator                DEBUG    Loading generator
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _load_generator                DEBUG    input_size: 64, output_shapes: [(64, 64, 3)]
07/21/2021 20:40:51 MainProcess     _training_0                    generator       __init__                       DEBUG    Initializing TrainingDataGenerator: (model_input_size: 64, model_output_shapes: [(64, 64, 3)], coverage_ratio: 0.6875, color_order: bgr, augment_color: True, no_flip: False, no_warp: False, warp_to_landmarks: False, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
07/21/2021 20:40:51 MainProcess     _training_0                    generator       __init__                       DEBUG    Initialized TrainingDataGenerator
07/21/2021 20:40:51 MainProcess     _training_0                    generator       minibatch_ab                   DEBUG    Queue batches: (image_count: 54, batchsize: 14, side: 'b', do_shuffle: True, is_preview, True, is_timelapse: False)
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 64, output_shapes: [(64, 64, 3)], coverage_ratio: 0.6875, config: {'centering': 'face', 'coverage': 68.75, 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'epsilon_exponent': -7, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'mask_loss_function': 'mse', 'l2_reg_term': 100, 'eye_multiplier': 3, 'mouth_multiplier': 2, 'penalized_mask_loss': True, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Output sizes: [64]
07/21/2021 20:40:51 MainProcess     _training_0                    augmentation    __init__                       DEBUG    Initialized ImageAugmentation
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  __init__                       DEBUG    Initialized BackgroundGenerator: '_run'
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread(s): '_run'
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 1 of 2: '_run_0'
07/21/2021 20:40:51 MainProcess     _run_0                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 54, side: 'b', do_shuffle: True)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Starting thread 2 of 2: '_run_1'
07/21/2021 20:40:51 MainProcess     _run_1                         generator       _minibatch                     DEBUG    Loading minibatch generator: (image_count: 54, side: 'b', do_shuffle: True)
07/21/2021 20:40:51 MainProcess     _training_0                    multithreading  start                          DEBUG    Started all threads '_run': 2
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _set_preview_feed              DEBUG    Set preview feed. Batchsize: 14
07/21/2021 20:40:51 MainProcess     _training_0                    _base           __init__                       DEBUG    Initialized _Feeder:
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Enabling TensorBoard Logging
07/21/2021 20:40:51 MainProcess     _training_0                    _base           _set_tensorboard               DEBUG    Setting up TensorBoard Logging
07/21/2021 20:43:55 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): Failed to create a directory: G:\Shared drives\VFX_PROJECTS_2021\MATTHEW_FACE SWAP\ASSETS_FROM CLIENT\test\MODELAB\original_logs\session_2/train; Invalid argument [Op:CreateSummaryFileWriter]
07/21/2021 20:43:56 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
07/21/2021 20:43:56 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
07/21/2021 20:43:56 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
07/21/2021 20:43:56 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
07/21/2021 20:43:56 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
07/21/2021 20:43:56 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
07/21/2021 20:43:56 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "C:\Users\Visionary2\faceswap\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "C:\Users\Visionary2\faceswap\scripts\train.py", line 190, in process
    self._end_thread(thread, err)
  File "C:\Users\Visionary2\faceswap\scripts\train.py", line 230, in _end_thread
    thread.join()
  File "C:\Users\Visionary2\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\Visionary2\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Visionary2\faceswap\scripts\train.py", line 252, in _training
    raise err
  File "C:\Users\Visionary2\faceswap\scripts\train.py", line 241, in _training
    trainer = self._load_trainer(model)
  File "C:\Users\Visionary2\faceswap\scripts\train.py", line 287, in _load_trainer
    trainer = trainer(model,
  File "C:\Users\Visionary2\faceswap\plugins\train\trainer\original.py", line 10, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\Visionary2\faceswap\plugins\train\trainer\_base.py", line 79, in __init__
    self._tensorboard = self._set_tensorboard()
  File "C:\Users\Visionary2\faceswap\plugins\train\trainer\_base.py", line 142, in _set_tensorboard
    tensorboard.set_model(self._model.model)
  File "C:\Users\Visionary2\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\callbacks.py", line 1980, in set_model
    self._write_keras_model_graph()
  File "C:\Users\Visionary2\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\callbacks.py", line 2009, in _write_keras_model_graph
    with self._train_writer.as_default():
  File "C:\Users\Visionary2\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\callbacks.py", line 1987, in _train_writer
    self._writers['train'] = summary_ops_v2.create_file_writer_v2(
  File "C:\Users\Visionary2\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py", line 412, in create_file_writer_v2
    return ResourceSummaryWriter(
  File "C:\Users\Visionary2\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py", line 242, in __init__
    self._init_op = init_op_fn(self._resource)
  File "C:\Users\Visionary2\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\ops\gen_summary_ops.py", line 148, in create_summary_file_writer
    _ops.raise_from_not_ok_status(e, name)
  File "C:\Users\Visionary2\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Failed to create a directory: G:\Shared drives\VFX_PROJECTS_2021\MATTHEW_FACE SWAP\ASSETS_FROM CLIENT\test\MODELAB\original_logs\session_2/train; Invalid argument [Op:CreateSummaryFileWriter]

============ System Information ============
encoding:            cp1252
git_branch:          Not Found
git_commits:         Not Found
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1080 Ti, GPU_1: GeForce GTX 1080 Ti
gpu_devices_active:  GPU_0, GPU_1
gpu_driver:          457.30
gpu_vram:            GPU_0: 11264MB, GPU_1: 11264MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.17763-SP0
os_release:          10
py_command:          C:\Users\Visionary2\faceswap\faceswap.py train -A G:/Shared drives/VFX_PROJECTS_2021/MATTHEW_FACE SWAP/ASSETS_FROM CLIENT/test/FACE_A -B G:/Shared drives/VFX_PROJECTS_2021/MATTHEW_FACE SWAP/ASSETS_FROM CLIENT/test/FACE_B -m G:/Shared drives/VFX_PROJECTS_2021/MATTHEW_FACE SWAP/ASSETS_FROM CLIENT/test/MODELAB -t original -bs 16 -it 1000000 -s 250 -ss 25000 -tia G:/Shared drives/VFX_PROJECTS_2021/MATTHEW_FACE SWAP/ASSETS_FROM CLIENT/test/FACE_A -tib G:/Shared drives/VFX_PROJECTS_2021/MATTHEW_FACE SWAP/ASSETS_FROM CLIENT/test/FACE_B -to G:/Shared drives/VFX_PROJECTS_2021/MATTHEW_FACE SWAP/ASSETS_FROM CLIENT/test/TIMELAPSEAB -ps 100 -L INFO -gui
py_conda_version:    conda 4.10.3
py_implementation:   CPython
py_version:          3.8.10
py_virtual_env:      True
sys_cores:           16
sys_processor:       Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
sys_ram:             Total: 130770MB, Available: 119277MB, Used: 11492MB, Free: 119277MB

=============== Pip Packages ===============
absl-py @ file:///C:/ci/absl-py_1623867339542/work
aiohttp @ file:///C:/ci/aiohttp_1614361024229/work
astor==0.8.1
astunparse==1.6.3
async-timeout==3.0.1
attrs @ file:///tmp/build/80754af9/attrs_1620827162558/work
beautifulsoup4==4.9.3
blinker==1.4
brotlipy==0.7.0
bs4==0.0.1
cachetools @ file:///tmp/build/80754af9/cachetools_1619597386817/work
certifi==2021.5.30
cffi @ file:///C:/ci/cffi_1625831749120/work
chardet @ file:///C:/ci/chardet_1605303225733/work
click @ file:///tmp/build/80754af9/click_1621604852318/work
coverage @ file:///C:/ci/coverage_1614615074147/work
cryptography @ file:///C:/ci/cryptography_1616769344312/work
cycler==0.10.0
Cython @ file:///C:/ci/cython_1626257071357/work
fastcluster==1.1.26
ffmpy==0.2.3
gast @ file:///tmp/build/80754af9/gast_1597433534803/work
google-auth @ file:///tmp/build/80754af9/google-auth_1626320605116/work
google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1617120569401/work
google-pasta==0.2.0
grpcio @ file:///C:/ci/grpcio_1614884412260/work
h5py==2.10.0
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
imageio @ file:///tmp/build/80754af9/imageio_1617700267927/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1621542018480/work
importlib-metadata @ file:///C:/ci/importlib-metadata_1617877484576/work
joblib @ file:///tmp/build/80754af9/joblib_1613502643832/work
Keras-Applications @ file:///tmp/build/80754af9/keras-applications_1594366238411/work
Keras-Preprocessing @ file:///tmp/build/80754af9/keras-preprocessing_1612283640596/work
kiwisolver @ file:///C:/ci/kiwisolver_1612282606037/work
Markdown @ file:///C:/ci/markdown_1614364121613/work
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.3.0
mkl-random==1.1.1
mkl-service==2.3.0
multidict @ file:///C:/ci/multidict_1607362065515/work
numpy==1.19.4
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib @ file:///tmp/build/80754af9/oauthlib_1623060228408/work
olefile==0.46
opencv-python==4.5.3.56
opt-einsum @ file:///tmp/build/80754af9/opt_einsum_1621500238896/work
pandas==1.2.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1625663293593/work
protobuf==3.14.0
psutil @ file:///C:/ci/psutil_1612298324802/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
PyJWT @ file:///C:/ci/pyjwt_1619651841937/work
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
pyreadline==2.1
PySocks @ file:///C:/ci/pysocks_1605287845585/work
python-dateutil==2.8.1
pywin32==227
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-oauthlib==1.3.0
rsa @ file:///tmp/build/80754af9/rsa_1614366226499/work
scikit-learn @ file:///C:/ci/scikit-learn_1622739500535/work
scipy @ file:///C:/ci/scipy_1616703433439/work
selenium==3.141.0
sip==4.19.13
six==1.15.0
soupsieve==2.1
tensorboard @ file:///home/builder/ktietz/aggregate/tensorflow_recipes/ci_te/tensorboard_1614593728657/work/tmp_pip_dir
tensorboard-plugin-wit==1.6.0
tensorflow==2.3.0
tensorflow-estimator @ file:///home/builder/ktietz/aggregate/tensorflow_recipes/ci_baze37/tensorflow-estimator_1622026529081/work/tensorflow_estimator-2.5.0-py2.py3-none-any.whl
termcolor==1.1.0
threadpoolctl @ file:///tmp/build/80754af9/threadpoolctl_1626115094421/work
tornado @ file:///C:/ci/tornado_1606942392901/work
tqdm @ file:///tmp/build/80754af9/tqdm_1625563689033/work
typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1624965014186/work
urllib3==1.26.2
Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/work
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
wincertstore==0.2
wrapt==1.12.1
yarl @ file:///C:/ci/yarl_1606940076464/work
zipp @ file:///tmp/build/80754af9/zipp_1625570634446/work

============== Conda Packages ==============
# packages in environment at C:\Users\Visionary2\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.3.0                       gpu  
absl-py                   0.13.0           py38haa95532_0  
aiohttp                   3.7.4            py38h2bbff1b_1  
astor                     0.8.1            py38haa95532_0  
astunparse                1.6.3                      py_0  
async-timeout             3.0.1            py38haa95532_0  
attrs                     21.2.0             pyhd3eb1b0_0  
blas                      1.0                         mkl  
blinker                   1.4              py38haa95532_0  
brotlipy                  0.7.0           py38h2bbff1b_1003  
ca-certificates           2021.5.30            h5b45459_0    conda-forge
cachetools                4.2.2              pyhd3eb1b0_0  
certifi                   2021.5.30        py38haa244fe_0    conda-forge
cffi                      1.14.6           py38h2bbff1b_0  
chardet                   3.0.4           py38haa95532_1003  
click                     8.0.1              pyhd3eb1b0_0  
coverage                  5.5              py38h2bbff1b_2  
cryptography              3.4.7            py38h71e12ea_0  
cudatoolkit               10.1.243             h74a9793_0  
cudnn                     7.6.5                cuda10.1_0  
cycler                    0.10.0                   py38_0  
cython                    0.29.24          py38hd77b12b_0  
fastcluster               1.1.26           py38h251f6bf_2    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
freetype                  2.10.4               hd328e21_0  
gast                      0.4.0                      py_0  
git                       2.23.0               h6bb4b03_0  
google-auth               1.33.0             pyhd3eb1b0_0  
google-auth-oauthlib      0.4.4              pyhd3eb1b0_0  
google-pasta              0.2.0                      py_0  
grpcio                    1.36.1           py38hc60d5dd_1  
h5py                      2.10.0           py38h5e291fa_0  
hdf5                      1.10.4               h7ebc959_0  
icc_rt                    2019.0.0             h0cc432a_1  
icu                       58.2                 ha925a31_3  
idna                      2.10               pyhd3eb1b0_0  
imageio                   2.9.0              pyhd3eb1b0_0  
imageio-ffmpeg            0.4.4              pyhd8ed1ab_0    conda-forge
importlib-metadata        3.10.0           py38haa95532_0  
intel-openmp              2021.3.0          haa95532_3372  
joblib                    1.0.1              pyhd3eb1b0_0  
jpeg                      9b                   hb83a4c4_2  
keras-applications        1.0.8                      py_1  
keras-preprocessing       1.1.2              pyhd3eb1b0_0  
kiwisolver                1.3.1            py38hd77b12b_0  
libpng                    1.6.37               h2a8f88b_0  
libprotobuf               3.14.0               h23ce68f_0  
libtiff                   4.2.0                hd0e1b90_0  
lz4-c                     1.9.3                h2bbff1b_0  
markdown                  3.3.4            py38haa95532_0  
matplotlib                3.2.2                         0  
matplotlib-base           3.2.2            py38h64f37c6_0  
mkl                       2020.2                      256  
mkl-service               2.3.0            py38h196d8e1_0  
mkl_fft                   1.3.0            py38h46781fe_0  
mkl_random                1.1.1            py38h47e9c7a_0  
multidict                 5.1.0            py38h2bbff1b_2  
numpy                     1.19.2           py38hadc3359_0  
numpy-base                1.19.2           py38ha3acd2a_0  
nvidia-ml-py3             7.352.1                  pypi_0    pypi
oauthlib                  3.1.1              pyhd3eb1b0_0  
olefile                   0.46                       py_0  
opencv-python             4.5.3.56                 pypi_0    pypi
openssl                   1.1.1k               h8ffe710_0    conda-forge
opt_einsum                3.3.0              pyhd3eb1b0_1  
pathlib                   1.0.1                      py_1  
pillow                    8.3.1            py38h4fa10fc_0  
pip                       21.1.3           py38haa95532_0  
protobuf                  3.14.0           py38hd77b12b_1  
psutil                    5.8.0            py38h2bbff1b_1  
pyasn1                    0.4.8                      py_0  
pyasn1-modules            0.2.8                      py_0  
pycparser                 2.20                       py_2  
pyjwt                     2.1.0            py38haa95532_0  
pyopenssl                 20.0.1             pyhd3eb1b0_1  
pyparsing                 2.4.7              pyhd3eb1b0_0  
pyqt                      5.9.2            py38ha925a31_4  
pyreadline                2.1                      py38_1  
pysocks                   1.7.1            py38haa95532_0  
python                    3.8.10               hdbf39b2_7  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python_abi                3.8                      2_cp38    conda-forge
pywin32                   227              py38he774522_1  
qt                        5.9.7            vc14h73c81de_0  
requests                  2.25.1             pyhd3eb1b0_0  
requests-oauthlib         1.3.0                      py_0  
rsa                       4.7.2              pyhd3eb1b0_1  
scikit-learn              0.24.2           py38hf11a4ad_1  
scipy                     1.6.2            py38h14eb087_0  
setuptools                52.0.0           py38haa95532_0  
sip                       4.19.13          py38ha925a31_0  
six                       1.16.0             pyhd3eb1b0_0  
sqlite                    3.36.0               h2bbff1b_0  
tensorboard               2.4.0              pyhc547734_0  
tensorboard-plugin-wit    1.6.0                      py_0  
tensorflow                2.3.0           mkl_py38h1fcfbd6_0  
tensorflow-base           2.3.0           gpu_py38h7339f5a_0  
tensorflow-estimator      2.5.0              pyh7b7c402_0  
tensorflow-gpu            2.3.0                he13fc11_0  
termcolor                 1.1.0            py38haa95532_1  
threadpoolctl             2.2.0              pyhb85f177_0  
tk                        8.6.10               he774522_0  
tornado                   6.1              py38h2bbff1b_0  
tqdm                      4.61.2             pyhd3eb1b0_1  
typing-extensions         3.10.0.0             hd3eb1b0_0  
typing_extensions         3.10.0.0           pyh06a4308_0  
urllib3                   1.26.6             pyhd3eb1b0_1  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
werkzeug                  1.0.1              pyhd3eb1b0_0  
wheel                     0.36.2             pyhd3eb1b0_0  
win_inet_pton             1.1.0            py38haa95532_0  
wincertstore              0.2                      py38_0  
wrapt                     1.12.1           py38he774522_1  
xz                        5.2.5                h62dcd97_0  
yarl                      1.6.3            py38h2bbff1b_0  
zipp                      3.5.0              pyhd3eb1b0_0  
zlib                      1.2.11               h62dcd97_4  
zstd                      1.4.9                h19a0ad4_0  

=============== State File =================
{
  "name": "original",
  "sessions": {
    "1": {
      "timestamp": 1626905271.0902123,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    }
  },
  "lowest_avg_loss": {
    "a": 0.42868366837501526,
    "b": 0.354293555021286
  },
  "iterations": 1,
  "config": {
    "centering": "face",
    "coverage": 68.75,
    "optimizer": "adam",
    "learning_rate": 5e-05,
    "epsilon_exponent": -7,
    "allow_growth": false,
    "mixed_precision": false,
    "nan_protection": true,
    "convert_batchsize": 16,
    "loss_function": "ssim",
    "mask_loss_function": "mse",
    "l2_reg_term": 100,
    "eye_multiplier": 3,
    "mouth_multiplier": 2,
    "penalized_mask_loss": true,
    "mask_type": "extended",
    "mask_blur_kernel": 3,
    "mask_threshold": 4,
    "learn_mask": false,
    "lowmem": false
  }
}

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               10

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.bisenet_fp]
batch-size:               8
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                face
coverage:                 68.75
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
mask_loss_function:       mse
l2_reg_term:              100
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.phaze_a]
output_size:              128
shared_fc:                none
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              40
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          none
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_norm:                 none
dec_min_filters:          64
dec_max_filters:          512
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Failure to create Tensorboard logging folder

Post by torzdf »

This is a problem with creating the Tensorboard log files. It may be a permissions issue, or a path location issue, so check those first.

Secondly, on Windows based machines it may be that it doesn't like the forward slash ' / ' in the path for the model folder. This may just be how the error report shows the path, but if not, try changing the forward slashes in the path to back slashes ' \ '.

If you cannot resolve the issue that way, then you can disable logging to skip the creation of that folder, but you will not be able to use any graphing/analysis tools.

My word is final

Locked