odd Training error
Ok here's the deal. Recently i've gone through the process of extracting my frames/faces from a video that i'm going to use as my base to put a face on. Then after that's done, i go in and manually adjust the alignments in a few trouble areas. Those extracted faces are then trashed and i use the extract function under the alignments tab to create the new face set. Of course after doing this, i need to mask the faces. I use Unet-DFL. At this point everything has gone smoothly. However, if i try to train at this point i'll be greeted with the below error (that file doesn't exist anymore, since i trashed the original face set), or it will tell me there are no alignments for certain faces, even though i've used the alignment file to extract those faces from the video. I'm obviously doing something wrong, can anyone point me in the right direction?
Code: Select all
03/10/2020 20:17:22 MainProcess _training_0 config check_exists DEBUG Config file exists: 'C:\Users\Ryan\faceswap\config\train.ini'
03/10/2020 20:17:22 MainProcess _training_0 config load_config VERBOSE Loading config: 'C:\Users\Ryan\faceswap\config\train.ini'
03/10/2020 20:17:22 MainProcess _training_0 config validate_config DEBUG Validating config
03/10/2020 20:17:22 MainProcess _training_0 config check_config_change DEBUG Default config has not changed
03/10/2020 20:17:22 MainProcess _training_0 config check_config_choices DEBUG Checking config choices
03/10/2020 20:17:22 MainProcess _training_0 config check_config_choices DEBUG Checked config choices
03/10/2020 20:17:22 MainProcess _training_0 config validate_config DEBUG Validated config
03/10/2020 20:17:22 MainProcess _training_0 config handle_config DEBUG Handled config
03/10/2020 20:17:22 MainProcess _training_0 config __init__ DEBUG Initialized: Config
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'coverage')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'float'>, value: 62.5)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'mask_type')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'str'>, value: unet-dfl)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'mask_blur_kernel')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 3)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'mask_threshold')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 4)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'learn_mask')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'icnr_init')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'conv_aware_init')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'subpixel_upscaling')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'reflect_padding')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'penalized_mask_loss')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'bool'>, value: True)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'loss_function')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'str'>, value: mae)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'global', option: 'learning_rate')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'float'>, value: 0.0001)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'trainer.original', option: 'preview_images')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 14)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'trainer.original', option: 'zoom_amount')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 5)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'trainer.original', option: 'rotation_range')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 10)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'trainer.original', option: 'shift_range')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 5)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'trainer.original', option: 'flip_chance')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 50)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'trainer.original', option: 'color_lightness')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 30)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'trainer.original', option: 'color_ab')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 8)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'trainer.original', option: 'color_clahe_chance')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 50)
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Getting config item: (section: 'trainer.original', option: 'color_clahe_max_size')
03/10/2020 20:17:22 MainProcess _training_0 config get DEBUG Returning item: (type: <class 'int'>, value: 4)
03/10/2020 20:17:22 MainProcess _training_0 _base add_session_batchsize DEBUG Adding session batchsize: 16
03/10/2020 20:17:22 MainProcess _training_0 _base _process_training_opts DEBUG {'alignments': {'a': 'E:\\TED Talk\\TED Talk\\X_20200310_201237.fsa', 'b': 'E:\\TED Talk\\Elon Musk\\alignments_merged_20200309_171916.fsa'}, 'preview_scaling': 0.5, 'warp_to_landmarks': False, 'augment_color': True, 'no_flip': False, 'pingpong': False, 'snapshot_interval': 25000, 'training_size': 256, 'no_logs': False, 'coverage_ratio': 0.625, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True}
03/10/2020 20:17:22 MainProcess _training_0 _base _landmarks_required DEBUG False
03/10/2020 20:17:22 MainProcess _training_0 _base _use_mask DEBUG True
03/10/2020 20:17:22 MainProcess _training_0 _base __init__ DEBUG Initializing TrainingAlignments: (training_opts: '{'alignments': {'a': 'E:\\TED Talk\\TED Talk\\X_20200310_201237.fsa', 'b': 'E:\\TED Talk\\Elon Musk\\alignments_merged_20200309_171916.fsa'}, 'preview_scaling': 0.5, 'warp_to_landmarks': False, 'augment_color': True, 'no_flip': False, 'pingpong': False, 'snapshot_interval': 25000, 'training_size': 256, 'no_logs': False, 'coverage_ratio': 0.625, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True}', image counts: {'a': 4553, 'b': 45363})
03/10/2020 20:17:22 MainProcess _training_0 _base _get_image_hashes DEBUG side: a, file count: 4553
03/10/2020 20:17:22 MainProcess _training_0 image read_image_hash_batch DEBUG Submitting 4553 items to executor
03/10/2020 20:17:22 MainProcess _training_0 image read_image_hash_batch DEBUG Succesfully submitted 4553 items to executor
03/10/2020 20:17:22 MainProcess _training_0 _base _get_image_hashes DEBUG side: b, file count: 45363
03/10/2020 20:17:22 MainProcess _training_0 image read_image_hash_batch DEBUG Submitting 45363 items to executor
03/10/2020 20:17:23 MainProcess _training_0 image read_image_hash_batch DEBUG Succesfully submitted 45363 items to executor
03/10/2020 20:17:29 MainProcess _training_0 _base _load_alignments DEBUG Loading alignments
03/10/2020 20:17:29 MainProcess _training_0 _base _load_alignments DEBUG side: 'a', path: 'E:\TED Talk\TED Talk\X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess _training_0 alignments __init__ DEBUG Initializing Alignments: (folder: 'E:\TED Talk\TED Talk', filename: 'X_20200310_201237.fsa')
03/10/2020 20:17:29 MainProcess _training_0 serializer get_serializer DEBUG <lib.serializer._PickleSerializer object at 0x0000029FA11E8F88>
03/10/2020 20:17:29 MainProcess _training_0 serializer get_serializer DEBUG <lib.serializer._CompressedSerializer object at 0x0000029FCC44C4C8>
03/10/2020 20:17:29 MainProcess _training_0 alignments _get_location DEBUG Getting location: (folder: 'E:\TED Talk\TED Talk', filename: 'X_20200310_201237.fsa')
03/10/2020 20:17:29 MainProcess _training_0 alignments _get_location DEBUG Valid Alignments filename provided: 'X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess _training_0 alignments _get_location VERBOSE Alignments filepath: 'E:\TED Talk\TED Talk\X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess _training_0 alignments _load DEBUG Loading alignments
03/10/2020 20:17:29 MainProcess _training_0 alignments _load INFO Reading alignments from: 'E:\TED Talk\TED Talk\X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess _training_0 serializer load DEBUG filename: E:\TED Talk\TED Talk\X_20200310_201237.fsa
03/10/2020 20:17:29 MainProcess _training_0 serializer load DEBUG stored data type: <class 'bytes'>
03/10/2020 20:17:29 MainProcess _training_0 serializer unmarshal DEBUG data type: <class 'bytes'>
03/10/2020 20:17:29 MainProcess _training_0 serializer unmarshal DEBUG returned data type: <class 'dict'>
03/10/2020 20:17:29 MainProcess _training_0 serializer load DEBUG data type: <class 'dict'>
03/10/2020 20:17:29 MainProcess _training_0 alignments _load DEBUG Loaded alignments
03/10/2020 20:17:29 MainProcess _training_0 alignments _has_legacy_structure DEBUG legacy structure: False
03/10/2020 20:17:29 MainProcess _training_0 alignments _has_legacy_landmarksxy DEBUG checking legacy landmarksXY
03/10/2020 20:17:29 MainProcess _training_0 alignments _has_legacy_landmarksxy DEBUG legacy landmarksXY: False
03/10/2020 20:17:29 MainProcess _training_0 alignments _has_legacy_landmarks_list DEBUG checking legacy landmarks as list
03/10/2020 20:17:29 MainProcess _training_0 alignments __init__ DEBUG Initialized Alignments
03/10/2020 20:17:29 MainProcess _training_0 _base _to_detected_faces DEBUG Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000192.png'
03/10/2020 20:17:29 MainProcess _training_0 _base _to_detected_faces DEBUG Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000192.png'
03/10/2020 20:17:29 MainProcess _training_0 _base _to_detected_faces DEBUG Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000193.png'
03/10/2020 20:17:29 MainProcess _training_0 _base _to_detected_faces DEBUG Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000266.png'
03/10/2020 20:17:29 MainProcess _training_0 _base _to_detected_faces DEBUG Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000267.png'
03/10/2020 20:17:29 MainProcess _training_0 multithreading run DEBUG Error in thread (_training_0): At least one of your faces does not have the mask `unet-dfl` stored for it.\nYou should run the Mask Tool to generate this mask for your faceset or select a different mask in the training configuration options.\nThe face that caused this failure was [side: `A`, frame: `Peter Weylands 2023 TED Talk_000267.png`, index: 0]. The masks that exist for this face are: [].\nBe aware that there are probably more faces without this Mask Type
03/10/2020 20:17:29 MainProcess MainThread train _monitor DEBUG Thread error detected
03/10/2020 20:17:29 MainProcess MainThread train _monitor DEBUG Closed Monitor
03/10/2020 20:17:29 MainProcess MainThread train _end_thread DEBUG Ending Training thread
03/10/2020 20:17:29 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...
03/10/2020 20:17:29 MainProcess MainThread multithreading join DEBUG Joining Threads: '_training'
03/10/2020 20:17:29 MainProcess MainThread multithreading join DEBUG Joining Thread: '_training_0'
03/10/2020 20:17:29 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training_0'
03/10/2020 20:17:29 MainProcess MainThread cli execute_script ERROR At least one of your faces does not have the mask `unet-dfl` stored for it.
03/10/2020 20:17:29 MainProcess MainThread cli execute_script ERROR You should run the Mask Tool to generate this mask for your faceset or select a different mask in the training configuration options.
03/10/2020 20:17:29 MainProcess MainThread cli execute_script ERROR The face that caused this failure was [side: `A`, frame: `Peter Weylands 2023 TED Talk_000267.png`, index: 0]. The masks that exist for this face are: [].
03/10/2020 20:17:29 MainProcess MainThread cli execute_script ERROR Be aware that there are probably more faces without this Mask Type
Traceback (most recent call last):
File "C:\Users\Ryan\faceswap\lib\cli.py", line 128, in execute_script
process.process()
File "C:\Users\Ryan\faceswap\scripts\train.py", line 159, in process
self._end_thread(thread, err)
File "C:\Users\Ryan\faceswap\scripts\train.py", line 199, in _end_thread
thread.join()
File "C:\Users\Ryan\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\Ryan\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Ryan\faceswap\scripts\train.py", line 224, in _training
raise err
File "C:\Users\Ryan\faceswap\scripts\train.py", line 213, in _training
trainer = self._load_trainer(model)
File "C:\Users\Ryan\faceswap\scripts\train.py", line 275, in _load_trainer
self._args.configfile)
File "C:\Users\Ryan\faceswap\plugins\train\trainer\original.py", line 10, in __init__
super().__init__(*args, **kwargs)
File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 114, in __init__
self._process_training_opts()
File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 169, in _process_training_opts
alignments = TrainingAlignments(self._model.training_opts, self._images)
File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1055, in __init__
self._detected_faces = self._load_alignments()
File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1186, in _load_alignments
retval[side] = self._to_detected_faces(alignments, side)
File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1217, in _to_detected_faces
if not self._validate_face(face, filename, idx, side, side_hashes):
File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1278, in _validate_face
raise FaceswapError(msg)
lib.utils.FaceswapError: At least one of your faces does not have the mask `unet-dfl` stored for it.
You should run the Mask Tool to generate this mask for your faceset or select a different mask in the training configuration options.
The face that caused this failure was [side: `A`, frame: `Peter Weylands 2023 TED Talk_000267.png`, index: 0]. The masks that exist for this face are: [].
Be aware that there are probably more faces without this Mask Type