odd Training error

Extracting faces for training and converting
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Locked
User avatar
skivvy
Posts: 23
Joined: Fri Dec 13, 2019 10:56 pm
Answers: 1
Has thanked: 10 times

odd Training error

Post by skivvy »

Ok here's the deal. Recently i've gone through the process of extracting my frames/faces from a video that i'm going to use as my base to put a face on. Then after that's done, i go in and manually adjust the alignments in a few trouble areas. Those extracted faces are then trashed and i use the extract function under the alignments tab to create the new face set. Of course after doing this, i need to mask the faces. I use Unet-DFL. At this point everything has gone smoothly. However, if i try to train at this point i'll be greeted with the below error (that file doesn't exist anymore, since i trashed the original face set), or it will tell me there are no alignments for certain faces, even though i've used the alignment file to extract those faces from the video. I'm obviously doing something wrong, can anyone point me in the right direction?

Code: Select all

03/10/2020 20:17:22 MainProcess     _training_0     config          check_exists              DEBUG    Config file exists: 'C:\Users\Ryan\faceswap\config\train.ini'
03/10/2020 20:17:22 MainProcess     _training_0     config          load_config               VERBOSE  Loading config: 'C:\Users\Ryan\faceswap\config\train.ini'
03/10/2020 20:17:22 MainProcess     _training_0     config          validate_config           DEBUG    Validating config
03/10/2020 20:17:22 MainProcess     _training_0     config          check_config_change       DEBUG    Default config has not changed
03/10/2020 20:17:22 MainProcess     _training_0     config          check_config_choices      DEBUG    Checking config choices
03/10/2020 20:17:22 MainProcess     _training_0     config          check_config_choices      DEBUG    Checked config choices
03/10/2020 20:17:22 MainProcess     _training_0     config          validate_config           DEBUG    Validated config
03/10/2020 20:17:22 MainProcess     _training_0     config          handle_config             DEBUG    Handled config
03/10/2020 20:17:22 MainProcess     _training_0     config          __init__                  DEBUG    Initialized: Config
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'coverage')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'float'>, value: 62.5)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'mask_type')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'str'>, value: unet-dfl)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'mask_blur_kernel')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 3)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'mask_threshold')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 4)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'learn_mask')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'icnr_init')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'conv_aware_init')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'subpixel_upscaling')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'reflect_padding')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'penalized_mask_loss')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: True)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'loss_function')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'str'>, value: mae)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'learning_rate')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'float'>, value: 0.0001)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'preview_images')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 14)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'zoom_amount')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 5)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'rotation_range')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 10)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'shift_range')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 5)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'flip_chance')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 50)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'color_lightness')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 30)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'color_ab')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 8)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'color_clahe_chance')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 50)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'color_clahe_max_size')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 4)
03/10/2020 20:17:22 MainProcess     _training_0     _base           add_session_batchsize     DEBUG    Adding session batchsize: 16
03/10/2020 20:17:22 MainProcess     _training_0     _base           _process_training_opts    DEBUG    {'alignments': {'a': 'E:\\TED Talk\\TED Talk\\X_20200310_201237.fsa', 'b': 'E:\\TED Talk\\Elon Musk\\alignments_merged_20200309_171916.fsa'}, 'preview_scaling': 0.5, 'warp_to_landmarks': False, 'augment_color': True, 'no_flip': False, 'pingpong': False, 'snapshot_interval': 25000, 'training_size': 256, 'no_logs': False, 'coverage_ratio': 0.625, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True}
03/10/2020 20:17:22 MainProcess     _training_0     _base           _landmarks_required       DEBUG    False
03/10/2020 20:17:22 MainProcess     _training_0     _base           _use_mask                 DEBUG    True
03/10/2020 20:17:22 MainProcess     _training_0     _base           __init__                  DEBUG    Initializing TrainingAlignments: (training_opts: '{'alignments': {'a': 'E:\\TED Talk\\TED Talk\\X_20200310_201237.fsa', 'b': 'E:\\TED Talk\\Elon Musk\\alignments_merged_20200309_171916.fsa'}, 'preview_scaling': 0.5, 'warp_to_landmarks': False, 'augment_color': True, 'no_flip': False, 'pingpong': False, 'snapshot_interval': 25000, 'training_size': 256, 'no_logs': False, 'coverage_ratio': 0.625, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True}', image counts: {'a': 4553, 'b': 45363})
03/10/2020 20:17:22 MainProcess     _training_0     _base           _get_image_hashes         DEBUG    side: a, file count: 4553
03/10/2020 20:17:22 MainProcess     _training_0     image           read_image_hash_batch     DEBUG    Submitting 4553 items to executor
03/10/2020 20:17:22 MainProcess     _training_0     image           read_image_hash_batch     DEBUG    Succesfully submitted 4553 items to executor
03/10/2020 20:17:22 MainProcess     _training_0     _base           _get_image_hashes         DEBUG    side: b, file count: 45363
03/10/2020 20:17:22 MainProcess     _training_0     image           read_image_hash_batch     DEBUG    Submitting 45363 items to executor
03/10/2020 20:17:23 MainProcess     _training_0     image           read_image_hash_batch     DEBUG    Succesfully submitted 45363 items to executor
03/10/2020 20:17:29 MainProcess     _training_0     _base           _load_alignments          DEBUG    Loading alignments
03/10/2020 20:17:29 MainProcess     _training_0     _base           _load_alignments          DEBUG    side: 'a', path: 'E:\TED Talk\TED Talk\X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess     _training_0     alignments      __init__                  DEBUG    Initializing Alignments: (folder: 'E:\TED Talk\TED Talk', filename: 'X_20200310_201237.fsa')
03/10/2020 20:17:29 MainProcess     _training_0     serializer      get_serializer            DEBUG    <lib.serializer._PickleSerializer object at 0x0000029FA11E8F88>
03/10/2020 20:17:29 MainProcess     _training_0     serializer      get_serializer            DEBUG    <lib.serializer._CompressedSerializer object at 0x0000029FCC44C4C8>
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _get_location             DEBUG    Getting location: (folder: 'E:\TED Talk\TED Talk', filename: 'X_20200310_201237.fsa')
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _get_location             DEBUG    Valid Alignments filename provided: 'X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _get_location             VERBOSE  Alignments filepath: 'E:\TED Talk\TED Talk\X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _load                     DEBUG    Loading alignments
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _load                     INFO     Reading alignments from: 'E:\TED Talk\TED Talk\X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess     _training_0     serializer      load                      DEBUG    filename: E:\TED Talk\TED Talk\X_20200310_201237.fsa
03/10/2020 20:17:29 MainProcess     _training_0     serializer      load                      DEBUG    stored data type: <class 'bytes'>
03/10/2020 20:17:29 MainProcess     _training_0     serializer      unmarshal                 DEBUG    data type: <class 'bytes'>
03/10/2020 20:17:29 MainProcess     _training_0     serializer      unmarshal                 DEBUG    returned data type: <class 'dict'>
03/10/2020 20:17:29 MainProcess     _training_0     serializer      load                      DEBUG    data type: <class 'dict'>
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _load                     DEBUG    Loaded alignments
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _has_legacy_structure     DEBUG    legacy structure: False
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _has_legacy_landmarksxy   DEBUG    checking legacy landmarksXY
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _has_legacy_landmarksxy   DEBUG    legacy landmarksXY: False
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _has_legacy_landmarks_list DEBUG    checking legacy landmarks as list
03/10/2020 20:17:29 MainProcess     _training_0     alignments      __init__                  DEBUG    Initialized Alignments
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000192.png'
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000192.png'
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000193.png'
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000266.png'
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000267.png'
03/10/2020 20:17:29 MainProcess     _training_0     multithreading  run                       DEBUG    Error in thread (_training_0): At least one of your faces does not have the mask `unet-dfl` stored for it.\nYou should run the Mask Tool to generate this mask for your faceset or select a different mask in the training configuration options.\nThe face that caused this failure was [side: `A`, frame: `Peter Weylands 2023 TED Talk_000267.png`, index: 0]. The masks that exist for this face are: [].\nBe aware that there are probably more faces without this Mask Type
03/10/2020 20:17:29 MainProcess     MainThread      train           _monitor                  DEBUG    Thread error detected
03/10/2020 20:17:29 MainProcess     MainThread      train           _monitor                  DEBUG    Closed Monitor
03/10/2020 20:17:29 MainProcess     MainThread      train           _end_thread               DEBUG    Ending Training thread
03/10/2020 20:17:29 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
03/10/2020 20:17:29 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Threads: '_training'
03/10/2020 20:17:29 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Thread: '_training_0'
03/10/2020 20:17:29 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
03/10/2020 20:17:29 MainProcess     MainThread      cli             execute_script            ERROR    At least one of your faces does not have the mask `unet-dfl` stored for it.
03/10/2020 20:17:29 MainProcess     MainThread      cli             execute_script            ERROR    You should run the Mask Tool to generate this mask for your faceset or select a different mask in the training configuration options.
03/10/2020 20:17:29 MainProcess     MainThread      cli             execute_script            ERROR    The face that caused this failure was [side: `A`, frame: `Peter Weylands 2023 TED Talk_000267.png`, index: 0]. The masks that exist for this face are: [].
03/10/2020 20:17:29 MainProcess     MainThread      cli             execute_script            ERROR    Be aware that there are probably more faces without this Mask Type
Traceback (most recent call last):
  File "C:\Users\Ryan\faceswap\lib\cli.py", line 128, in execute_script
    process.process()
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 159, in process
    self._end_thread(thread, err)
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 199, in _end_thread
    thread.join()
  File "C:\Users\Ryan\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\Ryan\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 224, in _training
    raise err
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 213, in _training
    trainer = self._load_trainer(model)
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 275, in _load_trainer
    self._args.configfile)
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\original.py", line 10, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 114, in __init__
    self._process_training_opts()
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 169, in _process_training_opts
    alignments = TrainingAlignments(self._model.training_opts, self._images)
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1055, in __init__
    self._detected_faces = self._load_alignments()
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1186, in _load_alignments
    retval[side] = self._to_detected_faces(alignments, side)
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1217, in _to_detected_faces
    if not self._validate_face(face, filename, idx, side, side_hashes):
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1278, in _validate_face
    raise FaceswapError(msg)
lib.utils.FaceswapError: At least one of your faces does not have the mask `unet-dfl` stored for it.
You should run the Mask Tool to generate this mask for your faceset or select a different mask in the training configuration options.
The face that caused this failure was [side: `A`, frame: `Peter Weylands 2023 TED Talk_000267.png`, index: 0]. The masks that exist for this face are: [].
Be aware that there are probably more faces without this Mask Type
by pfakanator » Wed Mar 11, 2020 5:24 am

It is likely that during manual alignment you have copied an alignment from a frame without modifying it.

Go to full post

User avatar
deephomage
Posts: 27
Joined: Fri Jul 12, 2019 6:09 pm
Answers: 1
Has thanked: 2 times
Been thanked: 6 times

Re: odd Training error

Post by deephomage »

The error message tells you what you should do (near the end). You should run the Mask Tool to generate the Unet-DFL mask for your faceset, or select a different mask in the training configuration options. Look under Tools, Mask and select the Unet-DFL option. Provide the alignments.fsa location and choose frames or faces and the tool will generate the missing Unet-DFL masks for your face set. You should then be able to train with the mask you wanted.


User avatar
skivvy
Posts: 23
Joined: Fri Dec 13, 2019 10:56 pm
Answers: 1
Has thanked: 10 times

Re: odd Training error

Post by skivvy »

as i said earlier, i've done that, and i still get the errors i described. is Unet-DFL not compatible, or is there some kind of bug i'm not aware.

skivvy wrote: Wed Mar 11, 2020 1:23 am

Of course after doing this, i need to mask the faces. I use Unet-DFL.


User avatar
bryanlyon
Site Admin
Posts: 496
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 120 times
Contact:

Re: odd Training error

Post by bryanlyon »

Try again, chances are it errored out the first time and didn't actually generate the masks.


User avatar
skivvy
Posts: 23
Joined: Fri Dec 13, 2019 10:56 pm
Answers: 1
Has thanked: 10 times

Re: odd Training error

Post by skivvy »

i've quite literally done this several times throughout the day trying all different variables and nothing seems to change. is there something that i should look out for


User avatar
bryanlyon
Site Admin
Posts: 496
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 120 times
Contact:

Re: odd Training error

Post by bryanlyon »

Every time the mask tool runs, it creates a backup of the old alignments file. Make sure you're using the correct ones for all the tasks. Make sure that none of the processes fail or throw an error (make sure that the mask tool actually creates a new FSA and it should be larger than the previous one since it'll include masks).

If you try all this and still experience problems, please include a crash log and copy of the alignments file for us to investigate.


User avatar
skivvy
Posts: 23
Joined: Fri Dec 13, 2019 10:56 pm
Answers: 1
Has thanked: 10 times

Re: odd Training error

Post by skivvy »

Will do, i'll get it over to you in the AM, i'm burnt out troubleshooting


User avatar
skivvy
Posts: 23
Joined: Fri Dec 13, 2019 10:56 pm
Answers: 1
Has thanked: 10 times

Re: odd Training error

Post by skivvy »

how can i get you an .fsa file, not permitted in the attachments


User avatar
bryanlyon
Site Admin
Posts: 496
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 120 times
Contact:

Re: odd Training error

Post by bryanlyon »

You can put it on Google Drive/Dropbox or similar. Feel free to PM a link if you don't want it public.


User avatar
skivvy
Posts: 23
Joined: Fri Dec 13, 2019 10:56 pm
Answers: 1
Has thanked: 10 times

Re: odd Training error

Post by skivvy »

Sent it your way


User avatar
pfakanator
Posts: 25
Joined: Thu Jul 18, 2019 5:02 pm
Answers: 1
Has thanked: 1 time
Been thanked: 11 times

Re: odd Training error

Post by pfakanator »

It is likely that during manual alignment you have copied an alignment from a frame without modifying it.


User avatar
skivvy
Posts: 23
Joined: Fri Dec 13, 2019 10:56 pm
Answers: 1
Has thanked: 10 times

Re: odd Training error

Post by skivvy »

That makes a lot of sense. Didn't know the alignments had to be modified.


User avatar
skivvy
Posts: 23
Joined: Fri Dec 13, 2019 10:56 pm
Answers: 1
Has thanked: 10 times

Re: odd Training error

Post by skivvy »

that's what did it! each frame alignment must be modified and not just copied


User avatar
bryanlyon
Site Admin
Posts: 496
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 120 times
Contact:

Re: odd Training error

Post by bryanlyon »

Glad you've got a work-around. This is unlikely to be directly fixed. The manual tool is currently under a complete rework so the current tool will not likely see any fixes or changes as the new one should be here soon and will fix issues like this.


Locked