Page 1 of 1

odd Training error

Posted: Wed Mar 11, 2020 1:23 am
by skivvy

Ok here's the deal. Recently i've gone through the process of extracting my frames/faces from a video that i'm going to use as my base to put a face on. Then after that's done, i go in and manually adjust the alignments in a few trouble areas. Those extracted faces are then trashed and i use the extract function under the alignments tab to create the new face set. Of course after doing this, i need to mask the faces. I use Unet-DFL. At this point everything has gone smoothly. However, if i try to train at this point i'll be greeted with the below error (that file doesn't exist anymore, since i trashed the original face set), or it will tell me there are no alignments for certain faces, even though i've used the alignment file to extract those faces from the video. I'm obviously doing something wrong, can anyone point me in the right direction?

Code: Select all

03/10/2020 20:17:22 MainProcess     _training_0     config          check_exists              DEBUG    Config file exists: 'C:\Users\Ryan\faceswap\config\train.ini'
03/10/2020 20:17:22 MainProcess     _training_0     config          load_config               VERBOSE  Loading config: 'C:\Users\Ryan\faceswap\config\train.ini'
03/10/2020 20:17:22 MainProcess     _training_0     config          validate_config           DEBUG    Validating config
03/10/2020 20:17:22 MainProcess     _training_0     config          check_config_change       DEBUG    Default config has not changed
03/10/2020 20:17:22 MainProcess     _training_0     config          check_config_choices      DEBUG    Checking config choices
03/10/2020 20:17:22 MainProcess     _training_0     config          check_config_choices      DEBUG    Checked config choices
03/10/2020 20:17:22 MainProcess     _training_0     config          validate_config           DEBUG    Validated config
03/10/2020 20:17:22 MainProcess     _training_0     config          handle_config             DEBUG    Handled config
03/10/2020 20:17:22 MainProcess     _training_0     config          __init__                  DEBUG    Initialized: Config
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'coverage')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'float'>, value: 62.5)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'mask_type')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'str'>, value: unet-dfl)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'mask_blur_kernel')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 3)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'mask_threshold')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 4)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'learn_mask')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'icnr_init')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'conv_aware_init')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'subpixel_upscaling')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'reflect_padding')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: False)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'penalized_mask_loss')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'bool'>, value: True)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'loss_function')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'str'>, value: mae)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'global', option: 'learning_rate')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'float'>, value: 0.0001)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'preview_images')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 14)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'zoom_amount')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 5)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'rotation_range')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 10)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'shift_range')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 5)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'flip_chance')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 50)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'color_lightness')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 30)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'color_ab')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 8)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'color_clahe_chance')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 50)
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Getting config item: (section: 'trainer.original', option: 'color_clahe_max_size')
03/10/2020 20:17:22 MainProcess     _training_0     config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 4)
03/10/2020 20:17:22 MainProcess     _training_0     _base           add_session_batchsize     DEBUG    Adding session batchsize: 16
03/10/2020 20:17:22 MainProcess     _training_0     _base           _process_training_opts    DEBUG    {'alignments': {'a': 'E:\\TED Talk\\TED Talk\\X_20200310_201237.fsa', 'b': 'E:\\TED Talk\\Elon Musk\\alignments_merged_20200309_171916.fsa'}, 'preview_scaling': 0.5, 'warp_to_landmarks': False, 'augment_color': True, 'no_flip': False, 'pingpong': False, 'snapshot_interval': 25000, 'training_size': 256, 'no_logs': False, 'coverage_ratio': 0.625, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True}
03/10/2020 20:17:22 MainProcess     _training_0     _base           _landmarks_required       DEBUG    False
03/10/2020 20:17:22 MainProcess     _training_0     _base           _use_mask                 DEBUG    True
03/10/2020 20:17:22 MainProcess     _training_0     _base           __init__                  DEBUG    Initializing TrainingAlignments: (training_opts: '{'alignments': {'a': 'E:\\TED Talk\\TED Talk\\X_20200310_201237.fsa', 'b': 'E:\\TED Talk\\Elon Musk\\alignments_merged_20200309_171916.fsa'}, 'preview_scaling': 0.5, 'warp_to_landmarks': False, 'augment_color': True, 'no_flip': False, 'pingpong': False, 'snapshot_interval': 25000, 'training_size': 256, 'no_logs': False, 'coverage_ratio': 0.625, 'mask_type': 'unet-dfl', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True}', image counts: {'a': 4553, 'b': 45363})
03/10/2020 20:17:22 MainProcess     _training_0     _base           _get_image_hashes         DEBUG    side: a, file count: 4553
03/10/2020 20:17:22 MainProcess     _training_0     image           read_image_hash_batch     DEBUG    Submitting 4553 items to executor
03/10/2020 20:17:22 MainProcess     _training_0     image           read_image_hash_batch     DEBUG    Succesfully submitted 4553 items to executor
03/10/2020 20:17:22 MainProcess     _training_0     _base           _get_image_hashes         DEBUG    side: b, file count: 45363
03/10/2020 20:17:22 MainProcess     _training_0     image           read_image_hash_batch     DEBUG    Submitting 45363 items to executor
03/10/2020 20:17:23 MainProcess     _training_0     image           read_image_hash_batch     DEBUG    Succesfully submitted 45363 items to executor
03/10/2020 20:17:29 MainProcess     _training_0     _base           _load_alignments          DEBUG    Loading alignments
03/10/2020 20:17:29 MainProcess     _training_0     _base           _load_alignments          DEBUG    side: 'a', path: 'E:\TED Talk\TED Talk\X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess     _training_0     alignments      __init__                  DEBUG    Initializing Alignments: (folder: 'E:\TED Talk\TED Talk', filename: 'X_20200310_201237.fsa')
03/10/2020 20:17:29 MainProcess     _training_0     serializer      get_serializer            DEBUG    <lib.serializer._PickleSerializer object at 0x0000029FA11E8F88>
03/10/2020 20:17:29 MainProcess     _training_0     serializer      get_serializer            DEBUG    <lib.serializer._CompressedSerializer object at 0x0000029FCC44C4C8>
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _get_location             DEBUG    Getting location: (folder: 'E:\TED Talk\TED Talk', filename: 'X_20200310_201237.fsa')
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _get_location             DEBUG    Valid Alignments filename provided: 'X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _get_location             VERBOSE  Alignments filepath: 'E:\TED Talk\TED Talk\X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _load                     DEBUG    Loading alignments
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _load                     INFO     Reading alignments from: 'E:\TED Talk\TED Talk\X_20200310_201237.fsa'
03/10/2020 20:17:29 MainProcess     _training_0     serializer      load                      DEBUG    filename: E:\TED Talk\TED Talk\X_20200310_201237.fsa
03/10/2020 20:17:29 MainProcess     _training_0     serializer      load                      DEBUG    stored data type: <class 'bytes'>
03/10/2020 20:17:29 MainProcess     _training_0     serializer      unmarshal                 DEBUG    data type: <class 'bytes'>
03/10/2020 20:17:29 MainProcess     _training_0     serializer      unmarshal                 DEBUG    returned data type: <class 'dict'>
03/10/2020 20:17:29 MainProcess     _training_0     serializer      load                      DEBUG    data type: <class 'dict'>
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _load                     DEBUG    Loaded alignments
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _has_legacy_structure     DEBUG    legacy structure: False
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _has_legacy_landmarksxy   DEBUG    checking legacy landmarksXY
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _has_legacy_landmarksxy   DEBUG    legacy landmarksXY: False
03/10/2020 20:17:29 MainProcess     _training_0     alignments      _has_legacy_landmarks_list DEBUG    checking legacy landmarks as list
03/10/2020 20:17:29 MainProcess     _training_0     alignments      __init__                  DEBUG    Initialized Alignments
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000192.png'
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000192.png'
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000193.png'
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000266.png'
03/10/2020 20:17:29 MainProcess     _training_0     _base           _to_detected_faces        DEBUG    Face already exists, skipping: 'Peter Weylands 2023 TED Talk_000267.png'
03/10/2020 20:17:29 MainProcess     _training_0     multithreading  run                       DEBUG    Error in thread (_training_0): At least one of your faces does not have the mask `unet-dfl` stored for it.\nYou should run the Mask Tool to generate this mask for your faceset or select a different mask in the training configuration options.\nThe face that caused this failure was [side: `A`, frame: `Peter Weylands 2023 TED Talk_000267.png`, index: 0]. The masks that exist for this face are: [].\nBe aware that there are probably more faces without this Mask Type
03/10/2020 20:17:29 MainProcess     MainThread      train           _monitor                  DEBUG    Thread error detected
03/10/2020 20:17:29 MainProcess     MainThread      train           _monitor                  DEBUG    Closed Monitor
03/10/2020 20:17:29 MainProcess     MainThread      train           _end_thread               DEBUG    Ending Training thread
03/10/2020 20:17:29 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
03/10/2020 20:17:29 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Threads: '_training'
03/10/2020 20:17:29 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Thread: '_training_0'
03/10/2020 20:17:29 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
03/10/2020 20:17:29 MainProcess     MainThread      cli             execute_script            ERROR    At least one of your faces does not have the mask `unet-dfl` stored for it.
03/10/2020 20:17:29 MainProcess     MainThread      cli             execute_script            ERROR    You should run the Mask Tool to generate this mask for your faceset or select a different mask in the training configuration options.
03/10/2020 20:17:29 MainProcess     MainThread      cli             execute_script            ERROR    The face that caused this failure was [side: `A`, frame: `Peter Weylands 2023 TED Talk_000267.png`, index: 0]. The masks that exist for this face are: [].
03/10/2020 20:17:29 MainProcess     MainThread      cli             execute_script            ERROR    Be aware that there are probably more faces without this Mask Type
Traceback (most recent call last):
  File "C:\Users\Ryan\faceswap\lib\cli.py", line 128, in execute_script
    process.process()
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 159, in process
    self._end_thread(thread, err)
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 199, in _end_thread
    thread.join()
  File "C:\Users\Ryan\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\Ryan\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 224, in _training
    raise err
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 213, in _training
    trainer = self._load_trainer(model)
  File "C:\Users\Ryan\faceswap\scripts\train.py", line 275, in _load_trainer
    self._args.configfile)
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\original.py", line 10, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 114, in __init__
    self._process_training_opts()
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 169, in _process_training_opts
    alignments = TrainingAlignments(self._model.training_opts, self._images)
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1055, in __init__
    self._detected_faces = self._load_alignments()
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1186, in _load_alignments
    retval[side] = self._to_detected_faces(alignments, side)
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1217, in _to_detected_faces
    if not self._validate_face(face, filename, idx, side, side_hashes):
  File "C:\Users\Ryan\faceswap\plugins\train\trainer\_base.py", line 1278, in _validate_face
    raise FaceswapError(msg)
lib.utils.FaceswapError: At least one of your faces does not have the mask `unet-dfl` stored for it.
You should run the Mask Tool to generate this mask for your faceset or select a different mask in the training configuration options.
The face that caused this failure was [side: `A`, frame: `Peter Weylands 2023 TED Talk_000267.png`, index: 0]. The masks that exist for this face are: [].
Be aware that there are probably more faces without this Mask Type

Re: odd Training error

Posted: Wed Mar 11, 2020 2:23 am
by deephomage

The error message tells you what you should do (near the end). You should run the Mask Tool to generate the Unet-DFL mask for your faceset, or select a different mask in the training configuration options. Look under Tools, Mask and select the Unet-DFL option. Provide the alignments.fsa location and choose frames or faces and the tool will generate the missing Unet-DFL masks for your face set. You should then be able to train with the mask you wanted.


Re: odd Training error

Posted: Wed Mar 11, 2020 2:29 am
by skivvy

as i said earlier, i've done that, and i still get the errors i described. is Unet-DFL not compatible, or is there some kind of bug i'm not aware.

skivvy wrote: Wed Mar 11, 2020 1:23 am

Of course after doing this, i need to mask the faces. I use Unet-DFL.


Re: odd Training error

Posted: Wed Mar 11, 2020 2:31 am
by bryanlyon

Try again, chances are it errored out the first time and didn't actually generate the masks.


Re: odd Training error

Posted: Wed Mar 11, 2020 2:39 am
by skivvy

i've quite literally done this several times throughout the day trying all different variables and nothing seems to change. is there something that i should look out for


Re: odd Training error

Posted: Wed Mar 11, 2020 2:42 am
by bryanlyon

Every time the mask tool runs, it creates a backup of the old alignments file. Make sure you're using the correct ones for all the tasks. Make sure that none of the processes fail or throw an error (make sure that the mask tool actually creates a new FSA and it should be larger than the previous one since it'll include masks).

If you try all this and still experience problems, please include a crash log and copy of the alignments file for us to investigate.


Re: odd Training error

Posted: Wed Mar 11, 2020 2:46 am
by skivvy

Will do, i'll get it over to you in the AM, i'm burnt out troubleshooting


Re: odd Training error

Posted: Wed Mar 11, 2020 2:56 am
by skivvy

how can i get you an .fsa file, not permitted in the attachments


Re: odd Training error

Posted: Wed Mar 11, 2020 2:57 am
by bryanlyon

You can put it on Google Drive/Dropbox or similar. Feel free to PM a link if you don't want it public.


Re: odd Training error

Posted: Wed Mar 11, 2020 3:04 am
by skivvy

Sent it your way


Re: odd Training error

Posted: Wed Mar 11, 2020 5:24 am
by pfakanator

It is likely that during manual alignment you have copied an alignment from a frame without modifying it.


Re: odd Training error

Posted: Wed Mar 11, 2020 2:00 pm
by skivvy

That makes a lot of sense. Didn't know the alignments had to be modified.


Re: odd Training error

Posted: Thu Mar 12, 2020 6:16 pm
by skivvy

that's what did it! each frame alignment must be modified and not just copied


Re: odd Training error

Posted: Thu Mar 12, 2020 6:25 pm
by bryanlyon

Glad you've got a work-around. This is unlikely to be directly fixed. The manual tool is currently under a complete rework so the current tool will not likely see any fixes or changes as the new one should be here soon and will fix issues like this.