Training crash related alignment file missing?

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
cosmico
Posts: 95
Joined: Sat Jan 18, 2020 6:32 pm
Has thanked: 13 times
Been thanked: 35 times

Training crash related alignment file missing?

Post by cosmico »

Every time I try to train I get instantly hit with
08/18/2020 18:05:20 CRITICAL Error caught! Exiting...
08/18/2020 18:05:20 ERROR Caught exception in thread: '_training_0'
08/18/2020 18:05:20 ERROR Alignments file does not exist: D:\Nueral Network programs\[i]My input B photo set[/i]\alignments.fsa

No crashlog or anything. I didnt think i was working with actual alignment files when I trained. This happens on every single type of model, trained models, new models, and many combinations of training settings. Reading some past comments on here, I attempted to see if I had a image that was different sized in my input a or b, but sorting by file size in both folders reveals 256x256 at top and bottom -so I dont think so.

Other info: While updating faceswap, it tried to update or install something related to cudnn and failed and told me to do it separately, and then faceswap would crash on command prompt before the gui. Not knowing what to do, I just uninstalled faceswap and mini cuda then reinstalled them, and while this solved the command prompt crash, I cant get anything to train. Extracting works.

by torzdf » Wed Aug 19, 2020 6:27 am

The reason for this is that "Penalized Mask Loss" is enabled by default (where it wasn't in the old world), and this requires an alignments file. You can turn this off in Model Settings.

Go to full post
User avatar
Hossler
Posts: 1
Joined: Wed Aug 19, 2020 2:34 am

Re: Training crash related alignment file missing?

Post by Hossler »

The .fsa file is created in the folder with the file from which the faces are extracted. Rename this file to "alignments.fsa" and put it in the folder "D: \ Nueral Network programs \ My input B".
Sorry for the punctuation, English is not my first language.

User avatar
torzdf
Posts: 2681
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 133 times
Been thanked: 625 times

Re: Training crash related alignment file missing?

Post by torzdf »

The reason for this is that "Penalized Mask Loss" is enabled by default (where it wasn't in the old world), and this requires an alignments file. You can turn this off in Model Settings.

My word is final

User avatar
cosmico
Posts: 95
Joined: Sat Jan 18, 2020 6:32 pm
Has thanked: 13 times
Been thanked: 35 times

Re: Training crash related alignment file missing?

Post by cosmico »

torzdf wrote: Wed Aug 19, 2020 6:27 am

The reason for this is that "Penalized Mask Loss" is enabled by default (where it wasn't in the old world), and this requires an alignments file. You can turn this off in Model Settings.

This fixed my problem. Thank you very much. I'll admit that while I tinkered with lots of settings trying to get it to work, this is probably one of the few settings I didn't mess with.
Some follow up questions: is isn't penalized mask loss a good thing that I should want on if it didnt crash my training? and also second question: will the next update have this off by default like it used to be?

User avatar
torzdf
Posts: 2681
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 133 times
Been thanked: 625 times

Re: Training crash related alignment file missing?

Post by torzdf »

This will probably remain on by default, but your settings are saved.

Penalized mask loss basically means that the Neural Network gives less importance to parts of the image that aren't face (e.g. background etc.). This has 2 effects. It can lead to faster training, and it means that latent space within the model is not being wasted with details that are not important, which should result in a better final result.

I appreciate that it may trip some people up during this transition period, but I really want the default settings to be as close to getting the "best" result as possible.

My word is final

Locked