Improving video alignments

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Diamondback
Posts: 9
Joined: Fri May 14, 2021 4:50 pm
Has thanked: 4 times
Been thanked: 2 times

Improving video alignments

Post by Diamondback »

Fixing video alignments is, by far, the most tedious part of setting up a swap. For me anyway.

The Fan aligner works well, though not perfect. It seems like any face that is not full forward, or within about 1/4 in any direction, results in alignments being skewed. Eyes and eyebrows tend to stretch away from the face. And hair or backgrounds are often included in the alignment. I will literally spend hours, or even days, re-aligning video alignments, often performing very similar adjustments to thousands of frames.

I realize that this is complicated. But, is there a way to improve the automatic alignments?

  • Can the aligner be configured to prefer a smaller bounding box?

Or, can the manual alignment tool be augmented?

  • Can adjustments be batched, such that thousands of frames requiring the same, minor adjustment, can all be adjusted en-mass?
  • Can alignments be re-calculated, possibly using a key frame for reference?

I have contemplated grabbing the code and attempting to implement some helpers to the alignments process. But, I am not a Python programmer and I do not consider myself very good at software in general. But, I thought I would ask if anyone has some advice. Thanks.

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 176 times
Been thanked: 13 times

Re: Improving video alignments

Post by MaxHunter »

I second this! 😁

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Improving video alignments

Post by torzdf »

Diamondback wrote: Tue May 31, 2022 6:18 pm

I realize that this is complicated. But, is there a way to improve the automatic alignments?

Yes, the best way would be to retrain the model (we use the original model weights). The main problem with this is it needs data.... lots and lots of perfectly annotated data. I carried out a similar task to improve the masking solution. It took many months, and annotating for segmentation masking is definitely easier than annotating landmarks.

Diamondback wrote: Tue May 31, 2022 6:18 pm
  • Can the aligner be configured to prefer a smaller bounding box?

Yes it could, however, the size of the bounding box does not appear to be so much the issue, but where it is centered. It also brings with it 'the law of unintended consequences'. Ultimately you can play with the code to adjust the bounding box and see if you can find a 'sweet' spot. My experience when testing this was that it improves some alignments, but then makes others (which were fine) worse. YMMV. I would welcome feedback on any centering/scaling/cropping which produces reproducibly better results.

Diamondback wrote: Tue May 31, 2022 6:18 pm

Or, can the manual alignment tool be augmented?

  • Can adjustments be batched, such that thousands of frames requiring the same, minor adjustment, can all be adjusted en-mass?
  • Can alignments be re-calculated, possibly using a key frame for reference?

On a conceptual level, yes. Am I likely to implement? No. The manual tool is a fairly complex bit of code. More complex than I was ever meant to make it! Doing either of these things, whilst possible, would not be easy. I would welcome PRs in this direction, but it is unlikely to be something I focus on, sadly.

My word is final

Locked