Ensuring Both Faces are Extracted From a Frame (Even if they are the same - VR 3D Footage)

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Ensuring Both Faces are Extracted From a Frame (Even if they are the same - VR 3D Footage)

Post by ianstephens »

Hi there,

We are still playing around with some VR projects at the moment.

We're noticing something with regards to extracting faces/alignments.

Some of the footage we extract for both training and conversion is 3D side-by-side VR footage. Essentially, two videos side by side - one for the left eye and one for the right eye.

When extracting, and when the faces are quite far away from the capturing lens, the faces look almost identical and it seems that the extraction process thinks they are the same face and only extracts one of them.

This is fine for training sets but when it comes to the final conversion, we need both faces swapped out (on both sides of the video).

Are there any tweaks we can use to force the extract process to capture all faces even if they look identical?

Thinking about it - in theory, we could split the video into left/right eyes and process them individually... this is an option of course but it would increase workload.

Thank you for your help in advance.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Ensuring Both Faces are Extracted From a Frame (Even if they are the same - VR 3D Footage)

Post by torzdf »

The extraction process makes no comparison on faces. It will extract all faces it finds in a video. If it's not extracting one of the faces then it's not finding the face.

Unfortunately I have no experience faceswapping with VR, so this is about the extent that I can help, I'm afraid.

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Ensuring Both Faces are Extracted From a Frame (Even if they are the same - VR 3D Footage)

Post by ianstephens »

Thank you for that reply [mention]torzdf[/mention] and confirming the extraction process doesn't only extract the one (same) face.

That's also interesting information and puzzles me as to why only one of the faces is extracted - because both sides of the video - left/right clearly have a very visible face on both of them.

We are going to try splitting the video down the middle and running two extraction processes (left/right) to see if we get different results.

Will update this post with results for anyone interested.

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Ensuring Both Faces are Extracted From a Frame (Even if they are the same - VR 3D Footage)

Post by ianstephens »

Just an update:

Splitting the video into two separate videos has fixed the issue - now extract picks up the "missing" faces. Perhaps it was also something to do with the frame size of the input video (5K).

Locked