Reface analogue

Talk about AI and Deep Learning
Post Reply
User avatar
sintanial
Posts: 1
Joined: Tue Nov 03, 2020 1:52 pm

Reface analogue

Post by sintanial »

I try to create app like Reface App https://reface.ai/.

But I don't understand how face swapping works so quickly for them? For a faceswap, you need to train a model for more than 24 hours, and they (reface) do this face change "on the fly" !


User avatar
torzdf
Posts: 1495
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 51 times
Been thanked: 287 times

Re: Reface analogue

Post by torzdf »

They use a different technique, which is a lot quicker, but a lot more limited in what it can do.

My word is final


User avatar
lineage
Posts: 5
Joined: Tue Jan 05, 2021 3:36 am

Re: Reface analogue

Post by lineage »

realize this is old and probably a dead end ... but yah, same thought here.

juding by your comment on the different but limited ... Im curious of you have any more insight to share?

it just seems there is a clear and decisive difference in approach, a mere few seconds to get acceptable quality with any face photo to select 15 secs of video.

Im no expert, but clearly cloud crunching/processing with what appears like a 3-d head object per frame for the select videos and the face photo is a skin on top of that 3-d head object (im sure theres more to it, just my neophyte observation)

the eye blinks, smiles and facial gestures get really interesting ... but I imagine its similar to the motion tracking approach used by big studios. the software has more points to track for the movement, but the select 15 seconds would be "simple" to crunch. In fact, I suspect it is based on that core tech which makes movies (except purely software based with no actual sensors on an actors face)... hence the many hollywood vids made available for users to swap with.

curious if any faceswap member did any work towards reface :)

and of course the biggest difference is they have money... hollywood money... nuff said? :P


User avatar
bryanlyon
Site Admin
Posts: 634
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 161 times
Contact:

Re: Reface analogue

Post by bryanlyon »

Pretty sure that Reface doesn't do anything in 3d at all. They using a morphing/latent space search for their face swapping. Basically they take the features of each face and morph them together and put them back.


User avatar
lineage
Posts: 5
Joined: Tue Jan 05, 2021 3:36 am

Re: Reface analogue

Post by lineage »

that kinda sounds like what faceswap is doing but within seconds and acceptable quality...

tbh I was hoping that was the case here ... ie train like crazy on a source vid, clean the face extract/maps/landmarks... then you'll be able to use any static photo and put that face on our trained source vid.

or am I missing something? again, neophyte talking here ... appreciate the discussion :)


User avatar
bryanlyon
Site Admin
Posts: 634
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 161 times
Contact:

Re: Reface analogue

Post by bryanlyon »

This is nothing like what Faceswap is doing.

Faceswap is "learning" the faces and an encoding that corresponds to the input pose, expression, and lighting (though still technically not as a 3d object). Thats what the training is, building an understanding of what each face "is" and what it looks like in the various contexts. Because of this intense training, Faceswap is capable of much more than just a surface level copy of facial features onto the original video. It's able to match things like HOW an actor might "blink" or do other actions. That's how you're able to get tics and actions that one person does into a video.

If all you're looking for is surface facial features copied, then yes reface is one of the tools out there that does that but it's techniques are radically different from how Faceswap works and what Faceswap tries to accomplish.


Post Reply