Questions about a project.

[Plumtree3D]

Hello, I'm a french developper and artist.
I have been requested to work on a project and since I am not very well versed on deepfakes I'd like to ask a few questions before taking any more steps.

The project is a documentary in which a lot of the people filmed must remain anonymous (at least no be immediately identifiable). The director don't want to blur or censor their faces as it would kinda ruin the point of the movie so the idea of digitally altering their faces was quickly brought to the table. The problems are there are a lot of subjects to alter, several shots contain several faces. They sometimes turn around, move quickly, wear glasses, have objects passing in front of their faces, are out of field, etc....

I can't say much more about the project as it is still in production.

I did some quick tests using deepfacelab,2 in which I tried to replace the face of a character from a movie scene (not THE movie) with the face of a 3D model generated with meta human creator. The tests where widely inconclusive. I plan to do some more tests using faceswap to change faces from actual clips from the documentary and using a friend as the model. But before going further I wanted to check on the feasability of the project.

• I don't care that the result doesn't look like the model but I do need the result to NOT look like the source.
• I also need to preserve the feelings and subtles expressions as much as possible.
• It's not a big deal if it looks digitally altered

Is faceswap or the deepfake technology a good way to achieve this?
If so, are there things to know to get there faster? What would be a good model?

Thanks in advance,sorry for the vague subject, I hope this is allowed and in the right discussion, sorry for the inconvenience if it's not.

(Additionnally, I was kinda reluctant about working with deepfakes as I mostly saw it being used to produce sxual content of people without their consent so I really appreciate the ethics of this community)*

[torzdf]

Faceswap would definitely work for your usecase, but I'm not entirely convinced it is the best way to go, for a few reasons:

The problems are there are a lot of subjects to alter, several shots contain several faces. They sometimes turn around, move quickly, wear glasses, have objects passing in front of their faces, are out of field, etc....

Faceswap works as 1 to 1 swaps, so you would need to train a model for each and every subject. There are ways to shortcut this, but I would not say that this is a viable solution for doing swaps at scale. The other issues you list should be ok to deal with, but will require some work in pre/post

It is worth bearing in mind that faceswap has a fairly steep learning curve. I say to everyone that their first swap is always bad (it is), so if you have a relatively tight timeframe and no experience, I would not commit to being able to deliver this.

• I don't care that the result doesn't look like the model but I do need the result to NOT look like the source.
• I also need to preserve the feelings and subtles expressions as much as possible.
• It's not a big deal if it looks digitally altered

This should not be a problem, however, identity leakage is definitely a thing, and can happen more or less on a project to project basis, and it is impossible to predict whether this will be an issue prior to starting.

On top of this, and I confess that it is unlikely, but it is worth mentioning, Deepfakes tend to have 'tells' that the image has been been digitally altered. Whilst it is not possible, currently, to reverse a deepfake, I would say that there is a chance (albeit very slim) that the process could be reversed in the future, which would make me reluctant to use this method for hiding identities for at risk subjects.

Something like Roop or First Order Motion model may work better for your usecase, although I'm not sure they will give you a high enough fidelity for a broadcast quality product.

[EthnTempest]

It seems like deepfake technology could potentially work for your project, but there are significant challenges, especially with complex movements, multiple faces, and subtle expressions. While preserving emotions and expressions can be difficult, the technology can help anonymize individuals. To achieve better results, using high-quality footage and ensuring proper training with faceswap models might help. However, given the complexity, manual VFX techniques like rotoscoping or procedural masking might be more reliable in certain cases.