Training for a face vs scene / reusing a model

[JimmyBoy]

Hi,

Im trying to understand the training process. I believe that you are training for a face and not a scene per se and you want as many facial expressions, angles and different lighting conditions for both face A and B.

So, using the ubiquitous cage to trump, lets say I have the following videos of both trump and cage.

Code: Select all

C:\videos\trump1.mp4
C:\videos\trump2.mp4
C:\videos\trump3.mp4
C:\videos\cage1.mp4
C:\videos\cage2.mp4
C:\videos\cage3.mp4
C:\videos\trump-speech.mp4
C:\videos\trump-inauguration.mp4

The trump1, trump2, trump3, cage1, cage2, cage3 video files are from various sources (high quality) with many facial expressions, lighting etc.... trump-speech and trump-inauguration are my 'target' videos, I want cage's face to be put onto trump-speech and trump-inauguration (two separate faceswap projects / videos).

I then create a 'montage' video for both trump and cage. ie. load trump1,trump2,trump3 into video editor, cut out all non trump scenes and export, theoretically making the extraction process quicker and less junk to be deleted after extraction. I have this folder structure

Code: Select all

C:\faceswap\extract\trump\faces\
C:\faceswap\extract\trump\trump-montage.mp4
C:\faceswap\extract\trump\trump-montage_alignments.fsa
C:\faceswap\extract\cage\faces\
C:\faceswap\extract\cage\cage-montage.mp4
C:\faceswap\extract\cage\cage-montage_alignments.fsa

After examining the output (blurry faces, covered faces etc) and cleaning up the alignments file accordingly, I now have a dataset of both trump and cage faces.

Now, I create a 'Model' for cage to trump and train using trump-montage_alignments.fsa and cage-montage_alignments.fsa

Code: Select all

C:\faceswap\train\cage-to-trump\model

My logic says, this model now contains data to put cage's face onto trump's, am I correct?

If so, theoretically I can now use the 'C:\faceswap\train\cage-to-trump\model' to put cage's face on to trump's in ANY video? such as the 'trump-speech.mp4' and 'trump-inauguration.mp4' videos ?

If I am correct, im unsure how to do the last part - ie. how do I use my 'C:\faceswap\train\cage-to-trump\model' on 'C:\videos\trump-speech.mp4'

Have I completely misunderstood the concept ? what folder structure to others use ?

[torzdf]

That all looks correct to me.

To put Cage's face onto other Trump videos, you will need to generate an Alignments file for each Trump video that you want to swap, so that the process knows where the faces are in the video.

Then you just run the convert command with your trained model and your Trump videos.

[kamapisachihd]

Thanx @ tarzdf for bringing forward this query,
I have same problem as I have trained model with multiple random faces of Person A and Person B.
Once model has finally trained I have tried extracting faces from Target Clip of Person B and convert it with Model trained. Unfortunately conversion never happens as there is no matching face alignment found in the model file.

[torzdf]

The model doesn't contain alignments. It just contains algorithms for decoding and encoding face images.

You don't run conversion on extracted faces. You run conversion on videos/full frame images