Manual Landmarks

Extracting faces for training and converting
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Post Reply
User avatar
lorenzosale
Posts: 11
Joined: Thu May 28, 2020 11:58 am
Has thanked: 1 time

Manual Landmarks

Post by lorenzosale »

Hello,

I am new to the forum and I have a quick question please.

I am looking to create a very good deepfake, and I read that for that it was necessary to set up the landmarks manually.

(Sorry if the guide already explains it but I do not speak very good English and I find it difficult to understand certain points)

First question:

Should the manual implementation of landmarks be done on the source photos / videos, or on the destination video? Or both?

Second question:

Does the manual setting of landmarks have to be done on all frames and must it be perfect? (stick perfectly to the facial features)

Third question:

How do you do on videos that last several minutes? (+15000 frames) By hand it really takes a lot of time, unless you have a tip to do it faster? (except taking 10 second videos of course :))

Fourth and last question:

If I use "Warp to landmarks" for training, is it still necessary to extract beforehand with a mask? Or is it one or the other?

Many thanks in advance for your help!
by torzdf » Fri Jul 10, 2020 9:16 am
lorenzosale wrote:
Thu Jul 09, 2020 9:06 am
Should the manual implementation of landmarks be done on the source photos / videos, or on the destination video? Or both?
Destination.
Does the manual setting of landmarks have to be done on all frames and must it be perfect? (stick perfectly to the facial features)
No. The alignments are not directly used in the swap process (outside of mask generation). However they are used to "align" the face. This means that from frame to frame they should be as close as possible to remove "jitter".
How do you do on videos that last several minutes? (+15000 frames) By hand it really takes a lot of time, unless you have a tip to do it faster? (except taking 10 second videos of course :))
Unfortunately it does take time, but you should probably only concentrate on alignments that are obviously wrong.
If I use "Warp to landmarks" for training, is it still necessary to extract beforehand with a mask? Or is it one or the other?
You don't need a mask for "Warp to landmarks" you just need an alignments file for the faces you are training (on both sides). However, I don't think Warp to Landmarks really offers any benefit, so I wouldn't bother using it.
Go to full post

User avatar
torzdf
Posts: 699
Joined: Fri Jul 12, 2019 12:53 am
Answers: 99
Has thanked: 19 times
Been thanked: 143 times

Re: Manual Landmarks

Post by torzdf »

lorenzosale wrote:
Thu Jul 09, 2020 9:06 am
Should the manual implementation of landmarks be done on the source photos / videos, or on the destination video? Or both?
Destination.
Does the manual setting of landmarks have to be done on all frames and must it be perfect? (stick perfectly to the facial features)
No. The alignments are not directly used in the swap process (outside of mask generation). However they are used to "align" the face. This means that from frame to frame they should be as close as possible to remove "jitter".
How do you do on videos that last several minutes? (+15000 frames) By hand it really takes a lot of time, unless you have a tip to do it faster? (except taking 10 second videos of course :))
Unfortunately it does take time, but you should probably only concentrate on alignments that are obviously wrong.
If I use "Warp to landmarks" for training, is it still necessary to extract beforehand with a mask? Or is it one or the other?
You don't need a mask for "Warp to landmarks" you just need an alignments file for the faces you are training (on both sides). However, I don't think Warp to Landmarks really offers any benefit, so I wouldn't bother using it.
My word is final

User avatar
lorenzosale
Posts: 11
Joined: Thu May 28, 2020 11:58 am
Has thanked: 1 time

Re: Manual Landmarks

Post by lorenzosale »

Thank you very much for your detailed answer, i can see more clearly now!

User avatar
calipheron
Posts: 15
Joined: Thu May 14, 2020 7:39 pm
Has thanked: 1 time

Re: Manual Landmarks

Post by calipheron »

I looked for a way to assist editing / clearing multiple garbage faces from long videos using the manual tool.
I ended up making an AutoHotKey script for this purpose.
As this post is in a similar ballpark, I thought I would post it here:

Code: Select all

^j::
Loop 100
{
Send {1}
Sleep 10
Send {d}
Sleep 10
Send {x}
Sleep 10
}
Send {Space}
Sleep 10
return
This script is designed to remove extra face landmarks from videos in large batches.
Not saying this is the best way to do it, but this is what I use.
I commonly find that after Faceswap has made an alignments file, where there are multiple faces in a frame, face 0 is fine, but faces 1 onwards are garbage.
So...

This script selects face landmark 1, deletes it, moves to the next frame. It then does that 100 times, as denoted by the Loop command. You can change this to taste. Once the loop is complete, the script sends a space bar press to save the alignments file.

Here's how I use it.
Enter the Manual landmarks editor. Set it to look for multiple faces in a frame. Set it to edit mode.
Run AutoHotkey and send it this script. Make sure the Manual editor has focus - I make sure to click once on the title bar of the video preview.
Hit CTRL+J (you can change this if you like).

You may have to go through the whole video a few times if there are any frames with more than 2 faces.
Once the alignments file is "cleaned", convert.
I've just experimented with the new spatial option, and then re-masked afterwards.
Hopefully this will help people make cleaner conversions.

Edit: Keep in mind, this may also remove legitimate / correct face landmarks but for me I have only very rarely encountered this. This does not distinguish between faces, it's a "dumb script". So keep that in mind before using!

User avatar
bryanlyon
Site Admin
Posts: 349
Joined: Fri Jul 12, 2019 12:49 am
Answers: 28
Location: San Francisco
Has thanked: 3 times
Been thanked: 93 times
Contact:

Re: Manual Landmarks

Post by bryanlyon »

calipheron wrote:
Mon Jul 13, 2020 5:12 pm
I looked for a way to assist editing / clearing multiple garbage faces from long videos using the manual tool.
I ended up making an AutoHotKey script for this purpose.
As this post is in a similar ballpark, I thought I would post it here:
The manual tool is definitely not the best tool for this. Instead I recommend using the "sort by face" option which will almost always put all the "garbage faces" together in one chunk and make it extremely easy to delete them en-masse.

User avatar
calipheron
Posts: 15
Joined: Thu May 14, 2020 7:39 pm
Has thanked: 1 time

Re: Manual Landmarks

Post by calipheron »

bryanlyon wrote:
Thu Jul 16, 2020 3:48 pm
calipheron wrote:
Mon Jul 13, 2020 5:12 pm
I looked for a way to assist editing / clearing multiple garbage faces from long videos using the manual tool.
I ended up making an AutoHotKey script for this purpose.
As this post is in a similar ballpark, I thought I would post it here:
The manual tool is definitely not the best tool for this. Instead I recommend using the "sort by face" option which will almost always put all the "garbage faces" together in one chunk and make it extremely easy to delete them en-masse.
I had no idea about this feature, I'll give it a go. Thanks Bryan.

Post Reply