Here's the quick and dirty bullet list of the entire process, start to finish as I understand it (am I missing anything?

Discussions about research, Faceswapping and things that don't fit in the other categories here.


Locked
User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Here's the quick and dirty bullet list of the entire process, start to finish as I understand it (am I missing anything?

Post by unkempt »

Scenario: Subject is the person who's face I want to be able to put on videos of other people.

  1. I gather source videos and photos of them that clearly show their face from various angles.
  2. Using the extraction guide, I let faceswap pull faces from the folder, sort out the bad ones, and am left with faces of various kinds

Question: I can't use the manual tool on photos so I have to use clear and simple photos without crazy angles and obstructions over the face, yes?

  1. Get target video (the face to be replaced).
  2. Extract target - make sure not to forget to skip frames so you don't end up with billions of them. Leave the mask OFF unless you think you won't need to do any manual adjustments (like when they have something over their face). -- edit. Nevermind. If you don't select your mask at this step, it might not be an option when doing manual later.
  3. Use the manual tool to look for faces that got weird at strong angles. Mask tool to deal with face obstructions(like if they're drinking from a bottle or put their hand in front of their face) - make sure to select your mask type right away or you'll do a lot of work for nothing)
  4. SAVE the changes!
  5. Delete the original faces you extracted and use the "Extract to folder" button next to the save to re-extract the same faces, but with the improved geometry and masks.
  6. Close the manual tool and open the MASK tool
  7. Use "only missing" to apply the same mask selected during the manual process to images you didn't fiddle with - target the folder of extracted faces

With this I now have my source folder of faces and my target folder of faces and can train.

QUESTION: would I want to mess with masking during the training phase? Training doesn't seem to care about custom masks at all

TRAINING

  1. Set inputs
  2. Run with "warp to landmarks" for a long time. Pretty much until you can clearly see eyes/teeth (or they're clearly not improving).
  3. Stop, deselect "warp to landmarks". Select "no warp" instead. Run again for a while.

QUESTION: Despite doing the manual process above, I'm not seeing the proper mask in the previews. It's as if my masks don't exist. Why? Aren't the masks part of the files? Or do they only come into play during conversion?

CONVERSION

  1. Repeat the process for extraction, except choose every frame instead of skipping

Question: if I use skip existing while extracting, it's telling me all fames are present, but when I open the manual tool, only about 100 are. What am I doing wrong?

  1. Use the manual tool to look for poor geometry and fix them
  2. Use the masking tool (this could take forever depending on the number of frames and how many have a face obstruction)
  3. Save. No need to extract this time because you're not using images; the alignments and masking are stored in the alignments file.
  4. Set conversion settings and roll. YOLO
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Here's the quick and dirty bullet list of the entire process, start to finish as I understand it (am I missing anyth

Post by torzdf »

unkempt wrote: Mon Jan 16, 2023 1:59 pm

Question: I can't use the manual tool on photos so I have to use clear and simple photos without crazy angles and obstructions over the face, yes?

Why not? The manual tool should work on both source photos/frames or a video. The only thing it won't work with is extracted faces.

  1. Extract target - make sure not to forget to skip frames so you don't end up with billions of them. Leave the mask OFF unless you think you won't need to do any manual adjustments (like when they have something over their face). -- edit. Nevermind. If you don't select your mask at this step, it might not be an option when doing manual later.

I think you're missing something here. I never generate excess masks at extract time. I extract -> clean -> fix alignments. Then I use the Mask Tool to generate the masks I want to use, and use the manual tool to fix up masks before regenerating the fixed face-set

On your points 5-7:
Personally I would not re-extract faces until I have completed fixing + masking.... Anything that the manual tool shows you is what exists in the alignments file, and what will ultimately be reflected in your extracted face png-headers when you re-extract. With this is mind, the alignments file is your "master" document. With that known, I would be

  • extracting
  • deleting junk/fixing alignments + making sure alignments file is correct in terms of face landmarks
  • Using the mask tool with the source video/frames and updating the mask I want into the alignments file
  • Use the manual tool to fix up any masks (as, at this point, you have already fixed all your landmarks and are just fixing masks, you should not need to worry about any masks getting deleted)
    • In the event that I have had to make some landmark changes during the mask-editing session, I would run the Mask Tool against the source frames/video + alignments file, to get the missing masks put into the alignments file (the master document)
  • Finally, use the alignments tool to re-extract the face set with your corrected data

QUESTION: would I want to mess with masking during the training phase? Training doesn't seem to care about custom masks at all

Not sure what you mean about this. Training definitely does care about masks. You very much want to train with the same mask type that you plan to convert with. With that says, I won't spend anywhere near as much time cleaning up masks for training as I will for convert. My general view is that bisenet-fp performs pretty well (on obstructions and all), and with the amount of data that the model is seeing, small masking errors should get averaged out over time.

  1. Run with "warp to landmarks" for a long time. Pretty much until you can clearly see eyes/teeth (or they're clearly not improving).

Personally, I wouldn't worry about warp-to-landmarks. It is just an alternate method for applying the warp augmentation, but it does come at a bit of a speed overhead. Some people swear by it, but I have never seen any benefit over the standard warp method.

  1. Stop, deselect "warp to landmarks". Select "no warp" instead. Run again for a while.

See above You want to be enabling "no-warp" not deselecting "warp to landmarks". All the latter method is doing is changing the method uses to warp, not disable warp altogether.

QUESTION: Despite doing the manual process above, I'm not seeing the proper mask in the previews. It's as if my masks don't exist. Why? Aren't the masks part of the files? Or do they only come into play during conversion?

Most likely you have selected the wrong mask type in training options. I have never seen the wrong mask just showing.

Question: if I use skip existing while extracting, it's telling me all fames are present, but when I open the manual tool, only about 100 are. What am I doing wrong?

I don't use skip-existing. If I were to guess though, I would say its because all the frames get reflected in the alignments file, just 1 in 10 have faces populated (assuming an EEN of 10). skip-existing-faces would probably resolve that issue. HOWEVER. I'm not sure why you would need to go back with the same alignments file and select "skip-existing". A better work flow (for a video you intend to convert + train with) is to:

  • do the initial extraction at EEN=1 (ie extract every frame)
  • clean alignments/mask (as earlier)
  • Use the alignments tool to select a subset of these cleaned faces for training (extract job + een)
  1. Use the masking tool (this could take forever depending on the number of frames and how many have a face obstruction)

You've said this a few times (re: obstructions). Masking is definitely the most time consuming part, but bisenet-fp should do a pretty could job of getting you there. Which masker are you using?

My word is final

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Here's the quick and dirty bullet list of the entire process, start to finish as I understand it (am I missing anyth

Post by unkempt »

Ok, I definitely missed a few things in there. For example, after cleaning all the masks and alignments, I didn't use the alignments tool to get faces. And I always used exactly one of the warping options, not both. Let me try again...

Also, thank you!

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Here's the quick and dirty bullet list of the entire process, start to finish as I understand it (am I missing anyth

Post by unkempt »

Well my masks are all gone. they've been reset somehow... I'll have to start over again :(

Hopefully I figure this out eventually. It's a lot of work when you get something wrong. I don't suppose there's an idea out there to update the software to warn if any action you're taking would overwrite the settings in your alignments file?

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Here's the quick and dirty bullet list of the entire process, start to finish as I understand it (am I missing anyth

Post by unkempt »

I've been having the most luck with the mask tool not getting confused with obstructions with unetdfl. You're saying that Bisnet-fp will work ok for that? Like if someone is resting their head on their fist and the fist is partially covering their face.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Here's the quick and dirty bullet list of the entire process, start to finish as I understand it (am I missing anyth

Post by torzdf »

My word is final

Locked