How can i merge converted face to the original frame?

Got questions or tips about the Conversion process? This is the place to discuss them.


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Converting faces from your trained model.

If you are having issues with the Convert process not working as you would expect, then you should post in the Convert Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
carolYL
Posts: 3
Joined: Fri Apr 10, 2020 5:51 pm

How can i merge converted face to the original frame?

Post by carolYL »

Hi there,
I used faceswap to extract face images for person A and person B from two videos. I trained my own GAN models using the face images extracted and was able to use the model to get person B's face based on person A's face. Now I need to merge the converted faces to the original video frames. Can i somehow use faceswap's utilities for that purpose?

Thanks!

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: How can i merge converted face to the original frame?

Post by bryanlyon »

Faceswap only works with the built in models for convert. The conversion tool doesn't ever actually use the extracted face images, those are only used for training. There is no way to merge external swaps using Faceswap. You'd have to significantly modify how Faceswap handles conversion to be able to recomposite those images back.

User avatar
carolYL
Posts: 3
Joined: Fri Apr 10, 2020 5:51 pm

Re: How can i merge converted face to the original frame?

Post by carolYL »

Thank you very much for the quick reply. Is there anyway i can take advantage of faceswap's converter using my own model(my model was written in PyTorch)?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: How can i merge converted face to the original frame?

Post by bryanlyon »

You'd have to do some serious modifications to the convert process. If you have knowledge and experience with Python, you should be able to pull it off. Feel free to see how it works, start with scripts/convert and follow the process.

User avatar
carolYL
Posts: 3
Joined: Fri Apr 10, 2020 5:51 pm

Re: How can i merge converted face to the original frame?

Post by carolYL »

I have a question about _queue_out_frames function in class Predict(). This function seems to be putting the swapped faces back to the original frames, and it seems to be using item["swapped_faces"] = swapped_faces[pointer:pointer + num_faces] to do that, but i do not know where/what function does the actual merging. Could you please elaborate how the swapped faces are put back to the frame? (like how the alignment file is being used and which function is used to do the merging(where the color adjustment option, mask type option, scaling option are used))Thanks!

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: How can i merge converted face to the original frame?

Post by torzdf »

My word is final

Locked