Preview Tool out of sync with Convert?

Got questions or tips about the Conversion process? This is the place to discuss them.


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Converting faces from your trained model.

If you are having issues with the Convert process not working as you would expect, then you should post in the Convert Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Diamondback
Posts: 9
Joined: Fri May 14, 2021 4:50 pm
Has thanked: 4 times
Been thanked: 2 times

Preview Tool out of sync with Convert?

Post by Diamondback »

tl;dr
My video conversion has an artifact on the forehead that kind of looks like hair, but is mostly a dark splotch.

I am not sure what is going on. The swaps during training look quite good. And, the preview tool shows decent swaps as well. However, when I duplicate all my settings from the preview tool, the conversion results in most of the faces having a dark splotch across the forehead of the face. Sometimes it looks like hair. I suspect this may be a masking issue (using Bisenet-Fp Face) where the hairline of Face B is bleeding over onto the the larger forehead of Face A.

My bigger concern is that I do not see this defect in the Preview Tool, so, it is very difficult to correct. Maybe I am missing something, but it seems like the Preview and the Convert are not working with the same data or possibly a different algorithm.? TBH, I have struggled with the Preview tool. Many of the adjustments seem to produce little to no effect on the swapped faces. But, a few of them do. I am a bit color-blind so maybe I just cannot see the shifts. But, I definitely see the problems in the converted video, so ???

I will play with my masks and may follow up if I find something significant.

P.S. - I attempted to convert using Bisenet-Fp Head, but I receive an error that the mask is not stored in the alignments file. I created my Alignments with Bisenet-Fp, and was even sure to regenerate the masks with the Mask Tool. But, I do not see an option for 'Face' or 'Head' anywhere except in the Convert panel. How does one generate Bisenet-Fp Head masks?

User avatar
Diamondback
Posts: 9
Joined: Fri May 14, 2021 4:50 pm
Has thanked: 4 times
Been thanked: 2 times

Re: Preview Tool out of sync with Convert?

Post by Diamondback »

Follow up -

I tried my conversion with the Extended mask and the results were much more satisfying. I suspect that the Preview Tool only uses the extended mask and does not incorporate any of the other mask data in the previews.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Preview Tool out of sync with Convert?

Post by torzdf »

I suspect this may be a masking issue (using Bisenet-Fp Face) where the hairline of Face B is bleeding over onto the the larger forehead of Face A.

This is exactly what this is. Welcome to the law of unintended consequences. Implementing full face masking = a good thing. Unintended consequence: swapping to someone with a lower hairline presents challenges. I am looking at potential 'directional mask erosion' solutions for this.

You have identified the most likely main issue that you are facing, so I will skip that, but...

Diamondback wrote: Mon Jun 06, 2022 2:26 pm

I tried my conversion with the Extended mask and the results were much more satisfying. I suspect that the Preview Tool only uses the extended mask and does not incorporate any of the other mask data in the previews.

Any mask available in convert is available in the preview tool, but you need to select it:
Image

This is covered in convert guide, so you may need a refresher.

How does one generate Bisenet-Fp Head masks?

A bisenet-fp-face mask is generated when hair is not selected for inclusion in the mask's plugin settings. When you choose to include hair, then a bisenet-fp-head mask is generated.

It is only worth using if you have explicitly done full head training, with a mask that includes the hair. This brings with it its own challenges though.

My word is final

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Preview Tool out of sync with Convert?

Post by torzdf »

FWIW, I have pushed an update now which let's you erode the mask from any of 4 sides, so you can pull the mask in from the hairline now.

My word is final

User avatar
Diamondback
Posts: 9
Joined: Fri May 14, 2021 4:50 pm
Has thanked: 4 times
Been thanked: 2 times

Re: Preview Tool out of sync with Convert?

Post by Diamondback »

[quote=torzdf post_id=6960 time=1654593420 user_id=48]
Any mask available in convert is available in the preview tool, but you need to select it:
Image

This is covered in convert guide, so you may need a refresher.

[quote]

Ah yes. Thanks. It was staring me in the face. Duh.

User avatar
imanoven
Posts: 3
Joined: Mon Jun 13, 2022 11:33 pm
Has thanked: 2 times
Been thanked: 1 time

Re: Preview Tool out of sync with Convert?

Post by imanoven »

Hi,

New to the forum. Just wanted to add that there seems to be an issue with this new option. The ability to dilate individual directions of the mask would be amazing, but doesn't seem to be working. I can get the erosion slider to erode the face, but when going into negative values, no dilation occurs. :(

Would love to see this up and running as I have swapped face that is bigger than the target and need dilation to occur.

If there's not an issue with this and I'm just using the tool incorrectly, my apologies.

Thanks!

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Preview Tool out of sync with Convert?

Post by torzdf »

if there is any way you can post a short video demonstrating what you mean, that would be useful, or give me exact steps to reproduce. It seems to work for me ok in testing.

My word is final

User avatar
imanoven
Posts: 3
Joined: Mon Jun 13, 2022 11:33 pm
Has thanked: 2 times
Been thanked: 1 time

Re: Preview Tool out of sync with Convert?

Post by imanoven »

Thanks so much for looking into it. I will try to explain in greater detail.

I'm running the Preview tool and when I use the Mask Blend option to erode/dilate, it will erode the mask, but not dilate it. Positive values of the Erosion slider will erode the mask, but when going into (-) negative values, no dilation occurs. Dilation of the entire mask had worked before this most recent update--but now is not--at least not for me. All of the directional (top, bottom, right, left) erosion sliders produce an erosion of the mask as well. However, none of the sliders produce any dilation when moved into negative values.

I've tried this on a several Realface models and Phaze-A models. I have tried this with Extended, Component, and Bisnet-Face selected in Mask Type. I'm running the newest update of Faceswap. Hopefully that makes more sense?

Thanks again!

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Preview Tool out of sync with Convert?

Post by torzdf »

I've pushed an update which hopefully fixes this. Please update and let me know.

My word is final

User avatar
imanoven
Posts: 3
Joined: Mon Jun 13, 2022 11:33 pm
Has thanked: 2 times
Been thanked: 1 time

Re: Preview Tool out of sync with Convert?

Post by imanoven »

Yes-awesome! That fixed it--it now dilates! Thanks very much!

Locked