Page 1 of 1

Preview Tool out of sync with Convert?

Posted: Mon Jun 06, 2022 2:17 pm
by Diamondback

tl;dr
My video conversion has an artifact on the forehead that kind of looks like hair, but is mostly a dark splotch.

I am not sure what is going on. The swaps during training look quite good. And, the preview tool shows decent swaps as well. However, when I duplicate all my settings from the preview tool, the conversion results in most of the faces having a dark splotch across the forehead of the face. Sometimes it looks like hair. I suspect this may be a masking issue (using Bisenet-Fp Face) where the hairline of Face B is bleeding over onto the the larger forehead of Face A.

My bigger concern is that I do not see this defect in the Preview Tool, so, it is very difficult to correct. Maybe I am missing something, but it seems like the Preview and the Convert are not working with the same data or possibly a different algorithm.? TBH, I have struggled with the Preview tool. Many of the adjustments seem to produce little to no effect on the swapped faces. But, a few of them do. I am a bit color-blind so maybe I just cannot see the shifts. But, I definitely see the problems in the converted video, so ???

I will play with my masks and may follow up if I find something significant.

P.S. - I attempted to convert using Bisenet-Fp Head, but I receive an error that the mask is not stored in the alignments file. I created my Alignments with Bisenet-Fp, and was even sure to regenerate the masks with the Mask Tool. But, I do not see an option for 'Face' or 'Head' anywhere except in the Convert panel. How does one generate Bisenet-Fp Head masks?


Re: Preview Tool out of sync with Convert?

Posted: Mon Jun 06, 2022 2:26 pm
by Diamondback

Follow up -

I tried my conversion with the Extended mask and the results were much more satisfying. I suspect that the Preview Tool only uses the extended mask and does not incorporate any of the other mask data in the previews.


Re: Preview Tool out of sync with Convert?

Posted: Tue Jun 07, 2022 9:17 am
by torzdf

I suspect this may be a masking issue (using Bisenet-Fp Face) where the hairline of Face B is bleeding over onto the the larger forehead of Face A.

This is exactly what this is. Welcome to the law of unintended consequences. Implementing full face masking = a good thing. Unintended consequence: swapping to someone with a lower hairline presents challenges. I am looking at potential 'directional mask erosion' solutions for this.

You have identified the most likely main issue that you are facing, so I will skip that, but...

Diamondback wrote: Mon Jun 06, 2022 2:26 pm

I tried my conversion with the Extended mask and the results were much more satisfying. I suspect that the Preview Tool only uses the extended mask and does not incorporate any of the other mask data in the previews.

Any mask available in convert is available in the preview tool, but you need to select it:
Image

This is covered in convert guide, so you may need a refresher.

How does one generate Bisenet-Fp Head masks?

A bisenet-fp-face mask is generated when hair is not selected for inclusion in the mask's plugin settings. When you choose to include hair, then a bisenet-fp-head mask is generated.

It is only worth using if you have explicitly done full head training, with a mask that includes the hair. This brings with it its own challenges though.


Re: Preview Tool out of sync with Convert?

Posted: Tue Jun 07, 2022 6:37 pm
by torzdf

FWIW, I have pushed an update now which let's you erode the mask from any of 4 sides, so you can pull the mask in from the hairline now.


Re: Preview Tool out of sync with Convert?

Posted: Wed Jun 08, 2022 4:45 pm
by Diamondback

[quote=torzdf post_id=6960 time=1654593420 user_id=48]
Any mask available in convert is available in the preview tool, but you need to select it:
Image

This is covered in convert guide, so you may need a refresher.

[quote]

Ah yes. Thanks. It was staring me in the face. Duh.


Re: Preview Tool out of sync with Convert?

Posted: Mon Jun 13, 2022 11:44 pm
by imanoven

Hi,

New to the forum. Just wanted to add that there seems to be an issue with this new option. The ability to dilate individual directions of the mask would be amazing, but doesn't seem to be working. I can get the erosion slider to erode the face, but when going into negative values, no dilation occurs. :(

Would love to see this up and running as I have swapped face that is bigger than the target and need dilation to occur.

If there's not an issue with this and I'm just using the tool incorrectly, my apologies.

Thanks!


Re: Preview Tool out of sync with Convert?

Posted: Tue Jun 14, 2022 8:06 am
by torzdf

if there is any way you can post a short video demonstrating what you mean, that would be useful, or give me exact steps to reproduce. It seems to work for me ok in testing.


Re: Preview Tool out of sync with Convert?

Posted: Wed Jun 15, 2022 12:39 am
by imanoven

Thanks so much for looking into it. I will try to explain in greater detail.

I'm running the Preview tool and when I use the Mask Blend option to erode/dilate, it will erode the mask, but not dilate it. Positive values of the Erosion slider will erode the mask, but when going into (-) negative values, no dilation occurs. Dilation of the entire mask had worked before this most recent update--but now is not--at least not for me. All of the directional (top, bottom, right, left) erosion sliders produce an erosion of the mask as well. However, none of the sliders produce any dilation when moved into negative values.

I've tried this on a several Realface models and Phaze-A models. I have tried this with Extended, Component, and Bisnet-Face selected in Mask Type. I'm running the newest update of Faceswap. Hopefully that makes more sense?

Thanks again!


Re: Preview Tool out of sync with Convert?

Posted: Thu Jun 16, 2022 1:21 am
by torzdf

I've pushed an update which hopefully fixes this. Please update and let me know.


Re: Preview Tool out of sync with Convert?

Posted: Fri Jun 17, 2022 7:16 am
by imanoven

Yes-awesome! That fixed it--it now dilates! Thanks very much!