Faceswap Forum

in my converted video I'm noticing 3 key issues

1- The face is shaky/jitters. I managed to fix this a little with the spatial alignments tool, but its still a very noticeable problem. Is this caused by too many of a similar reference in the training set?

2- The features blur during particular facial expressions. Particularly the bottom lip and mouth blur and fade when opening wide. I figure this would be a lack of data for that particular expression on one of/both sets, but that shape definitely exists in both sets.

3- The inside mouth and tongue are unconvincing. The video is up close, so you can tell when their lips and tongue don't quite make the correct shape for that word. I have lots of up close talking footage of both actors in the sets, is there anything to keep in mind to improve this?

I'm unsure of how to "properly" build a training set. I'm worried I don't have enough images, but also that I have too many of the angle. How can I tell what might be lacking and what might be too much?

CProdDigital wrote: ↑Thu Nov 10, 2022 7:25 pm

1- The face is shaky/jitters. I managed to fix this a little with the spatial alignments tool, but its still a very noticeable problem. Is this caused by too many of a similar reference in the training set?

This is likely to be a combination of data and shaky alignments.... For the latter, see the re-feed section in the Extract guide

2- The features blur during particular facial expressions. Particularly the bottom lip and mouth blur and fade when opening wide. I figure this would be a lack of data for that particular expression on one of/both sets, but that shape definitely exists in both sets.

3- The inside mouth and tongue are unconvincing. The video is up close, so you can tell when their lips and tongue don't quite make the correct shape for that word. I have lots of up close talking footage of both actors in the sets, is there anything to keep in mind to improve this?

These second 2 are most likely data issues, especially if the variety of data is of a similar quality to that which you posted the other day.

I'm unsure of how to "properly" build a training set. I'm worried I don't have enough images, but also that I have too many of the angle. How can I tell what might be lacking and what might be too much?

The sad fact is this comes with experience. If your data doesn't look highly varied in terms of pose, expression, color, lighting etc. then it probably isn't. The model will always learn the features of an image best where it sees most examples of that type of image.

Thank you so much for all your help dawg, I notice youre the one replying to everyone (Mine) posts and being so helpful I really appreciate it.

torzdf wrote: ↑Fri Nov 11, 2022 9:28 am

The sad fact is this comes with experience. If your data doesn't look highly varied in terms of pose, expression, color, lighting etc. then it probably isn't. The model will always learn the features of an image best where it sees most examples of that type of image.

so is using Head mounted camera footage bad in general? Because extraction will result in many many similar faces? I was hoping that having so much of the head-on footage would help it since it would have reference for basically every facial expression from a clear perspective. Is having lots of HMC frames actively detrimental to the training, even if I have other angles and stuff aswell? I thought maybe the jitter was caused by having too many of the same image, as in it confused and going back and forth between many similar references.

Id say I have about 800 front-on HMC frames of basically the same lighting etc, and then 300 frames gathered from other videos and a handful of high-ish res images from a variety of angles and lighting. about the same variety for both A and B actors.

CProdDigital wrote: ↑Mon Nov 14, 2022 9:23 pm

I thought maybe the jitter was caused by having too many of the same image, as in it confused and going back and forth between many similar references.

This is possible, but I have done no experiments in the area, so couldn't confirm one way or the other.

Id say I have about 800 front-on HMC frames of basically the same lighting etc, and then 300 frames gathered from other videos and a handful of high-ish res images from a variety of angles and lighting. about the same variety for both A and B actors.

I would try and balance out that distribution a bit, Also, try to get more footage if possible.