Swapping Jeff Bezos and Elon Musk: blurry result

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
john
Posts: 2
Joined: Wed Jan 08, 2020 2:11 pm
Has thanked: 1 time

Swapping Jeff Bezos and Elon Musk: blurry result

Post by john »

The last couple of days I've been playing with Faceswap and created a couple of demo videos in which I swap the face of Jeff Bezos with that of Elon Musk. You can see the result here: https://streamable.com/7w1g6.

The videos I used to train the model are:

I extracted frames in which only the face of the subject occurs and randomly selected about 8000 frames from the video. Then I extracted the faces using:

Code: Select all

python faceswap.py extract -i <input> -o <output> -D s3fd -A fan -nm hist

The model used for training is the Villain model:

Code: Select all

python faceswap.py train -A <A> -ala <A fsa> -B <B> -alb <B fsa> -m <model> -t villain -bs 32 -nl -g 4

I changed the following parameters in train.ini

Code: Select all

coverage = 70.0
loss_function = ssim
reflect_padding = True

Training was done on an AWS p3.8xlarge instance containing 4 Tesla V100 GPU's for 10,175 iterations. The lowest average loss dropped to 0.01620 for A and 0.01586 for B.

To me the result still looks more blurry than I expected. Do you feel the same? Can this be caused by the resolution of the faces that are swapped? In the demo video they are around 275x275 pixels while the output of the network is 128x128. Is it possible to train the Villain network with a larger in- and output size? Or do you have any other suggestions to make the result less blurry?

Much appreciated!

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Swapping Jeff Bezos and Elon Musk: blurry result

Post by bryanlyon »

Some suggestions:

1: Use more than one video for each face.

The more the better, having just one video per side is going to lead to poor results

2: Use non-youtube videos.

Youtube videos are highly compressed and lose a lot of detail. It's fine to use them as part of your dataset, but it's best if you can find higher quality sources.

3: Find videos with closer views of the faces

Videos with small faces will lead to poor quality results, try to find larger faces where the face takes up more of the shot.

See viewtopic.php?f=6&t=74 where I go more into depth on these points and others to help ensure you get a good dataset.

User avatar
john
Posts: 2
Joined: Wed Jan 08, 2020 2:11 pm
Has thanked: 1 time

Re: Swapping Jeff Bezos and Elon Musk: blurry result

Post by john »

Thanks for your reply [mention]bryanlyon[/mention]! Can you explain how "close ups" are improving the quality of the swap? The output of the Villain network is 128x128 pixels, so this will be upscaled to the size of the face in the result, right?

And do you think increasing the in- and output size of the Villain network is feasible?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Swapping Jeff Bezos and Elon Musk: blurry result

Post by bryanlyon »

You want the training faces to be at least that 128x or you're training to lower quality faces and you wont get full quality for the swap.

Villain is already a very heavy model. Without some exotic hardware with a lot of vram, it's pretty much as big as it can get.

Locked