First swap - Looking for advice (MKBHD x Michael Jordan)

Display post links to interesting swaps


Locked
User avatar
jebanke
Posts: 5
Joined: Tue Sep 14, 2021 5:04 pm
Been thanked: 3 times

First swap - Looking for advice (MKBHD x Michael Jordan)

Post by jebanke »

Hello faceswap community,

I want to surprise a friend of mine who plays basketball by swapping Michael Jordan's face onto his head for his next game. I've never done deepfakes before, hence me posting here and showing you my progress. This is my first post, and I bet you'll see at least a few more detailing my progress because I want to do things right.

What was my goal for this first attempt :

The goal of this first deepfake was to understand the workflow, the concept, etc... But most importantly to test something very specific : whether I can get away with very little footage of MKBHD/my friend.
Why? Because if it works well enough, it would mean that I can just film my friend using a simple lighting setup, and get the "shoot" over with in less than half an hour - so as to not raise any suspicion on his part.

Here's what I did for my extract :

  • For the MKBHD footage, I used a simple video of just him talking about a product. Here is that video.

  • For the Michael Jordan footage, I used snippets of "The Last Dance" as well as footage from "Space Jam (1996)". Here is that video.

  • If my memory serves me well, I extracted every third frame out of these two .MP4s, which yielded 5353 frames of MKBHD and 5095 frames of Michael Jordan after cleaning up.

As per the guide says, I cleaned up my extraction :

  • For the Michael Jordan footage, I sorted and deleted unwanted faces, cleaned the alignements, then went into manual mode, cleaned some more by removing duplicate faces and other faces that were too far gone to be salvaged. I also tried to reposition the bounding-box whenever I felt it needed to be. I did not do much masking at all, in the case of the MJ I felt that it was mostly good enough, so I left some misaligned faces and obstructed faces in there. Here is an overview of what my alignments looked like, captured from the Manual tool (play it at 0.25x speed):

  • The MKBHD footage aligned properly from the get-go, I just had to remove some unwanted faces and cleaned the alignments file accordingly. I did not video-capture the alignments from the Manual tool for this reason.

Here's what I did for my training :

I'm running two RTX 2060S on my computer. From what I understand, this means I can use up to 8Gb of VRAM.

I tried running Villain with default settings, but Faceswap ran out of VRAM.
Then I (foolishly) tried running Phaze-A with default settings, but Faceswap ran out of VRAM too.
Then my finger slipped and I ran the Original model with default settings for 250k iterations. That seemed to hold up. It did crash after 12 hours of running but progress was saved so all was good.

Here's an overview of my graph when all was said and done, along with a timelapse of the training (nice feature) :

Image

Here's what the end swap looks like :

"This is the best deepfake that's ever been created", said no one. But hey, I wasn't going to ace it anyway. This is a very iterative process we're talking about. Right?

Now onto my observations and conclusions, which I hope somebody can help me validate/invalidate :

Observations :

  • It just kinda looks like a blurry MKBHD. They don't bear a significant resemblance IRL, but in my deepfake, I feel like I'm looking at Marques' weird blurry cousin. I don't really "see" Michael Jordan that much. (I did train the right way around, trust me)

  • The image is very mushy, even from a far.

  • The hair is disctracting as hell. I think that if MKBHD didn't have that little beard and that hair, it would've been a little more convincing.

  • There is some flickering going on in the top corners of the deepfake.

Conclusions :

  • I should have used a bald guy for my second face. In some shots, if I cover up Marques' hair with my thumb on my screen, it starts to resemble MJ.

  • I should probably only select the sharpest, most crisp footage of MJ, even if it means I only have 2000~ images to feed Faceswap when it comes to training.

  • Checking the "No-warp" helped towards the end.

  • I should find a way to output something bigger than 64px. In that regard, will lowering the batch size of - for example - the Villain model, enable me to use it?

  • Having only a little bit of footage of MKBHD seemed to work well enough for what I wanted to do. EDIT: Nevermind, it seems that having only a little bit of footage of MKBHD might be one of the (if not THE) reason that my swap looks like crap

Here's what I'm going to do for my next test (open to suggestions):

  • Find a black basket ball player who happens to have no hair and no facial hair to source footage from.

  • Trim down my source footage of Michael Jordan, while trying to keep only the best and sharpest footage.

  • Better understand the Manual tool, to better cleanup my MJ data set.

  • Try another model (open to suggestions), with a lower batch size (if that's what's going to enable me to use another model)

Last edited by jebanke on Wed Sep 15, 2021 5:34 pm, edited 1 time in total.
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: First swap - Looking for advice (MKBHD x Michael Jordan)

Post by torzdf »

Thanks for sharing your journey. I think/hope this will be helpful for other users.

re: Phaze-A. Default settings should work on 4GB with Mixed Precision enabled, so I would guess that Windows is taking more VRAM than it should. If you can try on a Linux distro, then I would recommend this.

A note about dual GPUs. The model itself still needs to be able to fit on 1 of the GPUs (so 4GB in your case). Villain definitely won't. I deliberately targeted 4GB when creating the Phaze-A default. Having multiple GPUs then let's you increase the batchsize beyond what you could hold on 1 card.

My word is final

User avatar
jebanke
Posts: 5
Joined: Tue Sep 14, 2021 5:04 pm
Been thanked: 3 times

Re: First swap - Looking for advice (MKBHD x Michael Jordan)

Post by jebanke »

Thank you for your answer.

So I guess I will have to try Phaze-A, I'm not quite sure what would be taking up so much VRAM on my computer, I'll see how I can optimize that.

On the topic of the swap itself, do you think the blurriness/overall bad fidelity to MJ's features/flickering is down to using an old model? Or do you think it's also down to my source footage? And if it's the latter, would you have any advice for me regarding the source footage (be even more selective in what I feed the model/give it some part of the faceset to begin training, and sharper frames to finish/something else)?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: First swap - Looking for advice (MKBHD x Michael Jordan)

Post by bryanlyon »

Variety. Quite simply, variety is 100% required for a good swap and you don't have enough. You won't get a great result with that data no matter what model you use our how long you train it.

You need more lighting and pose at a minimum.

The more variety the better.

User avatar
jebanke
Posts: 5
Joined: Tue Sep 14, 2021 5:04 pm
Been thanked: 3 times

Re: First swap - Looking for advice (MKBHD x Michael Jordan)

Post by jebanke »

Thank you for your answer!

The Michael Jordan footage seemed relatively fine to me, in terms of variety. When you say I don't have enough data, are you refering to the MKBHD footage then? (If yes, then am I correct in assuming that the less-varied MKBHD footage would be the reason that the end swap looks more like MKBHD? I have a hard time wrapping my head around that, to be honest!)

Just so you see both sources :

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: First swap - Looking for advice (MKBHD x Michael Jordan)

Post by bryanlyon »

Both sides need variety. The B side also needs to worry more about quality than the A side. It'd be easier to get a good swap ONTO Michael than from him due to the age and limitations of your data.

I'd suggest cleaning up your Michael data, for a fit train, though you may need to use more sources (Maybe some interviews or "Behind the Scenes" content are available for both movies? Or maybe some photos can be used to get some high quality sources.) Sometimes you just need to get more data, even if you think it's bad, and then just remove it for the fit training.

For MKBHD I'd at a minimum grab some additional videos. I know it's hard to get variety of youtube presenters like him since they're in studios so often. Maybe some of his car reviews where he's in a car so the angle and lighting is different? Or VLOGs where he's at a trade show?

Locked