Should I ever expect convergence? Couple of questions.

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
shambler74
Posts: 1
Joined: Fri Mar 06, 2020 12:23 pm

Should I ever expect convergence? Couple of questions.

Post by shambler74 »

Good morning! New to DeepFakes and FaceSwap.py in general. I have a couple questions and was hoping to hear from others in the community about them.

1). Will I ever get convergence?
--- Context: Face A has 7000 images, with multiple light sources and 100's of images from every direction including deep side and up/down. Face B has 2400 images with multiple light sources and not as many as face A side to side/up to down but has a decent amount. Knowing the above, should I expect convergence? To clarify, convergence is when I watch the sample images and none of the swapped faces are blurry, right?

2). Iterations for Original/Villain
--- I've read the other threads and know to use my eyes more than iterations or the graph, but I also read I should expect at least 400k iterations for original before convergence becomes a reality. Is this soft limit still in place? If not, are my eyes the true guiding light, when I see no blurry images from A->B in the sample/preview pane?

3). What type of monster machine is needed to run Villain on > 8 batch size?
--- Looking for more of "what you run and how you run it" in a response here. My main gaming rig is a 64 core AMD Threadripper with an nVidia Titan V. I can run 5k games with no problem, cruch ETH mining at 110MH/s with the unclocked Titan V rails and yet Villain will only run at batch size 8. Any larger and it will OOM me. Is this a VRAM size thing? The titan V has 12 GB of vram so I should be able to do batch sizes of 16 right? Why then does it cop out under that?

Thanks for your time reading this thread!

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Should I ever expect convergence? Couple of questions.

Post by bryanlyon »

On your questions:

1: Convergence isn't where the face is blurry, but where the model has learned all that it could. Either because you've hit the limit of your data, or the model. The face could still be blurry.

2: Original will probably converge in a couple days, and just not learn more. Villain can take a week or more to reach convergence. Don't look at iteration count as that's effectively a nonsense number.

3: It's a simple matter of vram. Honestly I'm surprised you can get a BS of 8 on Villain. People tend to struggle with it as it's a very heavy model.

User avatar
tiazgriff
Posts: 7
Joined: Mon Jan 20, 2020 8:45 pm

Re: Should I ever expect convergence? Couple of questions.

Post by tiazgriff »

Im running Villain on 1080 with batch size 5 no lowmem and playin Wow at the same time. Starting to look good at 300k iters

User avatar
Metalstylite
Posts: 1
Joined: Wed May 13, 2020 7:46 am

Re: Should I ever expect convergence? Couple of questions.

Post by Metalstylite »

tiazgriff wrote: Fri Mar 06, 2020 7:35 pm

Im running Villain on 1080 with batch size 5 no lowmem and playin Wow at the same time. Starting to look good at 300k iters

I used 1080 too.
But with side A 5k imgs, side B 2k2 imgs, it can handle with bs 4 no lowmem setting and 6 with lowmem.
I dont want trade off speed with bs 8 (memory saving). How can you do it when training and play WoW at sametime ? I can only surf web or view movie only while training. :cry:

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Should I ever expect convergence? Couple of questions.

Post by bryanlyon »

1080 can easily handle larger bs than that, make sure to close chrome or any other applications using vram.

Locked