RTX 4090 too slow?

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
alaric
Posts: 2
Joined: Tue Apr 18, 2023 8:37 pm

RTX 4090 too slow?

Post by alaric »

Hello all,
I am still quite a beginner but I wonder the following:
I have started a test project.
During the training I used the original model with a batch size of 16. Snapshot interval is at 250.

Faceswap is training at about 1k iterations per minute. Is this normal? I am using the RTX 4090, the GPU seems to be quite bored at least according to task manager. So it looks like the available resources are not being used optimally.

Or am I expecting too much?

Thanks for your assessment

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: RTX 4090 too slow?

Post by bryanlyon »

Task manager is an awful way to verify GPU usage. See app.php/faqpage#f0r3 for more info.

Original on a 4090 is probably a bit of a bottleneck. Honestly I'd recommend trying out DFaker or Phaze A (stojo preset) instead. They both have much higher quality and can give you better results. Original uses just 64x64 images so it's quite possible that loading and preparing the images is becoming the bottleneck in that situation.

Also, setting a snapshot to every 250 iterations means you're going to fill up your system with snapshots. I highly recommend you keep that at default.

User avatar
alaric
Posts: 2
Joined: Tue Apr 18, 2023 8:37 pm

Re: RTX 4090 too slow?

Post by alaric »

thank you for the quick reply. Of course, this explains why I still get blurry results after over 200k iterations, even though the facial structures are clearly visible.

Is there a way to incorporate the already trained model into the new model?

Or would it be better at this point to train from scratch when the trainer has been changed?

User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: RTX 4090 too slow?

Post by torzdf »

alaric wrote: Tue Apr 18, 2023 9:17 pm

Is there a way to incorporate the already trained model into the new model?

Sadly not. Model structures are unique, and are not transferable. You will need to start from scratch.

alaric wrote: Tue Apr 18, 2023 8:43 pm

During the training I used the original model with a batch size of 16. Snapshot interval is at 250.

I assume you mean save interval is at 250 (and snapshot interval is at 25,000). Snapshot at 250 would slow things down as you would be taking a full backup copy of your model every 250 iterations.

My word is final

User avatar
Ryzen1988
Posts: 57
Joined: Thu Aug 11, 2022 8:31 am
Location: Netherlands
Has thanked: 8 times
Been thanked: 28 times

Re: RTX 4090 too slow?

Post by Ryzen1988 »

alaric wrote: Tue Apr 18, 2023 8:43 pm

Faceswap is training at about 1k iterations per minute. Is this normal? I am using the RTX 4090, the GPU seems to be quite bored at least according to task manager. So it looks like the available resources are not being used optimally.

Or am I expecting too much?
Thanks for your assessment

Haha, 1k per minute. Rome wasn't built in a day man.
Patience young grasshopper

It ofcourse all depends on the models, amount of paraments etc.
I personally try and keep the models that i can hit above 100 It's per minute on my 4090, but then again i like complex phaze models :lol:
Only if power was cheaper here in the old europe :cry:

Locked