Exclude GPU working in opposite function

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
bba20
Posts: 11
Joined: Tue Sep 07, 2021 3:29 pm
Has thanked: 3 times
Been thanked: 2 times

Exclude GPU working in opposite function

Post by bba20 »

I installed my old videocard in my extra slot for other work and tried to exclude it from being used by faceswap. But when I click the checkmark, task manager shows that faceswap is using my older card for training. Likewise it did the same with my newer card when I clicked the checkmark to exclude it. When I click both checkmarks it defaults to cpu.

Last edited by torzdf on Fri Oct 06, 2023 11:10 am, edited 2 times in total.

Tags:
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Exclude GPU working in opposite function

Post by torzdf »

Thanks for the report.

I'm not sure there is much we can do about this, sadly. We talk to the Nvidia driver directly to get information about connected GPUs. The Nvidia driver will return each connected GPU in a particular order (which is what dictates GPU0, GPU1 etc). We then use this identifier to tell Tensorflow to exclude a certain GPU.

It looks like TF doesn't necessarily honour the order that Nvidia reports GPUs, but we can't check this as we have to exclude any unwanted GPUs prior to initializing tensorflow in any way.

In the meantime you'll just need to make a mental note that the order of GPUs is reversed.

I have added a Bug tag to this post, to remind me to look for a solution to this if I get some time.

My word is final

Locked