[Downgrading Tensorflow] - Training fails w/ 3-GPUs

Training your model
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Locked
User avatar
Chach_Pls
Posts: 1
Joined: Mon Dec 02, 2019 1:08 am

[Downgrading Tensorflow] - Training fails w/ 3-GPUs

Post by Chach_Pls »

I recently added a 3rd GPU to my system and have been experiencing a general CUDA failure to load all 3 cards. All of my GPUs worked individually and in pairs, just not all 3.

How would I go about downgrading the tensorflow library manually (without having to rebuild from scratch)?
by torzdf » Mon Dec 02, 2019 1:41 am
I assume you are on Windows?

If so, and assuming that you kept your virtual environment name the default faceswap

Start > Anaconda Prompt

Code: Select all

conda activate faceswap

conda remove tensorflow

conda install tensorflow-gpu==1.13.1

cd faceswap

python update_deps.py
The last command is required because uninstalling Tensorflow also removes some libs that Faceswap depends on, so this puts them back.

If you are not on Windows, then these commands should just work from a standard terminal.
Go to full post

User avatar
torzdf
Posts: 656
Joined: Fri Jul 12, 2019 12:53 am
Answers: 96
Has thanked: 17 times
Been thanked: 132 times

Re: [Downgrading Tensorflow] - Training fails w/ 3-GPUs

Post by torzdf »

I assume you are on Windows?

If so, and assuming that you kept your virtual environment name the default faceswap

Start > Anaconda Prompt

Code: Select all

conda activate faceswap

conda remove tensorflow

conda install tensorflow-gpu==1.13.1

cd faceswap

python update_deps.py
The last command is required because uninstalling Tensorflow also removes some libs that Faceswap depends on, so this puts them back.

If you are not on Windows, then these commands should just work from a standard terminal.
My word is final

User avatar
BLACKPINK
Posts: 13
Joined: Mon Dec 09, 2019 2:06 pm
Has thanked: 3 times

Re: [Downgrading Tensorflow] - Training fails w/ 3-GPUs

Post by BLACKPINK »

I m running my gpu at base block and i have tried tensorflow 1.12 and 1.13.1. All doesnt work :( Hope theres a solution

User avatar
Khilos
Posts: 3
Joined: Mon Jul 22, 2019 7:29 pm

Re: [Downgrading Tensorflow] - Training fails w/ 3-GPUs

Post by Khilos »

What Keras, Tensorflow, and CUDA versions are you on? There was a similiar issue on the GitHub and user fixed some issues by downgrading to

CUDA 10.0
tensorflow==1.13.1
tensorflow-gpu==1.13.1
keras==2.3 .1
keras-gpu==2.3.1

As just downgrading to TF alone didn't solve the issue, but needed the combination above.

Locked