[Downgrading Tensorflow] - Training fails w/ 3-GPUs

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Chach_Pls
Posts: 1
Joined: Mon Dec 02, 2019 1:08 am

[Downgrading Tensorflow] - Training fails w/ 3-GPUs

Post by Chach_Pls »

I recently added a 3rd GPU to my system and have been experiencing a general CUDA failure to load all 3 cards. All of my GPUs worked individually and in pairs, just not all 3.

How would I go about downgrading the tensorflow library manually (without having to rebuild from scratch)?

by torzdf » Mon Dec 02, 2019 1:41 am

I assume you are on Windows?

If so, and assuming that you kept your virtual environment name the default faceswap

Start > Anaconda Prompt

Code: Select all

conda activate faceswap

conda remove tensorflow

conda install tensorflow-gpu==1.13.1

cd faceswap

python update_deps.py

The last command is required because uninstalling Tensorflow also removes some libs that Faceswap depends on, so this puts them back.

If you are not on Windows, then these commands should just work from a standard terminal.

Go to full post
User avatar
torzdf
Posts: 2671
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: [Downgrading Tensorflow] - Training fails w/ 3-GPUs

Post by torzdf »

I assume you are on Windows?

If so, and assuming that you kept your virtual environment name the default faceswap

Start > Anaconda Prompt

Code: Select all

conda activate faceswap

conda remove tensorflow

conda install tensorflow-gpu==1.13.1

cd faceswap

python update_deps.py

The last command is required because uninstalling Tensorflow also removes some libs that Faceswap depends on, so this puts them back.

If you are not on Windows, then these commands should just work from a standard terminal.

My word is final

User avatar
BLACKPINK
Posts: 13
Joined: Mon Dec 09, 2019 2:06 pm
Has thanked: 3 times

Re: [Downgrading Tensorflow] - Training fails w/ 3-GPUs

Post by BLACKPINK »

I m running my gpu at base block and i have tried tensorflow 1.12 and 1.13.1. All doesnt work :( Hope theres a solution

User avatar
Khilos
Posts: 3
Joined: Mon Jul 22, 2019 7:29 pm

Re: [Downgrading Tensorflow] - Training fails w/ 3-GPUs

Post by Khilos »

What Keras, Tensorflow, and CUDA versions are you on? There was a similiar issue on the GitHub and user fixed some issues by downgrading to

CUDA 10.0
tensorflow==1.13.1
tensorflow-gpu==1.13.1
keras==2.3 .1
keras-gpu==2.3.1

As just downgrading to TF alone didn't solve the issue, but needed the combination above.

Locked