Has deepfaking for long periods caused anyone long-term hardware issues?

Talk about Hardware used for Deep Learning


Locked
User avatar
cosmico
Posts: 95
Joined: Sat Jan 18, 2020 6:32 pm
Has thanked: 13 times
Been thanked: 35 times

Has deepfaking for long periods caused anyone long-term hardware issues?

Post by cosmico »

I'm asking this more as a pre-emptive question and really the crux of the issue here is I'm honestly not very computer smart or savy. ...So I guess this is just as much a deepfaking question as it is a "how to properly manage my hardware in general question". Has anyone has had long term issues with their computer as a result of deepfaking extensively.... To be clear, I'm not talking from deepfaking once, or twice, I mean constant deepfaking projects for many months to ever years at a time.

When I deepfake, my cpu-temp and GPU-temp are all with in moderate range. My GPU load is constantly around 100% and my memory is normally around 50%. Now if I played a video game and I got these levels, I wouldn't worry because I'm at max only playing for an hour or two at a time, but with deepfaking, I'm sometimes maintaining these levels for 48-76 hour periods straight. Is this alright? Am I jeopardizing my long term computer health by subjecting my computer to these conditions for such long periods?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Has deepfaking for long periods caused anyone long-term hardware issues?

Post by bryanlyon »

Most cards should be thermally safe at their maximum operation speed. They should be okay indefinitely. That said, if you can add some cooling to reduce the temperature it's not a bad idea.

If your card hovers in the 90c range you might want to underclock it to keep it cooler.

There shouldn't be any long term effects, but it could cause your fan to wear out quicker or something like that.

User avatar
cosmico
Posts: 95
Joined: Sat Jan 18, 2020 6:32 pm
Has thanked: 13 times
Been thanked: 35 times

Re: Has deepfaking for long periods caused anyone long-term hardware issues?

Post by cosmico »

Thanks for the answer. I'm ok with fan wearout, that seems like a really easy and cheap fix.

User avatar
Linhaohoward
Posts: 23
Joined: Sat Dec 21, 2019 1:23 pm
Has thanked: 3 times

Re: Has deepfaking for long periods caused anyone long-term hardware issues?

Post by Linhaohoward »

bryanlyon wrote: Sat Jan 25, 2020 8:57 pm

Most cards should be thermally safe at their maximum operation speed. They should be okay indefinitely. That said, if you can add some cooling to reduce the temperature it's not a bad idea.

If your card hovers in the 90c range you might want to underclock it to keep it cooler.

There shouldn't be any long term effects, but it could cause your fan to wear out quicker or something like that.

Hi bryanlyon, i heard overclocking might cause the model to collapse once in awhile. Will underclocking have this issue too?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Has deepfaking for long periods caused anyone long-term hardware issues?

Post by bryanlyon »

No, overclocking stresses the chip and can cause it to artifact or glitch. For gaming and graphics that's not a big deal since a frame with a glitched polygon probably wont even be noticeable, but a glitch during CUDA processing can corrupt that whole calculation, which we can't catch so at the wrong point can corrupt your whole model.

User avatar
skivvy
Posts: 24
Joined: Fri Dec 13, 2019 10:56 pm
Answers: 1
Has thanked: 10 times

Re: Has deepfaking for long periods caused anyone long-term hardware issues?

Post by skivvy »

FWIW

I overclocked the bejesus out of my 2060 6GB and have had zero problems

HOWEVER

the second i started overclocking the memory, THAT's when weird corruptions on the models occurred.

Also i have my fan pegged at 100% and in an air conditioned room at around 72F or 22C. I've had i crunching away for probably the past two months on various models. My card never goes above 45C and haven't had any issues (knocks on wood).

Maybe i got lucky in the silicon lottery?

User avatar
Linhaohoward
Posts: 23
Joined: Sat Dec 21, 2019 1:23 pm
Has thanked: 3 times

Re: Has deepfaking for long periods caused anyone long-term hardware issues?

Post by Linhaohoward »

bryanlyon wrote: Fri Jan 31, 2020 5:45 pm

No, overclocking stresses the chip and can cause it to artifact or glitch. For gaming and graphics that's not a big deal since a frame with a glitched polygon probably wont even be noticeable, but a glitch during CUDA processing can corrupt that whole calculation, which we can't catch so at the wrong point can corrupt your whole model.

Hi bryan, so i suppose your "No" means that its totally fine to underclock? Since it reduces chip stress instead of increasing them. Reason i'm asking is that at room temperature in Singapore, my ryzen 9 3900x is hitting 85 degreeC for afew seconds every minute or so. And my GPU 2x 2070super are airfan models instead of blower. Thus gpu 1 is constantly receiving hot air from gpu 2. Gpu 1 is constantly at 80-83 degreeC.

Thanks in advice for all your knowledge

Tim3311

Re: Has deepfaking for long periods caused anyone long-term hardware issues?

Post by Tim3311 »

Linhaohoward wrote: Tue Feb 04, 2020 2:04 pm

Hi bryan, so i suppose your "No" means that its totally fine to underclock? Since it reduces chip stress instead of increasing them. Reason i'm asking is that at room temperature in Singapore, my ryzen 9 3900x is hitting 85 degreeC for afew seconds every minute or so. And my GPU 2x 2070super are airfan models instead of blower. Thus gpu 1 is constantly receiving hot air from gpu 2. Gpu 1 is constantly at 80-83 degreeC.

Thanks in advice for all your knowledge

Hi, I just can tell you from my experience that I had no negative consequences from underclocking. Actually I had to underclock my CPU, because my computer would shut down with faceswap because of the high temperature.
Also I only had the problem that my computer wouldn't wake up from energy saving mode - probably because I set some voltages too low.

User avatar
MIZOWIJO
Posts: 6
Joined: Sun Aug 02, 2020 9:11 pm
Has thanked: 8 times

Re: Has deepfaking for long periods caused anyone long-term hardware issues?

Post by MIZOWIJO »

Even though semiconductor devices are fine, substrates will be cured by constant heat and will affect GPU life.

User avatar
abigflea
Posts: 182
Joined: Sat Feb 22, 2020 10:59 pm
Answers: 2
Has thanked: 20 times
Been thanked: 62 times

Re: Has deepfaking for long periods caused anyone long-term hardware issues?

Post by abigflea »

Like Bryanlyon said overclocking will make it glitch.
If you were playing a game and four frames out of 60 frames per second didn't render a texture correctly you would never notice and you wouldn't care.

if you were doing a very long and involved math problem and you got one number wrong... the whole calculation would not be a good thing.

The way face swap works it can handle glitch, but could easily just make the whole thing come apart and be just a useless mess.

Underclocking fine. I have overclocked and under clock and not seen a considerable difference in training. So feel free to drop your GPU speed 5% and just let it run cooler

:o I dunno what I'm doing :shock:
2X RTX 3090 : RTX 3080 : RTX: 2060 : 2x RTX 2080 Super : Ghetto 1060

Locked