Using Cloud GPU Hardware

Talk about Hardware used for Deep Learning
Post Reply
User avatar
adleonis
Posts: 1
Joined: Fri Aug 02, 2019 3:19 pm
Has thanked: 1 time
Been thanked: 1 time

Using Cloud GPU Hardware

Post by adleonis »

So I am a totally new to ML & Faceswapping, but I wanted to give it a go and have no access to a decent GPU. I decided to try using Google Cloud GPUs. It worked, so hopefully I can get some feedback on improving my process from some more knowledgeable tweakers. Hopefully someone knowledgeable in hardware like bryanlyon can provide an opinion.

For my first try I provisioned a GCP VM with an NVIDIA Tesla P100 GPU. GCP also has K80, P4 and T4. I don't know the difference and I wanted something quick, so I actually just picked the preconfigured "MXNet 1 Python 3.6 NVidia GPU Production" from Jetware in their marketplace choices. It's a one-click install with CUDA & Python already installed. (here's the link: https://console.cloud.google.com/market ... are-public) If anyone can recommend a better preconfigured VM I appreciate it.

I left it training overnight, for a total of 12 hours, and it "cost me" $12. Not bad, especially since Google give you $300 credit for each account you have.

I learned quickly that the data transfer time would be a killer, but luckily you can provision an S3 bucket for pennies on the GB, load up all your images to the cloud bucket ahead of time from somewhere you have a strong connection (work?), and simply mount the bucket as a drive on your provisioned GPU-enabled VM...voila, instant training when you provision the VM and very cheap.

I'm getting decent results, but doing some very basic short clips, fairly low res until I build up the experience. Ideas on improving this workflow?

One downside of the GCP option for newbies like me is you can only use the command line, so you can't really tell how the training is going visually nor use the gui. For a cloud GUI-enabled option perhaps you could provision an AWS Graphics Pro workspace(NVIDIA Tesla M60 GPU)...it's more expensive and it doesn't fall within the free tier ($66/month + $11.62/hour or $999 a month). IF anyone's tried it let me know.


User avatar
torzdf
Posts: 926
Joined: Fri Jul 12, 2019 12:53 am
Answers: 126
Has thanked: 26 times
Been thanked: 182 times

Re: Using Cloud GPU Hardware

Post by torzdf »

I can't talk for cloud based options, as I don't use them, but as to your "preview" issue, if you use the -w flag, it will write out a preview to your faceswap folder every save iteration, that you can pull down and check the progress.

My word is final


User avatar
madGANs
Posts: 4
Joined: Sat Aug 24, 2019 1:08 am
Has thanked: 1 time

Re: Using Cloud GPU Hardware

Post by madGANs »

There may be a solution out there where you can view your cloud server files similar to File Explorer. One that comes to mind is pCloud. I have never used it personally.


User avatar
bryanlyon
Site Admin
Posts: 466
Joined: Fri Jul 12, 2019 12:49 am
Answers: 39
Location: San Francisco
Has thanked: 3 times
Been thanked: 112 times
Contact:

Re: Using Cloud GPU Hardware

Post by bryanlyon »

madGANs wrote: Sun Aug 25, 2019 10:15 pm

There may be a solution out there where you can view your cloud server files similar to File Explorer. One that comes to mind is pCloud. I have never used it personally.

No need for that, GCP connects to Google Drive natively so you might as well use it's native image viewing.


User avatar
bruno29
Posts: 2
Joined: Sat Sep 28, 2019 9:19 pm
Has thanked: 1 time

Re: Using Cloud GPU Hardware

Post by bruno29 »

Adleonis, I am trying to replicate your experience, that is running a GC instance to speed up the training of a model. I am launching this configuration https://console.cloud.google.com/market ... on3-cuda91 with a single powerful GPU, but I am having some issues. I would like to ask some assistance if possible. I do the following:

  1. Run the instance

  2. Download and install Anaconda

  3. Clone the repo and run setup.py

  4. answer no to AMD and Docker, yes to CUDA questions

At this point I encounter the issue, which is that setup.py does not find a local installation of tensorflow-gpu, nor CUDA or cuDNN, which is puzzling because both should be included in the instance by default: the instance should include already all aforementioned libraries. Does anyone know how to work this out?

Thank you in advance for any help!


User avatar
bryanlyon
Site Admin
Posts: 466
Joined: Fri Jul 12, 2019 12:49 am
Answers: 39
Location: San Francisco
Has thanked: 3 times
Been thanked: 112 times
Contact:

Re: Using Cloud GPU Hardware

Post by bryanlyon »

Setup.py should install those libraries. If it's not, please make sure you're actually inside the environment conda activate faceswap and if you are, post the text from the setup.py.


User avatar
ganma
Posts: 1
Joined: Tue Dec 17, 2019 2:57 pm

Re: Using Cloud GPU Hardware

Post by ganma »

Hi, I followed the steps above and was able to use GCP to train.
However, I got this error: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)
Even though it was running, it trained very slowly and I believe it failed to use GPU when training the network. I also started another VM with the exact same setup except I used CPU instead of GPU and it trained faster than the GPU setup, so I'm pretty sure GPU wasn't being used.

If anyone knows how to fix this please help, really appreciate it.


User avatar
koroep
Posts: 3
Joined: Sun May 17, 2020 10:39 am

Re: Using Cloud GPU Hardware

Post by koroep »

DId you get this resolved?

Any suggestions for the best bang for the buck cloud service machines?


User avatar
yn1989
Posts: 1
Joined: Fri Oct 23, 2020 2:04 pm

Re: Using Cloud GPU Hardware

Post by yn1989 »

Maybe this link could help someone here to save money on cloud GPU https://puzl.ee/gpu-cloud

Message from moderator

This may be spam, but it does help answer the question and seems like a legit service, so am giving you the benefit of the doubt.... for now.


Post Reply