Page 1 of 1

Using Cloud GPU Hardware

Posted: Fri Aug 02, 2019 4:00 pm
by adleonis

So I am a totally new to ML & Faceswapping, but I wanted to give it a go and have no access to a decent GPU. I decided to try using Google Cloud GPUs. It worked, so hopefully I can get some feedback on improving my process from some more knowledgeable tweakers. Hopefully someone knowledgeable in hardware like bryanlyon can provide an opinion.

For my first try I provisioned a GCP VM with an NVIDIA Tesla P100 GPU. GCP also has K80, P4 and T4. I don't know the difference and I wanted something quick, so I actually just picked the preconfigured "MXNet 1 Python 3.6 NVidia GPU Production" from Jetware in their marketplace choices. It's a one-click install with CUDA & Python already installed. (here's the link: https://console.cloud.google.com/market ... are-public) If anyone can recommend a better preconfigured VM I appreciate it.

I left it training overnight, for a total of 12 hours, and it "cost me" $12. Not bad, especially since Google give you $300 credit for each account you have.

I learned quickly that the data transfer time would be a killer, but luckily you can provision an S3 bucket for pennies on the GB, load up all your images to the cloud bucket ahead of time from somewhere you have a strong connection (work?), and simply mount the bucket as a drive on your provisioned GPU-enabled VM...voila, instant training when you provision the VM and very cheap.

I'm getting decent results, but doing some very basic short clips, fairly low res until I build up the experience. Ideas on improving this workflow?

One downside of the GCP option for newbies like me is you can only use the command line, so you can't really tell how the training is going visually nor use the gui. For a cloud GUI-enabled option perhaps you could provision an AWS Graphics Pro workspace(NVIDIA Tesla M60 GPU)...it's more expensive and it doesn't fall within the free tier ($66/month + $11.62/hour or $999 a month). IF anyone's tried it let me know.


Re: Using Cloud GPU Hardware

Posted: Fri Aug 02, 2019 5:24 pm
by torzdf

I can't talk for cloud based options, as I don't use them, but as to your "preview" issue, if you use the -w flag, it will write out a preview to your faceswap folder every save iteration, that you can pull down and check the progress.


Re: Using Cloud GPU Hardware

Posted: Sun Aug 25, 2019 10:15 pm
by madGANs

There may be a solution out there where you can view your cloud server files similar to File Explorer. One that comes to mind is pCloud. I have never used it personally.


Re: Using Cloud GPU Hardware

Posted: Mon Aug 26, 2019 1:29 am
by bryanlyon
madGANs wrote: Sun Aug 25, 2019 10:15 pm

There may be a solution out there where you can view your cloud server files similar to File Explorer. One that comes to mind is pCloud. I have never used it personally.

No need for that, GCP connects to Google Drive natively so you might as well use it's native image viewing.


Re: Using Cloud GPU Hardware

Posted: Sat Sep 28, 2019 9:32 pm
by bruno29

Adleonis, I am trying to replicate your experience, that is running a GC instance to speed up the training of a model. I am launching this configuration https://console.cloud.google.com/market ... on3-cuda91 with a single powerful GPU, but I am having some issues. I would like to ask some assistance if possible. I do the following:

  1. Run the instance

  2. Download and install Anaconda

  3. Clone the repo and run setup.py

  4. answer no to AMD and Docker, yes to CUDA questions

At this point I encounter the issue, which is that setup.py does not find a local installation of tensorflow-gpu, nor CUDA or cuDNN, which is puzzling because both should be included in the instance by default: the instance should include already all aforementioned libraries. Does anyone know how to work this out?

Thank you in advance for any help!


Re: Using Cloud GPU Hardware

Posted: Sun Sep 29, 2019 3:04 am
by bryanlyon

Setup.py should install those libraries. If it's not, please make sure you're actually inside the environment conda activate faceswap and if you are, post the text from the setup.py.


Re: Using Cloud GPU Hardware

Posted: Tue Dec 17, 2019 3:05 pm
by ganma

Hi, I followed the steps above and was able to use GCP to train.
However, I got this error: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)
Even though it was running, it trained very slowly and I believe it failed to use GPU when training the network. I also started another VM with the exact same setup except I used CPU instead of GPU and it trained faster than the GPU setup, so I'm pretty sure GPU wasn't being used.

If anyone knows how to fix this please help, really appreciate it.


Re: Using Cloud GPU Hardware

Posted: Sun May 17, 2020 11:10 am
by koroep

DId you get this resolved?

Any suggestions for the best bang for the buck cloud service machines?


Re: Using Cloud GPU Hardware

Posted: Fri Oct 23, 2020 2:12 pm
by yn1989

Maybe this link could help someone here to save money on cloud GPU https://puzl.ee/gpu-cloud

[mod]This may be spam, but it does help answer the question and seems like a legit service, so am giving you the benefit of the doubt.... for now.[/mod]