[Guide] Vast.ai setup script

Training your model
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Post Reply
User avatar
zzz
Posts: 2
Joined: Sat Feb 22, 2020 4:10 pm
Has thanked: 2 times
Been thanked: 3 times

[Guide] Vast.ai setup script

Post by zzz »

Had some issues with my home computer so used vast.ai to rent instances. A GTX 1080-TI runs about 0.1$/hour.

When creating instances use the tensorflow/tensorflow:1.15.2-gpu image

Here's the script I run to setup the enviroment, put it in a .sh file. Open a terminal and do a "chmod +x" on the file and then run the file. Zip is not necessary but useful when downloading the model.

Upload faces and alignment files.

Code: Select all

#!/bin/bash

pip install -U pip
apt-get install -y zip
apt-get install -y git
apt-get install -y libsm6 libxext6 libxrender-dev
git clone https://github.com/deepfakes/faceswap.git
cd faceswap
pip install -r requirements.txt
Run "python faceswap/faceswap.py" once to setup faceswap for NVIDIA and then run the training.
Edit faceswap/config/training.ini to setup which mask to use and other parameters.

Here's my example training command

Code: Select all

python ./faceswap/faceswap.py train -A ./A -ala ./A.fsa -B ./B -alb ./B.fsa -m ./model -t dfl-sae -bs 32 -it 1000000 -g 1 -s 100 -ss 25000 -ps 75 -w -ag -L INFO
Check the training_preview.jpg to evaluate progress.

Tags:

User avatar
kamapisachihd
Posts: 3
Joined: Sun Jun 07, 2020 12:39 am

Re: [Guide] Vast.ai setup script

Post by kamapisachihd »

First of all thank you for sharing this. I have successfully started training my model on remote GPU with Vast.ai although I have noticed that the GPU capacity of the instance labelled on the site is not what you get. Its actually not even a fraction of the GPU capacity that is being offered. I have RTX 2080 in my rig and I can get decent 9.2 EGs/sec, on the other hand I tried to train the model on Vast.ai and i barely got ~150 iterations @ 0.5 EG/sec in 3 hrs of my training time. ( was hiring RTX 2080Ti), i dont know whats going in there but seems like all the systems labelled are place holder names except true processing power on the other end. As much as I like the ease of the site and affordability system rented it meant nothing at the end. I dont know if I am doing something wrong or wondering if you had better experience?

User avatar
bryanlyon
Site Admin
Posts: 322
Joined: Fri Jul 12, 2019 12:49 am
Answers: 27
Location: San Francisco
Has thanked: 3 times
Been thanked: 87 times
Contact:

Re: [Guide] Vast.ai setup script

Post by bryanlyon »

Vast AI is a marketplace, not a single provider. When you buy a system, you're buying from some individual and not a company like Amazon/Google. Some of those people have found ways to "hack" the system to falsify hardware availability and fool Vast's software trying to detect it Report it to Vast and they'll check it and if they find the fake system they'll remove it and generally refund you pretty quickly.

User avatar
police.bike
Posts: 12
Joined: Tue Jun 30, 2020 3:37 pm
Has thanked: 6 times

Re: [Guide] Vast.ai setup script

Post by police.bike »

zzz wrote:
Wed May 27, 2020 12:19 pm
Had some issues with my home computer so used vast.ai to rent instances. A GTX 1080-TI runs about 0.1$/hour.

When creating instances use the tensorflow/tensorflow:1.15.2-gpu image

Here's the script I run to setup the enviroment, put it in a .sh file. Open a terminal and do a "chmod +x" on the file and then run the file. Zip is not necessary but useful when downloading the model.

Upload faces and alignment files.

Code: Select all

#!/bin/bash

pip install -U pip
apt-get install -y zip
apt-get install -y git
apt-get install -y libsm6 libxext6 libxrender-dev
git clone https://github.com/deepfakes/faceswap.git
cd faceswap
pip install -r requirements.txt
Run "python faceswap/faceswap.py" once to setup faceswap for NVIDIA and then run the training.
Edit faceswap/config/training.ini to setup which mask to use and other parameters.

Here's my example training command

Code: Select all

python ./faceswap/faceswap.py train -A ./A -ala ./A.fsa -B ./B -alb ./B.fsa -m ./model -t dfl-sae -bs 32 -it 1000000 -g 1 -s 100 -ss 25000 -ps 75 -w -ag -L INFO
Check the training_preview.jpg to evaluate progress.
excellent site and ultra easy. thank you !

Post Reply