problems with my setup

Installing and setting up FaceSwap
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Post Reply
User avatar
lexten
Posts: 2
Joined: Tue Nov 05, 2019 4:56 pm

problems with my setup

Post by lexten » Tue Nov 05, 2019 5:50 pm

Hi, everyone.
I am new into faceswap.
I ML in a other part but for about 2 weeks i am also interested in this.
Here is my problem now and i hope anyone can help me to have a smooth working FS rig.

i use the prefered linux installation guide.
after installing some missing dependencies, i am in the fs gui.
if i do anything(extract, train, etc.) i got some cudnn errors and it stopped.
In the log it say that it happens if cudnn is not configured right.
after figuring out, that my prim/sec gpu is not cuda compatible, i forced fs to just use my sec gpu with
CUDA_VISIBLE_DEVICES=0

Extract was possible now.
But for training i haved to set the 'allow growth' flag.
Else i got the cudnn errors again.
But preview doesn't work. i think because there is no 'allow growth' flag for it, because of the same cudnn errors.

For preview i start fs without any gpus so it just use my cpu for preview.

I should also mention, that i got on every start, and this and then some cuda allocate error:
tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 3.29G (3529156096 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

With Debug/Trace log i got no more informations about this problem.

Everything works on some way, but i think it's not how it should work ;)

I use Ubuntu 18.04

I am training now. If you need detailed informations i will get it tomorrow.

Here is the system information output from fs with the GPU_0 only active GPU:
GPU_1 is my active GPU.
GPU_0 is for ML.

Code: Select all

============ System Information ============
encoding:            UTF-8
git_branch:          master
git_commits:         11ab910 Reinstate fast frame count for convert and extract. Exit early if count is wrong.. 8085b4a lib.logger - Remove newlines from log messages. b1dd0db bugfix: lib.image - Ensure video frame count is always read correctly. f0a0bba Bugfix: lib.alignments. Check for .fsa file extension. 68109fc bugfix: plugins.extract.pipeline - Exclude CPU plugins from vram calculations
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce RTX 2080 SUPER, GPU_1: GeForce GTX 1060 6GB
gpu_devices_active:  GPU_0
gpu_driver:          430.50
gpu_vram:            GPU_0: 7982MB, GPU_1: 6075MB
os_machine:          x86_64
os_platform:         Linux-4.15.0-66-generic-x86_64-with-debian-buster-sid
os_release:          4.15.0-66-generic
py_command:          faceswap.py gui
py_conda_version:    conda 4.7.12
py_implementation:   CPython
py_version:          3.6.9
py_virtual_env:      True
sys_cores:           32
sys_processor:       x86_64
sys_ram:             Total: 32106MB, Available: 19887MB, Used: 9375MB, Free: 6256MB

=============== Pip Packages ===============
absl-py==0.8.0
astor==0.8.0
astroid==2.0.4
cachetools==3.1.1
certifi==2019.9.11
chardet==3.0.4
Click==7.0
colorama==0.4.1
cudnnenv==0.6.6
cycler==0.10.0
decorator==4.4.0
fastcluster==1.1.25
ffmpy==0.2.2
Flask==1.1.1
Flask-HTTPAuth==3.3.0
future==0.17.1
gast==0.3.2
google-auth==1.6.3
google-auth-oauthlib==0.4.1
google-pasta==0.1.7
grpcio==1.16.1
h5py==2.9.0
idna==2.8
imageio==2.5.0
imageio-ffmpeg==0.3.0
intelhex==2.2.1
intervaltree==3.0.2
isort==4.3.4
itsdangerous==1.1.0
joblib==0.14.0
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lazy-object-proxy==1.3.1
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==2.2.2
mccabe==0.6.1
mkl-fft==1.0.14
mkl-random==1.1.0
mkl-service==2.3.0
mock==3.0.5
networkx==2.4
numpy==1.16.2
nvidia-ml-py3==7.352.1
oauthlib==3.1.0
opencv-python==4.1.1.26
opt-einsum==3.1.0
pathlib==1.0.1
Pillow==6.1.0
protobuf==3.9.2
psutil==5.6.3
pyasn1==0.4.7
pyasn1-modules==0.2.7
pyelftools==0.25
pylint==2.1.1
pynvx==1.0.0
pyocd==0.18.0
pyparsing==2.4.2
python-dateutil==2.8.0
pytz==2019.3
pyusb==1.0.2
PyWavelets==1.1.1
PyYAML==4.2b4
requests==2.22.0
requests-oauthlib==1.2.0
rsa==4.0
scikit-image==0.16.1
scikit-learn==0.21.3
scipy==1.3.1
six==1.12.0
sortedcontainers==2.1.0
sysinfo==1.0.0
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
termcolor==1.1.0
tk==0.1.0
toposort==1.5
tqdm==4.36.1
typed-ast==1.1.0
urllib3==1.25.6
websocket-client==0.56.0
Werkzeug==0.16.0
wrapt==1.11.2

============== Conda Packages ==============
# packages in environment at /home/alex/miniconda3/envs/faceswap:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_tflow_select             2.1.0                       gpu  
absl-py                   0.8.0                    py36_0  
astor                     0.8.0                    py36_0  
blas                      1.0                         mkl  
c-ares                    1.15.0            h7b6447c_1001  
ca-certificates           2019.10.16                    0  
cachetools                3.1.1                    pypi_0    pypi
certifi                   2019.9.11                py36_0  
chardet                   3.0.4                 py36_1003  
cudatoolkit               10.1.243             h6bb024c_0  
cudnn                     7.6.0                cuda10.1_0  
cudnnenv                  0.6.6                    pypi_0    pypi
cupti                     10.1.168                      0  
gast                      0.3.2                      py_0  
google-auth               1.6.3                    pypi_0    pypi
google-auth-oauthlib      0.4.1                    pypi_0    pypi
google-pasta              0.1.7                      py_0  
grpcio                    1.16.1           py36hf8bcb03_1  
h5py                      2.9.0            py36h7918eee_0  
hdf5                      1.10.4               hb1b8bf9_0  
idna                      2.8                      pypi_0    pypi
intel-openmp              2019.4                      243  
keras                     2.2.4                         0  
keras-applications        1.0.8                      py_0  
keras-base                2.2.4                    py36_0  
keras-preprocessing       1.1.0                      py_1  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
libprotobuf               3.9.2                hd408876_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
markdown                  3.1.1                    py36_0  
mkl                       2019.4                      243  
mkl-service               2.3.0            py36he904b0f_0  
mkl_fft                   1.0.14           py36ha843d7b_0  
mkl_random                1.1.0            py36hd6b4f25_0  
mock                      3.0.5                    pypi_0    pypi
ncurses                   6.1                  he6710b0_1  
numpy                     1.16.2           py36h7e9f1db_0  
numpy-base                1.16.2           py36hde5b4d6_0  
oauthlib                  3.1.0                    pypi_0    pypi
openssl                   1.1.1d               h7b6447c_3  
opt-einsum                3.1.0                    pypi_0    pypi
pip                       19.3.1                   py36_0  
protobuf                  3.9.2            py36he6710b0_0  
pyasn1                    0.4.7                    pypi_0    pypi
pyasn1-modules            0.2.7                    pypi_0    pypi
pynvx                     1.0.0                    pypi_0    pypi
python                    3.6.9                h265db76_0  
python-dateutil           2.8.0                    pypi_0    pypi
pytz                      2019.3                   pypi_0    pypi
pyyaml                    5.1.2            py36h7b6447c_0  
readline                  7.0                  h7b6447c_5  
requests                  2.22.0                   pypi_0    pypi
requests-oauthlib         1.2.0                    pypi_0    pypi
rsa                       4.0                      pypi_0    pypi
scipy                     1.3.1            py36h7c811a0_0  
setuptools                41.6.0                   py36_0  
six                       1.12.0                   py36_0  
sqlite                    3.30.1               h7b6447c_0  
sysinfo                   1.0.0                    pypi_0    pypi
tensorboard               1.14.0           py36hf484d3e_0  
tensorflow                1.14.0          gpu_py36h3fb9ad6_0  
tensorflow-base           1.14.0          gpu_py36he45bfe2_0  
tensorflow-estimator      1.14.0                     py_0  
tensorflow-gpu            1.14.0               h0d30ee6_0  
termcolor                 1.1.0                    py36_1  
tk                        8.6.8                hbc83047_0  
urllib3                   1.25.6                   pypi_0    pypi
werkzeug                  0.16.0                     py_0  
wheel                     0.33.6                   py36_0  
wrapt                     1.11.2           py36h7b6447c_0  
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
zlib                      1.2.11               h7b6447c_3  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
font:                     default
font_size:                9

--------- convert.ini ---------

[color.color_transfer]
preserve_paper:           True
clip:                     True

[color.manual_balance]
contrast:                 0.0
brightness:               0.0
balance_3:                0.0
balance_2:                0.0
balance_1:                0.0
colorspace:               HSV

[color.match_hist]
threshold:                99.0

[mask.box_blend]
radius:                   5.0
passes:                   1
distance:                 11.0
type:                     gaussian

[mask.mask_blend]
radius:                   3.0
passes:                   4
erosion:                  0.0
type:                     normalized

[scaling.sharpen]
radius:                   2.3
threshold:                5.8
method:                   gaussian
amount:                   150

[writer.ffmpeg]
crf:                      23
container:                mp4
tune:                     none
level:                    auto
profile:                  auto
codec:                    libx264
preset:                   medium

[writer.gif]
subrectangles:            False
palettesize:              256
fps:                      25
loop:                     0

[writer.opencv]
format:                   png
jpg_quality:              75
draw_transparent:         False
png_compress_level:       3

[writer.pillow]
draw_transparent:         False
optimize:                 False
format:                   png
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate
gif_interlace:            True

--------- extract.ini ---------

[global]
allow_growth:             True

[align.fan]
batch-size:               8

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7
scalefactor:              0.709
batch-size:               8

[detect.s3fd]
confidence:               50
batch-size:               8

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- train.ini ---------

[global]
coverage:                 68.75
mask_type:                components
mask_blur:                True
icnr_init:                False
conv_aware_init:          False
subpixel_upscaling:       False
reflect_padding:          False
penalized_mask_loss:      True
loss_function:            mae
learning_rate:            5e-05

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       True

[model.original]
lowmem:                   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4
nvidia-smi:
With newer drivers i had more problems. 435 and 440 is also available

Code: Select all

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:0B:00.0 Off |                  N/A |
| 53%   67C    P2   193W / 250W |   7746MiB /  7982MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 106...  Off  | 00000000:43:00.0  On |                  N/A |
| 33%   47C    P8     8W / 120W |    821MiB /  6075MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     67226      C   ...lex/miniconda3/envs/faceswap/bin/python  7731MiB |
|    1      1820      G   /usr/lib/xorg/Xorg                            18MiB |
|    1      2004      G   /usr/bin/gnome-shell                          48MiB |
|    1      2922      G   /usr/lib/xorg/Xorg                           198MiB |
|    1      3058      G   /usr/bin/gnome-shell                         193MiB |
|    1     43928      G   ...uest-channel-token=16488666261292168793   359MiB |
+-----------------------------------------------------------------------------+
Also i am scared to update fs or use the linux installer :D
Thanks for will to help me get a stable rig
Lex

User avatar
torzdf
Posts: 226
Joined: Fri Jul 12, 2019 12:53 am
Answers: 57
Has thanked: 8 times
Been thanked: 43 times

Re: problems with my setup

Post by torzdf » Sun Nov 10, 2019 7:23 pm

CUDNN errors sometimes occur on some GPU/OS/Driver combinations. There doesn't seem to be any pattern to it. They can normally be averted by enabling the "allow_growth" option.

We are planning to put better multi-gpu handling into Faceswap soon.
My word is final

User avatar
lexten
Posts: 2
Joined: Tue Nov 05, 2019 4:56 pm

Re: problems with my setup

Post by lexten » Tue Dec 03, 2019 7:44 am

After updating fs some time ago, i have less problems.

Only warnings after starting jobs, like:
The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
That doesn't bother me.

But on some sorting jobs, i need to set
CUDA_VISIBLE_DEVICES=''
so that cpu will sort.
But set backend to GPU :D
I think it because there is no 'Allow Growth' option there.

Btw, i am very happy with this tool

User avatar
torzdf
Posts: 226
Joined: Fri Jul 12, 2019 12:53 am
Answers: 57
Has thanked: 8 times
Been thanked: 43 times

Re: problems with my setup

Post by torzdf » Tue Dec 03, 2019 9:58 am

I'm glad you're enjoying it :)

We have plans to centralize all of the GPU code into a single place so that the "Allow Growth" option is universal.

Unfortunately we need to rewrite a significant amount of code to do this, so it's not currently a priority.

You can force allow growth from the command line with the following environment variable:
https://www.tensorflow.org/guide/gpu wrote:Another way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.
My word is final

Post Reply