problems with my setup

Installing and setting up FaceSwap


Forum rules

Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
lexten
Posts: 2
Joined: Tue Nov 05, 2019 4:56 pm

problems with my setup

Post by lexten »

Hi, everyone.
I am new into faceswap.
I ML in a other part but for about 2 weeks i am also interested in this.
Here is my problem now and i hope anyone can help me to have a smooth working FS rig.

i use the prefered linux installation guide.
after installing some missing dependencies, i am in the fs gui.
if i do anything(extract, train, etc.) i got some cudnn errors and it stopped.
In the log it say that it happens if cudnn is not configured right.
after figuring out, that my prim/sec gpu is not cuda compatible, i forced fs to just use my sec gpu with
CUDA_VISIBLE_DEVICES=0

Extract was possible now.
But for training i haved to set the 'allow growth' flag.
Else i got the cudnn errors again.
But preview doesn't work. i think because there is no 'allow growth' flag for it, because of the same cudnn errors.

For preview i start fs without any gpus so it just use my cpu for preview.

I should also mention, that i got on every start, and this and then some cuda allocate error:
tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 3.29G (3529156096 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

With Debug/Trace log i got no more informations about this problem.

Everything works on some way, but i think it's not how it should work ;)

I use Ubuntu 18.04

I am training now. If you need detailed informations i will get it tomorrow.

Here is the system information output from fs with the GPU_0 only active GPU:
GPU_1 is my active GPU.
GPU_0 is for ML.

Code: Select all

============ System Information ============
encoding:            UTF-8
git_branch:          master
git_commits:         11ab910 Reinstate fast frame count for convert and extract. Exit early if count is wrong.. 8085b4a lib.logger - Remove newlines from log messages. b1dd0db bugfix: lib.image - Ensure video frame count is always read correctly. f0a0bba Bugfix: lib.alignments. Check for .fsa file extension. 68109fc bugfix: plugins.extract.pipeline - Exclude CPU plugins from vram calculations
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce RTX 2080 SUPER, GPU_1: GeForce GTX 1060 6GB
gpu_devices_active:  GPU_0
gpu_driver:          430.50
gpu_vram:            GPU_0: 7982MB, GPU_1: 6075MB
os_machine:          x86_64
os_platform:         Linux-4.15.0-66-generic-x86_64-with-debian-buster-sid
os_release:          4.15.0-66-generic
py_command:          faceswap.py gui
py_conda_version:    conda 4.7.12
py_implementation:   CPython
py_version:          3.6.9
py_virtual_env:      True
sys_cores:           32
sys_processor:       x86_64
sys_ram:             Total: 32106MB, Available: 19887MB, Used: 9375MB, Free: 6256MB

=============== Pip Packages ===============
absl-py==0.8.0
astor==0.8.0
astroid==2.0.4
cachetools==3.1.1
certifi==2019.9.11
chardet==3.0.4
Click==7.0
colorama==0.4.1
cudnnenv==0.6.6
cycler==0.10.0
decorator==4.4.0
fastcluster==1.1.25
ffmpy==0.2.2
Flask==1.1.1
Flask-HTTPAuth==3.3.0
future==0.17.1
gast==0.3.2
google-auth==1.6.3
google-auth-oauthlib==0.4.1
google-pasta==0.1.7
grpcio==1.16.1
h5py==2.9.0
idna==2.8
imageio==2.5.0
imageio-ffmpeg==0.3.0
intelhex==2.2.1
intervaltree==3.0.2
isort==4.3.4
itsdangerous==1.1.0
joblib==0.14.0
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lazy-object-proxy==1.3.1
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==2.2.2
mccabe==0.6.1
mkl-fft==1.0.14
mkl-random==1.1.0
mkl-service==2.3.0
mock==3.0.5
networkx==2.4
numpy==1.16.2
nvidia-ml-py3==7.352.1
oauthlib==3.1.0
opencv-python==4.1.1.26
opt-einsum==3.1.0
pathlib==1.0.1
Pillow==6.1.0
protobuf==3.9.2
psutil==5.6.3
pyasn1==0.4.7
pyasn1-modules==0.2.7
pyelftools==0.25
pylint==2.1.1
pynvx==1.0.0
pyocd==0.18.0
pyparsing==2.4.2
python-dateutil==2.8.0
pytz==2019.3
pyusb==1.0.2
PyWavelets==1.1.1
PyYAML==4.2b4
requests==2.22.0
requests-oauthlib==1.2.0
rsa==4.0
scikit-image==0.16.1
scikit-learn==0.21.3
scipy==1.3.1
six==1.12.0
sortedcontainers==2.1.0
sysinfo==1.0.0
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
termcolor==1.1.0
tk==0.1.0
toposort==1.5
tqdm==4.36.1
typed-ast==1.1.0
urllib3==1.25.6
websocket-client==0.56.0
Werkzeug==0.16.0
wrapt==1.11.2

============== Conda Packages ==============
# packages in environment at /home/alex/miniconda3/envs/faceswap:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_tflow_select 2.1.0 gpu
absl-py 0.8.0 py36_0
astor 0.8.0 py36_0
blas 1.0 mkl
c-ares 1.15.0 h7b6447c_1001
ca-certificates 2019.10.16 0
cachetools 3.1.1 pypi_0 pypi certifi 2019.9.11 py36_0
chardet 3.0.4 py36_1003
cudatoolkit 10.1.243 h6bb024c_0
cudnn 7.6.0 cuda10.1_0
cudnnenv 0.6.6 pypi_0 pypi cupti 10.1.168 0
gast 0.3.2 py_0
google-auth 1.6.3 pypi_0 pypi google-auth-oauthlib 0.4.1 pypi_0 pypi google-pasta 0.1.7 py_0
grpcio 1.16.1 py36hf8bcb03_1
h5py 2.9.0 py36h7918eee_0
hdf5 1.10.4 hb1b8bf9_0
idna 2.8 pypi_0 pypi intel-openmp 2019.4 243
keras 2.2.4 0
keras-applications 1.0.8 py_0
keras-base 2.2.4 py36_0
keras-preprocessing 1.1.0 py_1
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libprotobuf 3.9.2 hd408876_0
libstdcxx-ng 9.1.0 hdf63c60_0
markdown 3.1.1 py36_0
mkl 2019.4 243
mkl-service 2.3.0 py36he904b0f_0
mkl_fft 1.0.14 py36ha843d7b_0
mkl_random 1.1.0 py36hd6b4f25_0
mock 3.0.5 pypi_0 pypi ncurses 6.1 he6710b0_1
numpy 1.16.2 py36h7e9f1db_0
numpy-base 1.16.2 py36hde5b4d6_0
oauthlib 3.1.0 pypi_0 pypi openssl 1.1.1d h7b6447c_3
opt-einsum 3.1.0 pypi_0 pypi pip 19.3.1 py36_0
protobuf 3.9.2 py36he6710b0_0
pyasn1 0.4.7 pypi_0 pypi pyasn1-modules 0.2.7 pypi_0 pypi pynvx 1.0.0 pypi_0 pypi python 3.6.9 h265db76_0
python-dateutil 2.8.0 pypi_0 pypi pytz 2019.3 pypi_0 pypi pyyaml 5.1.2 py36h7b6447c_0
readline 7.0 h7b6447c_5
requests 2.22.0 pypi_0 pypi requests-oauthlib 1.2.0 pypi_0 pypi rsa 4.0 pypi_0 pypi scipy 1.3.1 py36h7c811a0_0
setuptools 41.6.0 py36_0
six 1.12.0 py36_0
sqlite 3.30.1 h7b6447c_0
sysinfo 1.0.0 pypi_0 pypi tensorboard 1.14.0 py36hf484d3e_0
tensorflow 1.14.0 gpu_py36h3fb9ad6_0
tensorflow-base 1.14.0 gpu_py36he45bfe2_0
tensorflow-estimator 1.14.0 py_0
tensorflow-gpu 1.14.0 h0d30ee6_0
termcolor 1.1.0 py36_1
tk 8.6.8 hbc83047_0
urllib3 1.25.6 pypi_0 pypi werkzeug 0.16.0 py_0
wheel 0.33.6 py36_0
wrapt 1.11.2 py36h7b6447c_0
xz 5.2.4 h14c3975_4
yaml 0.1.7 had09818_2
zlib 1.2.11 h7b6447c_3 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 font: default font_size: 9 --------- convert.ini --------- [color.color_transfer] preserve_paper: True clip: True [color.manual_balance] contrast: 0.0 brightness: 0.0 balance_3: 0.0 balance_2: 0.0 balance_1: 0.0 colorspace: HSV [color.match_hist] threshold: 99.0 [mask.box_blend] radius: 5.0 passes: 1 distance: 11.0 type: gaussian [mask.mask_blend] radius: 3.0 passes: 4 erosion: 0.0 type: normalized [scaling.sharpen] radius: 2.3 threshold: 5.8 method: gaussian amount: 150 [writer.ffmpeg] crf: 23 container: mp4 tune: none level: auto profile: auto codec: libx264 preset: medium [writer.gif] subrectangles: False palettesize: 256 fps: 25 loop: 0 [writer.opencv] format: png jpg_quality: 75 draw_transparent: False png_compress_level: 3 [writer.pillow] draw_transparent: False optimize: False format: png jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate gif_interlace: True --------- extract.ini --------- [global] allow_growth: True [align.fan] batch-size: 8 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 50 batch-size: 8 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- train.ini --------- [global] coverage: 68.75 mask_type: components mask_blur: True icnr_init: False conv_aware_init: False subpixel_upscaling: False reflect_padding: False penalized_mask_loss: True loss_function: mae learning_rate: 5e-05 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: True [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4

nvidia-smi:
With newer drivers i had more problems. 435 and 440 is also available

Code: Select all

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:0B:00.0 Off |                  N/A |
| 53%   67C    P2   193W / 250W |   7746MiB /  7982MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 106...  Off  | 00000000:43:00.0  On |                  N/A |
| 33%   47C    P8     8W / 120W |    821MiB /  6075MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 67226 C ...lex/miniconda3/envs/faceswap/bin/python 7731MiB | | 1 1820 G /usr/lib/xorg/Xorg 18MiB | | 1 2004 G /usr/bin/gnome-shell 48MiB | | 1 2922 G /usr/lib/xorg/Xorg 198MiB | | 1 3058 G /usr/bin/gnome-shell 193MiB | | 1 43928 G ...uest-channel-token=16488666261292168793 359MiB | +-----------------------------------------------------------------------------+

Also i am scared to update fs or use the linux installer :D
Thanks for will to help me get a stable rig
Lex

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: problems with my setup

Post by torzdf »

CUDNN errors sometimes occur on some GPU/OS/Driver combinations. There doesn't seem to be any pattern to it. They can normally be averted by enabling the "allow_growth" option.

We are planning to put better multi-gpu handling into Faceswap soon.

My word is final

User avatar
lexten
Posts: 2
Joined: Tue Nov 05, 2019 4:56 pm

Re: problems with my setup

Post by lexten »

After updating fs some time ago, i have less problems.

Only warnings after starting jobs, like:
The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
That doesn't bother me.

But on some sorting jobs, i need to set
CUDA_VISIBLE_DEVICES=''
so that cpu will sort.
But set backend to GPU :D
I think it because there is no 'Allow Growth' option there.

Btw, i am very happy with this tool

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: problems with my setup

Post by torzdf »

I'm glad you're enjoying it :)

We have plans to centralize all of the GPU code into a single place so that the "Allow Growth" option is universal.

Unfortunately we need to rewrite a significant amount of code to do this, so it's not currently a priority.

You can force allow growth from the command line with the following environment variable:

https://www.tensorflow.org/guide/gpu wrote:

Another way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.

My word is final

Locked