[Guide] Using Faceswap on Nvidia RTX 30xx cards

Installing and setting up FaceSwap


Forum rules

Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.

User avatar
torzdf
Posts: 1868
Joined: Fri Jul 12, 2019 12:53 am
Answers: 137
Has thanked: 82 times
Been thanked: 379 times

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by torzdf »

joezi wrote: Mon Jul 12, 2021 2:58 am
torzdf wrote: Mon Jul 12, 2021 12:17 am

Enable "Allow-Growth" in Global Model Settings.

Thanks, but no change. Same errors.

Any news on the official support? It seems conda has added tensorflow 2.5 to their package repo week a go. Updating faceswap to support 2.5 would allow 11.2 cuda/8.2 cudnn combo to work.

edit: manaaged to get it to run once by having some background load on the card when starting training...

edit2: yeah, related to vram. I can get it to run by having something running in the background and then stopping it in the right point when the training process is running so that it wont go OOM.

Yeah, it's VRAM related for sure.

Whilst TF2.5 has been released by Anaconda for Windows, it's still not out for Linux. Once it is, I will test and update.

My word is final


Tags:

User avatar
joezi
Posts: 8
Joined: Sat Jul 10, 2021 7:44 pm

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by joezi »

torzdf wrote: Mon Jul 12, 2021 11:44 am

Yeah, it's VRAM related for sure.

Whilst TF2.5 has been released by Anaconda for Windows, it's still not out for Linux. Once it is, I will test and update.

Ok, good to know.

If anyone else runs into this beofre that, I finally managed to get it to work properly by setting environment variable TF_FORCE_GPU_ALLOW_GROWTH=true. Seems like the tf.config.experimental.set_memory_growth() doesnt work at all with this card and windows.


User avatar
babala
Posts: 13
Joined: Sun Jul 11, 2021 10:16 pm
Has thanked: 3 times

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by babala »

so.. is this normal training for 3080 cards?

Phaze-A stojo preset
Mixed precision on
Color Augmentaion - Color ab 15

and nothing touched.

1st training attempt was good when training starts those blank area happend but goin on 150k ite smooth.

next and 3rd is just still shows blank areas to 10k ite or more.

every time I restart program and reboot with doing delete files in model folder.

should I wait for more ite to get mask or somethin to process?

Attachments
1626257189-crop.jpg
1626257189-crop.jpg (208.49 KiB) Viewed 19519 times
1626255942-crop.jpg
1626255942-crop.jpg (232.18 KiB) Viewed 19519 times
1626255528-crop.jpg
1626255528-crop.jpg (216.89 KiB) Viewed 19519 times

User avatar
torzdf
Posts: 1868
Joined: Fri Jul 12, 2019 12:53 am
Answers: 137
Has thanked: 82 times
Been thanked: 379 times

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by torzdf »

The solid colors mean model corruption. You can try lowering the learning rate. That preset is particularly complex, so can be susceptible to collapse at higher learning rate.

Also, if using mixed precision, I highly recommend setting the Epsilon Exponent to -5 for that preset.

My word is final


User avatar
Hollywood
Posts: 18
Joined: Thu Jun 11, 2020 2:52 pm
Has thanked: 2 times

Process Exited No Clear Error Message

Post by Hollywood »

Note: I am using a 3000 series card and I know it isn't officially supported but I'm hoping someone may be able to help. I installed using this guide:viewtopic.php?p=5466#p5466

When I try and extract I get this readout...

Code: Select all

Loading...
Setting Faceswap backend to NVIDIA
07/19/2021 23:34:18 INFO     Log level set to: DEBUG
2021-07-19 23:34:18.743051: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
07/19/2021 23:34:20 INFO     Output Directory: G:\Deepfake\Frames\Dwight

07/19/2021 23:34:21 VERBOSE  Alignments filepath: 'G:\Deepfake\Originals\Dwight\Dwight_alignments.fsa'
07/19/2021 23:34:21 INFO     Loading Detect from S3Fd plugin...
07/19/2021 23:34:21 VERBOSE  Loading config: 'C:\Users\(removed)\faceswap\config\extract.ini'
07/19/2021 23:34:21 INFO     Loading Align from Fan plugin...
07/19/2021 23:34:21 VERBOSE  Loading config: 'C:\Users\(removed)\faceswap\config\extract.ini'
07/19/2021 23:34:21 INFO     Loading Mask from Components plugin...
07/19/2021 23:34:21 VERBOSE  Loading config: 'C:\Users\(removed)\faceswap\config\extract.ini'
07/19/2021 23:34:21 INFO     Loading Mask from Extended plugin...
07/19/2021 23:34:21 VERBOSE  Loading config: 'C:\Users\(removed)\faceswap\config\extract.ini'
07/19/2021 23:34:21 VERBOSE  NVIDIA GeForce RTX 3060 - 10687MB free of 12288MB
07/19/2021 23:34:21 INFO     Starting, this may take a while...
07/19/2021 23:34:21 INFO     Initializing S3FD (Detect)...

2021-07-19 23:34:21.489190: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-07-19 23:34:21.489681: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-07-19 23:34:21.503693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: NVIDIA GeForce RTX 3060 computeCapability: 8.6
coreClock: 1.837GHz coreCount: 28 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 335.32GiB/s
2021-07-19 23:34:21.503961: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-07-19 23:34:21.553612: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-07-19 23:34:21.553780: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-07-19 23:34:21.583953: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-07-19 23:34:21.590398: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-07-19 23:34:21.620782: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-07-19 23:34:21.652701: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-07-19 23:34:21.653485: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
07/19/2021 23:34:21 INFO     Setting allow growth for GPU: PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
2021-07-19 23:34:21.653647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-07-19 23:34:21.663254: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-07-19 23:34:21.664255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: NVIDIA GeForce RTX 3060 computeCapability: 8.6
coreClock: 1.837GHz coreCount: 28 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 335.32GiB/s
2021-07-19 23:34:21.664469: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-07-19 23:34:21.664561: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-07-19 23:34:21.664645: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-07-19 23:34:21.664720: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-07-19 23:34:21.664795: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-07-19 23:34:21.664874: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-07-19 23:34:21.664951: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-07-19 23:34:21.665039: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-07-19 23:34:21.665123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-07-19 23:34:22.048924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-19 23:34:22.049094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2021-07-19 23:34:22.049146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2021-07-19 23:34:22.049321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10491 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:0a:00.0, compute capability: 8.6)
2021-07-19 23:34:22.049798: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
07/19/2021 23:34:22 VERBOSE  Initializing plugin model: S3FD
07/19/2021 23:34:22 INFO     Initialized S3FD (Detect) with batchsize of 4
07/19/2021 23:34:22 INFO     Initializing FAN (Align)...
07/19/2021 23:34:22 INFO     Setting allow growth for GPU: PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
07/19/2021 23:34:22 VERBOSE  Initializing plugin model: FAN
2021-07-19 23:34:23.134322: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-07-19 23:34:23.649479: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
Process exited.

I know it has to do with the card because when I try using the CPU I get a similar readout but it works instead of saying "process exited"

Many thanks in advance


User avatar
torzdf
Posts: 1868
Joined: Fri Jul 12, 2019 12:53 am
Answers: 137
Has thanked: 82 times
Been thanked: 379 times

Re: Process Exited No Clear Error Message

Post by torzdf »

Hollywood wrote: Tue Jul 20, 2021 3:36 am

I know it has to do with the card because when I try using the CPU I get a similar readout but it works instead of saying "process exited"

Many thanks in advance

Unfortunately that output doesn't show any indication of why it might be exiting. Can you post your system info (output from Help > Output System Information)

My word is final


User avatar
Hollywood
Posts: 18
Joined: Thu Jun 11, 2020 2:52 pm
Has thanked: 2 times

Re: Process Exited No Clear Error Message

Post by Hollywood »

torzdf wrote: Tue Jul 20, 2021 9:53 am

Unfortunately that output doesn't show any indication of why it might be exiting. Can you post your system info (output from Help > Output System Information)

Yeah here's the output from that:

Code: Select all

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         55bb723 New Model: Phaze-A
gpu_cuda:            11.2
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce RTX 3060
gpu_devices_active:  GPU_0
gpu_driver:          471.41
gpu_vram:            GPU_0: 12288MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.22000-SP0
os_release:          10
py_command:          C:\Users\(removed)\faceswap/faceswap.py gui
py_conda_version:    conda 4.10.3
py_implementation:   CPython
py_version:          3.8.10
py_virtual_env:      True
sys_cores:           12
sys_processor:       AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
sys_ram:             Total: 16329MB, Available: 8518MB, Used: 7811MB, Free: 8518MB

=============== Pip Packages ===============
absl-py==0.13.0
astunparse==1.6.3
brotlipy==0.7.0
cachetools==4.2.2
certifi==2020.6.20
cffi @ file:///C:/ci/cffi_1625831749120/work
charset-normalizer==2.0.3
cryptography @ file:///C:/ci/cryptography_1616769344312/work
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
flatbuffers==1.12
gast==0.3.3
google-auth==1.33.0
google-auth-oauthlib==0.4.4
google-pasta==0.2.0
grpcio==1.32.0
h5py==2.10.0
idna @ file:///tmp/build/80754af9/idna_1622654382723/work
imageio @ file:///tmp/build/80754af9/imageio_1617700267927/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1621542018480/work
joblib @ file:///tmp/build/80754af9/joblib_1613502643832/work
keras-nightly==2.5.0.dev2021032900
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/ci/kiwisolver_1612282606037/work
Markdown==3.3.4
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.3.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1603466732592/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.1
olefile==0.46
opencv-python==4.5.3.56
opt-einsum==3.3.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1625663293593/work
protobuf==3.17.3
psutil @ file:///C:/ci/psutil_1612298324802/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
PySocks @ file:///C:/ci/pysocks_1605287845585/work
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==227
requests==2.26.0
requests-oauthlib==1.3.0
rsa==4.7.2
scikit-learn @ file:///C:/ci/scikit-learn_1622739500535/work
scipy @ file:///C:/ci/scipy_1616703433439/work
sip==4.19.13
six==1.15.0
tensorboard==2.5.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow-estimator==2.4.0
tensorflow-gpu==2.4.1
termcolor==1.1.0
threadpoolctl @ file:///tmp/build/80754af9/threadpoolctl_1626115094421/work
tornado @ file:///C:/ci/tornado_1606942392901/work
tqdm @ file:///tmp/build/80754af9/tqdm_1625563689033/work
typing-extensions==3.7.4.3
urllib3 @ file:///tmp/build/80754af9/urllib3_1603305693037/work
Werkzeug==2.0.1
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\(removed)\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   0.13.0                   pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
brotli 1.0.9 ha925a31_2
brotlipy 0.7.0 py38h2bbff1b_1003
ca-certificates 2020.10.14 0 anaconda cachetools 4.2.2 pypi_0 pypi certifi 2020.6.20 py38_0 anaconda cffi 1.14.6 py38h2bbff1b_0
charset-normalizer 2.0.3 pypi_0 pypi cryptography 3.4.7 py38h71e12ea_0
cudatoolkit 10.1.243 h74a9793_0
cudnn 7.6.5 cuda10.1_0
cycler 0.10.0 py38_0
fastcluster 1.1.26 py38h251f6bf_2 conda-forge ffmpeg 4.2.2 he774522_0
ffmpy 0.2.3 pypi_0 pypi flatbuffers 1.12 pypi_0 pypi freetype 2.10.4 hd328e21_0
gast 0.3.3 pypi_0 pypi git 2.23.0 h6bb4b03_0
google-auth 1.33.0 pypi_0 pypi google-auth-oauthlib 0.4.4 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.32.0 pypi_0 pypi h5py 2.10.0 pypi_0 pypi icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 3.2 pyhd3eb1b0_0
imageio 2.9.0 pyhd3eb1b0_0
imageio-ffmpeg 0.4.4 pyhd8ed1ab_0 conda-forge intel-openmp 2021.3.0 haa95532_3372
joblib 1.0.1 pyhd3eb1b0_0
jpeg 9b hb83a4c4_2
keras-nightly 2.5.0.dev2021032900 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.3.1 py38hd77b12b_0
libpng 1.6.37 h2a8f88b_0
libtiff 4.2.0 hd0e1b90_0
lz4-c 1.9.3 h2bbff1b_0
markdown 3.3.4 pypi_0 pypi matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38h64f37c6_0
mkl 2020.2 256
mkl-service 2.3.0 py38h196d8e1_0
mkl_fft 1.3.0 py38h46781fe_0
mkl_random 1.1.1 py38h47e9c7a_0
numpy 1.19.2 py38hadc3359_0
numpy-base 1.19.2 py38ha3acd2a_0
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.1 pypi_0 pypi olefile 0.46 py_0
opencv-python 4.5.3.56 pypi_0 pypi openssl 1.1.1k h2bbff1b_0
opt-einsum 3.3.0 pypi_0 pypi pathlib 1.0.1 py_1
pillow 8.3.1 py38h4fa10fc_0
pip 21.1.3 py38haa95532_0
protobuf 3.17.3 pypi_0 pypi psutil 5.8.0 py38h2bbff1b_1
pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycparser 2.20 py_2
pyopenssl 20.0.1 pyhd3eb1b0_1
pyparsing 2.4.7 pyhd3eb1b0_0
pyqt 5.9.2 py38ha925a31_4
pysocks 1.7.1 py38haa95532_0
python 3.8.10 hdbf39b2_7
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.8 2_cp38 conda-forge pywin32 227 py38he774522_1
qt 5.9.7 vc14h73c81de_0
requests 2.26.0 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rsa 4.7.2 pypi_0 pypi scikit-learn 0.24.2 py38hf11a4ad_1
scipy 1.6.2 py38h14eb087_0
setuptools 52.0.0 py38haa95532_0
sip 4.19.13 py38ha925a31_0
six 1.15.0 pypi_0 pypi sqlite 3.36.0 h2bbff1b_0
tensorboard 2.5.0 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.0 pypi_0 pypi tensorflow-estimator 2.4.0 pypi_0 pypi tensorflow-gpu 2.4.1 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi threadpoolctl 2.2.0 pyhb85f177_0
tk 8.6.10 he774522_0
tornado 6.1 py38h2bbff1b_0
tqdm 4.61.2 pyhd3eb1b0_1
typing-extensions 3.7.4.3 pypi_0 pypi urllib3 1.25.11 py_0 anaconda vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 2.0.1 pypi_0 pypi wheel 0.36.2 pyhd3eb1b0_0
win_inet_pton 1.1.0 py38haa95532_0
wincertstore 0.2 py38_0
wrapt 1.12.1 pypi_0 pypi xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.9 h19a0ad4_0 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: none amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: True [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 scalefactor: 0.709 batch-size: 8 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 [detect.s3fd] confidence: 70 batch-size: 4 [mask.bisenet_fp] batch-size: 8 include_ears: False include_hair: False include_glasses: True [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] centering: face coverage: 68.75 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 epsilon_exponent: -7 reflect_padding: False allow_growth: False mixed_precision: False nan_protection: True convert_batchsize: 16 [global.loss] loss_function: ssim mask_loss_function: mse l2_reg_term: 100 eye_multiplier: 3 mouth_multiplier: 2 penalized_mask_loss: True mask_type: extended mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False [model.dfaker] output_size: 128 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.phaze_a] output_size: 128 shared_fc: none enable_gblock: True split_fc: True split_gblock: False split_decoders: False enc_architecture: fs_original enc_scaling: 40 enc_load_weights: True bottleneck_type: dense bottleneck_norm: none bottleneck_size: 1024 bottleneck_in_encoder: True fc_depth: 1 fc_min_filters: 1024 fc_max_filters: 1024 fc_dimensions: 4 fc_filter_slope: -0.5 fc_dropout: 0.0 fc_upsampler: upsample2d fc_upsamples: 1 fc_upsample_filters: 512 fc_gblock_depth: 3 fc_gblock_min_nodes: 512 fc_gblock_max_nodes: 512 fc_gblock_filter_slope: -0.5 fc_gblock_dropout: 0.0 dec_upscale_method: subpixel dec_norm: none dec_min_filters: 64 dec_max_filters: 512 dec_filter_slope: -0.45 dec_res_blocks: 1 dec_output_kernel: 5 dec_gaussian: True dec_skip_last_residual: True freeze_layers: keras_encoder load_layers: encoder fs_original_depth: 4 fs_original_min_filters: 128 fs_original_max_filters: 1024 mobilenet_width: 1.0 mobilenet_depth: 1 mobilenet_dropout: 0.001 [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4

User avatar
joezi
Posts: 8
Joined: Sat Jul 10, 2021 7:44 pm

Re: Process Exited No Clear Error Message

Post by joezi »

Hollywood wrote: Tue Jul 20, 2021 1:50 pm

Yeah here's the output from that:

Code: Select all

gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN

There's atleast part of your problem.


User avatar
Hollywood
Posts: 18
Joined: Thu Jun 11, 2020 2:52 pm
Has thanked: 2 times

Re: Process Exited No Clear Error Message

Post by Hollywood »

There's atleast part of your problem.

Yeah I was thinking that could be it. I have cudnn 8.1.1 installed but I'll try and reinstall it.


User avatar
Hollywood
Posts: 18
Joined: Thu Jun 11, 2020 2:52 pm
Has thanked: 2 times

Re: Process Exited No Clear Error Message

Post by Hollywood »

Yep that fixed it. I manually added the cudNN libs into the faceswap environment and it works now. Thanks for the help!


User avatar
Faceswappnooob
Posts: 1
Joined: Wed Sep 08, 2021 3:51 pm
Has thanked: 1 time

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by Faceswappnooob »

I can confirm that the automated install now works with RTX 3080 ROG STRIX Card.

I needed to uninstall MiniPython and perform install to a new folder, it gave a warning when trying to overwrite existing install.
I was able to continue training with existing model .
Upgrade from a 2070 at 1940 iterations in 5 mins to
2080 at 2476 iterations.

Extration of a sample video was also about 40% faster


User avatar
torzdf
Posts: 1868
Joined: Fri Jul 12, 2019 12:53 am
Answers: 137
Has thanked: 82 times
Been thanked: 379 times

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by torzdf »

Thanks for the feedback, that is useful information :)

My word is final


User avatar
paruru715
Posts: 3
Joined: Sun Sep 20, 2020 5:47 am
Been thanked: 1 time

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by paruru715 »

The new version works right away with RTX 3070.
There are two issues however,

  1. It won't pick up where I left off from an old project:

    Code: Select all

    Loading...
    Setting Faceswap backend to NVIDIA
    09/24/2021 02:39:40 INFO     Log level set to: INFO
    09/24/2021 02:39:43 ERROR    The input folder '[Redacted FACE A directory here]' contains images that are not extracted faces.
    09/24/2021 02:39:43 ERROR    You can only train a model on faces generated from Faceswap's extract process. Please check your sources and try again.
    Process exited.

Seems like it can't recognize extracted face from an older version of Faceswap.

  1. At 30,000 iterations, on Dlight, the Swap side of the preview is not developing at all. Not sure if it is a Dlight problem or compatibility/corruption problem that has to do with the RTX 3070.

The Swap (B) side for the Swap > Original is not developing as shown below. However, everything else looks as expected.

Image


User avatar
babala
Posts: 13
Joined: Sun Jul 11, 2021 10:16 pm
Has thanked: 3 times

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by babala »

I've been using app 2 month without any problems now, thanks to Mods.

I just checked update that support 30xx cards.

should I remove current apps and update for better performence?

like I got training Phaze-A 12.8 EGs/sec now, update and then go to 20 EGs/sec maybe?

edit : so I re-install latest vesion app and work find. I got an 'custom mask etc' error and checked viewtopic.php?f=6&t=1744 this thread.
but 25% ish slower now.
batch 4 Phaze-A 12.5 EGs/sec back then, now batch 4 Phaze-A 9.0 EGs/sec.

I think TF 2.6 is the problem, acording that thread.
should I downgrades TF 2.6 to 2.4 ?


User avatar
torzdf
Posts: 1868
Joined: Fri Jul 12, 2019 12:53 am
Answers: 137
Has thanked: 82 times
Been thanked: 379 times

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by torzdf »

paruru715 wrote: Thu Sep 23, 2021 6:48 pm

The new version works right away with RTX 3070.
There are two issues however,

  1. It won't pick up where I left off from an old project:

    Code: Select all

    Loading...
    Setting Faceswap backend to NVIDIA
    09/24/2021 02:39:40 INFO     Log level set to: INFO
    09/24/2021 02:39:43 ERROR    The input folder '[Redacted FACE A directory here]' contains images that are not extracted faces.
    09/24/2021 02:39:43 ERROR    You can only train a model on faces generated from Faceswap's extract process. Please check your sources and try again.
    Process exited.

Seems like it can't recognize extracted face from an older version of Faceswap.

Yeah, your last version must be super old. Putting alignment information into PNG headers was something implemented a long time ago. We did have an interim period when facesets got auto-upgraded, but sadly no help to you as that time has long since past

  1. At 30,000 iterations, on Dlight, the Swap side of the preview is not developing at all. Not sure if it is a Dlight problem or compatibility/corruption problem that has to do with the RTX 3070.

The Swap (B) side for the Swap > Original is not developing as shown below. However, everything else looks as expected.

Image

That may be how dlight likes to work. I can't remember to be honest, but it is unbalanced towards the B side, so it seems possible.

My word is final


User avatar
torzdf
Posts: 1868
Joined: Fri Jul 12, 2019 12:53 am
Answers: 137
Has thanked: 82 times
Been thanked: 379 times

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by torzdf »

babala wrote: Sat Sep 25, 2021 2:36 am

I think TF 2.6 is the problem, acording that thread.
should I downgrades TF 2.6 to 2.4 ?

Whichever is easiest for you to be honest. If you have a 30xx card, then you will need to stay on 2.6, otherwise downgrading to 2,4 is fine as it is fully supported for earlier gpus

My word is final


User avatar
babala
Posts: 13
Joined: Sun Jul 11, 2021 10:16 pm
Has thanked: 3 times

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by babala »

torzdf wrote: Mon Sep 27, 2021 9:08 am

Whichever is easiest for you to be honest. If you have a 30xx card, then you will need to stay on 2.6, otherwise downgrading to 2,4 is fine as it is fully supported for earlier gpus

tried re-install manually several times of TF-CPU and TF-GPU ways, but ends up with no detect GPU problem one.

and your latest methods too, so ;)

I'll just stick with TF 2.6 now for any updates. 25% slower tho

I can still training, thank you so much :D


User avatar
Barnuble
Posts: 6
Joined: Tue Jan 19, 2021 2:42 pm

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by Barnuble »

Hi Everybody,

I own an RTX 3090.
I reinstalled FaceSwap recently with the new installer and saw about a 12% slowdown (with TensorFlow 2.6).
I have just managed to switch back to version 2.4 and I found the performance that I had before.
For those interested, here's how I did it :

First, I followed the guide from torzdf :

conda env remove -n faceswap
conda create -n faceswap python=3.8
conda activate faceswap
cd faceswap
conda install git
git fetch --unshallow
git checkout c7d85f89e69c74e97bf7485b064c07487d31faae
python setup.py

I restarted FaceSwap but training was not working anymore... I decided to reinstall once TF 2.4 in this way :

conda activate faceswap
conda remove tensorflow
conda install brotli
conda install urllib3
conda install -c anaconda urllib3
pip install tensorflow-gpu==2.4.1

And now, it works like before without error message !!!

For information (Testing my two cards : RTX 3090 / RTX 2080Ti) :
RTX 3090 - TF 2.6 / RealFace @Batch 256 (Mixed-Precision) = 364.5 Egs/s
RTX 3090 - TF 2.4 / RealFace @Batch 256 (Mixed-Precision) = 410.2 Egs/s
RTX 2080Ti - TF 2.4 / RealFace @Batch 96 (Mixed-Precision) = 257.4 Egs/s

I am very happy and I hope this will be usefull for someone ;)

Last edited by Barnuble on Sat Oct 09, 2021 8:37 am, edited 1 time in total.

User avatar
babala
Posts: 13
Joined: Sun Jul 11, 2021 10:16 pm
Has thanked: 3 times

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by babala »

Barnuble wrote: Mon Oct 04, 2021 5:16 pm

I own an RTX 3090.
I reinstalled FaceSwap recently with the new installer and saw about a 12% slowdown (with TensorFlow 2.6).
I have just managed to switch back to version 2.4 and I found the performance that I had before.

in my case, I've got cuda 11.2 and cudnn 8.1.1.33 for TF 2.4.1 not support rtx 30xx version of FaceSwap app.

I saw your guide and followed it, neither guides not working with some error when training

Error in thread (_training_0): ('Keyword argument not understood:', 'keepdims')
or
Not found: No algorithm worked!

so are you installed global cuda and cudnn for tensorflow 2.4 ?


User avatar
Barnuble
Posts: 6
Joined: Tue Jan 19, 2021 2:42 pm

Re: [Guide] Using Faceswap on Nvidia RTX 30xx cards

Post by Barnuble »

Here's my System Information (FaceSwap/Help/Output System Information) :

============ System Information ============
...
gpu_cuda: 11.4
gpu_cudnn: 8.2.4
..
gpu_driver: 472.12
...
py_conda_version: conda 4.10.3
...
py_version: 3.8.11

=============== Pip Packages ===============
...
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow-estimator==2.4.0
tensorflow-gpu==2.4.1
...

Install Cuda 11.4 and Cudnn 8.2.4
(Restart your computer...)
Start : Windows Menu/Anaconda3/Anaconda Prompt
Force TF-GPU to 2.4.1 installation in PIP Packages using the following commands :

conda activate faceswap
conda remove tensorflow
conda install brotli
conda install urllib3
conda install -c anaconda urllib3
pip install tensorflow-gpu==2.4.1

Exit Anaconda prompt and Restart FaceSwap...
Everything should work !

If it still does not works :

  • Uninstall FaceSwap
  • Uninstall Cuda and Cudnn
  • Clean registry keys using softwares like "CCleaner"
  • Restart Computer
  • Install Cuda 11.4 and Cudnn 8.2.4
  • Install FaceSwap
  • Force TF-GPU to 2.4.1 using the guide described above (Conda activate...., .....)

Hope it will help you...


Locked