Any news on the official support? It seems conda has added tensorflow 2.5 to their package repo week a go. Updating faceswap to support 2.5 would allow 11.2 cuda/8.2 cudnn combo to work.
edit: manaaged to get it to run once by having some background load on the card when starting training...
edit2: yeah, related to vram. I can get it to run by having something running in the background and then stopping it in the right point when the training process is running so that it wont go OOM.
Yeah, it's VRAM related for sure.
Whilst TF2.5 has been released by Anaconda for Windows, it's still not out for Linux. Once it is, I will test and update.
Whilst TF2.5 has been released by Anaconda for Windows, it's still not out for Linux. Once it is, I will test and update.
Ok, good to know.
If anyone else runs into this beofre that, I finally managed to get it to work properly by setting environment variable TF_FORCE_GPU_ALLOW_GROWTH=true. Seems like the tf.config.experimental.set_memory_growth() doesnt work at all with this card and windows.
The solid colors mean model corruption. You can try lowering the learning rate. That preset is particularly complex, so can be susceptible to collapse at higher learning rate.
Also, if using mixed precision, I highly recommend setting the Epsilon Exponent to -5 for that preset.
Note: I am using a 3000 series card and I know it isn't officially supported but I'm hoping someone may be able to help. I installed using this guide:viewtopic.php?p=5466#p5466
Loading...
Setting Faceswap backend to NVIDIA
07/19/2021 23:34:18 INFO Log level set to: DEBUG
2021-07-19 23:34:18.743051: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
07/19/2021 23:34:20 INFO Output Directory: G:\Deepfake\Frames\Dwight
07/19/2021 23:34:21 VERBOSE Alignments filepath: 'G:\Deepfake\Originals\Dwight\Dwight_alignments.fsa'
07/19/2021 23:34:21 INFO Loading Detect from S3Fd plugin...
07/19/2021 23:34:21 VERBOSE Loading config: 'C:\Users\(removed)\faceswap\config\extract.ini'
07/19/2021 23:34:21 INFO Loading Align from Fan plugin...
07/19/2021 23:34:21 VERBOSE Loading config: 'C:\Users\(removed)\faceswap\config\extract.ini'
07/19/2021 23:34:21 INFO Loading Mask from Components plugin...
07/19/2021 23:34:21 VERBOSE Loading config: 'C:\Users\(removed)\faceswap\config\extract.ini'
07/19/2021 23:34:21 INFO Loading Mask from Extended plugin...
07/19/2021 23:34:21 VERBOSE Loading config: 'C:\Users\(removed)\faceswap\config\extract.ini'
07/19/2021 23:34:21 VERBOSE NVIDIA GeForce RTX 3060 - 10687MB free of 12288MB
07/19/2021 23:34:21 INFO Starting, this may take a while...
07/19/2021 23:34:21 INFO Initializing S3FD (Detect)...
2021-07-19 23:34:21.489190: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-07-19 23:34:21.489681: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-07-19 23:34:21.503693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: NVIDIA GeForce RTX 3060 computeCapability: 8.6
coreClock: 1.837GHz coreCount: 28 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 335.32GiB/s
2021-07-19 23:34:21.503961: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-07-19 23:34:21.553612: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-07-19 23:34:21.553780: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-07-19 23:34:21.583953: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-07-19 23:34:21.590398: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-07-19 23:34:21.620782: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-07-19 23:34:21.652701: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-07-19 23:34:21.653485: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
07/19/2021 23:34:21 INFO Setting allow growth for GPU: PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
2021-07-19 23:34:21.653647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-07-19 23:34:21.663254: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-07-19 23:34:21.664255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: NVIDIA GeForce RTX 3060 computeCapability: 8.6
coreClock: 1.837GHz coreCount: 28 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 335.32GiB/s
2021-07-19 23:34:21.664469: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-07-19 23:34:21.664561: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-07-19 23:34:21.664645: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-07-19 23:34:21.664720: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-07-19 23:34:21.664795: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-07-19 23:34:21.664874: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-07-19 23:34:21.664951: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-07-19 23:34:21.665039: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-07-19 23:34:21.665123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-07-19 23:34:22.048924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-19 23:34:22.049094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-07-19 23:34:22.049146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-07-19 23:34:22.049321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10491 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:0a:00.0, compute capability: 8.6)
2021-07-19 23:34:22.049798: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
07/19/2021 23:34:22 VERBOSE Initializing plugin model: S3FD
07/19/2021 23:34:22 INFO Initialized S3FD (Detect) with batchsize of 4
07/19/2021 23:34:22 INFO Initializing FAN (Align)...
07/19/2021 23:34:22 INFO Setting allow growth for GPU: PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
07/19/2021 23:34:22 VERBOSE Initializing plugin model: FAN
2021-07-19 23:34:23.134322: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-07-19 23:34:23.649479: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
Process exited.
I know it has to do with the card because when I try using the CPU I get a similar readout but it works instead of saying "process exited"
I know it has to do with the card because when I try using the CPU I get a similar readout but it works instead of saying "process exited"
Many thanks in advance
Unfortunately that output doesn't show any indication of why it might be exiting. Can you post your system info (output from Help > Output System Information)
Unfortunately that output doesn't show any indication of why it might be exiting. Can you post your system info (output from Help > Output System Information)
I can confirm that the automated install now works with RTX 3080 ROG STRIX Card.
I needed to uninstall MiniPython and perform install to a new folder, it gave a warning when trying to overwrite existing install.
I was able to continue training with existing model .
Upgrade from a 2070 at 1940 iterations in 5 mins to
2080 at 2476 iterations.
Extration of a sample video was also about 40% faster
Loading...
Setting Faceswap backend to NVIDIA
09/24/2021 02:39:40 INFO Log level set to: INFO
09/24/2021 02:39:43 ERROR The input folder '[Redacted FACE A directory here]' contains images that are not extracted faces.
09/24/2021 02:39:43 ERROR You can only train a model on faces generated from Faceswap's extract process. Please check your sources and try again.
Process exited.
Seems like it can't recognize extracted face from an older version of Faceswap.
At 30,000 iterations, on Dlight, the Swap side of the preview is not developing at all. Not sure if it is a Dlight problem or compatibility/corruption problem that has to do with the RTX 3070.
The Swap (B) side for the Swap > Original is not developing as shown below. However, everything else looks as expected.
I've been using app 2 month without any problems now, thanks to Mods.
I just checked update that support 30xx cards.
should I remove current apps and update for better performence?
like I got training Phaze-A 12.8 EGs/sec now, update and then go to 20 EGs/sec maybe?
edit : so I re-install latest vesion app and work find. I got an 'custom mask etc' error and checked viewtopic.php?f=6&t=1744 this thread.
but 25% ish slower now.
batch 4 Phaze-A 12.5 EGs/sec back then, now batch 4 Phaze-A 9.0 EGs/sec.
I think TF 2.6 is the problem, acording that thread.
should I downgrades TF 2.6 to 2.4 ?
Loading...
Setting Faceswap backend to NVIDIA
09/24/2021 02:39:40 INFO Log level set to: INFO
09/24/2021 02:39:43 ERROR The input folder '[Redacted FACE A directory here]' contains images that are not extracted faces.
09/24/2021 02:39:43 ERROR You can only train a model on faces generated from Faceswap's extract process. Please check your sources and try again.
Process exited.
Seems like it can't recognize extracted face from an older version of Faceswap.
Yeah, your last version must be super old. Putting alignment information into PNG headers was something implemented a long time ago. We did have an interim period when facesets got auto-upgraded, but sadly no help to you as that time has long since past
At 30,000 iterations, on Dlight, the Swap side of the preview is not developing at all. Not sure if it is a Dlight problem or compatibility/corruption problem that has to do with the RTX 3070.
The Swap (B) side for the Swap > Original is not developing as shown below. However, everything else looks as expected.
That may be how dlight likes to work. I can't remember to be honest, but it is unbalanced towards the B side, so it seems possible.
I think TF 2.6 is the problem, acording that thread.
should I downgrades TF 2.6 to 2.4 ?
Whichever is easiest for you to be honest. If you have a 30xx card, then you will need to stay on 2.6, otherwise downgrading to 2,4 is fine as it is fully supported for earlier gpus
Whichever is easiest for you to be honest. If you have a 30xx card, then you will need to stay on 2.6, otherwise downgrading to 2,4 is fine as it is fully supported for earlier gpus
tried re-install manually several times of TF-CPU and TF-GPU ways, but ends up with no detect GPU problem one.
and your latest methods too, so
I'll just stick with TF 2.6 now for any updates. 25% slower tho
I own an RTX 3090.
I reinstalled FaceSwap recently with the new installer and saw about a 12% slowdown (with TensorFlow 2.6).
I have just managed to switch back to version 2.4 and I found the performance that I had before.
For those interested, here's how I did it :
I own an RTX 3090.
I reinstalled FaceSwap recently with the new installer and saw about a 12% slowdown (with TensorFlow 2.6).
I have just managed to switch back to version 2.4 and I found the performance that I had before.
in my case, I've got cuda 11.2 and cudnn 8.1.1.33 for TF 2.4.1 not support rtx 30xx version of FaceSwap app.
I saw your guide and followed it, neither guides not working with some error when training
Error in thread (_training_0): ('Keyword argument not understood:', 'keepdims')
or
Not found: No algorithm worked!
so are you installed global cuda and cudnn for tensorflow 2.4 ?
Install Cuda 11.4 and Cudnn 8.2.4
(Restart your computer...)
Start : Windows Menu/Anaconda3/Anaconda Prompt
Force TF-GPU to 2.4.1 installation in PIP Packages using the following commands :