Extraction simply hangs :(

The Extraction process failing on you, and you aren't getting an error back with clear instructions? Tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Extraction process. If you want to get tips, or better understand the Extract process, then you should look in the Extract Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
yking90
Posts: 17
Joined: Fri Jan 01, 2021 5:23 pm
Has thanked: 2 times

Extraction simply hangs :(

Post by yking90 »

It's really strange, I reinstalled FaceSwap from scratch and tried again...

During extract state, it does progress, and when I check the output directory - yes it does extract.

However, somewhere post 50-60% or so, everything just stalls and I have to abort the process.

Here's the System Output information of my system. Can someone please help?

Code: Select all


============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         15bbc95 Update Dockerfile (#1112)
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce RTX 3090, GPU_1: GeForce RTX 2080 Ti
gpu_devices_active:  GPU_0, GPU_1
gpu_driver:          457.51
gpu_vram:            GPU_0: 24576MB, GPU_1: 11264MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.19041-SP0
os_release:          10
py_command:          C:\Users\Dev\faceswap/faceswap.py gui
py_conda_version:    conda 4.9.2
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           20
sys_processor:       Intel64 Family 6 Model 165 Stepping 5, GenuineIntel
sys_ram:             Total: 130967MB, Available: 119254MB, Used: 11712MB, Free: 119254MB

=============== Pip Packages ===============
absl-py @ file:///tmp/build/80754af9/absl-py_1607439979954/work
aiohttp @ file:///C:/ci/aiohttp_1607109697839/work
astunparse==1.6.3
async-timeout==3.0.1
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
blinker==1.4
brotlipy==0.7.0
cachetools @ file:///tmp/build/80754af9/cachetools_1607706694405/work
certifi==2020.12.5
cffi @ file:///C:/ci/cffi_1606255208697/work
chardet @ file:///C:/ci/chardet_1605303225733/work
click==7.1.2
cryptography==2.9.2
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast @ file:///tmp/build/80754af9/gast_1597433534803/work
google-auth @ file:///tmp/build/80754af9/google-auth_1607969906642/work
google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1603929124518/work
google-pasta==0.2.0
grpcio @ file:///C:/ci/grpcio_1597406462198/work
h5py==2.10.0
idna @ file:///tmp/build/80754af9/idna_1593446292537/work
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1609799311556/work
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1602276842396/work
joblib @ file:///tmp/build/80754af9/joblib_1607970656719/work
Keras-Applications @ file:///tmp/build/80754af9/keras-applications_1594366238411/work
Keras-Preprocessing==1.1.0
kiwisolver @ file:///C:/ci/kiwisolver_1604014703538/work
Markdown @ file:///C:/ci/markdown_1605111189761/work
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
multidict @ file:///C:/ci/multidict_1600456481656/work
numpy @ file:///C:/ci/numpy_and_numpy_base_1603466732592/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.5.1.48
opt-einsum==3.1.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1609786840597/work
protobuf==3.13.0
psutil @ file:///C:/ci/psutil_1598370330503/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
PyJWT @ file:///C:/ci/pyjwt_1608658192037/work
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing==2.4.7
pyreadline==2.1
PySocks @ file:///C:/ci/pysocks_1605287845585/work
python-dateutil==2.8.1
pywin32==227
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-oauthlib==1.3.0
rsa @ file:///tmp/build/80754af9/rsa_1610483308194/work
scikit-learn @ file:///C:/ci/scikit-learn_1598377018496/work
scipy @ file:///C:/ci/scipy_1604596260408/work
sip==4.19.13
six @ file:///C:/ci/six_1605187374963/work
tensorboard @ file:///home/builder/ktietz/conda/conda-bld/tensorboard_1604313476433/work/tmp_pip_dir
tensorboard-plugin-wit==1.6.0
tensorflow==2.3.0
tensorflow-estimator @ file:///tmp/build/80754af9/tensorflow-estimator_1599136169057/work/whl_temp/tensorflow_estimator-2.3.0-py2.py3-none-any.whl
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado @ file:///C:/ci/tornado_1606942392901/work
tqdm @ file:///tmp/build/80754af9/tqdm_1609788246169/work
typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1598376058250/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1606938623459/work
Werkzeug==1.0.1
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
wincertstore==0.2
wrapt==1.12.1
yarl @ file:///C:/ci/yarl_1598045274898/work
zipp @ file:///tmp/build/80754af9/zipp_1604001098328/work

============== Conda Packages ==============
# packages in environment at C:\Users\Dev\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.3.0                       gpu  
absl-py                   0.11.0             pyhd3eb1b0_1  
aiohttp                   3.7.3            py38h2bbff1b_1  
astunparse                1.6.3                      py_0  
async-timeout             3.0.1                    py38_0  
attrs                     20.3.0             pyhd3eb1b0_0  
blas                      1.0                         mkl  
blinker                   1.4                      py38_0  
brotlipy                  0.7.0           py38h2bbff1b_1003  
ca-certificates           2020.12.8            haa95532_0  
cachetools                4.2.0              pyhd3eb1b0_0  
certifi                   2020.12.5        py38haa95532_0  
cffi                      1.14.4           py38hcd4344a_0  
chardet                   3.0.4           py38haa95532_1003  
click                     7.1.2                      py_0  
cryptography              2.9.2            py38h7a1dbc1_0  
cudatoolkit               10.1.243             h74a9793_0  
cudnn                     7.6.5                cuda10.1_0  
cycler                    0.10.0                   py38_0  
fastcluster               1.1.26           py38h251f6bf_2    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
freetype                  2.10.4               hd328e21_0  
gast                      0.4.0                      py_0  
git                       2.23.0               h6bb4b03_0  
google-auth               1.24.0             pyhd3eb1b0_0  
google-auth-oauthlib      0.4.2              pyhd3eb1b0_2  
google-pasta              0.2.0                      py_0  
grpcio                    1.31.0           py38he7da953_0  
h5py                      2.10.0           py38h5e291fa_0  
hdf5                      1.10.4               h7ebc959_0  
icc_rt                    2019.0.0             h0cc432a_1  
icu                       58.2                 ha925a31_3  
idna                      2.10                       py_0  
imageio                   2.9.0                      py_0  
imageio-ffmpeg            0.4.3              pyhd8ed1ab_0    conda-forge
importlib-metadata        2.0.0                      py_1  
intel-openmp              2020.2                      254  
joblib                    1.0.0              pyhd3eb1b0_0  
jpeg                      9b                   hb83a4c4_2  
keras-applications        1.0.8                      py_1  
keras-preprocessing       1.1.0                      py_1  
kiwisolver                1.3.0            py38hd77b12b_0  
libpng                    1.6.37               h2a8f88b_0  
libprotobuf               3.13.0.1             h200bbdf_0  
libtiff                   4.1.0                h56a325e_1  
lz4-c                     1.9.2                hf4a77e7_3  
markdown                  3.3.3            py38haa95532_0  
matplotlib                3.2.2                         0  
matplotlib-base           3.2.2            py38h64f37c6_0  
mkl                       2020.2                      256  
mkl-service               2.3.0            py38h196d8e1_0  
mkl_fft                   1.2.0            py38h45dec08_0  
mkl_random                1.1.1            py38h47e9c7a_0  
multidict                 4.7.6            py38he774522_1  
numpy                     1.19.2           py38hadc3359_0  
numpy-base                1.19.2           py38ha3acd2a_0  
nvidia-ml-py3             7.352.1                  pypi_0    pypi
oauthlib                  3.1.0                      py_0  
olefile                   0.46                       py_0  
opencv-python             4.5.1.48                 pypi_0    pypi
openssl                   1.1.1i               h2bbff1b_0  
opt_einsum                3.1.0                      py_0  
pathlib                   1.0.1                      py_1  
pillow                    8.1.0            py38h4fa10fc_0  
pip                       20.3.3           py38haa95532_0  
protobuf                  3.13.0.1         py38ha925a31_1  
psutil                    5.7.2            py38he774522_0  
pyasn1                    0.4.8                      py_0  
pyasn1-modules            0.2.8                      py_0  
pycparser                 2.20                       py_2  
pyjwt                     2.0.0            py38haa95532_0  
pyopenssl                 20.0.1             pyhd3eb1b0_1  
pyparsing                 2.4.7                      py_0  
pyqt                      5.9.2            py38ha925a31_4  
pyreadline                2.1                      py38_1  
pysocks                   1.7.1            py38haa95532_0  
python                    3.8.5                h5fd99cc_1  
python-dateutil           2.8.1                      py_0  
python_abi                3.8                      1_cp38    conda-forge
pywin32                   227              py38he774522_1  
qt                        5.9.7            vc14h73c81de_0  
requests                  2.25.1             pyhd3eb1b0_0  
requests-oauthlib         1.3.0                      py_0  
rsa                       4.7                pyhd3eb1b0_1  
scikit-learn              0.23.2           py38h47e9c7a_0  
scipy                     1.5.2            py38h14eb087_0  
setuptools                51.1.2           py38haa95532_4  
sip                       4.19.13          py38ha925a31_0  
six                       1.15.0           py38haa95532_0  
sqlite                    3.33.0               h2a8f88b_0  
tensorboard               2.3.0              pyh4dce500_0  
tensorboard-plugin-wit    1.6.0                      py_0  
tensorflow                2.3.0           mkl_py38h1fcfbd6_0  
tensorflow-base           2.3.0           gpu_py38h7339f5a_0  
tensorflow-estimator      2.3.0              pyheb71bc4_0  
tensorflow-gpu            2.3.0                he13fc11_0  
termcolor                 1.1.0                    py38_1  
threadpoolctl             2.1.0              pyh5ca1d4c_0  
tk                        8.6.10               he774522_0  
tornado                   6.1              py38h2bbff1b_0  
tqdm                      4.55.1             pyhd3eb1b0_0  
typing-extensions         3.7.4.3                       0  
typing_extensions         3.7.4.3                    py_0  
urllib3                   1.26.2             pyhd3eb1b0_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
werkzeug                  1.0.1                      py_0  
wheel                     0.36.2             pyhd3eb1b0_0  
win_inet_pton             1.1.0            py38haa95532_0  
wincertstore              0.2                      py38_0  
wrapt                     1.12.1           py38he774522_1  
xz                        5.2.5                h62dcd97_0  
yarl                      1.5.1            py38he774522_0  
zipp                      3.4.0              pyhd3eb1b0_0  
zlib                      1.2.11               h62dcd97_4  
zstd                      1.4.5                h04227a9_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7
scalefactor:              0.709
batch-size:               8

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                face
coverage:                 68.75
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
convert_batchsize:        16

[global.loss]
loss_function:            ssim
mask_loss_function:       mse
l2_reg_term:              100
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
disable_warp:             False
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

User avatar
torzdf
Posts: 1306
Joined: Fri Jul 12, 2019 12:53 am
Answers: 121
Has thanked: 42 times
Been thanked: 260 times

Re: Extraction simply hangs :(

Post by torzdf »

Which GPU are you using?

My word is final


User avatar
yking90
Posts: 17
Joined: Fri Jan 01, 2021 5:23 pm
Has thanked: 2 times

Re: Extraction simply hangs :(

Post by yking90 »

2080 ti

I've tried all attempts - either or, and even both together.... same issue...


User avatar
Barnuble
Posts: 2
Joined: Tue Jan 19, 2021 2:42 pm

Re: Extraction simply hangs :(

Post by Barnuble »

Hi ykin80

- Did you tried to update Nvidia Drivers ?
- Did you check you Hard drive/SSD for errors ?... Try to change destination drive for extraction...

I own 2080Ti and would like to buy 3090, would you please tell me what speed can i hope ? (I am using Villain model with batch size 24 / Mixed precision enabled).

Thanks in advance


User avatar
torzdf
Posts: 1306
Joined: Fri Jul 12, 2019 12:53 am
Answers: 121
Has thanked: 42 times
Been thanked: 260 times

Re: Extraction simply hangs :(

Post by torzdf »

yking90 wrote: Tue Jan 19, 2021 5:03 pm

2080 ti

I've tried all attempts - either or, and even both together.... same issue...

Ok. Well, I know that the symptoms seem to be the same as trying to run a 3xxx series card on Faceswap, so my guess is that just having that card connected is causing issues. My best suggestion (not ideal) is to disconnect the 3090 and see if the issue persists.

My word is final


User avatar
yking90
Posts: 17
Joined: Fri Jan 01, 2021 5:23 pm
Has thanked: 2 times

Re: Extraction simply hangs :(

Post by yking90 »

Hey mate - yes done both... ensuring that NVIDIA drivers are at 457 (not updated to 46x)

Unfortunately I've never been able to successfully run Faceswap with 3090 so I can't comment :(

That's why i got 2080ti as a second GPU.

As I said, it still extracts - but generally hangs... I have to quit the process and then I see that it's extracted quite a few faces already.

Barnuble wrote: Tue Jan 19, 2021 10:10 pm

Hi ykin80

- Did you tried to update Nvidia Drivers ?
- Did you check you Hard drive/SSD for errors ?... Try to change destination drive for extraction...

I own 2080Ti and would like to buy 3090, would you please tell me what speed can i hope ? (I am using Villain model with batch size 24 / Mixed precision enabled).

Thanks in advance


User avatar
Barnuble
Posts: 2
Joined: Tue Jan 19, 2021 2:42 pm

Re: Extraction simply hangs :(

Post by Barnuble »

Hi ykin80,

I read somewhere that extraction was problematic with RTX 30xx. Why not disabling RTX 3090 for extraction and enabling 3090 once done ? Personally i don't play any games, my GPU(s) are used for filtering (Neat video filter, VirtualDub) and Deepfakes (FaceSwap) so if i were you, i would move RTX 2080Ti to be card number 0 and 3090 to be card number 1 and disabling card 1 while extracting (3090).

Once you have succeeded, dont forget to enable "Mixed precision" before starting any project (Prefs/Training), this will speed up training with RTX having tensor cores (RTX 20xx and RTX 30xx) !

Without Mixed precision using 2080ti = 41 Egs @batchSize 12 (Villain)
With Mixed precision using 2080ti = 80 Egs @batchSize 24 (Villain)

Villain goes almost twice as fast, RealFace goes almost +73% faster at maximum batch size.
Mixed precision must be impressive with 24GB memory !

Have a nice day ;)


User avatar
yking90
Posts: 17
Joined: Fri Jan 01, 2021 5:23 pm
Has thanked: 2 times

Re: Extraction simply hangs :(

Post by yking90 »

Thank you mate - I'll try that and report back :)


Locked