OOM killer kills faceswap process because it uses all 32GB of installed RAM (Not VRAM).

The Extraction process failing on you, and you aren't getting an error back with clear instructions? Tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Extraction process. If you want to get tips, or better understand the Extract process, then you should look in the Extract Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
hs1985
Posts: 8
Joined: Fri Jan 13, 2023 4:36 pm
Has thanked: 2 times

OOM killer kills faceswap process because it uses all 32GB of installed RAM (Not VRAM).

Post by hs1985 »

Hello,

I am trying to extract faces from images. My input folder contains 25,000 images. Faceswap processes around 3,000 images and then the OOM killer kills the Faceswap python process because it uses too much RAM (Not VRAM). I read in "Hardware Requirements" topic that Faceswap does not need much RAM because it uses VRAM for calculations. I am trying to understand why the extraction processes is using 32GB of RAM. Can you guys please help me?

My PC specs are:
OS: Ubuntu 22.04 LTS
CPU: Intel Xeon Silver 4310 CPU @ 2.10GHz (24 cores)
RAM: 32GB
VRAM: 6GB (NVIDIA GTX 1060)

I checked the syslog and it logged following messages after OOM killer killed the Faceswap process:
Jan 12 00:49:21 Ubuntu kernel: [ 9904.476519] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-3.scope,task=python,pid=2017,uid=1000

Jan 12 00:49:21 Ubuntu kernel: [ 9904.476757] Out of memory: Killed process 2017 (python) total-vm:62482864kB, anon-rss:30951064kB, file-rss:66384kB, shmem-rss:23332kB, UID:1000 pgtables:66812kB oom_score_adj:0

I wrote a bash script that saves the RAM and VRAM usage info in a text file. It logged the following usage information right before the process was killed:
RAM:
total used free shared buff/cache available
Mem: 31713 31334 224 39 154 19
Swap: 511 511 0

VRAM:

Code: Select all

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.11    Driver Version: 525.60.11    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:51:00.0  On |                  N/A |
|  0%   49C    P5     9W / 180W |   5814MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1288      G   /usr/lib/xorg/Xorg                 95MiB |
|    0   N/A  N/A      4401      C   .../envs/faceswap/bin/python     5712MiB |
|    0   N/A  N/A   3168537      G   flurry                              2MiB |
+-----------------------------------------------------------------------------+

As you can above, Faceswap was using all available RAM when it was killed (Only 19MB of 32GB was free).

Following is the Faceswap log:

Code: Select all

01/12/2023 22:10:14 MainProcess     MainThread                     logger          log_setup                      INFO     Log level set to: INFO
01/12/2023 22:10:16 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Detect from S3Fd plugin...
01/12/2023 22:10:16 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Align from Fan plugin...
01/12/2023 22:10:16 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Recognition from Vgg_Face2 plugin...
01/12/2023 22:10:16 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Mask from Components plugin...
01/12/2023 22:10:16 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Mask from Extended plugin...
01/12/2023 22:10:16 MainProcess     MainThread                     plugin_loader   _import                        INFO     Loading Mask from Bisenet_Fp plugin...
01/12/2023 22:10:16 MainProcess     MainThread                     pipeline        _set_plugin_batchsize          INFO     Reset batch sizes due to available VRAM: Detect: 1, Align: 1, Recognition: 1, Mask: 1
01/12/2023 22:10:16 MainProcess     MainThread                     extract         process                        INFO     Starting, this may take a while...
01/12/2023 22:10:16 MainProcess     MainThread                     extract         __init__                       INFO     Output Directory: /media/guest/fee1c673-3f7f-4529-a986-ade8019f64b4/Extracted/B/dir_001
01/12/2023 22:10:16 MainProcess     MainThread                     _base           initialize                     INFO     Initializing S3FD (Detect)...
01/12/2023 22:10:19 MainProcess     MainThread                     _base           initialize                     INFO     Initialized S3FD (Detect) with batchsize of 1
01/12/2023 22:10:19 MainProcess     MainThread                     _base           initialize                     INFO     Initializing FAN (Align)...
01/12/2023 22:10:32 MainProcess     MainThread                     _base           initialize                     INFO     Initialized FAN (Align) with batchsize of 1
01/12/2023 22:10:32 MainProcess     MainThread                     _base           initialize                     INFO     Initializing VGGFace2 (Recognition)...
01/12/2023 22:10:39 MainProcess     MainThread                     _base           initialize                     INFO     Initialized VGGFace2 (Recognition) with batchsize of 1
01/12/2023 22:10:39 MainProcess     MainThread                     _base           initialize                     INFO     Initializing Components (Mask)...
01/12/2023 22:10:39 MainProcess     MainThread                     _base           initialize                     INFO     Initialized Components (Mask) with batchsize of 1
01/12/2023 22:10:39 MainProcess     MainThread                     _base           initialize                     INFO     Initializing Extended (Mask)...
01/12/2023 22:10:39 MainProcess     MainThread                     _base           initialize                     INFO     Initialized Extended (Mask) with batchsize of 1
01/12/2023 22:10:39 MainProcess     MainThread                     _base           initialize                     INFO     Initializing BiSeNet - Face Parsing (Mask)...
01/12/2023 22:10:43 MainProcess     MainThread                     _base           initialize                     INFO     Initialized BiSeNet - Face Parsing (Mask) with batchsize of 1

Following are the screenshots of global settings:
Image
Image
Image
Image
Image

Following are the screenshots of extract settings:
Image
Image

Thanks

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: OOM killer kills faceswap process because it uses all 32GB of installed RAM (Not VRAM).

Post by torzdf »

Ok, your setup is very similar to mine (I run on Kubuntu 22.04 and dev on Xubuntu 20.04) and I have not hit these kinds of issues at all....

There is clearly a memory leak going on somewhere as 32GB of RAM usage is insane by any measure.

A couple of questions:
What is the length and resolution of the video you are extracting fro,
Can you bring up System Monitor, switch to the RAM graph and run Extraction until you OOM. Screengrab the RAM graph for this period and send that over.

How much of this I'm going to be able to get to the bottom of, without being able to recreate the issue, I do not know, but those may (or may not) help point towards what may be happening.

My word is final

User avatar
hs1985
Posts: 8
Joined: Fri Jan 13, 2023 4:36 pm
Has thanked: 2 times

Re: OOM killer kills faceswap process because it uses all 32GB of installed RAM (Not VRAM).

Post by hs1985 »

The length of the video is 56 minutes and the resolution is 1080. I have tried different videos of different lengths and resolutions and OOM killer kills it after few hours due to high RAM usage. I even tried to extract faces from images instead of a video file and ran into the same issue.

I ran the Faceswap extraction again and took multiple screenshots of memory usage graph. This time the OOM killer didn't kill Faceswap. When the memory usage hit 100%, the system froze and I had to restart it. I have tested it multiple times. Usually, the system doesn't freeze when memory usage hits 100%. The OOM killer kills the Python process, but not this time.

I think you are right....It looks like a memory leak.

Image

Image

Image

Image

Image

Image

Image

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: OOM killer kills faceswap process because it uses all 32GB of installed RAM (Not VRAM).

Post by bryanlyon »

Memory leaks are usually always in the libraries we use (and with this only happening to you it seems exceptionally likely). I'd suggest trying to update FaceSwap using the GUI which also updates the conda environment to the latest versions. If that doesn't work you can try to reinstall so you get the latest libraries. If the problem persists I think we would also like a copy of your system info. To get these follow this:

  • GUI Users: Go to Help -> Output system information

  • CLI Users: From inside your virtual environment, inside your faceswap folder, run:
    python -c "from lib.sysinfo import sysinfo ; print(sysinfo)"

And post the log data here. That way we can look at the versions that you're running and see if there are any known leaks in those.

User avatar
hs1985
Posts: 8
Joined: Fri Jan 13, 2023 4:36 pm
Has thanked: 2 times

Re: OOM killer kills faceswap process because it uses all 32GB of installed RAM (Not VRAM).

Post by hs1985 »

bryanlyon wrote: Sun Jan 15, 2023 10:13 pm

Memory leaks are usually always in the libraries we use (and with this only happening to you it seems exceptionally likely). I'd suggest trying to update FaceSwap using the GUI which also updates the conda environment to the latest versions. If that doesn't work you can try to reinstall so you get the latest libraries. If the problem persists I think we would also like a copy of your system info. To get these follow this:

  • GUI Users: Go to Help -> Output system information

  • CLI Users: From inside your virtual environment, inside your faceswap folder, run:
    python -c "from lib.sysinfo import sysinfo ; print(sysinfo)"

And post the log data here. That way we can look at the versions that you're running and see if there are any known leaks in those.

I did a fresh install of Ubuntu and Faceswap last week. I also updated it using the GUI couple of days ago and that didn't help either. Following is the system information:

Code: Select all

============ System Information ============
backend:             nvidia
encoding:            UTF-8
git_branch:          Not Found
git_commits:         Not Found
gpu_cuda:            11.
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce GTX 1060 6GB
gpu_devices_active:  GPU_0
gpu_driver:          525.60.11
gpu_vram:            GPU_0: 6144MB (5857MB free)
os_machine:          x86_64
os_platform:         Linux-5.15.0-56-generic-x86_64-with-glibc2.35
os_release:          5.15.0-56-generic
py_command:          /home/harjot/faceswap/faceswap.py gui
py_conda_version:    conda 22.11.1
py_implementation:   CPython
py_version:          3.9.15
py_virtual_env:      True
sys_cores:           24
sys_processor:       x86_64
sys_ram:             Total: 31713MB, Available: 29598MB, Used: 1613MB, Free: 28093MB

=============== Pip Packages ===============
absl-py @ file:///croot/absl-py_1666362940888/work
astunparse==1.6.3
cachetools==5.2.0
certifi==2022.12.7
charset-normalizer==2.1.1
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1632508026186/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
dm-tree @ file:///croot/dm-tree_1671027432897/work
fastcluster @ file:///home/conda/feedstock_root/build_artifacts/fastcluster_1649783242764/work
ffmpy==0.3.0
flatbuffers==22.12.6
flit_core @ file:///opt/conda/conda-bld/flit-core_1644941570762/work/source/flit_core
fonttools==4.25.0
gast==0.4.0
google-auth==2.15.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.1
h5py==3.7.0
idna==3.4
imageio @ file:///tmp/abs_cd920173-f360-47c5-97b0-bf4d1076d5d4dvic0oys/croots/recipe/imageio_1658785036907/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1649960641006/work
importlib-metadata==5.2.0
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1663332044897/work
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///croot/kiwisolver_1672387140495/work
libclang==14.0.6
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib @ file:///opt/conda/conda-bld/matplotlib-suite_1660167928326/work
mkl-fft==1.3.1
mkl-random @ file:///tmp/build/80754af9/mkl_random_1626186066731/work
mkl-service==2.4.0
munkres==1.1.4
numexpr @ file:///croot/numexpr_1668713893690/work
numpy @ file:///croot/numpy_and_numpy_base_1672336185480/work
nvidia-ml-py==11.515.75
oauthlib==3.2.2
opencv-python==4.7.0.68
opt-einsum==3.3.0
packaging @ file:///croot/packaging_1671697413597/work
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
Pillow==9.3.0
ply==3.11
protobuf==3.19.6
psutil @ file:///opt/conda/conda-bld/psutil_1656431268089/work
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing @ file:///opt/conda/conda-bld/pyparsing_1661452539315/work
PyQt5-sip==12.11.0
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn @ file:///croot/scikit-learn_1667587546862/work
scipy==1.9.3
sip @ file:///tmp/abs_44cd77b_pu/croots/recipe/sip_1659012365470/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.10.0
tensorflow-gpu==2.10.1
tensorflow-io-gcs-filesystem==0.29.0
tensorflow-probability @ file:///tmp/build/80754af9/tensorflow-probability_1633017132682/work
termcolor==2.1.1
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
tornado @ file:///opt/conda/conda-bld/tornado_1662061693373/work
tqdm @ file:///opt/conda/conda-bld/tqdm_1664392687731/work
typing_extensions @ file:///croot/typing_extensions_1669924550328/work
urllib3==1.26.13
Werkzeug==2.2.2
wrapt==1.14.1
zipp==3.11.0

============== Conda Packages ==============
# packages in environment at /home/harjot/miniconda3/envs/faceswap:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex 5.1 1_gnu
absl-py 1.3.0 py39h06a4308_0
astunparse 1.6.3 pypi_0 pypi blas 1.0 mkl
brotli 1.0.9 h5eee18b_7
brotli-bin 1.0.9 h5eee18b_7
bzip2 1.0.8 h7b6447c_0
c-ares 1.18.1 h7f8727e_0
ca-certificates 2022.12.7 ha878542_0 conda-forge cachetools 5.2.0 pypi_0 pypi certifi 2022.12.7 pyhd8ed1ab_0 conda-forge charset-normalizer 2.1.1 pypi_0 pypi cloudpickle 2.0.0 pyhd3eb1b0_0
cudatoolkit 11.2.2 hbe64b41_10 conda-forge cudnn 8.1.0.77 h90431f1_0 conda-forge curl 7.86.0 h5eee18b_0
cycler 0.11.0 pyhd3eb1b0_0
dbus 1.13.18 hb2f20db_0
decorator 5.1.1 pyhd3eb1b0_0
dm-tree 0.1.7 py39h6a678d5_1
expat 2.4.9 h6a678d5_0
fastcluster 1.2.6 py39h1832856_1 conda-forge ffmpeg 4.2.2 h20bf706_0
ffmpy 0.3.0 pypi_0 pypi fftw 3.3.10 nompi_h77c792f_102 conda-forge flatbuffers 22.12.6 pypi_0 pypi flit-core 3.6.0 pyhd3eb1b0_0
fontconfig 2.14.1 h52c9d5c_1
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.12.1 h4a9f257_0
gast 0.4.0 pypi_0 pypi gdbm 1.18 hd4cb3f1_4
gettext 0.21.0 hf68c758_0
giflib 5.2.1 h7b6447c_0
git 2.34.1 pl5262hc120c5b_0
glib 2.69.1 he621ea3_2
gmp 6.2.1 h58526e2_0 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge google-auth 2.15.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.51.1 pypi_0 pypi gst-plugins-base 1.14.0 h8213a91_2
gstreamer 1.14.0 h28cd5cc_2
h5py 3.7.0 pypi_0 pypi icu 58.2 he6710b0_3
idna 3.4 pypi_0 pypi imageio 2.19.3 py39h06a4308_0
imageio-ffmpeg 0.4.7 pyhd8ed1ab_0 conda-forge importlib-metadata 5.2.0 pypi_0 pypi intel-openmp 2021.4.0 h06a4308_3561
joblib 1.2.0 pyhd8ed1ab_0 conda-forge jpeg 9e h7f8727e_0
keras 2.10.0 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.4.4 py39h6a678d5_0
krb5 1.19.2 hac12032_0
lame 3.100 h7f98852_1001 conda-forge lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libbrotlicommon 1.0.9 h5eee18b_7
libbrotlidec 1.0.9 h5eee18b_7
libbrotlienc 1.0.9 h5eee18b_7
libclang 14.0.6 pypi_0 pypi libcurl 7.86.0 h91b91d3_0
libdeflate 1.8 h7f8727e_5
libedit 3.1.20221030 h5eee18b_0
libev 4.33 h7f8727e_1
libevent 2.1.12 h8f2d780_0
libffi 3.4.2 h6a678d5_6
libgcc-ng 11.2.0 h1234567_1
libgfortran-ng 12.2.0 h69a702a_19 conda-forge libgfortran5 12.2.0 h337968e_19 conda-forge libgomp 11.2.0 h1234567_1
libllvm10 10.0.1 hbcb73fb_5
libnghttp2 1.46.0 hce63b2e_0
libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.37 hbc83047_0
libpq 12.9 h16c4e8d_3
libssh2 1.10.0 h8f2d780_0
libstdcxx-ng 11.2.0 h1234567_1
libtiff 4.4.0 hecacb30_2
libuuid 1.41.5 h5eee18b_0
libvpx 1.7.0 h439df22_0
libwebp 1.2.4 h11a3e52_0
libwebp-base 1.2.4 h5eee18b_0
libxcb 1.15 h7f8727e_0
libxkbcommon 1.0.1 hfa300c1_0
libxml2 2.9.14 h74e7548_0
libxslt 1.1.35 h4e12654_0
lz4-c 1.9.4 h6a678d5_0
markdown 3.4.1 pypi_0 pypi markupsafe 2.1.1 pypi_0 pypi matplotlib 3.5.2 py39h06a4308_0
matplotlib-base 3.5.2 py39hf590b9c_0
mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py39h7f8727e_0
mkl_fft 1.3.1 py39hd3c417c_0
mkl_random 1.2.2 py39h51133e4_0
munkres 1.1.4 py_0
ncurses 6.3 h5eee18b_3
nettle 3.6 he412f7d_0 conda-forge nspr 4.33 h295c915_0
nss 3.74 h0370c37_0
numexpr 2.8.4 py39he184ba9_0
numpy 1.23.5 py39h14f4228_0
numpy-base 1.23.5 py39h31eccc5_0
nvidia-ml-py 11.515.75 pypi_0 pypi oauthlib 3.2.2 pypi_0 pypi opencv-python 4.7.0.68 pypi_0 pypi openh264 2.1.1 h4ff587b_0
openssl 1.1.1s h7f8727e_0
opt-einsum 3.3.0 pypi_0 pypi packaging 22.0 py39h06a4308_0
pcre 8.45 h295c915_0
pcre2 10.37 he7ceb23_1
perl 5.34.0 h5eee18b_2
pexpect 4.8.0 pyhd3eb1b0_3
pillow 9.3.0 py39hace64e9_1
pip 22.3.1 py39h06a4308_0
ply 3.11 py39h06a4308_0
protobuf 3.19.6 pypi_0 pypi psutil 5.9.0 py39h5eee18b_0
ptyprocess 0.7.0 pyhd3eb1b0_2
pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pyparsing 3.0.9 py39h06a4308_0
pyqt 5.15.7 py39h6a678d5_1
pyqt5-sip 12.11.0 py39h6a678d5_1
python 3.9.15 h7a1cb2a_2
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.9 2_cp39 conda-forge qt-main 5.15.2 h327a75a_7
qt-webengine 5.15.9 hd2b0992_4
qtwebkit 5.212 h4eab89a_4
readline 8.2 h5eee18b_0
requests 2.28.1 pypi_0 pypi requests-oauthlib 1.3.1 pypi_0 pypi rsa 4.9 pypi_0 pypi scikit-learn 1.1.3 py39h6a678d5_0
scipy 1.9.3 py39h14f4228_0
setuptools 65.5.0 py39h06a4308_0
sip 6.6.2 py39h6a678d5_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.40.0 h5082296_0
tensorboard 2.10.1 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow-estimator 2.10.0 pypi_0 pypi tensorflow-gpu 2.10.1 pypi_0 pypi tensorflow-io-gcs-filesystem 0.29.0 pypi_0 pypi tensorflow-probability 0.14.0 pyhd3eb1b0_0
termcolor 2.1.1 pypi_0 pypi threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tk 8.6.12 h1ccaba5_0
toml 0.10.2 pyhd3eb1b0_0
tornado 6.2 py39h5eee18b_0
tqdm 4.64.1 py39h06a4308_0
typing-extensions 4.4.0 py39h06a4308_0
typing_extensions 4.4.0 py39h06a4308_0
tzdata 2022g h04d1e81_0
urllib3 1.26.13 pypi_0 pypi werkzeug 2.2.2 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0
wrapt 1.14.1 pypi_0 pypi x264 1!157.20191217 h7b6447c_0
xz 5.2.8 h5eee18b_0
zipp 3.11.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0
zstd 1.5.2 ha4553b6_0 ================= Configs ================== --------- train.ini --------- [global] centering: face coverage: 87.5 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 epsilon_exponent: -7 autoclip: False reflect_padding: False allow_growth: False mixed_precision: False nan_protection: True convert_batchsize: 16 [global.loss] loss_function: ssim loss_function_2: mse loss_weight_2: 100 loss_function_3: None loss_weight_3: 0 loss_function_4: None loss_weight_4: 0 mask_loss_function: mse eye_multiplier: 3 mouth_multiplier: 2 penalized_mask_loss: True mask_type: extended mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4 [model.original] lowmem: False [model.dfaker] output_size: 128 [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.dlight] features: best details: good output_size: 256 [model.phaze_a] output_size: 128 shared_fc: None enable_gblock: True split_fc: True split_gblock: False split_decoders: False enc_architecture: fs_original enc_scaling: 7 enc_load_weights: True bottleneck_type: dense bottleneck_norm: None bottleneck_size: 1024 bottleneck_in_encoder: True fc_depth: 1 fc_min_filters: 1024 fc_max_filters: 1024 fc_dimensions: 4 fc_filter_slope: -0.5 fc_dropout: 0.0 fc_upsampler: upsample2d fc_upsamples: 1 fc_upsample_filters: 512 fc_gblock_depth: 3 fc_gblock_min_nodes: 512 fc_gblock_max_nodes: 512 fc_gblock_filter_slope: -0.5 fc_gblock_dropout: 0.0 dec_upscale_method: subpixel dec_upscales_in_fc: 0 dec_norm: None dec_min_filters: 64 dec_max_filters: 512 dec_slope_mode: full dec_filter_slope: -0.45 dec_res_blocks: 1 dec_output_kernel: 5 dec_gaussian: True dec_skip_last_residual: True freeze_layers: keras_encoder load_layers: encoder fs_original_depth: 4 fs_original_min_filters: 128 fs_original_max_filters: 1024 fs_original_use_alt: False mobilenet_width: 1.0 mobilenet_depth: 1 mobilenet_dropout: 0.001 mobilenet_minimalistic: False [model.villain] lowmem: False [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False --------- gui.ini --------- [global] fullscreen: True tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 erosion_top: 0.0 erosion_bottom: 0.0 erosion_left: 0.0 erosion_right: 0.0 [scaling.sharpen] method: none amount: 150 radius: 0.3 threshold: 5.0 [color.match_hist] threshold: 99.0 [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.pillow] format: png draw_transparent: False separate_mask: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate [writer.opencv] format: png draw_transparent: False separate_mask: False jpg_quality: 75 png_compress_level: 3 --------- extract.ini --------- [global] allow_growth: False aligner_min_scale: 0.07 aligner_max_scale: 2.0 aligner_distance: 22.5 aligner_roll: 45.0 aligner_features: True filter_refeed: True save_filtered: False realign_refeeds: True filter_realign: True [recognition.vgg_face2] batch-size: 16 cpu: False [detect.mtcnn] minsize: 20 scalefactor: 0.709 batch-size: 8 cpu: True threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 [detect.cv2_dnn] confidence: 50 [detect.s3fd] confidence: 70 batch-size: 4 [align.fan] batch-size: 12 [mask.vgg_obstructed] batch-size: 2 [mask.vgg_clear] batch-size: 6 [mask.bisenet_fp] batch-size: 8 cpu: False weights: faceswap include_ears: False include_hair: False include_glasses: True [mask.unet_dfl] batch-size: 8 [mask.custom] batch-size: 8 centering: face fill: False
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: OOM killer kills faceswap process because it uses all 32GB of installed RAM (Not VRAM).

Post by bryanlyon »

Thanks, we'll look at those. But know that this is something we probably can't duplicate. I'd suggest giving it a week or so and trying again with a clean install. Most of the libraries we use are very popular and any memory leaks are likely to be fixed pretty quickly.

User avatar
hs1985
Posts: 8
Joined: Fri Jan 13, 2023 4:36 pm
Has thanked: 2 times

Re: OOM killer kills faceswap process because it uses all 32GB of installed RAM (Not VRAM).

Post by hs1985 »

Update: I got the extraction to work by implementing a workaround. I increased the swap to 32GB (Physical memory also 32GB). Faceswap used all 32GB of RAM. Swap was almost full (Need to increase it). It extracted the faces without running into OOM.

The memory leak issue is still there. I will do a fresh installation again in few weeks and check if it has been resolved.

Locked