NOOB - Very frustrated with OOM issues

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Zebra522020
Posts: 3
Joined: Mon Sep 21, 2020 8:49 pm
Been thanked: 1 time

NOOB - Very frustrated with OOM issues

Post by Zebra522020 »

Okay, yes I'm a Newbie to all of this. Had problems getting this program to run on my desktop, but can run it on my laptop just fine except very slow. I get it - no graphics card. So I bought a recommended graphics card for my desktop, installed it, and all was right with the world. For a while.
I saved a project after finishing it - kind of a practice run. Worked okay and I see how to get better results. Started to set up a new project, started an extraction (on something I had extracted before) and WHAM - VRAM Out of Memory, Can't get the extraction to run.

I know people have asked about OOM issues before, but I thought I had all the kinks worked out. Can't really find my particular problem in the Forum because I don't know what I am actually looking for.
Here's the log if someone can help... Thanks

Code: Select all

09/22/2020 20:38:52 MainProcess     MainThread      pipeline        _set_phases               DEBUG    Required: 2240.0, available: 3587. Adding phase 'mask_0' to current phase: ['align']
09/22/2020 20:38:52 MainProcess     MainThread      pipeline        _set_phases               DEBUG    Num plugins for phase: 1, scaling: 1.0, vram required: 2240.0
09/22/2020 20:38:52 MainProcess     MainThread      pipeline        _set_phases               DEBUG    Required: 2240.0, available: 3587. Adding phase 'mask_1' to current phase: ['align', 'mask_0']
09/22/2020 20:38:52 MainProcess     MainThread      pipeline        _set_phases               DEBUG    Total phases: 2, Phases: [['detect'], ['align', 'mask_0', 'mask_1']]
09/22/2020 20:38:52 MainProcess     MainThread      pipeline        _set_plugin_batchsize     DEBUG    requested_batchsizes: [4], batchsizes: [1], remaining vram: -733.0
09/22/2020 20:38:52 MainProcess     MainThread      pipeline        _set_plugin_batchsize     INFO     Reset batch sizes due to available VRAM: Detect: 1
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager adding: (name: 'extract0_detect_in', maxsize: 1)
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager added: (name: 'extract0_detect_in')
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'extract0_detect_in'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'extract0_detect_in'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager adding: (name: 'extract0_align_in', maxsize: 1)
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager added: (name: 'extract0_align_in')
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'extract0_align_in'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'extract0_align_in'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager adding: (name: 'extract0_mask_0_in', maxsize: 1)
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager added: (name: 'extract0_mask_0_in')
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'extract0_mask_0_in'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'extract0_mask_0_in'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager adding: (name: 'extract0_mask_1_in', maxsize: 1)
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager added: (name: 'extract0_mask_1_in')
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'extract0_mask_1_in'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'extract0_mask_1_in'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager adding: (name: 'extract0_mask_1_out', maxsize: 1)
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager added: (name: 'extract0_mask_1_out')
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'extract0_mask_1_out'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'extract0_mask_1_out'
09/22/2020 20:38:52 MainProcess     MainThread      pipeline        _add_queues               DEBUG    Queues: {'extract0_detect_in': <queue.Queue object at 0x000001BCEA7E72E0>, 'extract0_align_in': <queue.Queue object at 0x000001BCEA88C190>, 'extract0_mask_0_in': <queue.Queue object at 0x000001BCEA88CA60>, 'extract0_mask_1_in': <queue.Queue object at 0x000001BCEA88C820>, 'extract0_mask_1_out': <queue.Queue object at 0x000001BCEA88CDC0>}
09/22/2020 20:38:52 MainProcess     MainThread      pipeline        __init__                  DEBUG    Initialized Extractor
09/22/2020 20:38:52 MainProcess     MainThread      extract         __init__                  DEBUG    Initialized Extract
09/22/2020 20:38:52 MainProcess     MainThread      extract         process                   INFO     Starting, this may take a while...
09/22/2020 20:38:52 MainProcess     MainThread      extract         _threaded_redirector      DEBUG    Threading task: (Task: 'load')
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initializing MultiThread: (target: '_load', thread_count: 1)
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initialized MultiThread: '_load'
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread(s): '_load'
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 1 of 1: '_load_0'
09/22/2020 20:38:52 MainProcess     _load_0         extract         _load                     DEBUG    Load Images: Start
09/22/2020 20:38:52 MainProcess     _load_0         image           load                      DEBUG    Initializing Load Generator
09/22/2020 20:38:52 MainProcess     _load_0         image           _set_thread               DEBUG    Setting thread
09/22/2020 20:38:52 MainProcess     _load_0         multithreading  __init__                  DEBUG    Initializing MultiThread: (target: 'ImagesLoader', thread_count: 1)
09/22/2020 20:38:52 MainProcess     _load_0         multithreading  __init__                  DEBUG    Initialized MultiThread: 'ImagesLoader'
09/22/2020 20:38:52 MainProcess     _load_0         image           _set_thread               DEBUG    Set thread: <lib.multithreading.MultiThread object at 0x000001BCEA89E160>
09/22/2020 20:38:52 MainProcess     _load_0         multithreading  start                     DEBUG    Starting thread(s): 'ImagesLoader'
09/22/2020 20:38:52 MainProcess     _load_0         multithreading  start                     DEBUG    Starting thread 1 of 1: 'ImagesLoader_0'
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  start                     DEBUG    Started all threads '_load': 1
09/22/2020 20:38:52 MainProcess     MainThread      image           __init__                  DEBUG    Initializing ImagesSaver: (path: C:\Users\Arland\Desktop\mm new\FACE A, queue_size: 8, as_bytes: True)
09/22/2020 20:38:52 MainProcess     MainThread      image           __init__                  DEBUG    Initializing ImagesSaver: (path: C:\Users\Arland\Desktop\mm new\FACE A, queue_size: 8, args: None)
09/22/2020 20:38:52 MainProcess     ImagesLoader_0  image           _process                  DEBUG    Load iterator: <bound method ImagesLoader._from_video of <lib.image.ImagesLoader object at 0x000001BCD44FC580>>
09/22/2020 20:38:52 MainProcess     ImagesLoader_0  image           _from_video               DEBUG    Loading frames from video: 'C:\Users\Arland\Desktop\mm new\MELANIA SOURCE.mp4'
09/22/2020 20:38:52 MainProcess     _load_0         multithreading  start                     DEBUG    Started all threads 'ImagesLoader': 1
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'ImagesSaver'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager adding: (name: 'ImagesSaver', maxsize: 8)
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager added: (name: 'ImagesSaver')
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'ImagesSaver'
09/22/2020 20:38:52 MainProcess     MainThread      pipeline        _launch_plugin            DEBUG    Launching detect plugin
09/22/2020 20:38:52 MainProcess     MainThread      pipeline        _launch_plugin            DEBUG    in_qname: extract0_detect_in, out_qname: extract0_align_in
09/22/2020 20:38:52 MainProcess     MainThread      _base           initialize                DEBUG    initialize Detect: (args: (), kwargs: {'in_queue': <queue.Queue object at 0x000001BCEA7E72E0>, 'out_queue': <queue.Queue object at 0x000001BCEA88C190>})
09/22/2020 20:38:52 MainProcess     MainThread      _base           initialize                INFO     Initializing S3FD (Detect)...
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'detect0_predict_s3fd'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager adding: (name: 'detect0_predict_s3fd', maxsize: 1)
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager added: (name: 'detect0_predict_s3fd')
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'detect0_predict_s3fd'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'detect0_post_s3fd'
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager adding: (name: 'detect0_post_s3fd', maxsize: 1)
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   add_queue                 DEBUG    QueueManager added: (name: 'detect0_post_s3fd')
09/22/2020 20:38:52 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'detect0_post_s3fd'
09/22/2020 20:38:52 MainProcess     MainThread      _base           _compile_threads          DEBUG    Compiling detect threads
09/22/2020 20:38:52 MainProcess     MainThread      _base           _add_thread               DEBUG    Adding thread: (name: detect_s3fd_input, function: <bound method Detect.process_input of <plugins.extract.detect.s3fd.Detect object at 0x000001BCEA7E7370>>, in_queue: <queue.Queue object at 0x000001BCEA7E72E0>, out_queue: <queue.Queue object at 0x000001BCEA88C3D0>)
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initializing MultiThread: (target: 'detect_s3fd_input', thread_count: 1)
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initialized MultiThread: 'detect_s3fd_input'
09/22/2020 20:38:52 MainProcess     MainThread      _base           _add_thread               DEBUG    Added thread: detect_s3fd_input
09/22/2020 20:38:52 MainProcess     MainThread      _base           _add_thread               DEBUG    Adding thread: (name: detect_s3fd_predict, function: <bound method Detector._predict of <plugins.extract.detect.s3fd.Detect object at 0x000001BCEA7E7370>>, in_queue: <queue.Queue object at 0x000001BCEA88C3D0>, out_queue: <queue.Queue object at 0x000001BCEA88C0A0>)
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initializing MultiThread: (target: 'detect_s3fd_predict', thread_count: 1)
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initialized MultiThread: 'detect_s3fd_predict'
09/22/2020 20:38:52 MainProcess     MainThread      _base           _add_thread               DEBUG    Added thread: detect_s3fd_predict
09/22/2020 20:38:52 MainProcess     MainThread      _base           _add_thread               DEBUG    Adding thread: (name: detect_s3fd_output, function: <bound method Detect.process_output of <plugins.extract.detect.s3fd.Detect object at 0x000001BCEA7E7370>>, in_queue: <queue.Queue object at 0x000001BCEA88C0A0>, out_queue: <queue.Queue object at 0x000001BCEA88C190>)
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initializing MultiThread: (target: 'detect_s3fd_output', thread_count: 1)
09/22/2020 20:38:52 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initialized MultiThread: 'detect_s3fd_output'
09/22/2020 20:38:52 MainProcess     MainThread      _base           _add_thread               DEBUG    Added thread: detect_s3fd_output
09/22/2020 20:38:52 MainProcess     MainThread      _base           _compile_threads          DEBUG    Compiled detect threads: [<lib.multithreading.MultiThread object at 0x000001BCEA88C4C0>, <lib.multithreading.MultiThread object at 0x000001BCEA869DC0>, <lib.multithreading.MultiThread object at 0x000001BCEA869910>]
09/22/2020 20:38:52 MainProcess     MainThread      s3fd            __init__                  DEBUG    Initializing: S3fd: (model_path: 'C:\Users\Arland\faceswap\plugins\extract\detect\.cache\s3fd_keras_v1.h5', model_kwargs: {'custom_objects': {'O2K_Add': <class 'plugins.extract.detect.s3fd.AddO2K'>, 'O2K_Slice': <class 'plugins.extract.detect.s3fd.SliceO2K'>, 'O2K_Sum': <class 'plugins.extract.detect.s3fd.SumO2K'>, 'O2K_Sqrt': <class 'plugins.extract.detect.s3fd.SqrtO2K'>, 'O2K_Pow': <class 'plugins.extract.detect.s3fd.PowO2K'>, 'O2K_ConstantLayer': <class 'plugins.extract.detect.s3fd.ConstantLayerO2K'>, 'O2K_Div': <class 'plugins.extract.detect.s3fd.DivO2K'>}}, allow_growth: True, exclude_gpus: None, confidence: 0.7)
09/22/2020 20:38:52 MainProcess     MainThread      session         _set_session              INFO     Setting allow growth for GPU: PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
09/22/2020 20:38:52 MainProcess     MainThread      session         load_model                VERBOSE  Initializing plugin model: S3FD
09/22/2020 20:38:54 MainProcess     MainThread      s3fd            __init__                  DEBUG    Initialized: S3fd
09/22/2020 20:38:54 MainProcess     MainThread      _base           initialize                INFO     Initialized S3FD (Detect) with batchsize of 1
09/22/2020 20:38:54 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread(s): 'detect_s3fd_input'
09/22/2020 20:38:54 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 1 of 1: 'detect_s3fd_input_0'
09/22/2020 20:38:54 MainProcess     detect_s3fd_input_0 _base           _thread_process           DEBUG    threading: (function: 'process_input')
09/22/2020 20:38:54 MainProcess     MainThread      multithreading  start                     DEBUG    Started all threads 'detect_s3fd_input': 1
09/22/2020 20:38:54 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread(s): 'detect_s3fd_predict'
09/22/2020 20:38:54 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 1 of 1: 'detect_s3fd_predict_0'
09/22/2020 20:38:54 MainProcess     detect_s3fd_predict_0 _base           _thread_process           DEBUG    threading: (function: '_predict')
09/22/2020 20:38:54 MainProcess     MainThread      multithreading  start                     DEBUG    Started all threads 'detect_s3fd_predict': 1
09/22/2020 20:38:54 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread(s): 'detect_s3fd_output'
09/22/2020 20:38:54 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 1 of 1: 'detect_s3fd_output_0'
09/22/2020 20:38:54 MainProcess     detect_s3fd_output_0 _base           _thread_process           DEBUG    threading: (function: 'process_output')
09/22/2020 20:38:54 MainProcess     MainThread      multithreading  start                     DEBUG    Started all threads 'detect_s3fd_output': 1
09/22/2020 20:38:54 MainProcess     MainThread      pipeline        _launch_plugin            DEBUG    Launched detect plugin
09/22/2020 20:38:54 MainProcess     MainThread      pipeline        detected_faces            DEBUG    Running Detection. Phase: '['detect']'
09/22/2020 20:39:09 MainProcess     detect_s3fd_predict_0 multithreading  run                       DEBUG    Error in thread (detect_s3fd_predict_0):  OOM when allocating tensor with shape[64,642,642] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n	 [[node model_1/LAYER_2_pad/Pad (defined at \faceswap\lib\model\session.py:77) ]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n [Op:__inference_predict_function_1399]\n\nFunction call stack:\npredict_function\n
09/22/2020 20:39:09 MainProcess     MainThread      multithreading  check_and_raise_error     DEBUG    Thread error caught: [(<class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, ResourceExhaustedError(), <traceback object at 0x000001BCA0626D40>)]
Traceback (most recent call last):
  File "C:\Users\Arland\faceswap\lib\cli\launcher.py", line 156, in execute_script
    process.process()
  File "C:\Users\Arland\faceswap\scripts\extract.py", line 117, in process
    self._run_extraction()
  File "C:\Users\Arland\faceswap\scripts\extract.py", line 213, in _run_extraction
    for idx, extract_media in enumerate(status_bar):
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tqdm\std.py", line 1130, in __iter__
    for obj in iterable:
  File "C:\Users\Arland\faceswap\plugins\extract\pipeline.py", line 233, in detected_faces
    if self._check_and_raise_error():
  File "C:\Users\Arland\faceswap\plugins\extract\pipeline.py", line 655, in _check_and_raise_error
    if plugin.check_and_raise_error():
  File "C:\Users\Arland\faceswap\plugins\extract\_base.py", line 302, in check_and_raise_error
    err = thread.check_and_raise_error()
  File "C:\Users\Arland\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\Arland\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Arland\faceswap\plugins\extract\_base.py", line 420, in _thread_process
    batch = function(batch)
  File "C:\Users\Arland\faceswap\plugins\extract\detect\_base.py", line 208, in _predict
    batch = self.predict(batch)
  File "C:\Users\Arland\faceswap\plugins\extract\detect\s3fd.py", line 54, in predict
    predictions = self.model.predict(batch["feed"])
  File "C:\Users\Arland\faceswap\lib\model\session.py", line 77, in predict
    return self._model.predict(feed, batch_size=batch_size)
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py", line 88, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1268, in predict
    tmp_batch_outputs = predict_function(iterator)
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 618, in _call
    results = self._stateful_fn(*args, **kwds)
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 1661, in _filtered_call
    return self._call_flat(
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 1745, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 593, in call
    outputs = execute.execute(
  File "C:\Users\Arland\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError:  OOM when allocating tensor with shape[64,642,642] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[node model_1/LAYER_2_pad/Pad (defined at \faceswap\lib\model\session.py:77) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
 [Op:__inference_predict_function_1399]

Function call stack:
predict_function


============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         5530346 Bugfix - Pre-caching masks when warp-to-landmarks is disabled
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1650
gpu_devices_active:  GPU_0
gpu_driver:          441.66
gpu_vram:            GPU_0: 4096MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.18362-SP0
os_release:          10
py_command:          C:\Users\Arland\faceswap\faceswap.py extract -i C:/Users/Arland/Desktop/mm new/MELANIA SOURCE.mp4 -o C:/Users/Arland/Desktop/mm new/FACE A -D s3fd -A fan -nm none -min 20 -l 0.4 -een 1 -sz 256 -si 0 -L INFO -gui
py_conda_version:    conda 4.8.4
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           8
sys_processor:       Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
sys_ram:             Total: 16290MB, Available: 11803MB, Used: 4486MB, Free: 11803MB

=============== Pip Packages ===============
absl-py==0.10.0
astunparse==1.6.3
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.3.3
google-auth==1.21.2
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.32.0
h5py==2.10.0
idna==2.10
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1589202782679/work
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
Markdown==3.2.2
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.4.0.42
opt-einsum==3.3.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1594298230227/work
protobuf==3.13.0
psutil @ file:///C:/ci/psutil_1598370330503/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
pywin32==227
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-learn @ file:///C:/ci/scikit-learn_1598377018496/work
scipy==1.4.1
sip==4.19.13
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tensorflow-gpu==2.2.0
tensorflow-gpu-estimator==2.2.0
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1596810128862/work
urllib3==1.25.10
Werkzeug==1.0.1
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\Arland\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   0.10.0                   pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
ca-certificates 2020.7.22 0
cachetools 4.1.1 pypi_0 pypi certifi 2020.6.20 py38_0
chardet 3.0.4 pypi_0 pypi cudatoolkit 10.1.243 h74a9793_0
cudnn 7.6.5 cuda10.1_0
cycler 0.10.0 py38_0
fastcluster 1.1.26 py38hbe40bda_1 conda-forge ffmpeg 4.3.1 ha925a31_0 conda-forge ffmpy 0.2.3 pypi_0 pypi freetype 2.10.2 hd328e21_0
gast 0.3.3 pypi_0 pypi git 2.23.0 h6bb4b03_0
google-auth 1.21.2 pypi_0 pypi google-auth-oauthlib 0.4.1 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.32.0 pypi_0 pypi h5py 2.10.0 pypi_0 pypi icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 pypi_0 pypi imageio 2.9.0 py_0
imageio-ffmpeg 0.4.2 py_0 conda-forge intel-openmp 2020.2 254
joblib 0.16.0 py_0
jpeg 9b hb83a4c4_2
keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.2.0 py38h74a9793_0
libpng 1.6.37 h2a8f88b_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 h62dcd97_1
markdown 3.2.2 pypi_0 pypi matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38h64f37c6_0
mkl 2020.2 256
mkl-service 2.3.0 py38hb782905_0
mkl_fft 1.2.0 py38h45dec08_0
mkl_random 1.1.1 py38h47e9c7a_0
numpy 1.19.1 py38h5510c5b_0
numpy-base 1.19.1 py38ha3acd2a_0
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 pypi_0 pypi olefile 0.46 py_0
opencv-python 4.4.0.42 pypi_0 pypi openssl 1.1.1g he774522_1
opt-einsum 3.3.0 pypi_0 pypi pathlib 1.0.1 py_1
pillow 7.2.0 py38hcc1f983_0
pip 20.2.2 py38_0
protobuf 3.13.0 pypi_0 pypi psutil 5.7.2 py38he774522_0
pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pyparsing 2.4.7 py_0
pyqt 5.9.2 py38ha925a31_4
python 3.8.5 h5fd99cc_1
python-dateutil 2.8.1 py_0
python_abi 3.8 1_cp38 conda-forge pywin32 227 py38he774522_1
qt 5.9.7 vc14h73c81de_0
requests 2.24.0 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rsa 4.6 pypi_0 pypi scikit-learn 0.23.2 py38h47e9c7a_0
scipy 1.4.1 pypi_0 pypi setuptools 49.6.0 py38_0
sip 4.19.13 py38ha925a31_0
six 1.15.0 py_0
sqlite 3.33.0 h2a8f88b_0
tensorboard 2.2.2 pypi_0 pypi tensorboard-plugin-wit 1.7.0 pypi_0 pypi tensorflow-gpu 2.2.0 pypi_0 pypi tensorflow-gpu-estimator 2.2.0 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
tornado 6.0.4 py38he774522_1
tqdm 4.48.2 py_0
urllib3 1.25.10 pypi_0 pypi vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_3
werkzeug 1.0.1 pypi_0 pypi wheel 0.35.1 py_0
wincertstore 0.2 py38_0
wrapt 1.12.1 pypi_0 pypi xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.5 h04227a9_0 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: unsharp_mask amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: True [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 70 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] coverage: 68.75 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 reflect_padding: False allow_growth: False mixed_precision: False convert_batchsize: 16 [global.loss] loss_function: ssim mask_loss_function: mse l2_reg_term: 100 eye_multiplier: 6 mouth_multiplier: 4 penalized_mask_loss: True mask_type: extended mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 disable_warp: False color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
by bryanlyon » Wed Sep 23, 2020 1:34 am

Hey, welcome to the forums. OOMs happen whenever the GPU runs out of memory. Often this happens due to overly large BSs or too high of settings on your model. However, sometimes (as it appears in your case) the problem can actually be outside of Faceswap. Many applications use GPU Vram and can take vram that Faceswap might need. Windows reserves some (but this shouldn't be a new issue unless you just switched to Windows). But also programs such as browsers, video players, and games will all reserve some vram (and sometimes not give it back).

Try a reboot and running Faceswap to see if that helps at all. This will clear any programs that are using vram and let you use the maximum for Faceswap.

Go to full post
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: NOOB - VERY FRUSTRATED

Post by bryanlyon »

Hey, welcome to the forums. OOMs happen whenever the GPU runs out of memory. Often this happens due to overly large BSs or too high of settings on your model. However, sometimes (as it appears in your case) the problem can actually be outside of Faceswap. Many applications use GPU Vram and can take vram that Faceswap might need. Windows reserves some (but this shouldn't be a new issue unless you just switched to Windows). But also programs such as browsers, video players, and games will all reserve some vram (and sometimes not give it back).

Try a reboot and running Faceswap to see if that helps at all. This will clear any programs that are using vram and let you use the maximum for Faceswap.

User avatar
Zebra522020
Posts: 3
Joined: Mon Sep 21, 2020 8:49 pm
Been thanked: 1 time

Re: NOOB - VERY FRUSTRATED

Post by Zebra522020 »

Thank you. I had tried rebooting and got the same error. I will research further

User avatar
Zebra522020
Posts: 3
Joined: Mon Sep 21, 2020 8:49 pm
Been thanked: 1 time

Re: NOOB - VERY FRUSTRATED

Post by Zebra522020 »

Tried another re-boot and loaded an in-progress project that I had saved from my Laptop. Seems to have picked up on the iterations and is continuing with no issues yet this morning. Fingers crossed

Locked