OOM out of memory during convert but not training

Getting errors or found a bug when converting faces from a trained model? Post about them here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Convert process. If you want to get tips, or better understand the Convert process, then you should look in the Convert Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
tochan
Posts: 18
Joined: Sun Sep 22, 2019 8:17 am
Been thanked: 5 times

OOM out of memory during convert but not training

Post by tochan »

Hi,

i train with the Dlight Trainer (without problems) but when i want to use the converter.... :?

Can someone help her? Faceswap is up to date (2019.11.10)Win10 OS

Code: Select all

11/10/2019 08:38:17 MainProcess     MainThread      config          check_exists              DEBUG    Config file exists: 'C:\Users\denni\faceswap\config\convert.ini'
11/10/2019 08:38:17 MainProcess     MainThread      config          load_config               VERBOSE  Loading config: 'C:\Users\denni\faceswap\config\convert.ini'
11/10/2019 08:38:17 MainProcess     MainThread      config          validate_config           DEBUG    Validating config
11/10/2019 08:38:17 MainProcess     MainThread      config          check_config_change       DEBUG    Default config has not changed
11/10/2019 08:38:17 MainProcess     MainThread      config          check_config_choices      DEBUG    Checking config choices
11/10/2019 08:38:17 MainProcess     MainThread      config          check_config_choices      DEBUG    Checked config choices
11/10/2019 08:38:17 MainProcess     MainThread      config          validate_config           DEBUG    Validated config
11/10/2019 08:38:17 MainProcess     MainThread      config          handle_config             DEBUG    Handled config
11/10/2019 08:38:17 MainProcess     MainThread      config          __init__                  DEBUG    Initialized: Config
11/10/2019 08:38:17 MainProcess     MainThread      config          get                       DEBUG    Getting config item: (section: 'scaling.sharpen', option: 'method')
11/10/2019 08:38:17 MainProcess     MainThread      config          get                       DEBUG    Returning item: (type: <class 'str'>, value: gaussian)
11/10/2019 08:38:17 MainProcess     MainThread      config          get                       DEBUG    Getting config item: (section: 'scaling.sharpen', option: 'amount')
11/10/2019 08:38:17 MainProcess     MainThread      config          get                       DEBUG    Returning item: (type: <class 'int'>, value: 150)
11/10/2019 08:38:17 MainProcess     MainThread      config          get                       DEBUG    Getting config item: (section: 'scaling.sharpen', option: 'radius')
11/10/2019 08:38:17 MainProcess     MainThread      config          get                       DEBUG    Returning item: (type: <class 'float'>, value: 0.3)
11/10/2019 08:38:17 MainProcess     MainThread      config          get                       DEBUG    Getting config item: (section: 'scaling.sharpen', option: 'threshold')
11/10/2019 08:38:17 MainProcess     MainThread      config          get                       DEBUG    Returning item: (type: <class 'float'>, value: 5.0)
11/10/2019 08:38:17 MainProcess     MainThread      _base           set_config                DEBUG    Config: {'method': 'gaussian', 'amount': 150, 'radius': 0.3, 'threshold': 5.0}
11/10/2019 08:38:17 MainProcess     MainThread      _base           __init__                  DEBUG    config: {'method': 'gaussian', 'amount': 150, 'radius': 0.3, 'threshold': 5.0}
11/10/2019 08:38:17 MainProcess     MainThread      _base           __init__                  DEBUG    Initialized Scaling
11/10/2019 08:38:17 MainProcess     MainThread      convert         load_plugins              DEBUG    Loaded plugins: {'box': <plugins.convert.mask.box_blend.Mask object at 0x0000028D78266358>, 'mask': <plugins.convert.mask.mask_blend.Mask object at 0x0000028D78266D68>, 'color': <plugins.convert.color.avg_color.Color object at 0x0000028D782D8F60>, 'seamless': None, 'scaling': <plugins.convert.scaling.sharpen.Scaling object at 0x0000028D78302978>}
11/10/2019 08:38:17 MainProcess     MainThread      convert         __init__                  DEBUG    Initialized Converter
11/10/2019 08:38:17 MainProcess     MainThread      convert         __init__                  DEBUG    Initialized Convert
11/10/2019 08:38:17 MainProcess     MainThread      convert         process                   DEBUG    Starting Conversion
11/10/2019 08:38:17 MainProcess     MainThread      convert         convert_images            DEBUG    Converting images
11/10/2019 08:38:17 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'convert_out'
11/10/2019 08:38:17 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'convert_out'
11/10/2019 08:38:17 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'patch'
11/10/2019 08:38:17 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'patch'
11/10/2019 08:38:17 MainProcess     MainThread      convert         pool_processes            DEBUG    16
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initializing MultiThread: (target: 'patch', thread_count: 16)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initialized MultiThread: 'patch'
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread(s): 'patch'
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 1 of 16: 'patch_0'
11/10/2019 08:38:17 MainProcess     patch_0         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 2 of 16: 'patch_1'
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     patch_1         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 3 of 16: 'patch_2'
11/10/2019 08:38:17 MainProcess     patch_2         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 4 of 16: 'patch_3'
11/10/2019 08:38:17 MainProcess     patch_3         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 5 of 16: 'patch_4'
11/10/2019 08:38:17 MainProcess     patch_4         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 6 of 16: 'patch_5'
11/10/2019 08:38:17 MainProcess     patch_5         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 7 of 16: 'patch_6'
11/10/2019 08:38:17 MainProcess     patch_6         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 8 of 16: 'patch_7'
11/10/2019 08:38:17 MainProcess     patch_7         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 9 of 16: 'patch_8'
11/10/2019 08:38:17 MainProcess     patch_8         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 10 of 16: 'patch_9'
11/10/2019 08:38:17 MainProcess     patch_9         convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 11 of 16: 'patch_10'
11/10/2019 08:38:17 MainProcess     patch_10        convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 12 of 16: 'patch_11'
11/10/2019 08:38:17 MainProcess     patch_11        convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 13 of 16: 'patch_12'
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     patch_12        convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 14 of 16: 'patch_13'
11/10/2019 08:38:17 MainProcess     patch_13        convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 15 of 16: 'patch_14'
11/10/2019 08:38:17 MainProcess     patch_14        convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Starting thread 16 of 16: 'patch_15'
11/10/2019 08:38:17 MainProcess     patch_15        convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x0000028BB2297320>, out_queue: <queue.Queue object at 0x0000028BB22971D0>, completion_queue: None)
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  start                     DEBUG    Started all threads 'patch': 16
11/10/2019 08:38:17 MainProcess     MainThread      multithreading  completed                 DEBUG    False
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:17 MainProcess     predict_faces_0 _base           largest_face_index        DEBUG    0
11/10/2019 08:38:18 MainProcess     MainThread      multithreading  completed                 DEBUG    False
11/10/2019 08:38:18 MainProcess     predict_faces_0 multithreading  run                       DEBUG    Error in thread (predict_faces_0): cudnn PoolForward launch failed\n     [[{{node encoder/average_pooling2d_1/AvgPool}} = AvgPool[T=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="VALID", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:GPU:0"](encoder/average_pooling2d_1/AvgPool-0-TransposeNHWCToNCHW-LayoutOptimizer)]]\n     [[{{node decoder_b/face_out/Sigmoid/_761}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_971_decoder_b/face_out/Sigmoid", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
11/10/2019 08:38:19 MainProcess     MainThread      multithreading  check_and_raise_error     DEBUG    Thread error caught: [(<class 'tensorflow.python.framework.errors_impl.InternalError'>, InternalError(), <traceback object at 0x0000028D7831DA08>)]
Traceback (most recent call last):
  File "C:\Users\denni\faceswap\lib\cli.py", line 128, in execute_script
    process.process()
  File "C:\Users\denni\faceswap\scripts\convert.py", line 105, in process
    self.convert_images()
  File "C:\Users\denni\faceswap\scripts\convert.py", line 131, in convert_images
    self.check_thread_error()
  File "C:\Users\denni\faceswap\scripts\convert.py", line 151, in check_thread_error
    thread.check_and_raise_error()
  File "C:\Users\denni\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\denni\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\denni\faceswap\scripts\convert.py", line 572, in predict_faces
    predicted = self.predict(feed_faces, batch_size)
  File "C:\Users\denni\faceswap\scripts\convert.py", line 622, in predict
    predicted = self.predictor(feed, batch_size=batch_size)
  File "C:\Users\denni\MiniConda3\envs\faceswap\lib\site-packages\keras\engine\training.py", line 1169, in predict
    steps=steps)
  File "C:\Users\denni\MiniConda3\envs\faceswap\lib\site-packages\keras\engine\training_arrays.py", line 294, in predict_loop
    batch_outs = f(ins_batch)
  File "C:\Users\denni\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "C:\Users\denni\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "C:\Users\denni\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\client\session.py", line 1439, in __call__
    run_metadata_ptr)
  File "C:\Users\denni\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: cudnn PoolForward launch failed
     [[{{node encoder/average_pooling2d_1/AvgPool}} = AvgPool[T=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="VALID", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:GPU:0"](encoder/average_pooling2d_1/AvgPool-0-TransposeNHWCToNCHW-LayoutOptimizer)]]
     [[{{node decoder_b/face_out/Sigmoid/_761}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_971_decoder_b/face_out/Sigmoid", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         ffd3829 Added allow_growth argument for preview. 87ccdfa Merge pull request #929 from kilroythethird/fsa_conv_fixes. c2d9a27 Add json to the filter in the GUI alignment open dialog. bb90bcd Merge branch 'double_fsa_fix' into fsa_conv_fixes. 61497a9 Updart INSTALL.md
gpu_cuda:            9.0
gpu_cudnn:           7.0.5
gpu_devices:         GPU_0: GeForce RTX 2080, GPU_1: GeForce GTX 1080
gpu_devices_active:  GPU_0, GPU_1
gpu_driver:          441.12
gpu_vram:            GPU_0: 8192MB, GPU_1: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.18362-SP0
os_release:          10
py_command:          C:\Users\denni\faceswap\faceswap.py convert -i D:/In_progress/Face/Images_full_sets/Images -o D:/In_progress/Face/Images_full_sets/swap/Dlight/1_Sharpen -al D:/In_progress/Face/Images_full_sets/Images/alignments.fsa -m D:/In_progress/Face/Images_full_sets/Model/Dlight -c avg-color -M predicted -sc sharpen -w opencv -osc 100 -a D:/In_progress/Face/Images_full_sets/Faces -l 0.4 -j 0 -g 1 -L INFO -gui
py_conda_version:    conda 4.5.12
py_implementation:   CPython
py_version:          3.6.8
py_virtual_env:      True
sys_cores:           16
sys_processor:       AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
sys_ram:             Total: 65467MB, Available: 55716MB, Used: 9751MB, Free: 55716MB

=============== Pip Packages ===============
absl-py==0.7.0
astor==0.7.1
certifi==2019.6.16
Click==7.0
cloudpickle==0.8.0
cmake==3.13.3
cycler==0.10.0
cytoolz==0.9.0.1
dask==1.1.4
decorator==4.3.2
dlib==19.16.99
face-recognition==1.2.3
face-recognition-models==0.3.0
fastcluster==1.1.25
ffmpy==0.2.2
gast==0.2.2
grpcio==1.16.1
h5py==2.9.0
imageio==2.5.0
imageio-ffmpeg==0.3.0
Keras==2.2.4
Keras-Applications==1.0.7
Keras-Preprocessing==1.0.9
kiwisolver==1.0.1
Markdown==3.0.1
matplotlib==2.2.2
mkl-fft==1.0.10
mkl-random==1.0.2
mock==2.0.0
networkx==2.2
numpy==1.16.2
nvidia-ml-py3==7.352.1
olefile==0.46
opencv-python==4.1.1.26
pathlib==1.0.1
pbr==5.1.3
Pillow==6.1.0
protobuf==3.6.1
psutil==5.6.1
pyparsing==2.3.1
pyreadline==2.1
python-dateutil==2.8.0
pytz==2018.9
PyWavelets==1.0.2
pywin32==224
PyYAML==3.13
scikit-image==0.14.2
scikit-learn==0.20.3
scipy==1.2.1
six==1.12.0
tensorboard==1.12.2
tensorflow==1.12.0
tensorflow-estimator==1.13.0
termcolor==1.1.0
toolz==0.9.0
toposort==1.5
tornado==6.0.1
tqdm==4.31.1
Werkzeug==0.14.1
wincertstore==0.2

============== Conda Packages ==============
# packages in environment at C:\Users\denni\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu 
absl-py                   0.7.0                    py36_0 
astor                     0.7.1                    py36_0 
blas                      1.0                         mkl 
ca-certificates           2019.5.15                     0 
certifi                   2019.6.16                py36_1 
Click                     7.0                       <pip>
cloudpickle               0.8.0                    py36_0 
cmake                     3.13.3                    <pip>
cudatoolkit               9.0                           1 
cudnn                     7.3.1                 cuda9.0_0 
cycler                    0.10.0           py36h009560c_0 
cytoolz                   0.9.0.1          py36hfa6e2cd_1 
dask-core                 1.1.4                      py_0 
decorator                 4.3.2                    py36_0 
dlib                      19.16.99                  <pip>
face-recognition          1.2.3                     <pip>
face-recognition-models   0.3.0                     <pip>
fastcluster               1.1.25          py36h830ac7b_1000    conda-forge
ffmpeg                    4.1               h6538335_1002    conda-forge
ffmpy                     0.2.2                     <pip>
freetype                  2.9.1                ha9979f8_1 
gast                      0.2.2                    py36_0 
grpcio                    1.16.1           py36h351948d_1 
h5py                      2.9.0            py36h5e291fa_0 
hdf5                      1.10.4               h7ebc959_0 
icc_rt                    2019.0.0             h0cc432a_1 
icu                       58.2                 ha66f8fd_1 
imageio                   2.5.0                    py36_0 
imageio-ffmpeg            0.3.0                     <pip>
intel-openmp              2019.1                      144 
jpeg                      9c                hfa6e2cd_1001    conda-forge
keras                     2.2.4                         0 
keras-applications        1.0.7                      py_0 
keras-base                2.2.4                    py36_0 
keras-preprocessing       1.0.9                      py_0 
kiwisolver                1.0.1            py36h6538335_0 
libblas                   3.8.0                     8_mkl    conda-forge
libcblas                  3.8.0                     8_mkl    conda-forge
liblapack                 3.8.0                     8_mkl    conda-forge
liblapacke                3.8.0                     8_mkl    conda-forge
libmklml                  2019.0.3                      0 
libpng                    1.6.36               h2a8f88b_0 
libprotobuf               3.6.1                h7bd577a_0 
libtiff                   4.0.10               hb898794_2 
libwebp                   1.0.2                hfa6e2cd_2    conda-forge
markdown                  3.0.1                    py36_0 
matplotlib                2.2.2            py36had4c4a9_2 
mkl                       2019.1                      144 
mkl_fft                   1.0.10           py36h14836fe_0 
mkl_random                1.0.2            py36h343c172_0 
mock                      2.0.0            py36h9086845_0 
networkx                  2.2                      py36_1 
numpy                     1.16.2           py36h19fb1c0_0 
numpy-base                1.16.2           py36hc3f5095_0 
nvidia-ml-py3             7.352.1                   <pip>
olefile                   0.46                     py36_0 
opencv                    4.1.0            py36hb4945ee_5    conda-forge
opencv-python             4.0.0.21                  <pip>
opencv-python             4.1.1.26                  <pip>
openssl                   1.1.1c               he774522_1 
pathlib                   1.0.1                    py36_1 
pbr                       5.1.3                      py_0 
pillow                    6.1.0            py36hdc69c19_0 
pip                       19.0.3                   py36_0 
protobuf                  3.6.1            py36h33f27b4_0 
psutil                    5.6.1            py36he774522_0 
pyparsing                 2.3.1                    py36_0 
pyqt                      5.9.2            py36h6538335_2 
pyreadline                2.1                      py36_1 
python                    3.6.8                h9f7ef89_7 
python-dateutil           2.8.0                    py36_0 
pytz                      2018.9                   py36_0 
pywavelets                1.0.2            py36h8c2d366_0 
pywin32                   224                       <pip>
pyyaml                    3.13             py36hfa6e2cd_0 
qt                        5.9.7            vc14h73c81de_0 
scikit-image              0.14.2           py36ha925a31_0 
scikit-learn              0.20.3           py36h343c172_0 
scipy                     1.2.1            py36h29ff71c_0 
setuptools                40.8.0                   py36_0 
sip                       4.19.8           py36h6538335_0 
six                       1.12.0                   py36_0 
sqlite                    3.27.2               he774522_0 
tensorboard               1.12.2           py36h33f27b4_0 
tensorflow                1.12.0          gpu_py36ha5f9131_0 
tensorflow-base           1.12.0          gpu_py36h6e53903_0 
tensorflow-estimator      1.13.0                     py_0 
tensorflow-gpu            1.12.0               h0d30ee6_0 
termcolor                 1.1.0                    py36_1 
tk                        8.6.8                hfa6e2cd_0 
toolz                     0.9.0                    py36_0 
toposort                  1.5                       <pip>
tornado                   6.0.1            py36he774522_0 
tqdm                      4.31.1                     py_0 
vc                        14.1                 h0510ff6_4 
vs2015_runtime            14.15.26706          h3a45250_0 
werkzeug                  0.14.1                   py36_0 
wheel                     0.33.1                   py36_0 
wincertstore              0.2              py36h7fe50ca_0 
xz                        5.2.4                h2fa13f4_4 
yaml                      0.1.7                hc54c509_2 
zlib                      1.2.11               h62dcd97_3 
zstd                      1.3.7                h508b16e_0 

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     gaussian
radius:                   3.0
passes:                   4
erosion:                  0.0

[scaling.sharpen]
method:                   gaussian
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               8

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7
scalefactor:              0.709
batch-size:               8

[detect.s3fd]
confidence:               50
batch-size:               8

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      train
options_panel_width:      30
console_panel_height:     20
font:                     default
font_size:                9

--------- train.ini ---------

[global]
coverage:                 87.5
mask_type:                components
mask_blur:                True
icnr_init:                True
conv_aware_init:          False
subpixel_upscaling:       True
reflect_padding:          True
penalized_mask_loss:      True
loss_function:            mae
learning_rate:            5e-05

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4
by torzdf » Fri Jan 22, 2021 11:53 am

Yes, this is out of memory.

You should be able to resolve your issue by going into Settings > Train Settings and lowering convert batchsize

Go to full post

User avatar
torzdf
Posts: 1495
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 51 times
Been thanked: 287 times

Re: Dlight Convert Crash: report

Post by torzdf »

Can you try enabling the "allow_growth" option and see if that fixes the issue?

My word is final


User avatar
tochan
Posts: 18
Joined: Sun Sep 22, 2019 8:17 am
Been thanked: 5 times

Re: Dlight Convert Crash: report

Post by tochan »

Nope, not working....


User avatar
bryanlyon
Site Admin
Posts: 634
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 161 times
Contact:

Re: Dlight Convert Crash: report

Post by bryanlyon »

This is a very weird issue and doesn't make a whole lot of sense. Have you tried to run it after a reboot? Can you try convert using one of the snapshots to make sure it isn't a model that has gone corrupt? Does the Preview tool work?

If the problem persists after these steps, please set your logging to "Trace", try a convert, and send us a zipped copy of your faceswap.log (Not the crash report, but the faceswap.log in the same folder) and we'll look into it more.


User avatar
tochan
Posts: 18
Joined: Sun Sep 22, 2019 8:17 am
Been thanked: 5 times

Re: Dlight Convert Crash: report

Post by tochan »

I try a reboot, same crash.
I try a snapshot, same crash.

Then i set the trainer plugin to default, create a new model an it works!
Then i try the settings step by step with new Dlight models and the crash come, when the "Reflect Padding" is activ.

Not testet with other trainers to this moment, only with Dlight.

I hope i can help with this information.

And for me... Don't test a new trainer with days of training... test the convert after 100 iterations... ;)

second info: When i start Training with dual Gpu crashes (other report). When i start with one, save and try then 2 GPU's training works.


User avatar
superjj
Posts: 2
Joined: Sat Apr 04, 2020 9:59 pm

OOM out of memory during convert but not training

Post by superjj »

I've trained my model to about 91k iterations with no crashes. But when I try to convert, faceswap crashes with an OOM error in the logs. Conversion seems to work when I select a small range of frames to convert, maybe 15 frames at a time. But that's barely half a second of video.

Has anyone trained fine, but end up with OOM crashes during conversion?

I'm using a GTX 1650 Super 4gb, and training on the Dlight model with resource-saving options turned on.

Thanks!


User avatar
torzdf
Posts: 1495
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 51 times
Been thanked: 287 times

Re: OOM out of memory during convert but not training

Post by torzdf »

I have never seen this before, sadly.

My word is final


User avatar
bryanlyon
Site Admin
Posts: 634
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 161 times
Contact:

Re: OOM out of memory during convert but not training

Post by bryanlyon »

Were you having to use Allow Growth during training? In which case, you might be running into a weird issue we've noticed on some people's setups.


User avatar
PLAY-911
Posts: 6
Joined: Mon Apr 13, 2020 6:52 pm
Has thanked: 1 time

Re: OOM out of memory during convert but not training

Post by PLAY-911 »

superjj wrote: Tue Apr 14, 2020 10:27 pm

I've trained my model to about 91k iterations with no crashes. But when I try to convert, faceswap crashes with an OOM error in the logs. Conversion seems to work when I select a small range of frames to convert, maybe 15 frames at a time. But that's barely half a second of video.

Has anyone trained fine, but end up with OOM crashes during conversion?

I'm using a GTX 1650 Super 4gb, and training on the Dlight model with resource-saving options turned on.

Thanks!

Are you in Windows? I had problems with virtual memory assigned by windows


User avatar
superjj
Posts: 2
Joined: Sat Apr 04, 2020 9:59 pm

Re: OOM out of memory during convert but not training

Post by superjj »

bryanlyon wrote: Wed Apr 15, 2020 7:59 pm

Were you having to use Allow Growth during training? In which case, you might be running into a weird issue we've noticed on some people's setups.

Yes I had Allow Growth turned on during training.


User avatar
torzdf
Posts: 1495
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 51 times
Been thanked: 287 times

Re: OOM out of memory during convert but not training

Post by torzdf »

Make sure you select "Allow Growth" for convert too

My word is final


User avatar
mgolvach
Posts: 3
Joined: Sun May 17, 2020 2:01 am
Has thanked: 1 time

Re: OOM out of memory during convert but not training

Post by mgolvach »

Just in case it helps, I had a similar situation. Training with DFL-SAE at 128px (max I could do) was working fine, but conversion gave me the error:

Resource exhausted: OOM when allocating tensor with shape[16,130,130,126] and type float...

I had turned on "allow growth" for conversion, but found I did not have "allow growth" checked for training. Though it seemed counterintuitive, I turned off (unchecked) "allow growth" for conversion, and that solved the problem.

I think, essentially, with regard to the "allow growth" option, you need to be consistent with training and conversion. If you train with it on (or off), you must do the same for conversion.

This may not be the case for everyone. I'm certain more GPU power would probably solve the problem as well ;)

Thanks for this board's wealth of information and help!

Mike


User avatar
bryanlyon
Site Admin
Posts: 634
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 161 times
Contact:

Re: OOM out of memory during convert but not training

Post by bryanlyon »

Allow_growth does not affect your model in anyway, it only changes how Tensorflow allocates memory. You are likely running into a different issue. But we recommend leaving allow_growth off unless it's absolutely necessary to getting Faceswap running on your system.


User avatar
RahulRookie
Posts: 4
Joined: Sun May 31, 2020 9:31 am

Error: OOM when allocating tensor

Post by RahulRookie »

Hi,

Need help to debug this error

I am getting the following error when I run Convert function. Full video is not generated, it crashes in the mid way.
What is the cause of this issue and how to fix. Appreciate any support. I am using GPU

Code: Select all

Traceback (most recent call last):
File "D:\FACESWAP_NEW\faceswap\lib\cli\launcher.py", line 155, in execute_script
process.process()
File "D:\FACESWAP_NEW\faceswap\scripts\convert.py", line 157, in process
self._convert_images()
File "D:\FACESWAP_NEW\faceswap\scripts\convert.py", line 184, in _convert_images
self._check_thread_error()
File "D:\FACESWAP_NEW\faceswap\scripts\convert.py", line 204, in _check_thread_error
thread.check_and_raise_error()
File "D:\FACESWAP_NEW\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
raise error[1].with_traceback(error[2])
File "D:\FACESWAP_NEW\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "D:\FACESWAP_NEW\faceswap\scripts\convert.py", line 873, in _predict_faces
predicted = self._predict(feed_faces, batch_size)
File "D:\FACESWAP_NEW\faceswap\scripts\convert.py", line 958, in _predict
predicted = self._predictor(feed, batch_size=batch_size)
File "C:\Users\RR\MiniConda3\envs\faceswap\lib\site-packages\keras\engine\training.py", line 1169, in predict
steps=steps)
File "C:\Users\RR\MiniConda3\envs\faceswap\lib\site-packages\keras\engine\training_arrays.py", line 294, in predict_loop
batch_outs = f(ins_batch)
File "C:\Users\RR\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "C:\Users\RR\MiniConda3\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "C:\Users\RR\MiniConda3\envs\faceswap\lib\site-packages\tensorflow_core\python\client\session.py", line 1472, in __call__
run_metadata_ptr)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[256,128,5,5] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node encoder/conv_32_0_conv2d/convolution}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

[[decoder_b/face_out/Sigmoid/_185]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[256,128,5,5] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node encoder/conv_32_0_conv2d/convolution}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.
06/16/2020 12:28:31 CRITICAL An unexpected crash has occurred. Crash report written to 'D:\FACESWAP_NEW\faceswap\crash_report.2020.06.16.122825733693.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Process exited.

Script from Generate function

C:\Users\RR\MiniConda3\envs\faceswap\python.exe D:\FACESWAP_NEW\faceswap\faceswap.py convert -i D:/FACESWAP_PROJECTS/PROJ10/DST-101.mp4 -o D:/FACESWAP_PROJECTS/PROJ10/Output -m D:/FACESWAP_PROJECTS/PROJ10/ModelAB -c avg-color -M none -sc none -w ffmpeg -osc 100 -l 0.4 -j 0 -g 1 -ag -otf -L INFO


User avatar
djgs11
Posts: 1
Joined: Thu Jan 21, 2021 11:01 pm

OOM when allocating tensor

Post by djgs11 »

Hello

When converting i get this error.
Can any1 enlighten me as to what it is? Not enough VRAM?

Code: Select all

01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Added defaults: mask.mask_blend
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Adding defaults: (filename: sharpen_defaults.py, module_path: plugins.convert.scaling, plugin_type: scaling
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Importing defaults module: plugins.convert.scaling.sharpen_defaults
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_section                    DEBUG    Add section: (title: 'scaling.sharpen', info: 'Options for sharpening the face after placement')
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'scaling.sharpen', title: 'method', datatype: '<class 'str'>', default: 'none', info: 'The type of sharpening to use:\n	 none: Don't perform any sharpening.\n	 box: Fastest, but weakest method. Uses a box filter to assess edges.\n	 gaussian: Slower, but better than box. Uses a gaussian filter to assess edges.\n	 unsharp-mask: Slowest, but most tweakable. Uses the unsharp-mask method to assess edges.', rounding: 'None', min_max: None, choices: ['none', 'box', 'gaussian', 'unsharp_mask'], gui_radio: True, fixed: True, group: None)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'scaling.sharpen', title: 'amount', datatype: '<class 'int'>', default: '150', info: 'Percentage that controls the magnitude of each overshoot (how much darker and how much lighter the edge borders become).\nThis can also be thought of as how much contrast is added at the edges. It does not affect the width of the edge rims.', rounding: '1', min_max: (100, 500), choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'scaling.sharpen', title: 'radius', datatype: '<class 'float'>', default: '0.3', info: 'Affects the size of the edges to be enhanced or how wide the edge rims become, so a smaller radius enhances smaller-scale detail.\nRadius is set as a percentage of the final frame width and rounded to the nearest pixel. E.g for a 1280 width frame, a 0.6 percenatage will give a radius of 8px.\nHigher radius values can cause halos at the edges, a detectable faint light rim around objects. Fine detail needs a smaller radius. \nRadius and amount interact; reducing one allows more of the other.', rounding: '1', min_max: (0.1, 5.0), choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'scaling.sharpen', title: 'threshold', datatype: '<class 'float'>', default: '5.0', info: '[unsharp_mask only] Controls the minimal brightness change that will be sharpened or how far apart adjacent tonal values have to be before the filter does anything.\nThis lack of action is important to prevent smooth areas from becoming speckled. The threshold setting can be used to sharpen more pronounced edges, while leaving subtler edges untouched. \nLow values should sharpen more because fewer areas are excluded. \nHigher threshold values exclude areas of lower contrast.', rounding: '1', min_max: (1.0, 10.0), choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Added defaults: scaling.sharpen
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Adding defaults: (filename: ffmpeg_defaults.py, module_path: plugins.convert.writer, plugin_type: writer
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Importing defaults module: plugins.convert.writer.ffmpeg_defaults
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_section                    DEBUG    Add section: (title: 'writer.ffmpeg', info: 'Options for encoding converted frames to video.')
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.ffmpeg', title: 'container', datatype: '<class 'str'>', default: 'mp4', info: 'Video container to use.', rounding: 'None', min_max: None, choices: ['avi', 'flv', 'mkv', 'mov', 'mp4', 'mpeg', 'webm'], gui_radio: True, fixed: True, group: None)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.ffmpeg', title: 'codec', datatype: '<class 'str'>', default: 'libx264', info: 'Video codec to use:\n	 libx264: H.264. A widely supported and commonly used codec.\n	 libx265: H.265 / HEVC video encoder application library.', rounding: 'None', min_max: None, choices: ['libx264', 'libx265'], gui_radio: True, fixed: True, group: None)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.ffmpeg', title: 'crf', datatype: '<class 'int'>', default: '23', info: 'Constant Rate Factor:  0 is lossless and 51 is worst quality possible. A lower value generally leads to higher quality, and a subjectively sane range is 17-28. Consider 17 or 18 to be visually lossless or nearly so; it should look the same or nearly the same as the input but it isn't technically lossless.\nThe range is exponential, so increasing the CRF value +6 results in roughly half the bitrate / file size, while -6 leads to roughly twice the bitrate.', rounding: '1', min_max: (0, 51), choices: [], gui_radio: False, fixed: True, group: quality)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.ffmpeg', title: 'preset', datatype: '<class 'str'>', default: 'medium', info: 'A preset is a collection of options that will provide a certain encoding speed to compression ratio.\nA slower preset will provide better compression (compression is quality per filesize).\nUse the slowest preset that you have patience for.', rounding: 'None', min_max: None, choices: ['ultrafast', 'superfast', 'veryfast', 'faster', 'fast', 'medium', 'slow', 'slower', 'veryslow'], gui_radio: True, fixed: True, group: quality)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.ffmpeg', title: 'tune', datatype: '<class 'str'>', default: 'none', info: 'Change settings based upon the specifics of your input:\n	 none: Don't perform any additional tuning.\n	 film: [H.264 only] Use for high quality movie content; lowers deblocking.\n	 animation: [H.264 only] Good for cartoons; uses higher deblocking and more reference frames.\n	 grain: Preserves the grain structure in old, grainy film material.\n	 stillimage: [H.264 only] Good for slideshow-like content.\n	 fastdecode: Allows faster decoding by disabling certain filters.\n	 zerolatency: Good for fast encoding and low-latency streaming.', rounding: 'None', min_max: None, choices: ['none', 'film', 'animation', 'grain', 'stillimage', 'fastdecode', 'zerolatency'], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.ffmpeg', title: 'profile', datatype: '<class 'str'>', default: 'auto', info: '[H.264 Only] Limit the output to a specific H.264 profile. Don't change this unless your target device only supports a certain profile.', rounding: 'None', min_max: None, choices: ['auto', 'baseline', 'main', 'high', 'high10', 'high422', 'high444'], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.ffmpeg', title: 'level', datatype: '<class 'str'>', default: 'auto', info: '[H.264 Only] Set the encoder level, Don't change this unless your target device only supports a certain level.', rounding: 'None', min_max: None, choices: ['auto', '1', '1b', '1.1', '1.2', '1.3', '2', '2.1', '2.2', '3', '3.1', '3.2', '4', '4.1', '4.2', '5', '5.1', '5.2', '6', '6.1', '6.2'], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.ffmpeg', title: 'skip_mux', datatype: '<class 'bool'>', default: 'False', info: 'Skip muxing audio to the final video output. This will result in a video without an audio track.', rounding: 'None', min_max: None, choices: None, gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Added defaults: writer.ffmpeg
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Adding defaults: (filename: gif_defaults.py, module_path: plugins.convert.writer, plugin_type: writer
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Importing defaults module: plugins.convert.writer.gif_defaults
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_section                    DEBUG    Add section: (title: 'writer.gif', info: 'Options for outputting converted frames to an animated gif.')
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.gif', title: 'fps', datatype: '<class 'int'>', default: '25', info: 'Frames per Second.', rounding: '1', min_max: (1, 60), choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.gif', title: 'loop', datatype: '<class 'int'>', default: '0', info: 'The number of iterations. Set to 0 to loop indefinitely.', rounding: '1', min_max: (0, 100), choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.gif', title: 'palettesize', datatype: '<class 'str'>', default: '256', info: 'The number of colors to quantize the image to. Is rounded to the nearest power of two.', rounding: 'None', min_max: None, choices: ['2', '4', '8', '16', '32', '64', '128', '256'], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.gif', title: 'subrectangles', datatype: '<class 'bool'>', default: 'False', info: 'If True, will try and optimize the GIF by storing only the rectangular parts of each frame that change with respect to the previous.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Added defaults: writer.gif
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Adding defaults: (filename: opencv_defaults.py, module_path: plugins.convert.writer, plugin_type: writer
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Importing defaults module: plugins.convert.writer.opencv_defaults
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_section                    DEBUG    Add section: (title: 'writer.opencv', info: 'Options for outputting converted frames to a series of images using OpenCV\nOpenCV can be faster than other image writers, but lacks some configuration options and formats.')
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.opencv', title: 'format', datatype: '<class 'str'>', default: 'png', info: 'Image format to use:\n	 bmp: Windows bitmap\n	 jpg: JPEG format\n	 jp2: JPEG 2000 format\n	 png: Portable Network Graphics\n	 ppm: Portable Pixmap Format', rounding: 'None', min_max: None, choices: ['bmp', 'jpg', 'jp2', 'png', 'ppm'], gui_radio: True, fixed: True, group: None)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.opencv', title: 'draw_transparent', datatype: '<class 'bool'>', default: 'False', info: 'Place the swapped face on a transparent layer rather than the original frame.\nNB: This is only compatible with images saved in png format. If an incompatible format is selected then the image will be saved as a png.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.opencv', title: 'jpg_quality', datatype: '<class 'int'>', default: '75', info: '[jpg only] Set the jpg quality. 1 is worst 95 is best. Higher quality leads to larger file sizes.', rounding: '1', min_max: (1, 95), choices: [], gui_radio: False, fixed: True, group: compression)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.opencv', title: 'png_compress_level', datatype: '<class 'int'>', default: '3', info: '[png only] ZLIB compression level, 1 gives best speed, 9 gives best compression, 0 gives no compression at all.', rounding: '1', min_max: (0, 9), choices: [], gui_radio: False, fixed: True, group: compression)
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Added defaults: writer.opencv
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Adding defaults: (filename: pillow_defaults.py, module_path: plugins.convert.writer, plugin_type: writer
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Importing defaults module: plugins.convert.writer.pillow_defaults
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_section                    DEBUG    Add section: (title: 'writer.pillow', info: 'Options for outputting converted frames to a series of images using Pillow\nPillow is more feature rich than OpenCV but can be slower.')
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.pillow', title: 'format', datatype: '<class 'str'>', default: 'png', info: 'Image format to use:\n	 bmp: Windows bitmap\n	 gif: Graphics Interchange Format (NB: Not animated)\n	 jpg: JPEG format\n	 jp2: JPEG 2000 format\n	 png: Portable Network Graphics\n	 ppm: Portable Pixmap Format\n	 tif: Tag Image File Format', rounding: 'None', min_max: None, choices: ['bmp', 'gif', 'jpg', 'jp2', 'png', 'ppm', 'tif'], gui_radio: True, fixed: True, group: None)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.pillow', title: 'draw_transparent', datatype: '<class 'bool'>', default: 'False', info: 'Place the swapped face on a transparent layer rather than the original frame.\nNB: This is only compatible with images saved in png or tif format. If an incompatible format is selected then the image will be saved as a png.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.pillow', title: 'optimize', datatype: '<class 'bool'>', default: 'False', info: '[gif, jpg and png only] If enabled, indicates that the encoder should make an extra pass over the image in order to select optimal encoder settings.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.pillow', title: 'gif_interlace', datatype: '<class 'bool'>', default: 'True', info: '[gif only] Set whether to save the gif as interlaced or not.', rounding: 'None', min_max: None, choices: [], gui_radio: False, fixed: True, group: settings)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.pillow', title: 'jpg_quality', datatype: '<class 'int'>', default: '75', info: '[jpg only] Set the jpg quality. 1 is worst 95 is best. Higher quality leads to larger file sizes.', rounding: '1', min_max: (1, 95), choices: [], gui_radio: False, fixed: True, group: compression)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.pillow', title: 'png_compress_level', datatype: '<class 'int'>', default: '3', info: '[png only] ZLIB compression level, 1 gives best speed, 9 gives best compression, 0 gives no compression at all. When optimize option is set to True this has no effect (it is set to 9 regardless of a value passed).', rounding: '1', min_max: (0, 9), choices: [], gui_radio: False, fixed: True, group: compression)
01/21/2021 22:58:07 MainProcess     MainThread                     config          add_item                       DEBUG    Add item: (section: 'writer.pillow', title: 'tif_compression', datatype: '<class 'str'>', default: 'tiff_deflate', info: '[tif only] The desired compression method for the file.', rounding: 'None', min_max: None, choices: ['none', 'tiff_ccitt', 'group3', 'group4', 'tiff_jpeg', 'tiff_adobe_deflate', 'tiff_thunderscan', 'tiff_deflate', 'tiff_sgilog', 'tiff_sgilog24', 'tiff_raw_16'], gui_radio: False, fixed: True, group: compression)
01/21/2021 22:58:07 MainProcess     MainThread                     _config         load_module                    DEBUG    Added defaults: writer.pillow
01/21/2021 22:58:07 MainProcess     MainThread                     config          handle_config                  DEBUG    Handling config
01/21/2021 22:58:07 MainProcess     MainThread                     config          check_exists                   DEBUG    Config file exists: 'C:\Users\Duarte\faceswap\config\convert.ini'
01/21/2021 22:58:07 MainProcess     MainThread                     config          load_config                    VERBOSE  Loading config: 'C:\Users\Duarte\faceswap\config\convert.ini'
01/21/2021 22:58:07 MainProcess     MainThread                     config          validate_config                DEBUG    Validating config
01/21/2021 22:58:07 MainProcess     MainThread                     config          check_config_change            DEBUG    Default config has not changed
01/21/2021 22:58:07 MainProcess     MainThread                     config          check_config_choices           DEBUG    Checking config choices
01/21/2021 22:58:07 MainProcess     MainThread                     config          check_config_choices           DEBUG    Checked config choices
01/21/2021 22:58:07 MainProcess     MainThread                     config          validate_config                DEBUG    Validated config
01/21/2021 22:58:07 MainProcess     MainThread                     config          handle_config                  DEBUG    Handled config
01/21/2021 22:58:07 MainProcess     MainThread                     config          __init__                       DEBUG    Initialized: Config
01/21/2021 22:58:07 MainProcess     MainThread                     config          get                            DEBUG    Getting config item: (section: 'scaling.sharpen', option: 'method')
01/21/2021 22:58:07 MainProcess     MainThread                     config          get                            DEBUG    Returning item: (type: <class 'str'>, value: None)
01/21/2021 22:58:07 MainProcess     MainThread                     config          get                            DEBUG    Getting config item: (section: 'scaling.sharpen', option: 'amount')
01/21/2021 22:58:07 MainProcess     MainThread                     config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 150)
01/21/2021 22:58:07 MainProcess     MainThread                     config          get                            DEBUG    Getting config item: (section: 'scaling.sharpen', option: 'radius')
01/21/2021 22:58:07 MainProcess     MainThread                     config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 0.3)
01/21/2021 22:58:07 MainProcess     MainThread                     config          get                            DEBUG    Getting config item: (section: 'scaling.sharpen', option: 'threshold')
01/21/2021 22:58:07 MainProcess     MainThread                     config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 5.0)
01/21/2021 22:58:07 MainProcess     MainThread                     _base           set_config                     DEBUG    Config: {'method': None, 'amount': 150, 'radius': 0.3, 'threshold': 5.0}
01/21/2021 22:58:07 MainProcess     MainThread                     _base           __init__                       DEBUG    config: {'method': None, 'amount': 150, 'radius': 0.3, 'threshold': 5.0}
01/21/2021 22:58:07 MainProcess     MainThread                     _base           __init__                       DEBUG    Initialized Scaling
01/21/2021 22:58:07 MainProcess     MainThread                     convert         _load_plugins                  DEBUG    Loaded plugins: {'box': <plugins.convert.mask.box_blend.Mask object at 0x00000153A60C9D60>, 'mask': <plugins.convert.mask.mask_blend.Mask object at 0x00000153935FFA90>, 'color': <plugins.convert.color.match_hist.Color object at 0x000001539F8EF6D0>, 'seamless': None, 'sharpening': None}
01/21/2021 22:58:07 MainProcess     MainThread                     convert         __init__                       DEBUG    Initialized Converter
01/21/2021 22:58:07 MainProcess     MainThread                     convert         __init__                       DEBUG    Initialized Convert
01/21/2021 22:58:07 MainProcess     MainThread                     convert         process                        DEBUG    Starting Conversion
01/21/2021 22:58:07 MainProcess     MainThread                     convert         _convert_images                DEBUG    Converting images
01/21/2021 22:58:07 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager getting: 'convert_out'
01/21/2021 22:58:07 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager got: 'convert_out'
01/21/2021 22:58:07 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager getting: 'patch'
01/21/2021 22:58:07 MainProcess     MainThread                     queue_manager   get_queue                      DEBUG    QueueManager got: 'patch'
01/21/2021 22:58:07 MainProcess     MainThread                     convert         _pool_processes                DEBUG    1
01/21/2021 22:58:07 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initializing MultiThread: (target: 'patch', thread_count: 1)
01/21/2021 22:58:07 MainProcess     MainThread                     multithreading  __init__                       DEBUG    Initialized MultiThread: 'patch'
01/21/2021 22:58:07 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread(s): 'patch'
01/21/2021 22:58:07 MainProcess     MainThread                     multithreading  start                          DEBUG    Starting thread 1 of 1: 'patch_0'
01/21/2021 22:58:07 MainProcess     patch_0                        convert         process                        DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x000001539F408370>, out_queue: <queue.Queue object at 0x000001539F408340>)
01/21/2021 22:58:07 MainProcess     MainThread                     multithreading  start                          DEBUG    Started all threads 'patch': 1
01/21/2021 22:58:07 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:08 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:09 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:10 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:11 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:12 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:13 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:14 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:15 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:16 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:17 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:18 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:19 MainProcess     MainThread                     multithreading  completed                      DEBUG    False
01/21/2021 22:58:20 MainProcess     _predict_faces_0               multithreading  run                            DEBUG    Error in thread (_predict_faces_0):  OOM when allocating tensor with shape[16,126,128,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n	 [[{{node dfl_sae_df_inference/decoder_b/add_10/add-1-TransposeNHWCToNCHW-LayoutOptimizer}}]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n [Op:__inference_predict_function_2372]\n\nFunction call stack:\npredict_function\n
01/21/2021 22:58:20 MainProcess     MainThread                     multithreading  check_and_raise_error          DEBUG    Thread error caught: [(<class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, ResourceExhaustedError(), <traceback object at 0x00000153BE2B6580>)]
Traceback (most recent call last):
  File "C:\Users\Duarte\faceswap\lib\cli\launcher.py", line 182, in execute_script
    process.process()
  File "C:\Users\Duarte\faceswap\scripts\convert.py", line 161, in process
    self._convert_images()
  File "C:\Users\Duarte\faceswap\scripts\convert.py", line 188, in _convert_images
    self._check_thread_error()
  File "C:\Users\Duarte\faceswap\scripts\convert.py", line 208, in _check_thread_error
    thread.check_and_raise_error()
  File "C:\Users\Duarte\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\Duarte\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Duarte\faceswap\scripts\convert.py", line 875, in _predict_faces
    predicted = self._predict(feed_faces, batch_size)
  File "C:\Users\Duarte\faceswap\scripts\convert.py", line 968, in _predict
    predicted = self._model.model.predict(feed, batch_size=batch_size)
  File "C:\Users\Duarte\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py", line 130, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\Duarte\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1599, in predict
    tmp_batch_outputs = predict_function(iterator)
  File "C:\Users\Duarte\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\Duarte\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\def_function.py", line 846, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "C:\Users\Duarte\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 1843, in _filtered_call
    return self._call_flat(
  File "C:\Users\Duarte\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 1923, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\Duarte\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\function.py", line 545, in call
    outputs = execute.execute(
  File "C:\Users\Duarte\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError:  OOM when allocating tensor with shape[16,126,128,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{{node dfl_sae_df_inference/decoder_b/add_10/add-1-TransposeNHWCToNCHW-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
 [Op:__inference_predict_function_2372]

Function call stack:
predict_function


============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         15bbc95 Update Dockerfile (#1112)
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce RTX 2060
gpu_devices_active:  GPU_0
gpu_driver:          461.09
gpu_vram:            GPU_0: 6144MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.19041-SP0
os_release:          10
py_command:          C:\Users\Duarte\faceswap\faceswap.py convert -i D:/zarajordan/video/5ef8af4c720d92866e58b_source.mp4 -o C:/Users/Duarte/Desktop/DEEOFAKES/New folder -al C:/Users/Duarte/Desktop/DEEOFAKES/zaradata/5ef8af4c720d92866e58b_source_alignments.fsa -ref D:/zarajordan/video/5ef8af4c720d92866e58b_source.mp4 -m C:/Users/Duarte/Desktop/DEEOFAKES/SAE -c match-hist -M extended -w ffmpeg -osc 100 -l 0.4 -j 1 -L INFO -gui
py_conda_version:    conda 4.9.2
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           12
sys_processor:       AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD
sys_ram:             Total: 16333MB, Available: 10233MB, Used: 6100MB, Free: 10233MB

=============== Pip Packages ===============
absl-py @ file:///tmp/build/80754af9/absl-py_1607439979954/work
aiohttp @ file:///C:/ci/aiohttp_1607109697839/work
astunparse==1.6.3
async-timeout==3.0.1
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
blinker==1.4
brotlipy==0.7.0
cachetools @ file:///tmp/build/80754af9/cachetools_1607706694405/work
certifi==2020.12.5
cffi @ file:///C:/ci/cffi_1606255208697/work
chardet @ file:///C:/ci/chardet_1605303225733/work
click==7.1.2
cryptography==2.9.2
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast @ file:///tmp/build/80754af9/gast_1597433534803/work
google-auth @ file:///tmp/build/80754af9/google-auth_1607969906642/work
google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1603929124518/work
google-pasta==0.2.0
grpcio @ file:///C:/ci/grpcio_1597406462198/work
h5py==2.10.0
idna @ file:///tmp/build/80754af9/idna_1593446292537/work
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1609799311556/work
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1602276842396/work
joblib @ file:///tmp/build/80754af9/joblib_1607970656719/work
Keras-Applications @ file:///tmp/build/80754af9/keras-applications_1594366238411/work
Keras-Preprocessing==1.1.0
kiwisolver @ file:///C:/ci/kiwisolver_1604014703538/work
Markdown @ file:///C:/ci/markdown_1605111189761/work
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
multidict @ file:///C:/ci/multidict_1600456481656/work
numpy @ file:///C:/ci/numpy_and_numpy_base_1603466732592/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.5.1.48
opt-einsum==3.1.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1609786840597/work
protobuf==3.13.0
psutil @ file:///C:/ci/psutil_1598370330503/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
PyJWT @ file:///C:/ci/pyjwt_1608658192037/work
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing==2.4.7
pyreadline==2.1
PySocks @ file:///C:/ci/pysocks_1605287845585/work
python-dateutil==2.8.1
pywin32==227
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-oauthlib==1.3.0
rsa @ file:///tmp/build/80754af9/rsa_1610483308194/work
scikit-learn @ file:///C:/ci/scikit-learn_1598377018496/work
scipy @ file:///C:/ci/scipy_1604596260408/work
sip==4.19.13
six @ file:///C:/ci/six_1605187374963/work
tensorboard @ file:///home/builder/ktietz/conda/conda-bld/tensorboard_1604313476433/work/tmp_pip_dir
tensorboard-plugin-wit==1.6.0
tensorflow==2.3.0
tensorflow-estimator @ file:///tmp/build/80754af9/tensorflow-estimator_1599136169057/work/whl_temp/tensorflow_estimator-2.3.0-py2.py3-none-any.whl
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado @ file:///C:/ci/tornado_1606942392901/work
tqdm @ file:///tmp/build/80754af9/tqdm_1609788246169/work
typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1598376058250/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1606938623459/work
Werkzeug==1.0.1
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
wincertstore==0.2
wrapt==1.12.1
yarl @ file:///C:/ci/yarl_1598045274898/work
zipp @ file:///tmp/build/80754af9/zipp_1604001098328/work

============== Conda Packages ==============
# packages in environment at C:\Users\Duarte\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.3.0                       gpu  
absl-py 0.11.0 pyhd3eb1b0_1
aiohttp 3.7.3 py38h2bbff1b_1
astunparse 1.6.3 py_0
async-timeout 3.0.1 py38_0
attrs 20.3.0 pyhd3eb1b0_0
blas 1.0 mkl
blinker 1.4 py38_0
brotlipy 0.7.0 py38h2bbff1b_1003
ca-certificates 2020.12.8 haa95532_0
cachetools 4.2.0 pyhd3eb1b0_0
certifi 2020.12.5 py38haa95532_0
cffi 1.14.4 py38hcd4344a_0
chardet 3.0.4 py38haa95532_1003
click 7.1.2 py_0
cryptography 2.9.2 py38h7a1dbc1_0
cudatoolkit 10.1.243 h74a9793_0
cudnn 7.6.5 cuda10.1_0
cycler 0.10.0 py38_0
fastcluster 1.1.26 py38h251f6bf_2 conda-forge ffmpeg 4.3.1 ha925a31_0 conda-forge ffmpy 0.2.3 pypi_0 pypi freetype 2.10.4 hd328e21_0
gast 0.4.0 py_0
git 2.23.0 h6bb4b03_0
google-auth 1.24.0 pyhd3eb1b0_0
google-auth-oauthlib 0.4.2 pyhd3eb1b0_2
google-pasta 0.2.0 py_0
grpcio 1.31.0 py38he7da953_0
h5py 2.10.0 py38h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 py_0
imageio 2.9.0 py_0
imageio-ffmpeg 0.4.3 pyhd8ed1ab_0 conda-forge importlib-metadata 2.0.0 py_1
intel-openmp 2020.2 254
joblib 1.0.0 pyhd3eb1b0_0
jpeg 9b hb83a4c4_2
keras-applications 1.0.8 py_1
keras-preprocessing 1.1.0 py_1
kiwisolver 1.3.0 py38hd77b12b_0
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.13.0.1 h200bbdf_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 hf4a77e7_3
markdown 3.3.3 py38haa95532_0
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38h64f37c6_0
mkl 2020.2 256
mkl-service 2.3.0 py38h196d8e1_0
mkl_fft 1.2.0 py38h45dec08_0
mkl_random 1.1.1 py38h47e9c7a_0
multidict 4.7.6 py38he774522_1
numpy 1.19.2 py38hadc3359_0
numpy-base 1.19.2 py38ha3acd2a_0
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 py_0
olefile 0.46 py_0
opencv-python 4.5.1.48 pypi_0 pypi openssl 1.1.1i h2bbff1b_0
opt_einsum 3.1.0 py_0
pathlib 1.0.1 py_1
pillow 8.1.0 py38h4fa10fc_0
pip 20.3.3 py38haa95532_0
protobuf 3.13.0.1 py38ha925a31_1
psutil 5.7.2 py38he774522_0
pyasn1 0.4.8 py_0
pyasn1-modules 0.2.8 py_0
pycparser 2.20 py_2
pyjwt 2.0.0 py38haa95532_0
pyopenssl 20.0.1 pyhd3eb1b0_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py38ha925a31_4
pyreadline 2.1 py38_1
pysocks 1.7.1 py38haa95532_0
python 3.8.5 h5fd99cc_1
python-dateutil 2.8.1 py_0
python_abi 3.8 1_cp38 conda-forge pywin32 227 py38he774522_1
qt 5.9.7 vc14h73c81de_0
requests 2.25.1 pyhd3eb1b0_0
requests-oauthlib 1.3.0 py_0
rsa 4.7 pyhd3eb1b0_1
scikit-learn 0.23.2 py38h47e9c7a_0
scipy 1.5.2 py38h14eb087_0
setuptools 51.1.2 py38haa95532_4
sip 4.19.13 py38ha925a31_0
six 1.15.0 py38haa95532_0
sqlite 3.33.0 h2a8f88b_0
tensorboard 2.3.0 pyh4dce500_0
tensorboard-plugin-wit 1.6.0 py_0
tensorflow 2.3.0 mkl_py38h1fcfbd6_0
tensorflow-base 2.3.0 gpu_py38h7339f5a_0
tensorflow-estimator 2.3.0 pyheb71bc4_0
tensorflow-gpu 2.3.0 he13fc11_0
termcolor 1.1.0 py38_1
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
tornado 6.1 py38h2bbff1b_0
tqdm 4.55.1 pyhd3eb1b0_0
typing-extensions 3.7.4.3 0
typing_extensions 3.7.4.3 py_0
urllib3 1.26.2 pyhd3eb1b0_0
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 1.0.1 py_0
wheel 0.36.2 pyhd3eb1b0_0
win_inet_pton 1.1.0 py38haa95532_0
wincertstore 0.2 py38_0
wrapt 1.12.1 py38he774522_1
xz 5.2.5 h62dcd97_0
yarl 1.5.1 py38he774522_0
zipp 3.4.0 pyhd3eb1b0_0
zlib 1.2.11 h62dcd97_4
zstd 1.4.5 h04227a9_0 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: none amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: True [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 70 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] centering: face coverage: 68.75 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 reflect_padding: False allow_growth: True mixed_precision: False convert_batchsize: 16 [global.loss] loss_function: ssim mask_loss_function: mse l2_reg_term: 100 eye_multiplier: 3 mouth_multiplier: 2 penalized_mask_loss: True mask_type: extended mask_blur_kernel: 3 mask_threshold: 4 learn_mask: True [model.dfaker] output_size: 128 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 disable_warp: False color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4

User avatar
torzdf
Posts: 1495
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 51 times
Been thanked: 287 times

Re: CRITICAL An unexpected crash has occurred.

Post by torzdf »

Yes, this is out of memory.

You should be able to resolve your issue by going into Settings > Train Settings and lowering convert batchsize

My word is final


Locked