Made it about 15000 its in, then crashing every few hundred or so. File in use error?

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by unkempt »

So I'm getting a "file in use permission error" on a file that faceswap is using. Is it getting in it's own way? Note that I've had this problem before, but it was late in the training process so I switched to no warp and go no more errors. Now I'm doing a different set and it's way too early in the process to turn off warp, but it stops every few hundred or so.

Code: Select all

'R': Refresh Preview. 'M': Toggle Mask. 'F': Toggle Screen Fit-Actual Size. 'ENTER': Save and Quit)
05/25/2023 10:51:24 MainProcess     _training                      train           _show                          DEBUG    Generating preview for GUI
05/25/2023 10:51:24 MainProcess     _training                      train           _show                          DEBUG    Generated preview for GUI: 'D:\Nuff\faceswap2\lib\gui\.cache\preview\.gui_training_preview.png'
05/25/2023 10:51:24 MainProcess     _training                      train           _show                          DEBUG    Updated preview: (name: Training - 'S': Save Now. 'R': Refresh Preview. 'M': Toggle Mask. 'F': Toggle Screen Fit-Actual Size. 'ENTER': Save and Quit)
05/25/2023 10:51:24 MainProcess     _training                      train           _run_training_cycle            INFO     [Preview Updated]
05/25/2023 10:52:30 MainProcess     _run_0                         multithreading  run                            DEBUG    Error in thread (_run_0): [WinError 32] The process cannot access the file because it is being used by another process: b'C:\\Users\\J\\AppData\\Local\\Temp\\scipy-5w_cirki'
05/25/2023 10:52:30 MainProcess     _training                      multithreading  check_and_raise_error          DEBUG    Thread error caught: [(<class 'PermissionError'>, PermissionError(13, 'The process cannot access the file because it is being used by another process'), <traceback object at 0x000001DBA71D9D40>)]
05/25/2023 10:52:30 MainProcess     _training                      multithreading  run                            DEBUG    Error in thread (_training): [WinError 32] The process cannot access the file because it is being used by another process: b'C:\\Users\\J\\AppData\\Local\\Temp\\scipy-5w_cirki'
05/25/2023 10:52:31 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
05/25/2023 10:52:31 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
05/25/2023 10:52:31 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
05/25/2023 10:52:31 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
05/25/2023 10:52:31 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
05/25/2023 10:52:31 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training'
05/25/2023 10:52:31 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training'
Traceback (most recent call last):
  File "D:\Nuff\faceswap2\lib\cli\launcher.py", line 230, in execute_script
    process.process()
  File "D:\Nuff\faceswap2\scripts\train.py", line 213, in process
    self._end_thread(thread, err)
  File "D:\Nuff\faceswap2\scripts\train.py", line 253, in _end_thread
    thread.join()
  File "D:\Nuff\faceswap2\lib\multithreading.py", line 220, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "D:\Nuff\faceswap2\lib\multithreading.py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Nuff\faceswap2\scripts\train.py", line 275, in _training
    raise err
  File "D:\Nuff\faceswap2\scripts\train.py", line 265, in _training
    self._run_training_cycle(model, trainer)
  File "D:\Nuff\faceswap2\scripts\train.py", line 353, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "D:\Nuff\faceswap2\plugins\train\trainer\_base.py", line 217, in train_one_step
    model_inputs, model_targets = self._feeder.get_batch()
  File "D:\Nuff\faceswap2\plugins\train\trainer\_base.py", line 485, in get_batch
    side_feed, side_targets = next(self._feeds[side])
  File "D:\Nuff\faceswap2\lib\multithreading.py", line 292, in iterator
    self.check_and_raise_error()
  File "D:\Nuff\faceswap2\lib\multithreading.py", line 169, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "D:\Nuff\faceswap2\lib\multithreading.py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Nuff\faceswap2\lib\multithreading.py", line 275, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "D:\Nuff\faceswap2\lib\training\generator.py", line 222, in _minibatch
    retval = self._process_batch(img_paths)
  File "D:\Nuff\faceswap2\lib\training\generator.py", line 339, in _process_batch
    feed, targets = self.process_batch(filenames, raw_faces, detected_faces, batch)
  File "D:\Nuff\faceswap2\lib\training\generator.py", line 532, in process_batch
    warped = batch[..., :3] if self._no_warp else self._processing.warp(
  File "D:\Nuff\faceswap2\lib\training\augmentation.py", line 314, in warp
    return self._random_warp_landmarks(batch, **kwargs)
  File "D:\Nuff\faceswap2\lib\training\augmentation.py", line 389, in _random_warp_landmarks
    grid_z = np.array([griddata(dst, src, (grids[0], grids[1]), method="linear")
  File "D:\Nuff\faceswap2\lib\training\augmentation.py", line 389, in <listcomp>
    grid_z = np.array([griddata(dst, src, (grids[0], grids[1]), method="linear")
  File "C:\Users\J\MiniConda3\envs\faceswap\lib\site-packages\scipy\interpolate\_ndgriddata.py", line 260, in griddata
    ip = LinearNDInterpolator(points, values, fill_value=fill_value,
  File "interpnd.pyx", line 280, in scipy.interpolate.interpnd.LinearNDInterpolator.__init__
  File "_qhull.pyx", line 1848, in scipy.spatial._qhull.Delaunay.__init__
  File "_qhull.pyx", line 1560, in scipy.spatial._qhull._QhullUser.__init__
  File "_qhull.pyx", line 405, in scipy.spatial._qhull._Qhull.close
  File "messagestream.pyx", line 91, in scipy._lib.messagestream.MessageStream.close
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: b'C:\\Users\\J\\AppData\\Local\\Temp\\scipy-5w_cirki'

============ System Information ============
backend:             nvidia
encoding:            cp1252
git_branch:          master
git_commits:         e2ad3e2 add ru locale (#1311)
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce RTX 3070 Laptop GPU
gpu_devices_active:  GPU_0
gpu_driver:          512.36
gpu_vram:            GPU_0: 8192MB (427MB free)
os_machine:          AMD64
os_platform:         Windows-10-10.0.22621-SP0
os_release:          10
py_command:          D:\Nuff\faceswap2\faceswap.py train -A D:/Nuff/Virtual Machines/VM-Airlock/smm12/home/Home Faces all/subset -B D:/Nuff/Virtual Machines/VM-Airlock/smm12/H/hfaces_a+m -m D:/Nuff/Virtual Machines/VM-Airlock/smm12/home/Models/Realface/Realface -t realface -bs 10 -it 1000000 -D default -s 250 -ss 25000 -wl -L INFO -gui
py_conda_version:    conda 23.3.1
py_implementation:   CPython
py_version:          3.9.16
py_virtual_env:      True
sys_cores:           16
sys_processor:       AMD64 Family 25 Model 80 Stepping 0, AuthenticAMD
sys_ram:             Total: 32620MB, Available: 18126MB, Used: 14493MB, Free: 18126MB

=============== Pip Packages ===============
absl-py @ file:///C:/b/abs_5babsu7y5x/croot/absl-py_1666362945682/work
astunparse==1.6.3
cachetools==5.3.0
certifi==2023.5.7
charset-normalizer==3.1.0
cloudpickle @ file:///C:/b/abs_3796yxesic/croot/cloudpickle_1683040098851/work
colorama @ file:///C:/b/abs_a9ozq0l032/croot/colorama_1672387194846/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
dm-tree @ file:///C:/b/abs_10z0iy5knj/croot/dm-tree_1671027465819/work
fastcluster @ file:///D:/bld/fastcluster_1649783471014/work
ffmpy==0.3.0
flatbuffers==23.5.8
fonttools==4.25.0
gast==0.4.0
google-auth==2.17.3
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.54.0
h5py==3.8.0
idna==3.4
imageio @ file:///C:/b/abs_27kq2gy1us/croot/imageio_1677879918708/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1673483481485/work
importlib-metadata==6.6.0
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1663332044897/work
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/b/abs_88mdhvtahm/croot/kiwisolver_1672387921783/work
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.2
matplotlib @ file:///C:/b/abs_ae02atcfur/croot/matplotlib-suite_1667356722968/work
mkl-fft==1.3.6
mkl-random @ file:///C:/Users/dev-admin/mkl/mkl_random_1682977971003/work
mkl-service==2.4.0
munkres==1.1.4
numexpr @ file:///C:/b/abs_afm0oewmmt/croot/numexpr_1683221839116/work
numpy @ file:///C:/Users/dev-admin/mkl/numpy_and_numpy_base_1682981337988/work
nvidia-ml-py==11.525.112
oauthlib==3.2.2
opencv-python==4.7.0.72
opt-einsum==3.3.0
packaging @ file:///C:/b/abs_ed_kb9w6g4/croot/packaging_1678965418855/work
Pillow==9.4.0
ply==3.11
protobuf==3.19.6
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
pyasn1==0.5.0
pyasn1-modules==0.3.0
pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work
PyQt5==5.15.7
PyQt5-sip @ file:///C:/Windows/Temp/abs_d7gmd2jg8i/croots/recipe/pyqt-split_1659273064801/work/pyqt_sip
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==305.1
pywinpty @ file:///C:/ci_310/pywinpty_1644230983541/work/target/wheels/pywinpty-2.0.2-cp39-none-win_amd64.whl
requests==2.30.0
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn @ file:///C:/b/abs_a0emltxu06/croot/scikit-learn_1680198772090/work
scipy==1.9.3
sip @ file:///C:/Windows/Temp/abs_b8fxd17m2u/croots/recipe/sip_1659012372737/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.10.0
tensorflow-gpu==2.10.1
tensorflow-io-gcs-filesystem==0.31.0
tensorflow-probability @ file:///tmp/build/80754af9/tensorflow-probability_1633017132682/work
termcolor==2.3.0
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
tornado @ file:///C:/ci/tornado_1662458743919/work
tqdm @ file:///C:/b/abs_f76j9hg7pv/croot/tqdm_1679561871187/work
typing_extensions @ file:///C:/b/abs_a1bb332wcs/croot/typing_extensions_1681939523095/work
urllib3==2.0.2
Werkzeug==2.3.4
wrapt==1.15.0
zipp==3.15.0

============== Conda Packages ==============
# packages in environment at C:\Users\J\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   1.3.0            py39haa95532_0  
astunparse 1.6.3 pypi_0 pypi blas 1.0 mkl
brotli 1.0.9 h2bbff1b_7
brotli-bin 1.0.9 h2bbff1b_7
ca-certificates 2023.5.7 h56e8100_0 conda-forge cachetools 5.3.0 pypi_0 pypi certifi 2023.5.7 pypi_0 pypi charset-normalizer 3.1.0 pypi_0 pypi cloudpickle 2.2.1 py39haa95532_0
colorama 0.4.6 py39haa95532_0
cudatoolkit 11.2.2 h933977f_10 conda-forge cudnn 8.1.0.77 h3e0f4f4_0 conda-forge cycler 0.11.0 pyhd3eb1b0_0
decorator 5.1.1 pyhd3eb1b0_0
dm-tree 0.1.7 py39hd77b12b_1
fastcluster 1.2.6 py39h2e25243_1 conda-forge ffmpeg 4.3.1 ha925a31_0 conda-forge ffmpy 0.3.0 pypi_0 pypi flatbuffers 23.5.8 pypi_0 pypi fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.12.1 ha860e81_0
gast 0.4.0 pypi_0 pypi giflib 5.2.1 h8cc25b3_3
git 2.34.1 haa95532_0
glib 2.69.1 h5dc1a3c_2
google-auth 2.17.3 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.54.0 pypi_0 pypi gst-plugins-base 1.18.5 h9e645db_0
gstreamer 1.18.5 hd78058f_0
h5py 3.8.0 pypi_0 pypi icc_rt 2022.1.0 h6049295_2
icu 58.2 ha925a31_3
idna 3.4 pypi_0 pypi imageio 2.26.0 py39haa95532_0
imageio-ffmpeg 0.4.8 pyhd8ed1ab_0 conda-forge importlib-metadata 6.6.0 pypi_0 pypi intel-openmp 2023.1.0 h59b6b97_46319
joblib 1.2.0 pyhd8ed1ab_0 conda-forge jpeg 9e h2bbff1b_1
keras 2.10.0 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.4.4 py39hd77b12b_0
krb5 1.19.4 h5b6d351_0
lerc 3.0 hd77b12b_0
libbrotlicommon 1.0.9 h2bbff1b_7
libbrotlidec 1.0.9 h2bbff1b_7
libbrotlienc 1.0.9 h2bbff1b_7
libclang 16.0.0 pypi_0 pypi libclang13 14.0.6 default_h8e68704_1
libdeflate 1.17 h2bbff1b_0
libffi 3.4.2 hd77b12b_6
libiconv 1.16 h2bbff1b_2
libogg 1.3.5 h2bbff1b_1
libpng 1.6.39 h8cc25b3_0
libtiff 4.5.0 h6c2663c_2
libvorbis 1.3.7 he774522_0
libwebp 1.2.4 hbc33d0d_1
libwebp-base 1.2.4 h2bbff1b_1
libxml2 2.10.3 h0ad7f3c_0
libxslt 1.1.37 h2bbff1b_0
lz4-c 1.9.4 h2bbff1b_0
markdown 3.4.3 pypi_0 pypi markupsafe 2.1.2 pypi_0 pypi matplotlib 3.5.3 py39haa95532_0
matplotlib-base 3.5.3 py39hd77b12b_0
mkl 2023.1.0 h8bd8f75_46356
mkl-service 2.4.0 py39h2bbff1b_1
mkl_fft 1.3.6 py39hf11a4ad_1
mkl_random 1.2.2 py39hf11a4ad_1
munkres 1.1.4 py_0
numexpr 2.8.4 py39h7b80656_1
numpy 1.23.5 py39h6917f2d_1
numpy-base 1.23.5 py39h46c4fa8_1
nvidia-ml-py 11.525.112 pypi_0 pypi oauthlib 3.2.2 pypi_0 pypi opencv-python 4.7.0.72 pypi_0 pypi openssl 1.1.1t h2bbff1b_0
opt-einsum 3.3.0 pypi_0 pypi packaging 23.0 py39haa95532_0
pcre 8.45 hd77b12b_0
pillow 9.4.0 py39hd77b12b_0
pip 23.0.1 py39haa95532_0
ply 3.11 py39haa95532_0
protobuf 3.19.6 pypi_0 pypi psutil 5.9.0 py39h2bbff1b_0
pyasn1 0.5.0 pypi_0 pypi pyasn1-modules 0.3.0 pypi_0 pypi pyparsing 3.0.9 py39haa95532_0
pyqt 5.15.7 py39hd77b12b_0
pyqt5-sip 12.11.0 py39hd77b12b_0
python 3.9.16 h6244533_2
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.9 2_cp39 conda-forge pywin32 305 py39h2bbff1b_0
pywinpty 2.0.2 py39h5da7b33_0
qt-main 5.15.2 he8e5bd7_8
qt-webengine 5.15.9 hb9a9bb5_5
qtwebkit 5.212 h2bbfb41_5
requests 2.30.0 pypi_0 pypi requests-oauthlib 1.3.1 pypi_0 pypi rsa 4.9 pypi_0 pypi scikit-learn 1.2.2 py39hd77b12b_0
scipy 1.9.3 py39hdcfc7df_2
setuptools 66.0.0 py39haa95532_0
sip 6.6.2 py39hd77b12b_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.41.2 h2bbff1b_0
tbb 2021.8.0 h59b6b97_0
tensorboard 2.10.1 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow-estimator 2.10.0 pypi_0 pypi tensorflow-gpu 2.10.1 pypi_0 pypi tensorflow-io-gcs-filesystem 0.31.0 pypi_0 pypi tensorflow-probability 0.14.0 pyhd3eb1b0_0
termcolor 2.3.0 pypi_0 pypi threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tk 8.6.12 h2bbff1b_0
toml 0.10.2 pyhd3eb1b0_0
tornado 6.2 py39h2bbff1b_0
tqdm 4.65.0 py39hd4e2768_0
typing-extensions 4.5.0 py39haa95532_0
typing_extensions 4.5.0 py39haa95532_0
tzdata 2023c h04d1e81_0
urllib3 2.0.2 pypi_0 pypi vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 2.3.4 pypi_0 pypi wheel 0.38.4 py39haa95532_0
winpty 0.4.3 4
wrapt 1.15.0 pypi_0 pypi xz 5.4.2 h8cc25b3_0
zipp 3.15.0 pypi_0 pypi zlib 1.2.13 h8cc25b3_0
zstd 1.5.5 hd43e919_0 =============== State File ================= { "name": "realface", "sessions": { "1": { "timestamp": 1685026455.9237044, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 12, "iterations": 10750, "config": { "learning_rate": 5e-05, "epsilon_exponent": -7, "autoclip": false, "allow_growth": false, "mixed_precision": false, "nan_protection": true, "convert_batchsize": 16, "loss_function": "ssim", "loss_function_2": "mse", "loss_weight_2": 100, "loss_function_3": null, "loss_weight_3": 0, "loss_function_4": null, "loss_weight_4": 0, "mask_loss_function": "mse", "eye_multiplier": 3, "mouth_multiplier": 2 } }, "2": { "timestamp": 1685031155.8298178, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 12, "iterations": 4250, "config": { "learning_rate": 5e-05, "epsilon_exponent": -7, "autoclip": false, "allow_growth": false, "mixed_precision": false, "nan_protection": true, "convert_batchsize": 16, "loss_function": "ssim", "loss_function_2": "mse", "loss_weight_2": 100, "loss_function_3": null, "loss_weight_3": 0, "loss_function_4": null, "loss_weight_4": 0, "mask_loss_function": "mse", "eye_multiplier": 3, "mouth_multiplier": 2 } }, "3": { "timestamp": 1685032993.1724362, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 12, "iterations": 1, "config": { "learning_rate": 5e-05, "epsilon_exponent": -7, "autoclip": false, "allow_growth": false, "mixed_precision": false, "nan_protection": true, "convert_batchsize": 16, "loss_function": "ssim", "loss_function_2": "mse", "loss_weight_2": 100, "loss_function_3": null, "loss_weight_3": 0, "loss_function_4": null, "loss_weight_4": 0, "mask_loss_function": "mse", "eye_multiplier": 3, "mouth_multiplier": 2 } }, "4": { "timestamp": 1685033025.707565, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 10, "iterations": 250, "config": { "learning_rate": 5e-05, "epsilon_exponent": -7, "autoclip": false, "allow_growth": false, "mixed_precision": false, "nan_protection": true, "convert_batchsize": 16, "loss_function": "ssim", "loss_function_2": "mse", "loss_weight_2": 100, "loss_function_3": null, "loss_weight_3": 0, "loss_function_4": null, "loss_weight_4": 0, "mask_loss_function": "mse", "eye_multiplier": 3, "mouth_multiplier": 2 } }, "5": { "timestamp": 1685036423.2512548, "no_logs": false, "loss_names": [ "total", "face_a", "face_b" ], "batchsize": 10, "iterations": 2250, "config": { "learning_rate": 5e-05, "epsilon_exponent": -7, "autoclip": false, "allow_growth": false, "mixed_precision": false, "nan_protection": true, "convert_batchsize": 16, "loss_function": "ssim", "loss_function_2": "mse", "loss_weight_2": 100, "loss_function_3": null, "loss_weight_3": 0, "loss_function_4": null, "loss_weight_4": 0, "mask_loss_function": "mse", "eye_multiplier": 3, "mouth_multiplier": 2 } } }, "lowest_avg_loss": { "a": 0.029578834772109985, "b": 0.033040016889572144 }, "iterations": 17501, "mixed_precision_layers": [ "conv_128_0_conv2d", "leaky_re_lu", "residual_128_0_conv2d_0", "residual_128_0_leakyrelu_1", "residual_128_0_conv2d_1", "add", "residual_128_0_leakyrelu_3", "residual_128_1_conv2d_0", "residual_128_1_leakyrelu_1", "residual_128_1_conv2d_1", "add_1", "residual_128_1_leakyrelu_3", "conv_256_0_conv2d", "leaky_re_lu_1", "residual_256_0_conv2d_0", "residual_256_0_leakyrelu_1", "residual_256_0_conv2d_1", "add_2", "residual_256_0_leakyrelu_3", "residual_256_1_conv2d_0", "residual_256_1_leakyrelu_1", "residual_256_1_conv2d_1", "add_3", "residual_256_1_leakyrelu_3", "conv_512_0_conv2d", "leaky_re_lu_2", "residual_512_0_conv2d_0", "residual_512_0_leakyrelu_1", "residual_512_0_conv2d_1", "add_4", "residual_512_0_leakyrelu_3", "residual_512_1_conv2d_0", "residual_512_1_leakyrelu_1", "residual_512_1_conv2d_1", "add_5", "residual_512_1_leakyrelu_3", "conv_1024_0_conv2d", "conv_1024_0_leakyrelu", "flatten", "dense", "dense_1", "reshape", "upscale_682_0_conv2d_conv2d", "upscale_682_0_pixelshuffler", "leaky_re_lu_3", "residual_682_0_conv2d_0", "residual_682_0_leakyrelu_1", "residual_682_0_conv2d_1", "add_6", "residual_682_0_leakyrelu_3", "upscale_341_0_conv2d_conv2d", "upscale_341_0_conv2d_leakyrelu", "upscale_341_0_pixelshuffler", "upscale_170_0_conv2d_conv2d", "upscale_170_0_conv2d_leakyrelu", "upscale_170_0_pixelshuffler", "upscale_85_0_conv2d_conv2d", "upscale_85_0_conv2d_leakyrelu", "upscale_85_0_pixelshuffler", "upscale_42_0_conv2d_conv2d", "upscale_42_0_conv2d_leakyrelu", "upscale_42_0_pixelshuffler", "face_out_a_conv2d", "flatten_1", "dense_2", "dense_3", "reshape_1", "upscale_1024_0_conv2d_conv2d", "upscale_1024_0_pixelshuffler", "leaky_re_lu_4", "residual_1024_0_conv2d_0", "residual_1024_0_leakyrelu_1", "residual_1024_0_conv2d_1", "add_7", "residual_1024_0_leakyrelu_3", "upscale_512_0_conv2d_conv2d", "upscale_512_0_pixelshuffler", "leaky_re_lu_5", "residual_512_2_conv2d_0", "residual_512_2_leakyrelu_1", "residual_512_2_conv2d_1", "add_8", "residual_512_2_leakyrelu_3", "residual_512_3_conv2d_0", "residual_512_3_leakyrelu_1", "residual_512_3_conv2d_1", "add_9", "residual_512_3_leakyrelu_3", "upscale_256_0_conv2d_conv2d", "upscale_256_0_pixelshuffler", "leaky_re_lu_6", "residual_256_2_conv2d_0", "residual_256_2_leakyrelu_1", "residual_256_2_conv2d_1", "add_10", "residual_256_2_leakyrelu_3", "residual_256_3_conv2d_0", "residual_256_3_leakyrelu_1", "residual_256_3_conv2d_1", "add_11", "residual_256_3_leakyrelu_3", "upscale_128_0_conv2d_conv2d", "upscale_128_0_pixelshuffler", "leaky_re_lu_7", "residual_128_2_conv2d_0", "residual_128_2_leakyrelu_1", "residual_128_2_conv2d_1", "add_12", "residual_128_2_leakyrelu_3", "residual_128_3_conv2d_0", "residual_128_3_leakyrelu_1", "residual_128_3_conv2d_1", "add_13", "residual_128_3_leakyrelu_3", "upscale_64_0_conv2d_conv2d", "upscale_64_0_conv2d_leakyrelu", "upscale_64_0_pixelshuffler", "face_out_b_conv2d" ], "config": { "centering": "face", "coverage": 100.0, "optimizer": "adam", "learning_rate": 5e-05, "epsilon_exponent": -7, "autoclip": false, "allow_growth": false, "mixed_precision": false, "nan_protection": true, "convert_batchsize": 16, "loss_function": "ssim", "loss_function_2": "mse", "loss_weight_2": 100, "loss_function_3": null, "loss_weight_3": 0, "loss_function_4": null, "loss_weight_4": 0, "mask_loss_function": "mse", "eye_multiplier": 3, "mouth_multiplier": 2, "penalized_mask_loss": true, "mask_type": "bisenet-fp_face", "mask_blur_kernel": 3, "mask_threshold": 4, "learn_mask": false, "input_size": 64, "output_size": 128, "dense_nodes": 1536, "complexity_encoder": 128, "complexity_decoder": 512 } } ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 erosion_top: 0.0 erosion_bottom: 0.0 erosion_left: 0.0 erosion_right: 0.0 [scaling.sharpen] method: none amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto skip_mux: False [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False separate_mask: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False separate_mask: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False aligner_min_scale: 0.07 aligner_max_scale: 2.0 aligner_distance: 22.5 aligner_roll: 45.0 aligner_features: True filter_refeed: True save_filtered: False realign_refeeds: True filter_realign: True [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 scalefactor: 0.709 batch-size: 8 cpu: True threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 [detect.s3fd] confidence: 70 batch-size: 4 [mask.bisenet_fp] batch-size: 8 cpu: False weights: faceswap include_ears: False include_hair: False include_glasses: True [mask.custom] batch-size: 8 centering: face fill: False [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 [recognition.vgg_face2] batch-size: 16 cpu: False --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] centering: face coverage: 100.0 icnr_init: False conv_aware_init: False optimizer: adam learning_rate: 5e-05 epsilon_exponent: -7 autoclip: False reflect_padding: False allow_growth: False mixed_precision: False nan_protection: True convert_batchsize: 16 [global.loss] loss_function: ssim loss_function_2: mse loss_weight_2: 100 loss_function_3: None loss_weight_3: 0 loss_function_4: None loss_weight_4: 0 mask_loss_function: mse eye_multiplier: 3 mouth_multiplier: 2 penalized_mask_loss: True mask_type: bisenet-fp_face mask_blur_kernel: 3 mask_threshold: 4 learn_mask: False [model.dfaker] output_size: 128 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.phaze_a] output_size: 128 shared_fc: None enable_gblock: True split_fc: True split_gblock: False split_decoders: False enc_architecture: fs_original enc_scaling: 7 enc_load_weights: True bottleneck_type: dense bottleneck_norm: None bottleneck_size: 1024 bottleneck_in_encoder: True fc_depth: 1 fc_min_filters: 1024 fc_max_filters: 1024 fc_dimensions: 4 fc_filter_slope: -0.5 fc_dropout: 0.0 fc_upsampler: upsample2d fc_upsamples: 1 fc_upsample_filters: 512 fc_gblock_depth: 3 fc_gblock_min_nodes: 512 fc_gblock_max_nodes: 512 fc_gblock_filter_slope: -0.5 fc_gblock_dropout: 0.0 dec_upscale_method: subpixel dec_upscales_in_fc: 0 dec_norm: None dec_min_filters: 64 dec_max_filters: 512 dec_slope_mode: full dec_filter_slope: -0.45 dec_res_blocks: 1 dec_output_kernel: 5 dec_gaussian: True dec_skip_last_residual: True freeze_layers: keras_encoder load_layers: encoder fs_original_depth: 4 fs_original_min_filters: 128 fs_original_max_filters: 1024 fs_original_use_alt: False mobilenet_width: 1.0 mobilenet_depth: 1 mobilenet_dropout: 0.001 mobilenet_minimalistic: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 mask_opacity: 30 mask_color: #ff0000 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
Last edited by bryanlyon on Fri May 26, 2023 10:25 pm, edited 1 time in total.
User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by unkempt »

I've made it over 3000 its by disabling the "enable preview" option after I started training. I'd really like to keep previews though so that's sub-optimal.

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by unkempt »

I discovered that the preview is still generated, just not shown and I can click "enable preview" at any time to see the current status and then turn it off again. Still not optimal, but it works.

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by bryanlyon »

Probably caused by something reading the image at the same time we try to write it. Maybe thumbnailer?

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by unkempt »

Possibly, but weird it only happens when I have preview enabled and on warp. I've left preview on for nowarp and it was fine.

User avatar
torzdf
Posts: 2701
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 136 times
Been thanked: 632 times

Re: Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by torzdf »

warp/no-warp would have no impact on file permissions, so I suspect that is coincidental

My word is final

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by unkempt »

I'm running another training session and having the same issue. 2700 its and then dead due to file in use error.

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by unkempt »

I pushed the "save interval" out to 1000 on the hopes that the odds of it trying to access a file at a bad time would be lower with less instances and it seems to have helped. I got from 2000 its to 20,000 so far.

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by unkempt »

This has come back and is constantly getting in the way of my training:

Code: Select all

Traceback (most recent call last):
  File "C:\Users\J\faceswap\lib\cli\launcher.py", line 230, in execute_script
    process.process()
  File "C:\Users\J\faceswap\scripts\train.py", line 213, in process
    self._end_thread(thread, err)
  File "C:\Users\J\faceswap\scripts\train.py", line 253, in _end_thread
    thread.join()
  File "C:\Users\J\faceswap\lib\multithreading.py", line 220, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\J\faceswap\lib\multithreading.py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\J\faceswap\scripts\train.py", line 275, in _training
    raise err
  File "C:\Users\J\faceswap\scripts\train.py", line 265, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Users\J\faceswap\scripts\train.py", line 353, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "C:\Users\J\faceswap\plugins\train\trainer\_base.py", line 217, in train_one_step
    model_inputs, model_targets = self._feeder.get_batch()
  File "C:\Users\J\faceswap\plugins\train\trainer\_base.py", line 485, in get_batch
    side_feed, side_targets = next(self._feeds[side])
  File "C:\Users\J\faceswap\lib\multithreading.py", line 292, in iterator
    self.check_and_raise_error()
  File "C:\Users\J\faceswap\lib\multithreading.py", line 169, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\J\faceswap\lib\multithreading.py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\J\faceswap\lib\multithreading.py", line 275, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "C:\Users\J\faceswap\lib\training\generator.py", line 222, in _minibatch
    retval = self._process_batch(img_paths)
  File "C:\Users\J\faceswap\lib\training\generator.py", line 339, in _process_batch
    feed, targets = self.process_batch(filenames, raw_faces, detected_faces, batch)
  File "C:\Users\J\faceswap\lib\training\generator.py", line 532, in process_batch
    warped = batch[..., :3] if self._no_warp else self._processing.warp(
  File "C:\Users\J\faceswap\lib\training\augmentation.py", line 314, in warp
    return self._random_warp_landmarks(batch, **kwargs)
  File "C:\Users\J\faceswap\lib\training\augmentation.py", line 389, in _random_warp_landmarks
    grid_z = np.array([griddata(dst, src, (grids[0], grids[1]), method="linear")
  File "C:\Users\J\faceswap\lib\training\augmentation.py", line 389, in <listcomp>
    grid_z = np.array([griddata(dst, src, (grids[0], grids[1]), method="linear")
  File "C:\Users\J\MiniConda3\envs\faceswap\lib\site-packages\scipy\interpolate\_ndgriddata.py", line 264, in griddata
    ip = LinearNDInterpolator(points, values, fill_value=fill_value,
  File "interpnd.pyx", line 281, in scipy.interpolate.interpnd.LinearNDInterpolator.__init__
  File "_qhull.pyx", line 1843, in scipy.spatial._qhull.Delaunay.__init__
  File "_qhull.pyx", line 1554, in scipy.spatial._qhull._QhullUser.__init__
  File "_qhull.pyx", line 400, in scipy.spatial._qhull._Qhull.close
  File "messagestream.pyx", line 91, in scipy._lib.messagestream.MessageStream.close
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: b'C:\\Users\\J\\AppData\\Local\\Temp\\scipy-twmkmks2'
User avatar
torzdf
Posts: 2701
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 136 times
Been thanked: 632 times

Re: Made it about 15000 its in, then crashing every few hundred or so. File in use error?

Post by torzdf »

Ok, the problem file is a file generated by scipy which is placed in a temp folder: C:\Users\J\AppData\Local\Temp\

This file will have been generated, and subsequently managed by scipy itself. As this is happening upstream from us (in the scipy lib) there is nothing that can be done in Faceswap itself. As there are no other reports of this bug, it appears that the problem is unique to your system.

Ultimately, and most likely, something in Windows is accessing files in this folder that it shouldn't be. It may be a virus checker, or some kind of indexing, or it may be something else. You would need to find what is accessing this folder and stop it. Easier said than done.

Alternatively, it looks like this error is happening when 'warp to landmarks' is enabled. The first thing I would do is just not use 'warp to landmarks' and hope that solves the issue.

My word is final

Locked