ERROR Caught exception in thread: '_training'

[kocheko]

i hope someone could help, otherwise this would be 3 days of training going to the trash. i saved the project and now what i do is: open the project, wait for it to load and click TRAIN, and then this happens :

Code: Select all

03/21/2024 15:39:24 INFO     Mixed precision compatibility check (mixed_float16): OK\nYour GPU will likely run quickly with dtype policy mixed_float16 as it has compute capability of at least 7.0. Your GPU: NVIDIA GeForce RTX 3060, compute capability 8.6
03/21/2024 15:39:24 INFO     Enabling Mixed Precision Training.
03/21/2024 15:39:24 CRITICAL Error caught! Exiting...
03/21/2024 15:39:24 ERROR    Caught exception in thread: '_training'
03/21/2024 15:39:27 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "C:\Users\kocheko\faceswap\lib\cli\launcher.py", line 225, in execute_script
    process.process()
  File "C:\Users\kocheko\faceswap\scripts\train.py", line 209, in process
    self._end_thread(thread, err)
  File "C:\Users\kocheko\faceswap\scripts\train.py", line 249, in _end_thread
    thread.join()
  File "C:\Users\kocheko\faceswap\lib\multithreading.py", line 224, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\kocheko\faceswap\lib\multithreading.py", line 100, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\kocheko\faceswap\scripts\train.py", line 274, in _training
    raise err
  File "C:\Users\kocheko\faceswap\scripts\train.py", line 259, in _training
    model = self._load_model()
  File "C:\Users\kocheko\faceswap\scripts\train.py", line 290, in _load_model
    model.build()
  File "C:\Users\kocheko\faceswap\plugins\train\model\_base\model.py", line 255, in build
    model = self.io.load()
  File "C:\Users\kocheko\faceswap\plugins\train\model\_base\io.py", line 147, in load
    model = kmodels.load_model(self.filename, compile=False)
  File "C:\Users\kocheko\MiniConda3\envs\faceswap\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\kocheko\MiniConda3\envs\faceswap\lib\site-packages\h5py\_hl\files.py", line 562, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "C:\Users\kocheko\MiniConda3\envs\faceswap\lib\site-packages\h5py\_hl\files.py", line 235, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5f.pyx", line 102, in h5py.h5f.open
OSError: Unable to synchronously open file (bad object header version number)
03/21/2024 15:39:27 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\kocheko\faceswap\crash_report.2024.03.21.153924976721.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Process exited.

IS THE PROJECT LOST? almost 700.000 iterations

[torzdf]

Not lost, but you will need to roll back.

This error means that your model file is corrupted. Most likely training was interrupted during a save.

You have 2 options, you can either restore from backup (you can use the model tool for this), or you can delete your model folder rename your latest snapshot folder to your model folder's name and continue from there.

I suggest you check the timestamps of the backups (in the model folder) and the last snapshot, to decide which is the best way to go.