Page 1 of 1

crash when training,I tried to recover from the snapshot, but this problem occurred later: numpy.core._exceptions._Array

Posted: Sun Feb 19, 2023 10:28 pm
by icey

Code: Select all

02/20/2023 06:25:41 CRITICAL Error caught! Exiting...
02/20/2023 06:25:41 ERROR    Caught exception in thread: '_training'
02/20/2023 06:25:57 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "C:\Users\mat\faceswap\lib\cli\launcher.py", line 230, in execute_script
    process.process()
  File "C:\Users\mat\faceswap\scripts\train.py", line 213, in process
    self._end_thread(thread, err)
  File "C:\Users\mat\faceswap\scripts\train.py", line 253, in _end_thread
    thread.join()
  File "C:\Users\mat\faceswap\lib\multithreading.py", line 220, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\mat\faceswap\lib\multithreading.py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\mat\faceswap\scripts\train.py", line 275, in _training
    raise err
  File "C:\Users\mat\faceswap\scripts\train.py", line 265, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Users\mat\faceswap\scripts\train.py", line 367, in _run_training_cycle
    model.save(is_exit=False)
  File "C:\Users\mat\faceswap\plugins\train\model\_base\model.py", line 437, in save
    self._io.save(is_exit=is_exit)
  File "C:\Users\mat\faceswap\plugins\train\model\_base\io.py", line 207, in save
    self._plugin.model.save(self._filename, include_optimizer=include_optimizer)
  File "C:\ProgramData\Miniconda3\envs\faceswap\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\ProgramData\Miniconda3\envs\faceswap\lib\site-packages\keras\backend.py", line 4240, in <listcomp>
    return [x.numpy() for x in tensors]
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 64.0 MiB for an array with shape (1024, 16384) and data type float32
02/20/2023 06:25:57 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\mat\faceswap\crash_report.2023.02.20.062541546845.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

Re: crash when training,I tried to recover from the snapshot, but this problem occurred later

Posted: Sun Feb 19, 2023 11:26 pm
by torzdf

Firstly, if seeking help you must provide the crash report as stated in both the rules of the forum and the message that you have posted in your topic.

This is a memory error. It may be fixed with a reboot. If not, provide the relevant information.