crash when training,I tried to recover from the snapshot, but this problem occurred later: numpy.core._exceptions._Array

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
icey
Posts: 3
Joined: Tue Feb 14, 2023 3:14 pm

crash when training,I tried to recover from the snapshot, but this problem occurred later: numpy.core._exceptions._Array

Post by icey »

Code: Select all

02/20/2023 06:25:41 CRITICAL Error caught! Exiting...
02/20/2023 06:25:41 ERROR    Caught exception in thread: '_training'
02/20/2023 06:25:57 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "C:\Users\mat\faceswap\lib\cli\launcher.py", line 230, in execute_script
    process.process()
  File "C:\Users\mat\faceswap\scripts\train.py", line 213, in process
    self._end_thread(thread, err)
  File "C:\Users\mat\faceswap\scripts\train.py", line 253, in _end_thread
    thread.join()
  File "C:\Users\mat\faceswap\lib\multithreading.py", line 220, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\mat\faceswap\lib\multithreading.py", line 96, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\mat\faceswap\scripts\train.py", line 275, in _training
    raise err
  File "C:\Users\mat\faceswap\scripts\train.py", line 265, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Users\mat\faceswap\scripts\train.py", line 367, in _run_training_cycle
    model.save(is_exit=False)
  File "C:\Users\mat\faceswap\plugins\train\model\_base\model.py", line 437, in save
    self._io.save(is_exit=is_exit)
  File "C:\Users\mat\faceswap\plugins\train\model\_base\io.py", line 207, in save
    self._plugin.model.save(self._filename, include_optimizer=include_optimizer)
  File "C:\ProgramData\Miniconda3\envs\faceswap\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\ProgramData\Miniconda3\envs\faceswap\lib\site-packages\keras\backend.py", line 4240, in <listcomp>
    return [x.numpy() for x in tensors]
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 64.0 MiB for an array with shape (1024, 16384) and data type float32
02/20/2023 06:25:57 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\mat\faceswap\crash_report.2023.02.20.062541546845.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Last edited by torzdf on Sun Feb 19, 2023 11:27 pm, edited 2 times in total.
User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: crash when training,I tried to recover from the snapshot, but this problem occurred later

Post by torzdf »

Firstly, if seeking help you must provide the crash report as stated in both the rules of the forum and the message that you have posted in your topic.

This is a memory error. It may be fixed with a reboot. If not, provide the relevant information.

My word is final

Locked