Critical error when starting training

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Toto12345
Posts: 1
Joined: Wed Mar 03, 2021 5:38 pm

Critical error when starting training

Post by Toto12345 »

Code: Select all

03/03/2021 17:36:17 CRITICAL Error caught! Exiting...
03/03/2021 17:36:17 ERROR    Caught exception in thread: '_training_0'
03/03/2021 17:36:19 ERROR    Got Exception on main handler:
Traceback (most recent call last):
File "C:\Users\35484\faceswap\lib\cli\launcher.py", line 182, in execute_script
process.process()
File "C:\Users\35484\faceswap\scripts\train.py", line 170, in process
self._end_thread(thread, err)
File "C:\Users\35484\faceswap\scripts\train.py", line 210, in _end_thread
thread.join()
File "C:\Users\35484\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\35484\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\35484\faceswap\scripts\train.py", line 232, in _training
raise err
File "C:\Users\35484\faceswap\scripts\train.py", line 221, in _training
trainer = self._load_trainer(model)
File "C:\Users\35484\faceswap\scripts\train.py", line 267, in _load_trainer
trainer = trainer(model,
File "C:\Users\35484\faceswap\plugins\train\trainer\original.py", line 10, in __init__
super().__init__(*args, **kwargs)
File "C:\Users\35484\faceswap\plugins\train\trainer\_base.py", line 83, in __init__
alignment_data = self._get_alignments_data()
File "C:\Users\35484\faceswap\plugins\train\trainer\_base.py", line 143, in _get_alignments_data
alignments = _TrainingAlignments(self._model, self._images)
File "C:\Users\35484\faceswap\plugins\train\trainer\_base.py", line 1090, in __init__
self._detected_faces = self._load_detected_faces(image_list)
File "C:\Users\35484\faceswap\plugins\train\trainer\_base.py", line 1135, in _load_detected_faces
for filename, meta in tqdm(read_image_meta_batch(filelist),
File "C:\Users\35484\MiniConda3\envs\faceswap\lib\site-packages\tqdm\std.py", line 1166, in __iter__
for obj in iterable:
File "C:\Users\35484\faceswap\lib\image.py", line 444, in read_image_meta_batch
retval = (read_meta[future], future.result())
File "C:\Users\35484\MiniConda3\envs\faceswap\lib\concurrent\futures\_base.py", line 432, in result
return self.__get_result()
File "C:\Users\35484\MiniConda3\envs\faceswap\lib\concurrent\futures\_base.py", line 388, in __get_result
raise self._exception
File "C:\Users\35484\MiniConda3\envs\faceswap\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\35484\faceswap\lib\image.py", line 383, in read_image_meta
raise ValueError(f"Invalid header found in png: {filename}")
ValueError: Invalid header found in png: C:\Users\35484\Pictures\Ný mappa (2)\ezgif-3-2a26225ecbb8_001460_0.png
03/03/2021 17:36:19 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\35484\faceswap\crash_report.2021.03.03.173617198780.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

Process exited.
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Critical error when starting training

Post by torzdf »

For future reference:

You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

However, can you please provide me with the image: C:\Users\35484\Pictures\Ný mappa (2)\ezgif-3-2a26225ecbb8_001460_0.png (zipped, not directly attached)

My word is final

User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Re: Critical error when starting training

Post by gravitation »

I have the latest updated version. I tried to load the Manual Tool to do some fine tuning, and I got the same error on an input image. I removed that image from the directory and tried again, and turns out I have a lot...I can't proceed.

What are the header requirements? If you like, I can provide you with a sample image that has invalid header. But if I may I'd like to send you the picture via private message, not comfortable showing it on forum. Feel free to dm me for a follow up. I sure would like to help you debug this issue. And if you can help me identify the invalid header and show me how to remove it, I can do it for all the headers. Perhaps add a header integrity tool to your tool belt?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Critical error when starting training

Post by torzdf »

Are you using the manual tool correctly? It requires a source video/folder of frames NOT a folder of faces.

If so, please feel free to DM me a sample.

My word is final

User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Re: Critical error when starting training

Post by gravitation »

Yes sir I am! I have not confused the input frame folder for the output faces folder. Pat on the back for competency 8-)

Anyways, I ended up isolating each png that was causing those errors, ended up collecting over 20+ of them in a separate folder and began poking in them to see what I could find. After some surgery, I was able to figure out what the culprit was and I came up with a solution. A solution that, if you want, can add to your Tools belt.

The reason why it was throwing the invalid_header exception was simply because the file's extension was not what the file was originally. I found this out by using a tool called TweakPNG which will show you what the headers are in any png. It too threw a fit, so I decided to use a more broader forensic tool called trid. Trid is a file analyzing forensic tool that has an extensive definition library of all collected file types and it will scan the binary structure of the file and tell you what the file type is (without relying on what its extension is). As a result, all those png files were in fact jpg. Trid has an option to automatically rename the extensions, so I fed all those files through that.

Re added those files to my input folder, and ran missing extractions to add them into the alignments and remove the old png versions. Then I used the manual tool and voila! All is honky dorry. So now, as part of the extraction workflow, before I do anything, I run all the files through trid to make sure they are what they represent themselves to be.

I hope this will help anyone else seeing this issue and possibly have the developer add a contingency for this in his code.

Locked