Page 1 of 1

Critical error when starting training

Posted: Wed Mar 03, 2021 5:43 pm
by Toto12345

Code: Select all

03/03/2021 17:36:17 CRITICAL Error caught! Exiting...
03/03/2021 17:36:17 ERROR    Caught exception in thread: '_training_0'
03/03/2021 17:36:19 ERROR    Got Exception on main handler:
Traceback (most recent call last):
File "C:\Users\35484\faceswap\lib\cli\launcher.py", line 182, in execute_script
process.process()
File "C:\Users\35484\faceswap\scripts\train.py", line 170, in process
self._end_thread(thread, err)
File "C:\Users\35484\faceswap\scripts\train.py", line 210, in _end_thread
thread.join()
File "C:\Users\35484\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\35484\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\35484\faceswap\scripts\train.py", line 232, in _training
raise err
File "C:\Users\35484\faceswap\scripts\train.py", line 221, in _training
trainer = self._load_trainer(model)
File "C:\Users\35484\faceswap\scripts\train.py", line 267, in _load_trainer
trainer = trainer(model,
File "C:\Users\35484\faceswap\plugins\train\trainer\original.py", line 10, in __init__
super().__init__(*args, **kwargs)
File "C:\Users\35484\faceswap\plugins\train\trainer\_base.py", line 83, in __init__
alignment_data = self._get_alignments_data()
File "C:\Users\35484\faceswap\plugins\train\trainer\_base.py", line 143, in _get_alignments_data
alignments = _TrainingAlignments(self._model, self._images)
File "C:\Users\35484\faceswap\plugins\train\trainer\_base.py", line 1090, in __init__
self._detected_faces = self._load_detected_faces(image_list)
File "C:\Users\35484\faceswap\plugins\train\trainer\_base.py", line 1135, in _load_detected_faces
for filename, meta in tqdm(read_image_meta_batch(filelist),
File "C:\Users\35484\MiniConda3\envs\faceswap\lib\site-packages\tqdm\std.py", line 1166, in __iter__
for obj in iterable:
File "C:\Users\35484\faceswap\lib\image.py", line 444, in read_image_meta_batch
retval = (read_meta[future], future.result())
File "C:\Users\35484\MiniConda3\envs\faceswap\lib\concurrent\futures\_base.py", line 432, in result
return self.__get_result()
File "C:\Users\35484\MiniConda3\envs\faceswap\lib\concurrent\futures\_base.py", line 388, in __get_result
raise self._exception
File "C:\Users\35484\MiniConda3\envs\faceswap\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\35484\faceswap\lib\image.py", line 383, in read_image_meta
raise ValueError(f"Invalid header found in png: {filename}")
ValueError: Invalid header found in png: C:\Users\35484\Pictures\Ný mappa (2)\ezgif-3-2a26225ecbb8_001460_0.png
03/03/2021 17:36:19 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\35484\faceswap\crash_report.2021.03.03.173617198780.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

Process exited.

Re: Critical error when starting training

Posted: Wed Mar 03, 2021 7:57 pm
by torzdf

For future reference:

You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

However, can you please provide me with the image: C:\Users\35484\Pictures\Ný mappa (2)\ezgif-3-2a26225ecbb8_001460_0.png (zipped, not directly attached)


Re: Critical error when starting training

Posted: Sat May 08, 2021 5:12 am
by gravitation

I have the latest updated version. I tried to load the Manual Tool to do some fine tuning, and I got the same error on an input image. I removed that image from the directory and tried again, and turns out I have a lot...I can't proceed.

What are the header requirements? If you like, I can provide you with a sample image that has invalid header. But if I may I'd like to send you the picture via private message, not comfortable showing it on forum. Feel free to dm me for a follow up. I sure would like to help you debug this issue. And if you can help me identify the invalid header and show me how to remove it, I can do it for all the headers. Perhaps add a header integrity tool to your tool belt?


Re: Critical error when starting training

Posted: Sat May 08, 2021 10:44 am
by torzdf

Are you using the manual tool correctly? It requires a source video/folder of frames NOT a folder of faces.

If so, please feel free to DM me a sample.


Re: Critical error when starting training

Posted: Sat May 08, 2021 10:28 pm
by gravitation

Yes sir I am! I have not confused the input frame folder for the output faces folder. Pat on the back for competency 8-)

Anyways, I ended up isolating each png that was causing those errors, ended up collecting over 20+ of them in a separate folder and began poking in them to see what I could find. After some surgery, I was able to figure out what the culprit was and I came up with a solution. A solution that, if you want, can add to your Tools belt.

The reason why it was throwing the invalid_header exception was simply because the file's extension was not what the file was originally. I found this out by using a tool called TweakPNG which will show you what the headers are in any png. It too threw a fit, so I decided to use a more broader forensic tool called trid. Trid is a file analyzing forensic tool that has an extensive definition library of all collected file types and it will scan the binary structure of the file and tell you what the file type is (without relying on what its extension is). As a result, all those png files were in fact jpg. Trid has an option to automatically rename the extensions, so I fed all those files through that.

Re added those files to my input folder, and ran missing extractions to add them into the alignments and remove the old png versions. Then I used the manual tool and voila! All is honky dorry. So now, as part of the extraction workflow, before I do anything, I run all the files through trid to make sure they are what they represent themselves to be.

I hope this will help anyone else seeing this issue and possibly have the developer add a contingency for this in his code.