Error During training

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
cedenburn
Posts: 11
Joined: Sun Jan 02, 2022 3:11 am
Has thanked: 11 times

Error During training

Post by cedenburn »

Error during training . I removed some images from a sorted file and removed the faces from the alignment file. I then proceeded to training again and received an error. I can restore the models and it will go back some distance 0.01977. I then train back down to a certain point 0.00107 and then I receive another error. It then repeats the process. It appears the model file may have an issue. Sorry I failed to capture the original error message. Thank you.

Attachments
crash_report.2022.03.12.015223831594.log
(61.38 KiB) Downloaded 111 times
crash_report.2022.03.12.094730481349.log
(38.28 KiB) Downloaded 106 times
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Error During training

Post by torzdf »

I have never seen the first error before. It is basically Windows throwing an error on a simple print statement, which suggests an issue with your OS. It may be related to the 2nd issue, which is a permissions issue.

Code: Select all

PermissionError: [WinError 5] Access is denied: 'C:\\Users\\camer\\Desktop\\FS\\Model\\original_logs' -> 'C:\\Users\\camer\\Desktop\\FS\\Model\\original_archived_20220312_094730\\original_logs'

In the first instance, check your permissions. In the 2nd, make sure you haven't filled your hard disk up. If neither of these are the problem, then I am afraid I am at a loss to explain it.

My word is final

User avatar
cedenburn
Posts: 11
Joined: Sun Jan 02, 2022 3:11 am
Has thanked: 11 times

Re: Error During training

Post by cedenburn »

Re occurring problem. The program closes before I get the error message. I can restore the file but the same problem returns. This happens well over 5 meg interations so permissions are OK. The Hdd has plenty of space. Not sure why it would attempt to print all of the sudden in the same area after starting training all over again. Win 10. It may be a file issue in one of the folders.

File "C:\Users\camer\faceswap\lib\cli\launcher.py", line 182, in execute_script
process.process()
File "C:\Users\camer\faceswap\scripts\train.py", line 190, in process
self._end_thread(thread, err)
File "C:\Users\camer\faceswap\scripts\train.py", line 230, in end_thread
thread.join()
File "C:\Users\camer\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\camer\faceswap\lib\multithreading.py", line 37, in run
self.
target(*self.args, **self.kwargs)
File "C:\Users\camer\faceswap\scripts\train.py", line 252, in _training
raise err
File "C:\Users\camer\faceswap\scripts\train.py", line 242, in training
self.
run_training_cycle(model, trainer)
File "C:\Users\camer\faceswap\scripts\train.py", line 327, in run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "C:\Users\camer\faceswap\plugins\train\trainer\_base.py", line 224, in train_one_step
self.
print_loss(loss)
File "C:\Users\camer\faceswap\plugins\train\trainer\_base.py", line 304, in _print_loss
print("\r{}".format(output), end="")
OSError: [Errno 22] Invalid argument

Attachments
crash_report.2022.04.14.060347140969.log
(41.3 KiB) Downloaded 86 times
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Error During training

Post by torzdf »

Ok, that I really can't explain. A print statement shouldn't be raising an OS Error :/

My word is final

Locked