Getting file in use crash during training on scipy-zun5tqr4

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Getting file in use crash during training on scipy-zun5tqr4

Post by unkempt »

In my users/appdata/temp folder, the above listed file is showing as in use and crashes the process after 1500 or so iterations. Early phase, still on "warp". The file isn't there when I go look and seems to be related directly to faceswap. This was running while I was away from the computer for the last hour or so which is why I don't think any other programs got in the way. Nothing but some web browsers and such open.

I've had this problem before, but it appears to be consistent.

by torzdf » Thu Aug 10, 2023 6:36 pm

I do not know what that file is, but it will be generated by scipy (a library we use in data augmentation).

As it comes from a library we use, rather than from Faceswap itself, I'm not sure there is much we can do to mitigate the issue.

Searching the forum it appears you have raised this before:
viewtopic.php?p=9203#p9203

However, I have not heard of any other similar issues from other users, and as this appears in training, it is highly likely we would have heard about it if the problem was universal.

Ultimately, Windows is a massive PITA when it comes to file permissions/sharing/locking. The most likely cause of your issue is going to be some other service accessing the file, which locks Faceswap out from it. That may be a virus scanner, or a cloud manager or some other process that we have no control over.

Unfortunately this is all I can really give you. You would need to find what on your system is trying to access that file and either disable it or add an exception.

Go to full post
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Getting file in use crash during training on scipy-zun5tqr4

Post by torzdf »

I do not know what that file is, but it will be generated by scipy (a library we use in data augmentation).

As it comes from a library we use, rather than from Faceswap itself, I'm not sure there is much we can do to mitigate the issue.

Searching the forum it appears you have raised this before:
viewtopic.php?p=9203#p9203

However, I have not heard of any other similar issues from other users, and as this appears in training, it is highly likely we would have heard about it if the problem was universal.

Ultimately, Windows is a massive PITA when it comes to file permissions/sharing/locking. The most likely cause of your issue is going to be some other service accessing the file, which locks Faceswap out from it. That may be a virus scanner, or a cloud manager or some other process that we have no control over.

Unfortunately this is all I can really give you. You would need to find what on your system is trying to access that file and either disable it or add an exception.

My word is final

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Getting file in use crash during training on scipy-zun5tqr4

Post by unkempt »

It only made it about 100 more iterations before crashing again.

Again at 2001 its.

scipy-cgk40_w2 @ 3001 its

scipy-5ekc3fh2 @ 3252

And so on.

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Getting file in use crash during training on scipy-zun5tqr4

Post by unkempt »

torzdf wrote: Thu Aug 10, 2023 6:36 pm

I do not know what that file is, but it will be generated by scipy (a library we use in data augmentation).

As it comes from a library we use, rather than from Faceswap itself, I'm not sure there is much we can do to mitigate the issue.

Searching the forum it appears you have raised this before:
viewtopic.php?p=9203#p9203

However, I have not heard of any other similar issues from other users, and as this appears in training, it is highly likely we would have heard about it if the problem was universal.

Ultimately, Windows is a massive PITA when it comes to file permissions/sharing/locking. The most likely cause of your issue is going to be some other service accessing the file, which locks Faceswap out from it. That may be a virus scanner, or a cloud manager or some other process that we have no control over.

Unfortunately this is all I can really give you. You would need to find what on your system is trying to access that file and either disable it or add an exception.

Ok, that helps. I think I can debug a bit now and hopefully will find something. Thanks for the help :)

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Getting file in use crash during training on scipy-zun5tqr4

Post by unkempt »

For whatever reason, either setting python and faceswap to run as admin or setting it to save every 1000 iterations seems to have helped.

Locked