NaN while training, then invalid value encountered in rint patched_face = np.rint(patched_face,

Getting errors or found a bug when converting faces from a trained model? Post about them here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Convert process. If you want to get tips, or better understand the Convert process, then you should look in the Convert Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Diego427
Posts: 2
Joined: Fri Mar 19, 2021 8:52 pm

NaN while training, then invalid value encountered in rint patched_face = np.rint(patched_face,

Post by Diego427 »

Hello

After it trained for seven hours I left it going on for the night, but it ended up having NaN errors and a black rectangle instead of the face on the preview... :o
I tried to convert anyway, but it says :
faceswap\lib\convert.py:197: RuntimeWarning: invalid value encountered in rint
patched_face = np.rint(patched_face,

Is there a way to have FaceSwap ignore the bad corrupt data and convert using only the good data ?

Thanks

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: NaN while training, then invalid value encountered in rint patched_face = np.rint(patched_face,

Post by torzdf »

Sadly not.

Ultimately, if a NaN occurs in the model, your model is effectively dead. That one NaN will start propagating out to other weights and ultimately the model will collapse and fail.

I have recently (read: yesterday) added NaN protection for training. This will not solve your current issue, unfortunately (you'll need to roll-back to a pre NaN snapshot/backup). But in future, make sure it's enabled and if a NaN is encountered then the model will halt immediately. Your last save should be NaN free, and sometimes it's as simple as just kicking off training again.

My word is final

User avatar
Diego427
Posts: 2
Joined: Fri Mar 19, 2021 8:52 pm

Re: NaN while training, then invalid value encountered in rint patched_face = np.rint(patched_face,

Post by Diego427 »

Ok, too bad. :cry:
It made an auto snapshot at 25.000 but I tried to load it and it was already corrupt.
Maybe I can share my logs to help you find bugs or something ?
By the way, I was training in Lighweight mode on the CPU ( Ryzen 5 3600 + 32GB DDR4 ), for my graphic card is too old and incompatible (9500GT from 2008 :oops:). Maybe that explains the bug.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: NaN while training, then invalid value encountered in rint patched_face = np.rint(patched_face,

Post by torzdf »

Unfortunately NaNs can happen on any model (although some are more susceptible than others).

Training a model is "unbounded" which means that it is possible for a weight update to occur outside of the numerical range that a datatype (specifically float32) can hold. This can happen for numerous reasons and there isn't much you can do to protect against it.

My word is final

User avatar
230V
Posts: 20
Joined: Sun Jan 19, 2020 6:20 pm
Answers: 1
Has thanked: 7 times
Been thanked: 2 times

error when converting or preview

Post by 230V »

how to understand this log?

Loading...
Setting Faceswap backend to AMD
03/21/2021 02:22:13 INFO Log level set to: INFO
03/21/2021 02:22:14 INFO Setting up for PlaidML
03/21/2021 02:22:14 INFO Using GPU(s): ['Advanced Micro Devices, Inc. - Ellesmere (experimental)', 'Advanced Micro Devices, Inc. - Ellesmere (supported)']
03/21/2021 02:22:14 INFO Successfully set up for PlaidML

03/21/2021 02:22:16 INFO Reading alignments from: 'C:\fs\convert\edit_alignments.fsa'
03/21/2021 02:22:16 INFO Loading Writer from Opencv plugin...

Using plaidml.keras.backend backend.
03/21/2021 02:22:16 INFO Using configuration saved in state file
03/21/2021 02:22:16 INFO Opening device "opencl_amd_ellesmere.0"
03/21/2021 02:22:17 INFO Loaded model from disk: 'C:\fs\convert\edit\MODEL\original.h5'
03/21/2021 02:22:17 INFO Loading Mask from Box_Blend plugin...
03/21/2021 02:22:17 INFO Loading Mask from Mask_Blend plugin...
03/21/2021 02:22:17 INFO Loading Color from Avg_Color plugin...
03/21/2021 02:22:17 INFO Loading Scaling from Sharpen plugin...
C:\Users\fs\faceswap_amd\plugins\convert\color\avg_color.py:16: RuntimeWarning: invalid value encountered in true_divide
adjustment = avg_diff / np.sum(raw_mask, axis=(0, 1))
C:\Users\fs\faceswap_amd\plugins\convert\scaling\sharpen.py:57: RuntimeWarning: invalid value encountered in less
low_contrast_mask = (abs(new_face - blur) < threshold).astype("float32")
C:\Users\fs\faceswap_amd\lib\convert.py:197: RuntimeWarning: invalid value encountered in rint
patched_face = np.rint(patched_face,

Terminating Process...
Terminated
Process exited.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: error when converting or preview

Post by torzdf »

Does this happen immediately, or several faces into your convert?

My word is final

User avatar
230V
Posts: 20
Joined: Sun Jan 19, 2020 6:20 pm
Answers: 1
Has thanked: 7 times
Been thanked: 2 times

Re: error when converting or preview

Post by 230V »

it happens immediately and shoots out frames without faces (black squares where the swapface should be)

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: error when converting or preview

Post by torzdf »

Ok. So most likely there is a "NaN" in your model. I'm merging this topic with a similar discussion....

My word is final

Locked