Faceswap Forum

Hello

After it trained for seven hours I left it going on for the night, but it ended up having NaN errors and a black rectangle instead of the face on the preview...
I tried to convert anyway, but it says :
faceswap\lib\convert.py:197: RuntimeWarning: invalid value encountered in rint
patched_face = np.rint(patched_face,

Is there a way to have FaceSwap ignore the bad corrupt data and convert using only the good data ?

Thanks

Sadly not.

Ultimately, if a NaN occurs in the model, your model is effectively dead. That one NaN will start propagating out to other weights and ultimately the model will collapse and fail.

I have recently (read: yesterday) added NaN protection for training. This will not solve your current issue, unfortunately (you'll need to roll-back to a pre NaN snapshot/backup). But in future, make sure it's enabled and if a NaN is encountered then the model will halt immediately. Your last save should be NaN free, and sometimes it's as simple as just kicking off training again.

Ok, too bad.
It made an auto snapshot at 25.000 but I tried to load it and it was already corrupt.
Maybe I can share my logs to help you find bugs or something ?
By the way, I was training in Lighweight mode on the CPU ( Ryzen 5 3600 + 32GB DDR4 ), for my graphic card is too old and incompatible (9500GT from 2008 ). Maybe that explains the bug.

Unfortunately NaNs can happen on any model (although some are more susceptible than others).

Training a model is "unbounded" which means that it is possible for a weight update to occur outside of the numerical range that a datatype (specifically float32) can hold. This can happen for numerous reasons and there isn't much you can do to protect against it.

how to understand this log?

Loading...
Setting Faceswap backend to AMD
03/21/2021 02:22:13 INFO Log level set to: INFO
03/21/2021 02:22:14 INFO Setting up for PlaidML
03/21/2021 02:22:14 INFO Using GPU(s): ['Advanced Micro Devices, Inc. - Ellesmere (experimental)', 'Advanced Micro Devices, Inc. - Ellesmere (supported)']
03/21/2021 02:22:14 INFO Successfully set up for PlaidML

03/21/2021 02:22:16 INFO Reading alignments from: 'C:\fs\convert\edit_alignments.fsa'
03/21/2021 02:22:16 INFO Loading Writer from Opencv plugin...

Using plaidml.keras.backend backend.
03/21/2021 02:22:16 INFO Using configuration saved in state file
03/21/2021 02:22:16 INFO Opening device "opencl_amd_ellesmere.0"
03/21/2021 02:22:17 INFO Loaded model from disk: 'C:\fs\convert\edit\MODEL\original.h5'
03/21/2021 02:22:17 INFO Loading Mask from Box_Blend plugin...
03/21/2021 02:22:17 INFO Loading Mask from Mask_Blend plugin...
03/21/2021 02:22:17 INFO Loading Color from Avg_Color plugin...
03/21/2021 02:22:17 INFO Loading Scaling from Sharpen plugin...
C:\Users\fs\faceswap_amd\plugins\convert\color\avg_color.py:16: RuntimeWarning: invalid value encountered in true_divide
adjustment = avg_diff / np.sum(raw_mask, axis=(0, 1))
C:\Users\fs\faceswap_amd\plugins\convert\scaling\sharpen.py:57: RuntimeWarning: invalid value encountered in less
low_contrast_mask = (abs(new_face - blur) < threshold).astype("float32")
C:\Users\fs\faceswap_amd\lib\convert.py:197: RuntimeWarning: invalid value encountered in rint
patched_face = np.rint(patched_face,

Terminating Process...
Terminated
Process exited.

Does this happen immediately, or several faces into your convert?

it happens immediately and shoots out frames without faces (black squares where the swapface should be)

Ok. So most likely there is a "NaN" in your model. I'm merging this topic with a similar discussion....