Error on training

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
bba20
Posts: 11
Joined: Tue Sep 07, 2021 3:29 pm
Has thanked: 3 times
Been thanked: 2 times

Error on training

Post by bba20 »

I recently upgraded faceswap from a version I was using for a few months yesterday. I had to uninstall/reinstall to get it working because something happened during the update that kept crashing it.

Anyway I went back to my training file and now I get this error:

Code: Select all

C:\Users\XX\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py:494: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
warnings.warn('Custom mask layers require a config and must override '
2021-09-07 08:25:27.048652: E tensorflow/core/platform/windows/subprocess.cc:287] Call to CreateProcess failed. Error code: 2

Ive been using the same settings for months and never got this error, only on the latest version. I use the dfaker model with warp to landmarks checked. I dont use a custom-mask. Training seems to work but I noticed the session lterations seemed smaller compared to before in the same few hours I normally train in.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Error on training

Post by torzdf »

For the errors, you can safely ignore them. These have been introduced by Tensorflow 2.6

The custom_mask message is nothing to do with you and is most likely to do with a custom layer inside the faceswap application. It is a completely useless warning for the end user, but it has been decided by the powers that be to make it particularly tricky for me to suppress. It does not affect you in any way.

Similarly, the 2nd error message about Call to CreateProcess failed is also an un-suppressable warning message which Tensorflow, in their infinite wisdom, have decided to elevate to an error message. This failure has actually always happened in tf2, but you've just never known about it. It just means that your machine (and nearly all machines installing Cuda through Conda) is missing a dll file for an obscure part of Tensorflow that you don't need. It does not impact you at all. For now, ignore it.

EDIT
Both of these messages should now be suppressed.
/EDIT

As to the speed issue. That is more problematic. Somewhere between Nvidia and Conda there has been a massive screw-up in creating working Tensorflow packages for users with RTX30xx cards. I have been waiting for them to sort this out... and waiting, and waiting, and waiting. And it looked like Conda finally had. Only they made the totally mystifying decision to release a Tensorflow package that is basically thoroughly broken as it forces an install of Numpy that breaks Tensorflow... And it's not possinle to install a working version without Conda totally shitting the bed.

Basically, it's a total mess, and it has caused me a huge headache, as I try to make faceswap as easy to use and setup as possible, and I have had RTX30xx users complaining at me ad-infinitum that they can't get faceswap to work on their GPU.

So clearly something had to be done. I have attempted to work around the issue by creating an install that will work on RTX30xx cards. I'm sure it does work for them. The down side is that it seems to have totally screwed the speed for the rest of us.

All is not lost though. If you just upgrade Tensorflow from a known working version (instructions later), you can keep the latest faceswap updates, whilst also keeping Tensorflow 2.4 which, whilst not working on RTX30xx cards, will keep the earlier speed. I have deliberately made it that an upgrade will not force an upgrade of Tensorflow and Cuda/cuDNN.

RTX30xx users are just going to have to suck it up for now. They can nag Conda on my behalf as I have spent enough of my unpaid time getting this to work.

I will look at a way to work around this (probably in options at the install stage), but I really don't have time at the moment, and it is yet more for me to do to work around someone else's mess :/

Anyway, rant over.

To revert your Tensorflow to a working version you can do the following:

NB: For the cd faceswap command: if you have installed to a non-default location, you will need to put the correct path to your faceswap install folder at this location.

Start > Anaconda Prompt

Code: Select all

conda env remove -n faceswap
conda create -n faceswap python=3.8
conda activate faceswap
cd faceswap
conda install git
git fetch --unshallow
git checkout c7d85f89e69c74e97bf7485b064c07487d31faae
python setup.py

Follow the prompts in the setup script and allow it to install. At this point you will be back on the tf2.4 version. You can then launch faceswap in the usual way.

You should be able to update from the GUI just fine, and still keep your previous Cuda/cuDNN/Tensorflow versions that were installed from this checkpoint.

My word is final

Locked