Converting model to TF2.x mask problem

Installing and setting up FaceSwap


Forum rules

Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Lig789
Posts: 3
Joined: Tue Aug 18, 2020 2:25 pm
Has thanked: 1 time

Converting model to TF2.x mask problem

Post by Lig789 »

After updating FS 2.0, It started of converting model for tensor flow 2.x version, Kindly help me with solving this problem. I am stuck with alignment files when I never used it in original training (only needed when using mask etc).

when I created alignment file for conversions from original face folder, I get the error for "mask" but I didn't use any mask in original Dlight training. If I do not include any alignment file it asks for alignment file. If I include alignment it is giving me this error. I dont even know what mask if used any. But I didn't use any and it wants alignment file. should I go back to FS v1? It does go through all the images but when it reads alignment files for A and B it crushes.

Code: Select all

INFO     Reading alignments from A and B then 
Following error: 
08/18/2020 19:42:16 CRITICAL Error caught! Exiting...
08/18/2020 19:42:16 ERROR    Caught exception in thread: '_training_0'
08/18/2020 19:42:21 ERROR    Got Exception on main handler:
Traceback (most recent call last):
File "D:\faceswap\lib\cli\launcher.py", line 156, in execute_script
process.process()
File "D:\faceswap\scripts\train.py", line 165, in process
self._end_thread(thread, err)
File "D:\faceswap\scripts\train.py", line 205, in _end_thread
thread.join()
File "D:\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "D:\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "D:\faceswap\scripts\train.py", line 227, in _training
raise err
File "D:\faceswap\scripts\train.py", line 216, in _training
trainer = self._load_trainer(model)
File "D:\faceswap\scripts\train.py", line 263, in _load_trainer
trainer = trainer(model,
File "D:\faceswap\plugins\train\trainer\original.py", line 10, in __init__
super().__init__(*args, **kwargs)
File "D:\faceswap\plugins\train\trainer\_base.py", line 85, in __init__
self._get_alignments_data())
File "D:\faceswap\plugins\train\trainer\_base.py", line 120, in _get_alignments_data
retval["masks"] = alignments.masks
File "D:\faceswap\plugins\train\trainer\_base.py", line 1090, in masks
retval = {side: self._get_masks(side, detected_faces)
File "D:\faceswap\plugins\train\trainer\_base.py", line 1090, in <dictcomp>
retval = {side: self._get_masks(side, detected_faces)
File "D:\faceswap\plugins\train\trainer\_base.py", line 1114, in _get_masks
mask = face.mask[self._config["mask_type"]]
KeyError: None
08/18/2020 19:42:21 CRITICAL An unexpected crash has occurred. Crash report written to 'D:\faceswap\crash_report.2020.08.18.194216335184.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting.
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

Please provide this file:

D:\faceswap\crash_report.2020.08.18.194216335184.log

My word is final

User avatar
Lig789
Posts: 3
Joined: Tue Aug 18, 2020 2:25 pm
Has thanked: 1 time

Re: Converting model to TF2.x mask problem

Post by Lig789 »

Successfully, I get two files converted and they are "dlight.h5" and "dlight_state.json" and the rest is error. In original training I did not use any mask and left alignment files empty but in conversion for TF 2.X models I get this following error, when it reads newly created alignment file{note: FS 2.0, does not allow me to choose "None" as mask in global setting, it always goes back to "extended", when I tried to set none before clicking on train}:

Attachments
crash_report.2020.08.18.194037656493.log
(78.18 KiB) Downloaded 190 times
User avatar
Korben
Posts: 26
Joined: Wed Aug 19, 2020 3:17 pm
Has thanked: 2 times
Been thanked: 3 times

Re: Converting model to TF2.x mask problem

Post by Korben »

Same kind of problem, did not need alignments before updating.
Image

Image

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

[mention]Korben[/mention]
Please could you provide both of the <model_name>_state.json file from your new and archived model folders

My word is final

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

Lig789 wrote: Wed Aug 19, 2020 11:15 am

{note: FS 2.0, does not allow me to choose "None" as mask in global setting, it always goes back to "extended", when I tried to set none before clicking on train}:

This looks like a bug. I will investigate tomorrow.

My word is final

User avatar
Korben
Posts: 26
Joined: Wed Aug 19, 2020 3:17 pm
Has thanked: 2 times
Been thanked: 3 times

Re: Converting model to TF2.x mask problem

Post by Korben »

torzdf wrote: Thu Aug 20, 2020 12:08 am

@Korben
Please could you provide both of the <model_name>_state.json file from your new and archived model folders

Forgot to add the same thing lig789 just posted. I also noticed that if I tried to select none as mask it s always at extended when I come back to that screen.

But your post made me curious and I took a look the state files and saw that the new model was like this

Code: Select all

 "penalized_mask_loss": true,

compared to false in the original. So I tweaked it and it worked. I made an other test where I deleted the new model turned penalized_mask_loss off in the global config and redid the upgrade and it got rid of the error too.

BUT.......the model has zero training. Analysis only shows the new sessions starting at 22 in my case. Pretty sure I,m being blind right now but I dont see how to attach files.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

Ok, I suspect I know why this happens. Will review today.

My word is final

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

FYI: The mask reverting to "extended" issue was just a GUI display issue and has been fixed in the latest update.

I am still looking at the other issue.

My word is final

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

This issue should now be fixed....

The easiest way to get your model updated is to delete the new folder that was created before (and doesn't work).

Rename the "_archived" folder back to the the original folder name.

Run training again.

Any problems, let me know.

My word is final

User avatar
Korben
Posts: 26
Joined: Wed Aug 19, 2020 3:17 pm
Has thanked: 2 times
Been thanked: 3 times

Re: Converting model to TF2.x mask problem

Post by Korben »

ok so updated everything and now it gets the model running on first try.

But still in my case its like the first training. I dont know if its intended in the new version but dfl_h128_decoder_A.h5, and dfl_h128_decoder_B.h5 are not present in the new folder. Judging by the file size it all seems to be in the dfl_h128_encoder.h5 file now.

Attachments
Archived-dfl_h128_state.json.txt
Archived version
(7.17 KiB) Downloaded 190 times
NEW-dfl_h128_state.json.txt
new model with 1 new mini training session
(7.53 KiB) Downloaded 178 times
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

No, it should continue from where it left off, but if it's LIAE architecture, then it may be problematic....

If you can zip up the archived model folder and share it, I may be able to take a look.

My word is final

User avatar
Lig789
Posts: 3
Joined: Tue Aug 18, 2020 2:25 pm
Has thanked: 1 time

Re: Converting model to TF2.x mask problem

Post by Lig789 »

torzdf wrote: Thu Aug 20, 2020 11:27 am

FYI: The mask reverting to "extended" issue was just a GUI display issue and has been fixed in the latest update.

I am still looking at the other issue.

Thank you for updates, but there are new problems too. In GUI train tab, there used to be GPU saving section, and it is missing. Other problem is obviously posted above .json, I depend on GPU saving but I can not run without it.

For the time being, until these bugs are fixed, can I run FS v1 in parallel to FS v2 without influencing dependencies and packages (python38 in roaming folder, in anaconda naviagtor or miniconda3 method) which method is best to use both versions in same pc ?

Code: Select all

git clone --depth 1 https://github.com/deepfakes/faceswap.git

Using this I get latest but How to get FS V1 ? into the link above. or should I use archive installer with miniconda 3 while FS v2 running in Anaconda naviagtor ?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

We cannot currently bring the 2 VRAM saving options across to Tensorflow 2 as they are fundamentally incompatible.

However, you can enable "Mixed Precision" (In Settings > Train Options) to get a similar (or even better) level of VRAM savings. Try this, and see if it resolves your VRAM issues. Unfortunately this can only be set for new models at the moment (This is a Tensorflow, not a Faceswap limitation).

The best way to have both versions of Faceswap installed are to use the main installer for version 2, and the version 1 installer (https://github.com/deepfakes/faceswap/r ... tag/v1.0.0) for Version 1. Just make sure to select a different folder name and environment name for each of the installs.

My word is final

User avatar
Korben
Posts: 26
Joined: Wed Aug 19, 2020 3:17 pm
Has thanked: 2 times
Been thanked: 3 times

Re: Converting model to TF2.x mask problem

Post by Korben »

torzdf wrote: Thu Aug 20, 2020 3:00 pm

No, it should continue from where it left off, but if it's LIAE architecture, then it may be problematic....

If you can zip up the archived model folder and share it, I may be able to take a look.

I was able to reproduce it.
In faceswapV1 I make a Dfl-H128 model with no mask and then try to upgrade and it just ignores the previous progress. If I make it using a mask it seems to upgrade and resume correctly.

A small 2k iteration sample
removed link after bug was fixed

Last edited by Korben on Mon Sep 07, 2020 1:18 am, edited 1 time in total.
User avatar
Korben
Posts: 26
Joined: Wed Aug 19, 2020 3:17 pm
Has thanked: 2 times
Been thanked: 3 times

Re: Converting model to TF2.x mask problem

Post by Korben »

am I the only one with this problem? Models all resuming at correct iteration but loss starts from scratch.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

I have not had a chance to look at this yet. I may get a chance today, and will let you know if so.

My word is final

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Converting model to TF2.x mask problem

Post by torzdf »

There was a bug in dfl_h128 update code. This has been fixed in latest commit.

My word is final

User avatar
Korben
Posts: 26
Joined: Wed Aug 19, 2020 3:17 pm
Has thanked: 2 times
Been thanked: 3 times

Re: Converting model to TF2.x mask problem

Post by Korben »

torzdf wrote: Sun Aug 30, 2020 10:44 pm

There was a bug in dfl_h128 update code. This has been fixed in latest commit.

I can confirm the new update fixes the issue I had.

Locked