Possibility of Turning Off "Learn Mask" on an Active Training Model

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Possibility of Turning Off "Learn Mask" on an Active Training Model

Post by ianstephens »

We have been playing with different masks and settings recently.

We have a model that's at 1M iterations with "learn mask" enabled.

We'd like to switch off the learned mask and allow the trainer to work with the original mask in the PNG files.

Is this possible with an active training session?

We simply tried to set

Code: Select all

"learn_mask": false

in the existing JSON file but it hit us back with an error when we tried to continue training on the model.

Is this something that can be done?

Thanks for any help, suggestions, and tips in advance.

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Possibility of Turning Off "Learn Mask" on an Active Training Model

Post by bryanlyon »

Learn Mask changes the model itself in order to let it learn the mask. Unfortunately once a model is started, it can't be changed. You're going to be stuck with the mask being learned, that said, once convert comes up, you can just not use the predicted mask.

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Possibility of Turning Off "Learn Mask" on an Active Training Model

Post by ianstephens »

Thanks for that [mention]bryanlyon[/mention] - I did figure that was the case as it was throwing up errors about the model itself (can't remember the exact message) when we tried to manually edit the JSON config.

The issue we came across is that when using the original mask in the preview tool (after training with predicted), the model had only covered details learned what was inside the predicted mask.

And when using the predicted, there was a noticeable lack of coverage of the face area (the mask had been made smaller by the prediction algo).

Either way, it looks bad but the original mask is better (although has shading/black areas at the edges where the predicted learned model cuts off).

If you have any tips/suggestions that would be great - we're 1.2M iter. in and about to switch on nowarp.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Possibility of Turning Off "Learn Mask" on an Active Training Model

Post by torzdf »

There is a fix for this, but I can only tell you top level, I can't take you through this step-by-step as it's involved.

It is possible to create a new model which does not have "learn mask" enabled, and then port your required weights from your existing model into this new model.

If you can find someone with intermediate to advanced experience of tf/keras, they should be able to do this for you.

Alternatively, I'm currently taking contracts ;)

My word is final

Locked