Browsing datasets with masks

[BrassNucks]

Hi guys, I'm using Bisenet-fp for all my swaps these days, and have a good idea of what I'm doing now but have one real problem with troubleshooting my datasets.

How do we browse the extracted faces with masks intact (overlaid, like in the training session)? The older forum posts state that masks are in the alignments file but I know more recently they're all bound up in the PNG metadata. I checked the PNGs and they do not have an alpha channel, so I looked to the metadata and found the mask encoded in there, but have no idea how to browse them.

The reason I ask is that there are many faces that pop up in training with horribly wrong masks and generally terrible source images. The B face tries to adapt to those images and I know it's fouling my training session, so I cancel it and try to manually sift through the thousands of images to delete them.

So really, my question is twofold.
1) How can we browse our dataset folder in a similar fashion as the training previews, i.e. with the red mask overlaid on the image, and
2) If I had to, how can I decode the mask code inside the PNG to make my own viewer in Python etc?

Thanks so much for any info or tips

[torzdf]

1) How can we browse our dataset folder in a similar fashion as the training previews, i.e. with the red mask overlaid on the image, and

Manual tool. Lets you view and edit masks.

2) If I had to, how can I decode the mask code inside the PNG to make my own viewer in Python etc?

You can export masks with the mask tool, and also import (custom) masks with the mask tool.

Option 1 will be easier as masks are not stored at the same size/padding as the faces.

The reason we don't use the alpha channel is because at least 2 masks are stored in the PNG header (extended and components), plus any other masks that you choose to select, so alpha does not give us enough space to store all masks. Also, it is easier to browse facesets when they are unmasked,

Masks in the png header are zlib compressed.

[BrassNucks]

Thanks torzdf, that's what I tried but always get this when I select the faces folder:

04/10/2024 11:30:57 ERROR The input folder '\\blahblah\extracted512' contains extracted faces.
04/10/2024 11:30:57 ERROR The Manual Tool works with source frames or a video file, not extracted faces. Please update your input.

... which confuses me as that's where the masks are.

Should I just re-extract the faces from the extracted faces folder again (extract the extractions) to get an updated alignments file for all the faces? The faces folder has extractions from several different sessions typically, some from HD video etc. Or should I regenerate the masks again with the tool in the Mask panel? It just seems like an unnecessary step at the moment, so I'm trying to figure out where I'm going wrong.

[BrassNucks]

Masks in the png header are zlib compressed.

Thank you this is helpful, maybe I can extract the mask this way and make a quick and dirty faceset browser. I know you're extremely busy so it would be low on your priority list to ask for a new Browse panel

[torzdf]

Thanks torzdf, that's what I tried but always get this when I select the faces folder:

04/10/2024 11:30:57 ERROR The input folder '\\blahblah\extracted512' contains extracted faces.
04/10/2024 11:30:57 ERROR The Manual Tool works with source frames or a video file, not extracted faces. Please update your input.

... which confuses me as that's where the masks are.

The Manual tool is meant for working with frames. This is where you should be working prior to generating your final faceset. Think of frames + alignments file as the master document. The facesets themselves are entirely disposable and can be regenerated from your master document at any time.

Masks are stored in the alignments file. A copy of the information stored in the alignments file is saved to the PNG header when you generate your faceset... But it is just that. A copy.

Should I just re-extract the faces from the extracted faces folder again (extract the extractions) to get an updated alignments file for all the faces? The faces folder has extractions from several different sessions typically, some from HD video etc. Or should I regenerate the masks again with the tool in the Mask panel? It just seems like an unnecessary step at the moment, so I'm trying to figure out where I'm going wrong.

No.

The alignment tool will allow you to regenerate you alignments file if you no longer have it. Do that, and then go back to use the manual tool with the original frames. Once you've cleaned your masks, only then is it time to generate your faceset. You can do this either from the manual tool (convenient) or from the alignments tool (faster).

[torzdf]

Masks in the png header are zlib compressed.

Thank you this is helpful, maybe I can extract the mask this way and make a quick and dirty faceset browser. I know you're extremely busy so it would be low on your priority list to ask for a new Browse panel

The manual tool will never be adapted to work with facesets for the reasons described above. They are entirely disposable.

An example: You generate a faceset, you then clean up the masks in that faceset for training. You then want to run a convert against the source your faceset was generated from. Problem... you still have the unfixed masks, because you only fixed them in your training faceset, and not in the master document that created that faceset (ie, the alignments file you will be using for convert)

[BrassNucks]

Masks are stored in the alignments file. A copy of the information stored in the alignments file is saved to the PNG header when you generate your faceset... But it is just that. A copy.

Thanks for your replies torzdf, I think I understand where I need to adjust my workflow now. I didn't realize the copies were put into the PNG headers, that clarifies things. It's possible there are conflicting masks in the alignments file vs the current PNG in the master folder so I have some direction now where to tidy things up.