Masking

Discussions about research, Faceswapping and things that don't fit in the other categories here.


Locked
User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Masking

Post by gravitation »

Alright I think I got all the basics down. Now I'm ready to start taking it to the next level of realism. I have a feeling that I should wrap my head around the Masking aspect. I surfed the forums and didn't find much other than what each Masker does.

I was wondering if there is a more in-depth tutorial on it? Not on what it does, but geared more towards how to utilize it to get the most optimal results. From what I understood, you can use Masker techniques both for Converting and Training, but that's all I got.

What are the pro/cons for:
o Using Masker for Training?
o Using all 3 Maskers for Training?
o Using Masker for Converting?
o Using all 3 Maskers for Converting?

What are the general principles of which Masker to use and when? Can I just select all three and hit extract (or update the alignment file using the Mask tool)?

Please help, the detailed the explanations the better!

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Masking

Post by torzdf »

gravitation wrote: Fri May 14, 2021 12:40 am

What are the pro/cons for:
o Using Masker for Training?

This focuses training on the important part of the image. Parts that are masked in will be trained on, parts that are masked out will be ignored

o Using all 3 Maskers for Training?

You can only use 1 mask at a time. If you mean which masks to use, then use the one that works best for your situation. This will depend on your scene

o Using Masker for Converting?

Honestly, the easiest thing would be to run convert without a mask, and you'll see what it does (a picture says 1000 words etc, etc).

o Using all 3 Maskers for Converting?

See the training answer.

What are the general principles of which Masker to use and when? Can I just select all three and hit extract (or update the alignment file using the Mask tool)?

Yes. The alignments file can store any amount of masks that you want.

My word is final

User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Re: Masking

Post by gravitation »

...then use the one that works best for your situation. This will depend on your scene

What if my scene has frontal open (with and without obstructions) and side profile (with and without obstructions).
I am still stuck with picking only one mask? Or does this mean I have to break down my scene into multiple sources, apply appropriate masking, perform convert, and then combine them?

That is for converting, but for training I pressume using Vgg-obstructed will be optimal?

I'm just trying to decide on which route to take because I really don't want to perform training using the wrong mask (or any mask if needed) only to find out after a couple days that I should've taken a different route ;)

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Masking

Post by torzdf »

A mask is a mask. I'll be frank and say none of the existing masking solutions are optimal, and all fail in certain instances.

You will want to edit masks with the manual tool for convert, so really you want to use the mask which gives you the least amount of editing to do.

My word is final

User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Re: Masking

Post by gravitation »

I see. I got about 60,000 frames all cleaned and ready to go. It’s why I’m asking all this about masks.

My game plan now: I will not train with masks, I’ll just simply filter out any obstructed faces for the training set.

For convert, I’ll use the ogg-obstructed and review the frames again to fix the masks. That I think will be a pain in the ass though.

The Manual tool is awesome btw, the ability to copy previous/next frame is a life saver. I wish there was more tools though, like the ability to shift click to add landmark points to selection. Ability to rotate your selections instead of the entire landmark. The ability to rotate on the Z axis (in 3D Space). And a bunch more things I can suggest.

The alignment frame updates the preview once you click and let go. I found a trick to quickly align many frames in a row, but the previews don’t update because of this, even though they actually update. What you do is click and hold your auto aligner, and then press X to go to next frame. And if you do this while dragging the mouse around very slightly you can very quickly crunch out frame alignments. Then go back and single click to update preview. It would be nice if you are holding down the mouse and click X, the preview would update on frame change.

Another suggestion I might add is to change the shortcut keys for Manual Tool. Currently they are all over the place and either you have to drag your mouse to click instead of moving your keyboard hand all the way to get to the function buttons or delete. Might I suggest putting them all within reach of your left hand so your mouse is focused strictly on the frame alignments? It would drastically speed up the manual editing process. I know this because I have a mouse with many buttons and use macros for the short cuts.

Anyways sorry for the babble. Hopefully this insight will be useful.

Locked