Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Discussions about research, Faceswapping and things that don't fit in the other categories here.


User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

I made several adjustments to the alignment file including hundreds of manual mask updates on frames. I saved it, but noticed that it didn't seem to use the mask in the training phase. I assumed it would sort itself out later but when I try to convert, it says there are exactly 2514 components and 2514 extended masks (weird symmetry). It says I should use the Mask tool, but I don't want to overwrite my changes. What is the correct thing to do in this instance?

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by torzdf »

When you edit masks in the manual tool, you need to re-extract the faces for training so that the updated masks get stored in the PNG Header data.

With the mask tool, if you select the 'missing' option, it will only update masks where a mask does not pre-exist so your edits should be safe.

My word is final

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

Oh bummer. so 125k training iteration s down the drain :P

But thanks! Now I know.

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

torzdf wrote: Mon Jan 16, 2023 11:11 am

When you edit masks in the manual tool, you need to re-extract the faces for training so that the updated masks get stored in the PNG Header data.

With the mask tool, if you select the 'missing' option, it will only update masks where a mask does not pre-exist so your edits should be safe.

Ok. That didn't work. I got frustrated and deleted the faces and training folder and extracted each frame (luckily it's a short video so only about 1000). I manually adjusted hundreds of faces and did the masks too. I used the extract function to re-fill the faces folder and am doing training now, but the masks aren't correct. In the previews, I don't see the mask changes I made.

I assume the alignments and masks are stored in the alignments file itself at least... I assume that means it will work for conversion, but it doesn't seem to care about masks for training.

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by torzdf »

The masks not getting updated + used is not something I have heard of before. There might be a bug, but more likely you are making a mistake somewhere in the process.

  • Any changes made in the manual tool are saved into the alignments file
  • Any changes made to the alignments file means the faces have to be regenerated (by re-extracting in either the manual tool or the alignments tool)
  • Any change made to the landmarks, when in the manual tool, will delete all masks associated with that face (with the exception of components and extended), so these need to be regenerated with the mask tool

The only other thing I can think is that you have selected the incorrect mask type in the training settings.

My word is final

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

Hold up. If I fix alignment on ONE frame, ALL the masks for that face are destroyed? I'm guessing that's not what you meant.

Either way, I tried to update alignments and mask for hundreds of face and then extracted them with the manual tool after I was done and trained on it, but the mask adjustments I made didn't show.

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by torzdf »

unkempt wrote: Tue Jan 17, 2023 10:59 pm

Hold up. If I fix alignment on ONE frame, ALL the masks for that face are destroyed? I'm guessing that's not what you meant.

That's exactly what I mean. Just to be clear, I'm talking about all of the neural network generated masks for that particular face in that particular frame.

When you alter the landmarks of the face (be it through the landmark editor, bounding box editor or extract box editor) you alter the aligned face image that will be extracted to disk. As the neural-network masks were generated from the original image, they are no longer valid as that image has changed. There is not choice but to delete the masks because we can't generate new ones (maskers are not loaded in the manual tool) and leaving the mask in place would mean having a mask that does not correspond to the new aligned face.

The components and landmarks masks are not impacted, as they are just re-generated from the new landmarks after each update.

This is why it is not worth generating masks at extract time. It's better to generate the mask you want after you have sorted, deleted and cleaned up landmarks.

Either way, I tried to update alignments and mask for hundreds of face and then extracted them with the manual tool after I was done and trained on it, but the mask adjustments I made didn't show.

Honestly, I've done this loads of time and never had an issue. I'm not saying there isn't a bug, but I think its more likely you're going wrong somewhere in your extraction workflow.

If you could go through the steps you take to get you to the position that you find yourself in, that may be helpful.

My word is final

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

Ok, no worries. I already knew the mask was destroyed in a single FRAME when adjusting alignments.

1000% it's my fault I'm sure.

I'm going to start over again using this process:

1) Extract "to be replaced" faces from video at a skip level of 12 or so. S3Fd, Fan, no mask, Hist, Realign, Identity. Everything else default.
2) Because the video is short (53 seconds), I only have about 86 faces in total. If I understand correctly, there's no point in messing with them since I'm going to use the manual tool to correct alignments, remove faces, then EXTRACT them back to the folder with the new alignments.
3) Manual tool. Do everything I just said, end up with new folder of faces extracted AFTER my alignment fixes. Q: I assume that the lines should hug the cheeks and lines of the face pretty closely, yes? Or is there some measure of wiggle room there?
4) Extract faces to a folder from the manual tool.
5) Use mask tool to apply masks to the alignments file. It ALSO extracts photos so... did I wast time doing the extraction on the manual step?
6) Open manual tool again and do masks this time. Extract when done (for a third time?)

Then train using the images folder with the corrected alignments and masks. Yes?

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

Using the above, I can see the masks not being applied to the trainer. In another thread, someone said to extract using the alignments tool so I'm deleting the faces for the 4th time to extract with that tool isntead to see if it makes a difference.

Narrator: It made no difference.

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by torzdf »

Ok, I have now tested this extensively end to end...

My workflow:

  1. Extract all faces, no masks
  2. Sort faces and delete those not required
  3. Run alignments tool, remove-faces
  4. Run manual tool + fix up alignments
  5. Run mask tool to add bisenet-fp masks, using the original video as source, and the updated alignments file. (Tested with both process all and missing)
  6. Fire up manual tool to review masks (they are all there and my earlier edits to clean up alignments are also present)
  7. Deliberately edit some masks to be bad so I can identify if the edits come through:
    mask_test.jpg
    mask_test.jpg (235.24 KiB) Viewed 5859 times
  8. Re-extract the faceset with the alignments tool, so I have generated PNG images with the edited bisenet-fp mask included
  9. Run a sanity test with the mask tool. Input = newly extracted faces, process = output (i.e. let me review the masks). Confirm output is correct:
    mask_confirm.jpg
    mask_confirm.jpg (90.5 KiB) Viewed 5859 times
  10. Select the bisenet-fp mask in training settings, so that the mask is used.
  11. Start training, and see the edited masks are included:
    mask_train.jpg
    mask_train.jpg (183.23 KiB) Viewed 5859 times

So, at this point, I'm not sure what I can add. As demonstrated, the process is working as expected (at least in my testing).

re:

Q: I assume that the lines should hug the cheeks and lines of the face pretty closely, yes? Or is there some measure of wiggle room there?

I'll copy and paste a recent post I made on Discord which relates to the (point 4 specifically covers your question)

So a few high-level tips.

  1. Use bisenet-fp. It is by far and away the best mask in all circumstance. Particularly for handling obstructions
  2. Don't worry about editing individual landmark points. That only exists because there was demand for it. The reality is that it is very rare that you will need that fine grained control
  3. Don't fix masks until you have fixed your alignments. Any edits to the mask will get instantly erased (as will all NN generated masks for that face in that frame). Any landmarks edit, in any way, instantly invalidates masks
  4. If you are not using the 'extended' or 'components' mask (you aren't) then you only need to worry about the landmarks in the 'core' area. (that is eyes, nose mouth, eyebrows) as these are the points that are actually used to align the image. The jawline is entirely unused, so you can probably skip faces where the jawline 'flies away'
  5. Don't spend an inordinate amount of time fixing up training data. It should be generally correct, yes, but the model is seeing millions of faces, so a few incorrect alignments won't really hurt anything. It is easier to just delete bad training images than fix them all (unless data quality is at a premium)
  6. Fixing for convert is far more important.
  7. If you have a face which is particularly hard to get a 'lock' on the landmarks, first try switching between 'clahe' and 'hist' normalization in the manual tool. This can sometimes help. If that doesn't work, copy the alignments from the previous/next good face and use the "Extract box editor" (green box) to shift the good face's alignments to be an approximate match with the current face

I'm not sure what this means:

5) Use mask tool to apply masks to the alignments file. It ALSO extracts photos so... did I wast time doing the extraction on the manual step?

Mask tool does not extract anything. It just calculates masks and puts them into the alignments file and/or PNG header. However, I would say your second extraction is unnecessary, yes. I only extract twice. 1) The initial faceswap extraction. 2) After making all of my edits for alignments + masks. You only need faces for training, so it is only worth re-extracting them when they are final and ready to be used.

A few workflow things to check:

  1. Are you using the correct alignments file? Whenever an edit is made to the alignments file (be it using 'remove-faces' in the alignments tool, or edits in the manual tool), then a backup is taken (so you can roll back if you screw things up). The current alignments file will always be named <video>_alignments.fsa. Anything named with numbers at the end (eg <video>_alignments_20230120_121537.fsa) is a timestamped back-up (pre-edit) and can be safely deleted if you are happy with your edits. This particular example is timestamped 02 Jan 2023 - 12:15.37
  2. Are you definitely selecting the correct mask within the training options?
    mask_settings.jpg
    mask_settings.jpg (125.89 KiB) Viewed 5859 times

My word is final

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

First: you are my hero. Thank you so much for the detailed walkthrough! That was a lot of effort on your part and you answered SO MANY QUESTIONS!

The manual tool tip helps SO much! I was wasting a lot of time with jawline things, but I'll focus more on the other features going forward. I'll try your tips for switching up the hist etc and copying faces.

As for training masks, I never knew about that. I would have thought it would be in the training menu itself and not the global settings. I'll try this all and see what kind of results improvements I can get

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

On a side-note, what is the point of only doing a handful of faces for training? If you're going to end up extracting every frame and manually fixing them at some point, why not before the training step?

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

Also, why the "remove faces" step? If you use the manual tool and delete the faces you don't want, isn't that the same thing?

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by torzdf »

unkempt wrote: Fri Jan 20, 2023 7:44 pm

On a side-note, what is the point of only doing a handful of faces for training? If you're going to end up extracting every frame and manually fixing them at some point, why not before the training step?

I'm not sure what you mean.

Your training sets (A and B) will both come from a multitude of sources. 1 of the A sources will also be being used for convert (let's call this swap_source)
For all files other than swap_source you extract at -een 10/25/50 whatever is a sensible figure based on the source framerate and the length and number of sources that you have. For these sources, you will most likely just do a sort, delete unwanted faces/misaligns, clean the alignments file and move on.

For the swap_source file you will generate every frame (-een 1), you will do the same sort/delete/clean steps as before, but you will use the manual tool to clean the file up for convert.

From this cleansed alignment file, you can re-extract a sample of these faces to use for training (i.e alignments tool - extract job -een 10/25/50 etc)

unkempt wrote: Fri Jan 20, 2023 7:48 pm

Also, why the "remove faces" step? If you use the manual tool and delete the faces you don't want, isn't that the same thing?

Because it is far, far quicker to sort faces 'by face' and delete them directly from the hard disk than it is to delete them 1 by 1 in the manual tool

My word is final

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

Ok, so you're saying you would actually:

  • extract all frames from a source file (that have faces anyway)
  • Fix all alignments in the manual tool
  • Mask tool to add masks to alignments file
  • Fix all masks in the manual tool
  • Use the alignments tool to extract a certain number of frames (like every 15 for 30fps).
  • train on the extracted frames (because training on thousands is bad... for some reason?)
  • When training is done, convert with the alignments file that already has info for all face frames in the video.

Is that basically right?

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by torzdf »

Pretty much, except:

unkempt wrote: Sun Jan 22, 2023 6:23 am
  • train on the extracted frames (because training on thousands is bad... for some reason?)

No. Lots of data is good. The more varied data you can provide, the better the model will perform.

Lots of extremely similar data does not help at all though. Just causes bias towards the similar data, hence why we do not extract at every frame for training. Consecutive fames will have extremely similar faces.

My word is final

User avatar
unkempt
Posts: 81
Joined: Wed Dec 28, 2022 2:09 pm
Has thanked: 1 time
Been thanked: 9 times

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by unkempt »

Thank you!

User avatar
npittas
Posts: 1
Joined: Tue Jan 31, 2023 4:00 pm

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by npittas »

Quick note on extracting.
You can save time by not saving the images and not create the masks while extracting and just use the alignment file that will be generated to do all the sorting/alignment fix/masking, and then finally extracting the usuable frames for training.
Just use the "Skip Saving Frames" on the Extract Tab. It saves alot of extraction time and space.
You still need to designate a folder where your non saved files will rest, but it will be empty after the extraction, so your next steps should be:
Alignment fixes in manual tool,
Alignment removals, if needed, from the Tools/Alignment Tab
Mask creation from Masks Tab
Mask editing in Manual tool
Extract from Alingments tab
and finally
Sort from Sort tab and delete the bad frames.
Since the alignments are saved in the png headers and the alignment file, you can now use the extracted frames for training and the alignment file for convertion

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 215 times
Contact:

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by bryanlyon »

I find sorting first saves more time than writing it out later takes. But I guess your mileage may vary.

User avatar
lighting
Posts: 11
Joined: Thu Mar 02, 2023 6:54 pm
Has thanked: 1 time
Been thanked: 1 time

Re: Adjusted masks during manual adjustment - didn't seem to work with training - error on conversion.

Post by lighting »

torzdf wrote: Fri Jan 20, 2023 12:54 pm

The jawline is entirely unused, so you can probably skip faces where the jawline 'flies away'
Don't spend an inordinate amount of time fixing up training data. It should be generally correct, yes, but the model is seeing millions of faces, so a few incorrect alignments won't really hurt anything. It is easier to just delete bad training images than fix them all (unless data quality is at a premium)

Thanks for that. I think it worth to be added to FAQ.

Post Reply