import a mask made with XSeg?

The Extraction process failing on you, and you aren't getting an error back with clear instructions? Tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Extraction process. If you want to get tips, or better understand the Extract process, then you should look in the Extract Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
gregtorn
Posts: 12
Joined: Fri Aug 28, 2020 11:32 am

import a mask made with XSeg?

Post by gregtorn »

is there a way to import mask made with XSeg? the mask that is made after extracting is bad and changing mask for every frame manually is a pain in the ass(specially if its 20000+ frames) plus sometimes you even cant do it as it shows error.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: import a mask made with XSeg?

Post by torzdf »

Short answer, no. But would welcome PRs if people would find it useful.

However, we have not had complaints about the mask before so 2 questions:

  1. Which mask plugin are you using>?
  2. What does this mean? sometimes you even cant do it as it shows error. Can you be more specific?

My word is final

User avatar
gregtorn
Posts: 12
Joined: Fri Aug 28, 2020 11:32 am

Re: import a mask made with XSeg?

Post by gregtorn »

torzdf wrote: Tue Oct 25, 2022 5:45 pm

Short answer, no. But would welcome PRs if people would find it useful.

However, we have not had complaints about the mask before so 2 questions:

  1. Which mask plugin are you using>?
  2. What does this mean? sometimes you even cant do it as it shows error. Can you be more specific?

i use bisenet-fp. 99% of the time it masks space that isnt face. if face is slim it masks as face areas around face. if face is bigger it for some reason masks not full face. and if i get to start a manual mask that means i need to manually change mask for ALOT of frames. in this case for 32+ thousands on destination and 3+ thousands on source.
with XSeg you can mask 15+ frames on source 15-20 frames on destination and let it work for an hour to get a perfect mask(but i cant use it with deepfacelab because for some reason python crashes when i try to use it.)

2) i get this error on most of the videos when i try to go to manual
10/25/2022 21:30:35 ERROR There is a mismatch between the number of frames found in the video file (32217) and the number of frames found in the alignments file (32218).
10/25/2022 21:30:35 ERROR This can be caused by a number of issues:
10/25/2022 21:30:35 ERROR - The video has a Variable Frame Rate and FFMPEG is having a hard time calculating the correct number of frames.
10/25/2022 21:30:35 ERROR - You are working with a Merged Alignments file. This is not supported for your current use case.
10/25/2022 21:30:35 ERROR You should either extract the video to individual frames, re-encode the video at a constant frame rate and re-run extraction or work with a dedicated alignments file for your requested video.
i reencode video. i recut video. i reextract faces and remask and no mater what i do i get this type of error on most of the videos. when i try to start manual in tools

the things i do for source and destination videos- i first extract, then go to tools>sort, then delete bad faces or no faces, then go to tools>alignments>remove-faces, then tools>mask to reapply the mask. then train.

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 177 times
Been thanked: 13 times

Re: import a mask made with XSeg?

Post by MaxHunter »

Your second point is super common, in fact, I literally just fixed the same issue.

I believe the answer is, no. You have to re-encode the movie. Personally I use, Handbrake (I believe it's still free.). I think Torz said it has something to do with it not being in a constant frame rate. I usually re-encode at 24 fps at a constant frame rate with Handbrake, then re-extract because I don't know if the alignment files will still match. You can also use the ffmpeg tool to extract the frames delete the bad frame and then re-merge the frames into a single film.

User avatar
gregtorn
Posts: 12
Joined: Fri Aug 28, 2020 11:32 am

Re: import a mask made with XSeg?

Post by gregtorn »

MaxHunter wrote: Tue Oct 25, 2022 9:38 pm

Your second point is super common, in fact, I literally just fixed the same issue.

I believe the answer is, no. You have to re-encode the movie. Personally I use, Handbrake (I believe it's still free.). I think Torz said it has something to do with it not being in a constant frame rate. I usually re-encode at 24 fps at a constant frame rate with Handbrake, then re-extract because I don't know if the alignment files will still match. You can also use the ffmpeg tool to extract the frames delete the bad frame and then re-merge the frames into a single film.

i did that. i tried megui i tried xvid4psp and even davinci. the video is constant 30 fps.
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 30.000 FPS
Color space : YUV

but its not only with this video. but still even if that worked manually remask about 36000 frames is insane.

i really hope devs will think about implementing XSeg.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: import a mask made with XSeg?

Post by torzdf »

gregtorn wrote: Tue Oct 25, 2022 6:41 pm

i use bisenet-fp. 99% of the time it masks space that isnt face. if face is slim it masks as face areas around face. if face is bigger it for some reason masks not full face.

I find this very strange, and not something I've seen reported or experienced before. As a general mask, bisenet-fp is not perfect, but it is generally very good. If you can post some images to demonstrate what you mean, that may be useful to getting to a resolution.

2) i get this error on most of the videos when i try to go to manual
10/25/2022 21:30:35 ERROR There is a mismatch between the number of frames found in the video file (32217) and the number of frames found in the alignments file (32218).
10/25/2022 21:30:35 ERROR This can be caused by a number of issues:
10/25/2022 21:30:35 ERROR - The video has a Variable Frame Rate and FFMPEG is having a hard time calculating the correct number of frames.
10/25/2022 21:30:35 ERROR - You are working with a Merged Alignments file. This is not supported for your current use case.
10/25/2022 21:30:35 ERROR You should either extract the video to individual frames, re-encode the video at a constant frame rate and re-run extraction or work with a dedicated alignments file for your requested video.
i reencode video. i recut video. i reextract faces and remask and no mater what i do i get this type of error on most of the videos. when i try to start manual in tools

Yes, this happens sometimes. Working with video files is hard. Whilst the general faceswap process is ok with just iterating from first frame to last frame, the manual tool must have precise frame accuracy. Video is not built for precise frame accuracy and there is a distinct lack of standards that can cause this kind of issue (videos not cropped on keyframes, 3:2 pull-down, variable frame rates etc. etc.). I have put in many things to try to mitigate these issues, but there are always instances where this occurs. Ususally because of bad editing/encoding of the source video. Re-encoding to a known frame rate can help with this.

My word is final

User avatar
gregtorn
Posts: 12
Joined: Fri Aug 28, 2020 11:32 am

Re: import a mask made with XSeg?

Post by gregtorn »

torzdf wrote: Thu Oct 27, 2022 10:21 am

I find this very strange, and not something I've seen reported or experienced before. As a general mask, bisenet-fp is not perfect, but it is generally very good. If you can post some images to demonstrate what you mean, that may be useful to getting to a resolution.

this the only 2 video that i managed to open on manual. <LINK REMOVED>

Yes, this happens sometimes. Working with video files is hard. Whilst the general faceswap process is ok with just iterating from first frame to last frame, the manual tool must have precise frame accuracy. Video is not built for precise frame accuracy and there is a distinct lack of standards that can cause this kind of issue (videos not cropped on keyframes, 3:2 pull-down, variable frame rates etc. etc.). I have put in many things to try to mitigate these issues, but there are always instances where this occurs. Ususally because of bad editing/encoding of the source video. Re-encoding to a known frame rate can help with this.

i tried reencoding in megui in xvid4psp and even in davinci resolve. not helping. most of the videos i have are 30 or 60fps.
i really hope XSeg will be implemented to this software. its easy to use and takes an hour or two to make a decent mask.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: import a mask made with XSeg?

Post by torzdf »

gregtorn wrote: Thu Oct 27, 2022 4:13 pm

this the only 2 video that i managed to open on manual. <LINK REMOVED>

On those images you posted, that is not the mask, it is the alignments (and they look broadly ok to me). The mask icon is the 5th one down on the left-hand side (and then you would need to select the 'bisenet-fp' mask from the right hand menu). The mask should show up as a red overlay

My word is final

User avatar
gregtorn
Posts: 12
Joined: Fri Aug 28, 2020 11:32 am

Re: import a mask made with XSeg?

Post by gregtorn »

torzdf wrote: Thu Oct 27, 2022 6:06 pm
gregtorn wrote: Thu Oct 27, 2022 4:13 pm

this the only 2 video that i managed to open on manual. <LINK REMOVED>

On those images you posted, that is not the mask, it is the alignments (and they look broadly ok to me). The mask icon is the 5th one down on the left-hand side (and then you would need to select the 'bisenet-fp' mask from the right hand menu). The mask should show up as a red overlay

looks like this.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: import a mask made with XSeg?

Post by torzdf »

Not sure what the issue is with that. That looks like it is masked perfectly. You have "include hair" set, but assuming that's what you want, I'm failing to see the problem.

My word is final

User avatar
gregtorn
Posts: 12
Joined: Fri Aug 28, 2020 11:32 am

Re: import a mask made with XSeg?

Post by gregtorn »

I painted face. as you can see it looks like the source face went a bit smaller and tilted to the left. you can see a black outline of source face on the right and that it went on the hairs near forehead on the left. i went to 250000 iterations. it got chin perfect but then the applied face tilted. this can be fixed with manually redoing alignments and mask(i tried on a 10 seconds video(it took few hours to redo on each frame)) but if a video is more then 10 seconds its hell. this is 17 min. it will take days or weeks fixing frame by frame. with out option of mask training. the face type of video A is same as video B. they are pretty much same size, same width, same shape.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: import a mask made with XSeg?

Post by torzdf »

Ok, this is why images are useful. The issue here isn't the bisenet-fp mask, per-se. It's a data/full face issue.

Most likely what is happening is that your "B" face has either a much narrower face than "A", or (more likely) your "B" face does not have enough variety in the training data and nearly always has hair framing the face. This hair is then 'learned" by the model (as it is always there in the data) and then gets represented in the swap. You may be able to solve this by adjusting the erosion settings in the 'preview' tool.

You can read more about this issue, why it occurs, and how you can potentially solve it here:
viewtopic.php?t=2078

My word is final

User avatar
gregtorn
Posts: 12
Joined: Fri Aug 28, 2020 11:32 am

Re: import a mask made with XSeg?

Post by gregtorn »

will take a read. but think about XSeg implementation =)

Locked