Video Source Consolidation Workflow

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Video Source Consolidation Workflow

Post by gravitation »

I'm looking to make this process as optimal as possible. I have decided to try the following and I wanted to confirm some presumptions about it before I commit to this.

I have a collection of small clips, over 30+. Instead of using each video as an individual source, I will use the video tool to extract all of them into frames, and then generate a single combined video so that I would have a single source for all those videos. I will then proceed with the usual extraction, sorting, filtering, aligning, etc.

My presumption is that the alignment file generated for that single source video keeps track of each extracted frame by mapping it to the timestamp of the video. With this presumption, I can then assume that say later down the road, I make a few more clips and then extract them along with that single source, and then append the new clips so that single source video. By doing so (and make sure that the generated video keeps the same file name), the alignment file associated with it should not break because of the mapping to the time stamps.

Then all I would need to do is run another extraction but with Skip Existing Faces. Process those new frames, add them into the already processed batch, and then Remove-Faces to sync the alignment to the video.

Is this plausible? Or will I run into unforeseen issues?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Video Source Consolidation Workflow

Post by torzdf »

Yeah, I don't think that would work.

You can try, but don't expect success. The process does not track by time stamp, but by "image". So for a folder of images, that would be the filename of each image. For a video that is each frame.

My word is final

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Video Source Consolidation Workflow

Post by bryanlyon »

I think the big question is WHY are you trying to do it in this way? Are you trying to convert it all at once? Or are you using this collection for training? If the training, you don't need to merge the .fsa files, you just put the files you want to train with in a single directory.

If it's for conversion, I would not recommend this way, though it's possible it could work. There are a lot of potential problems it can raise.

Far better to try explaining why you are trying to do this, there might be a much better and easier (supported) way to accomplish what you're trying.

User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Re: Video Source Consolidation Workflow

Post by gravitation »

...is WHY are you trying to do it in this way?

Yes, it is because I want to perform the extraction process in as few steps a possible. From what I understood, you need to extract every 5-15 frames on videos in order to build your Training Set Portfolio. And then when you are ready to convert, you'll need to extract every frame for that. So for solely just training extraction, since I have over 30+ clips now and more are coming, I figured I'd consolidate them into one. Extract, clean up irregularities and alignments, and then re-extract them into a training set.

If I get 10 more clips say, and my consolidated video isn't too big, I can just append those 10 clips to that and extract using Skip Existing Faces and append them to the alignments.

The process does not track by time stamp, but by "image".

Here although my assumption was incorrect, the fact that it catalogs them by frame doesn't change my presumption that by appending new frames, the alignment file will not be broken. When I test this theory out, I will post the results.

It may seem like I'm adding extra work, but I think I am saving myself work in the long haul. I could not find a way to use the Tools to extract all videos and consolidate them as it needs a single video reference per generation. I also couldn't find any free good software to recommend for anyone else who wants to use this method either. (I tried about 10 different types and either they had watermarks, incorrect cropping aspect ratios, couldn't read certain file types, etc). So I ended up using my Sony Vegas Pro software to do it. It was very quick, I drop all the videos in, scrub through them at 4x speed, and cut out any extra frames that are unnecessary (or repetitive).

Then I hit render, out comes a single video, and then I extract that as a single source.

If it's for conversion, I would not recommend this way...

Here I agree with you 100%. On conversions I think I will stick to each clip individually just so the fine-tuning doesn't seem overwhelming.

Also, this is kind of veering off-topic: I'm already in the training phase and creating a model. The Example B I would like to reuse against other Example A's later. Is there a way to "jump-start" the training on a different Example A using Example B's previous model data? Or does each Ex A and B pair need to start from scratch?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Video Source Consolidation Workflow

Post by bryanlyon »

What I'd suggest is to learn batch scripting (or bash scripting in Linux) which can easily be used to extract multiple files at once.

On reusing part of the model: You can save the encoder for reuse, but the decoders work best from scratch.

User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Re: Video Source Consolidation Workflow

Post by gravitation »

You can save the encoder for reuse, but the decoders work best from scratch.

How would I go about doing that? Is it as simple as providing a h5 in the Load Weights under the Train > Model section?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Video Source Consolidation Workflow

Post by bryanlyon »

gravitation wrote: Tue May 11, 2021 1:30 am

You can save the encoder for reuse, but the decoders work best from scratch.

How would I go about doing that? Is it as simple as providing a h5 in the Load Weights under the Train > Model section?

Yup, exactly ;)

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Video Source Consolidation Workflow

Post by torzdf »

You will want to freeze the encoder too, whilst the decoders catch up.

My word is final

User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Re: Video Source Consolidation Workflow

Post by gravitation »

So when I start a new pair, I add the weights and select freeze. Is there any point I’ll need to disable freeze? Or will the training handle this intuitively?

Also, does this mean that by reusing Model B weights on the next model A’s it will build on top accumulatively? Meaning each time I reuse the weights, it will jump start even further then before?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Video Source Consolidation Workflow

Post by torzdf »

Probably best to start another thread for this as it's veering off topic and could be useful info.

My word is final

User avatar
gravitation
Posts: 15
Joined: Fri May 07, 2021 6:14 am
Has thanked: 10 times
Been thanked: 1 time

Re: Video Source Consolidation Workflow

Post by gravitation »

I agree. Can you help me by initiating it because I don’t know where to post this. I’d be happy to engage in it. :)

Locked