Page 1 of 1

Video Source Consolidation Workflow

Posted: Sun May 09, 2021 6:06 pm
by gravitation

I'm looking to make this process as optimal as possible. I have decided to try the following and I wanted to confirm some presumptions about it before I commit to this.

I have a collection of small clips, over 30+. Instead of using each video as an individual source, I will use the video tool to extract all of them into frames, and then generate a single combined video so that I would have a single source for all those videos. I will then proceed with the usual extraction, sorting, filtering, aligning, etc.

My presumption is that the alignment file generated for that single source video keeps track of each extracted frame by mapping it to the timestamp of the video. With this presumption, I can then assume that say later down the road, I make a few more clips and then extract them along with that single source, and then append the new clips so that single source video. By doing so (and make sure that the generated video keeps the same file name), the alignment file associated with it should not break because of the mapping to the time stamps.

Then all I would need to do is run another extraction but with Skip Existing Faces. Process those new frames, add them into the already processed batch, and then Remove-Faces to sync the alignment to the video.

Is this plausible? Or will I run into unforeseen issues?


Re: Video Source Consolidation Workflow

Posted: Mon May 10, 2021 10:24 am
by torzdf

Yeah, I don't think that would work.

You can try, but don't expect success. The process does not track by time stamp, but by "image". So for a folder of images, that would be the filename of each image. For a video that is each frame.


Re: Video Source Consolidation Workflow

Posted: Mon May 10, 2021 6:25 pm
by bryanlyon

I think the big question is WHY are you trying to do it in this way? Are you trying to convert it all at once? Or are you using this collection for training? If the training, you don't need to merge the .fsa files, you just put the files you want to train with in a single directory.

If it's for conversion, I would not recommend this way, though it's possible it could work. There are a lot of potential problems it can raise.

Far better to try explaining why you are trying to do this, there might be a much better and easier (supported) way to accomplish what you're trying.


Re: Video Source Consolidation Workflow

Posted: Tue May 11, 2021 12:42 am
by gravitation

...is WHY are you trying to do it in this way?

Yes, it is because I want to perform the extraction process in as few steps a possible. From what I understood, you need to extract every 5-15 frames on videos in order to build your Training Set Portfolio. And then when you are ready to convert, you'll need to extract every frame for that. So for solely just training extraction, since I have over 30+ clips now and more are coming, I figured I'd consolidate them into one. Extract, clean up irregularities and alignments, and then re-extract them into a training set.

If I get 10 more clips say, and my consolidated video isn't too big, I can just append those 10 clips to that and extract using Skip Existing Faces and append them to the alignments.

The process does not track by time stamp, but by "image".

Here although my assumption was incorrect, the fact that it catalogs them by frame doesn't change my presumption that by appending new frames, the alignment file will not be broken. When I test this theory out, I will post the results.

It may seem like I'm adding extra work, but I think I am saving myself work in the long haul. I could not find a way to use the Tools to extract all videos and consolidate them as it needs a single video reference per generation. I also couldn't find any free good software to recommend for anyone else who wants to use this method either. (I tried about 10 different types and either they had watermarks, incorrect cropping aspect ratios, couldn't read certain file types, etc). So I ended up using my Sony Vegas Pro software to do it. It was very quick, I drop all the videos in, scrub through them at 4x speed, and cut out any extra frames that are unnecessary (or repetitive).

Then I hit render, out comes a single video, and then I extract that as a single source.

If it's for conversion, I would not recommend this way...

Here I agree with you 100%. On conversions I think I will stick to each clip individually just so the fine-tuning doesn't seem overwhelming.

Also, this is kind of veering off-topic: I'm already in the training phase and creating a model. The Example B I would like to reuse against other Example A's later. Is there a way to "jump-start" the training on a different Example A using Example B's previous model data? Or does each Ex A and B pair need to start from scratch?


Re: Video Source Consolidation Workflow

Posted: Tue May 11, 2021 12:44 am
by bryanlyon

What I'd suggest is to learn batch scripting (or bash scripting in Linux) which can easily be used to extract multiple files at once.

On reusing part of the model: You can save the encoder for reuse, but the decoders work best from scratch.


Re: Video Source Consolidation Workflow

Posted: Tue May 11, 2021 1:30 am
by gravitation

You can save the encoder for reuse, but the decoders work best from scratch.

How would I go about doing that? Is it as simple as providing a h5 in the Load Weights under the Train > Model section?


Re: Video Source Consolidation Workflow

Posted: Tue May 11, 2021 3:20 am
by bryanlyon
gravitation wrote: Tue May 11, 2021 1:30 am

You can save the encoder for reuse, but the decoders work best from scratch.

How would I go about doing that? Is it as simple as providing a h5 in the Load Weights under the Train > Model section?

Yup, exactly ;)


Re: Video Source Consolidation Workflow

Posted: Tue May 11, 2021 10:39 am
by torzdf

You will want to freeze the encoder too, whilst the decoders catch up.


Re: Video Source Consolidation Workflow

Posted: Tue May 11, 2021 2:52 pm
by gravitation

So when I start a new pair, I add the weights and select freeze. Is there any point I’ll need to disable freeze? Or will the training handle this intuitively?

Also, does this mean that by reusing Model B weights on the next model A’s it will build on top accumulatively? Meaning each time I reuse the weights, it will jump start even further then before?


Re: Video Source Consolidation Workflow

Posted: Wed May 12, 2021 12:13 am
by torzdf

Probably best to start another thread for this as it's veering off topic and could be useful info.


Re: Video Source Consolidation Workflow

Posted: Wed May 12, 2021 12:45 am
by gravitation

I agree. Can you help me by initiating it because I don’t know where to post this. I’d be happy to engage in it. :)