Using multiple videos (upd)

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Lepiloff
Posts: 10
Joined: Wed Jan 08, 2020 2:32 pm
Answers: 1
Has thanked: 6 times
Been thanked: 1 time

Using multiple videos (upd)

Post by Lepiloff »

[mod=torzdf]Note: The information within this topic is now outdated. Alignments files are no longer required for training facesets. See here for more information: viewtopic.php?f=25&t=1377[/mod]



My old question viewtopic.php?f=5&t=330 is closed and I had to create a new one to clarify some details.

I found that some messages have tips for using multiple video sources to improve the quality of the result.
(this viewtopic.php?f=6&t=322 and this viewtopic.php?f=6&t=74)

Is it possible to combine several videos (source1 and source2) into one to get images and an alignments.fsa file. And then use only video source1 to get the processed video (end result)?

Сould you clear this up for me?

User avatar
tochan
Posts: 21
Joined: Sun Sep 22, 2019 8:17 am
Been thanked: 5 times

Re: Using multiple videos (upd)

Post by tochan »

  1. Extract the two videos (important: choose different file names for the videos !!! (Vid_1 / Vid_2, for example)

  2. Rename the two alignmets.fsa files (to alignments_vid_1.fsa / alignments_vid_2.fsa, as an example)

  3. Now copy the extracted Faces into one folder (when you use same video filenames, you'll overwrite the faces)

  4. Copy the two alignment files into this folder (when you don't rename the alignments files, you'll overwrite one)

  5. Go to Tools, Alignments, and select the "Merge" option.

  6. Go down to "Data" and select in "Alignments File" of the two alignment files you created (alignments_vid_1 / alignments_vid_2). You can select more than one alignment file here.

  7. Press the Alignments button. This will combine the two alignment files and create a new alignments.fsa file

  8. Go to Train, select the newly-created alignments file / and the newly created face folder with all faces from the videos (after cleaning)

To clean the alignment file and remove the wrong faces:

  1. go to Tools, Alignments, "Job" and select "Remove-Faces"

  2. Go down to "Data" and select in "Alignments File" the Alignments.fsa file

  3. Select under "Face Folder" the folder with the faces form the 2 videos (after cleaning the wrongly detected faces)

  4. Press the Alignments button. Now you will have a new alignments.fsa file with clean Face sources from the 2 videos

Now train the model with the new alignments.fsa file and combined faces folder. (Sorry for bad english skills ;) )

User avatar
Lepiloff
Posts: 10
Joined: Wed Jan 08, 2020 2:32 pm
Answers: 1
Has thanked: 6 times
Been thanked: 1 time

Re: Using multiple videos (upd)

Post by Lepiloff »

Thank for your advice, I'll try to implement it

User avatar
Lepiloff
Posts: 10
Joined: Wed Jan 08, 2020 2:32 pm
Answers: 1
Has thanked: 6 times
Been thanked: 1 time

Re: Using multiple videos (upd)

Post by Lepiloff »

Is it possible to use the resulting allignments.fsa file to convert another new video with the same person?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Using multiple videos (upd)

Post by torzdf »

You wouldn't really want to use a combined alignments file for that.

You would generate an alignments file specific to the video you're wanting to convert.

My word is final

User avatar
Tekniklee
Posts: 37
Joined: Fri Jan 31, 2020 6:03 pm
Has thanked: 7 times
Been thanked: 3 times

Re: Using multiple videos (upd)

Post by Tekniklee »

I'm posting to this thread because I think it refers most closely to what I am asking about.

I thought I was attempting to do something fairly simple when I began a project by extracting and working on separate groups of videos to obtain training images. I worked on the first group, extracting, sorting, culling and manually checking landmarks, then did the same with other groups, ending up with 6 groups. So I had original frames in Group1.mp4, Group1_alignments.fsa, the Group1 folder held the final images, and so on. All was well, and the idea was to then merge all of the alignment files together.

The problem was that I could no longer check on status of the final landmark alignments, because frames were in multiple MP4 files. I've had problems in the past with missing training faces, or (worse) alignments reverting to their original, often horrible landmarks because of some process error on my part. I wouldn't find out about these until watching training and seeing horribly disfigured faces. So I cycle through my entire final training set and confirm that all landmarks were good.

I tried combining all of the videos into one file using MP4Join, and pointing to the merged alignment file, but it found no alignments. I'm assuming because hashes/names no longer matched? I also tried using ffmpeg to write all of the frames to a folder (ffmpeg -i Group1.mpg c:\temp\Group1_%06d.png), which wrote the images out as "Group1_000001.png", etc. At this point, I lowered my troubleshooting expectations by running Manual using only a single group (Group1_alignments.fsa, with the Group1 frame images in c:\temp), so the only difference between Manual working (using frame images in Group1.mp4) and not working (using frame images in c:\temp) was where the frame images were coming from.

The Forum has topics on "Using Multiple Videos", which only discuss the merging of alignment and training faces - not the original frames file required to use the Manual option. I'm guessing that I can use the Update Hashes, which uses the names for matching, but there isn't much info on this.

I'm a bit lost in my understanding now - I hope I'm missing something obvious. Is there a way to merge frames so that a merged alignment file can access the landmarks properly?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Using multiple videos (upd)

Post by torzdf »

Not at present, no.....

Merging videos after generating facesets will almost definitely not work, as an alignments file is intrinsically linked to a video. If you create a new video by merging 2 together, then neither of the generated alignments files will be linked to this new video.

The manual tool is currently being rewritten from the ground up, so will work better with training images in future.

My word is final

User avatar
Tekniklee
Posts: 37
Joined: Fri Jan 31, 2020 6:03 pm
Has thanked: 7 times
Been thanked: 3 times

Re: Using multiple videos (upd)

Post by Tekniklee »

Thank you. I had been hoping that if ffmpeg or its OpenCV equivilant was the method of reading mp4 files into memory, and so may recognize the frames, or that I could at least force a reckoning between them by updating the hashes. Okay, so I guess this idea was a fail process-wise. Really looking forward to the new Manual tool - it seems to be where we spend most of the time. It would be nice to be able to move landmarks more easily than messing with bounding boxes. Like moving an entire eye, or morphing the corner of an eye into place, or a moving a dot point into place.

One thing that could be helpful shorter-term is to be able to just write out landmarks on the aligned faces instead of the frams. Everything needed is in the alignment file already.

Thanks again for everything you are doing. I used DFL before this, and the original OFS before that. The FS GUI is definately the way to go.

Locked