Workflow question, wasted three days

Discussions about research, Faceswapping and things that don't fit in the other categories here.


Locked
User avatar
TacoDance
Posts: 3
Joined: Wed Dec 30, 2020 9:05 pm

Workflow question, wasted three days

Post by TacoDance »

I think the problem (which I've been hit or miss on solving) is that sometimes an alignment file gets re-generated in another folder. Then you have two alignment files and the instructions are a bit crap at telling us which to target in the data sections so we get weird errors. I ran into several times where I did the extract then sorted, then removed faces manually, ran the remove faces program, and did an alignment extraction and would get "0 faces have been extracted". The system needs to have a way to keep alignment files updated through out the process a bit better. Also it should have an option on where to archive the files. The archives seem to send the trainer for a loop when there is two alignment files or more in the folder with the images. Took me awhile to figure out I'm better off axing the alignment files that it archives and just keeping the single alignment file.

However, my new problem is, I've made a working model, I can even preview the outcome. However when I run the convert I get a good several thousand messages saying the alignment for an image wasn't found. And before anyone assumes I didn't clean my alignment file, I've done it maybe three times on the same set of images these last few days. So I'm left to assume since I have group of images from a folder (we will call this data set A), and a group of images from a folder originally extracted from a video for a model/trainer (We will call this data set B) and then the original video (Data set C) and the original video's extracted files (Data set D) I'm a bit confused much of the time on which data set's alignment file or folder the whole thing wants me to point to. And even going back and looking at the instructions it gets confusing because it will mention one of the above data sets as "Where you extracted after blah" but not which type of extraction, not which steps were done before, and nowhere does it mention if the file placement is actually a requirement or not. There is one step in extraction with a "IMPORTANT***" tag that mentions moving a data set's alignment file to within the file. But when I do that it seems to give errors where it can't find anything. I assume because you shouldn't actually be moving it after some update or something, but I don't really know how to check the alignment file to see if there is some dead link somewhere that the alignment file points to a folder that no longer has the images because they were moved out during a sort or some other step.

I've gone back through the same data set each about three times and the best I can figure out is, after you sort and clean the file, assume the alignments file is useless and just start at the extraction step again (the very first extraction not the alignment extract) then go directly to the alignment extraction, do not pass sort, do not clean the file, because there isn't anything left to sort or clean. Then use that to do your model/training/convert.

That at least has worked up to the convert step.

My understanding on how the workflow "Should" be done is:
Extract> Sort> Clean >Remove faces> Alignment Ext. > Train> (Repeat all the prior over with a second set of data) then model

After that for the video you plan to swap into you have to do:
Extract > sort> clean> Remove faces> alignment extraction > use to convert

However, the work flow I seem to get dragged into through something going wrong is:
Extract> Sort> Clean >Remove faces>Stop here and start over with extract, skip sort and clean after> Alignment Ext. > Train> (Repeat all the prior over with a second set of data) then model

And after that for the video I plan to swap onto:
Extract > sort> clean> Remove faces> Stop here and go back to extract then skip sort and clean> alignment extraction > use to convert
But even after this workflow I get results where seemingly the first/last image the system looks at for the swap gets plastered on ever scene of the end video over top of whatever might have happened behind it. Or, I end up with a video that has had no swapping done at all. (And no, I checked, I did select the correct direction for my model to make the swap, and the correct direction for the convert. I didn't switch A & B around.)

If it matters I'm working from an .mp4 to an .mp4 with a lot of folders with files(pics and alignments) in between.

Hopefully my struggle busing along will help you get a little further. I'm still lost on the last few steps though, but I"m thinking that is because I wasn't using the manual tools to fix masking and such but idk, we'll see...

Anybody out there have any suggestions on how to get this to either work correctly or how to see better what is wrong? TIA

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Workflow question, wasted three days

Post by torzdf »

re: Alignments file. This is a workflow issue, and I accept that it is not the most intuitive, and welcome suggestions for improvements.

However, when an alignments file is generated, it is always generated in the same place (next to the source video/folder of images). This never changes. The problem comes from the fact that an alignments file serves 2 purposes. 1) it corresponds to a video to say where the faces are in a frame. 2) It can correspond to a folder of faces, which could be from multiple sources, to tell Faceswap how to train on each individual face. Unfortunately both these tasks need to be performed, but they do not play particularly nicely with each other.

Top level though... when you generate an alignments file it will always be associated with a source video.

TacoDance wrote: Wed Dec 30, 2020 9:34 pm

However when I run the convert I get a good several thousand messages saying the alignment for an image wasn't found.

This is almost always because an alignments file hasn't been generated properly for the video you want to swap on to (quite often it's because it has been generated with an Extract-Every-N setting higher than 1). First port of call will be to regenerate your alignments file for the video you want to swap on to.

Extracted faces can be entirely ignored for convert. They are not required. You just need a Video (where you want to swap faces). The alignments file that corresponds to this video, and the model that swaps the face in the video to the face you want.

I've gone back through the same data set each about three times and the best I can figure out is, after you sort and clean the file, assume the alignments file is useless and just start at the extraction step again

Not sure what this means.

Hopefully above will give you enough to go on to work out where things are going wrong, but if not, please feel free to ask for clarification.

My word is final

User avatar
TacoDance
Posts: 3
Joined: Wed Dec 30, 2020 9:05 pm

Re: Workflow question, wasted three days

Post by TacoDance »

I did some messing around and the biggest problem I encountered (and partially overcome) is that when I get files extracted and have to clean the files in the clean or sort step, is to not make a new folder for the sorted faces to go (this was my first mistake). That seems to keep the sort and cleaning steps from causing errors abound.
The new issue I've found is that using the manual tool and finding a face which isn't helpful and just right-clicking on it's bounding box and hitting "delete" causes more harm than good. Because after finding around 20+ of these mostly useless faces in the manual tool, the file becomes unable to be trained. It will hit an error rather than skip X number of faces (my assumption is that it has some maximum number of faces it can ignore, I know this because it will skip at least 1 in my current set and not err, and that when I hit "delete" in the manual tool it isn't deleting the image from the folder, or isn't deleting a space for information in the alignments file, idk which) however, by random guessing at which tool i can use to clean the alignment's file I've fixed this at least once successfully... I basically ran it through just about everything (missing faces, missing frames, missing alignments, leftover faces, whatever seemed like it may fix it, though I was doing this rather blindly since there isn't a good explanation I've seen on what those tools actually do really. So I just went with blind faith in running everything and hoped it's be okay.)

Also I noticed a big place where what the guide says is actually more confusing than what the tooltip in the program gives for an explanation. (this was a separate problem) I think mostly it revolves around the terms: "Faces" "Frames" and "input" when using each tool (my use of the word tool here is meant to include: the initial extraction, the alignment extraction, each sort and clean step, train, and convert etc.) It seems confusing how in one step a given set of data seems to be called the "faces folder" and in another it will be called the "frames folder" and in another we even see things like "training images" when best I can tell, I want to have every image in my training set unless I have a set of images beyond say 10,000. So I just select extract every N = 1.

My suggestions is to have a specific term for each folder/file and a separate term for a video file.

Example:
Data set A = Folder 1 or "original video 1"
Data set b = Folder 2 or "original video 2"
then other files that may be created:
"Sorted folder 1" or "sorted folder 2"
"Alignment file 1" (being from data set A) or ""Alignment file 2" (for B data set) or "Alignment file 3" (For the video to swap onto)
since for the case of swapping onto video we would always have a 3rd data set (we could call it data set C) which goes through a different process than the others we could avoid calling it anything other than the "Final swap video" or the "Final swap video's alignment file"

To me calling it a "face folder" or a "frames folder" gets needlessly confusing, and within the guide from step to step what was once faces in one step becomes frames for another so it gets very "wait whut?" to the newcomers.

The same goes for the input/output tags since the question can arise "Wait the output for this step? or from last step??"

Also, if there were a tool for looking not at the outcome of the alignments file (like the manual tool) but instead for seeing the actual important code within and being able to edit it. I may be able to decipher what exactly is going wrong when I use the delete function on a bounding box in the manual tool, or when I add a new bounding box on an image that didn't have one. (As that too was a problem)

I do have a guess at the solution though for how to take a lot of work out of the user's end when using the manual tool and fix a few potential issues it causes. When we hit the save key the manual tool should do a final check on all the images within the alignment file and be sure all the images are accounted for, if one isn't then it should remove it from the file. Also it should go through and add in any new alignments given to images which had none before. Then once it's gone through and fixed those issues, save all the data we made changes on that already existed.

Also I have found that some people mentioned crashes with the manual tool. I have info on the cause of those crashes.

For me if the images and the alignments have de-synced on a file (not a video) to where I have several hundred images with correct facial alignments in terms of the form of the face, however they are far off from where the actual face in the image lies, I can go through and correct them with the manual tool as needed, (however idk why they happen, I've had about 1100 out of a data set of 2300+ faces) but I have noticed while correcting them I tend to stay on the bounding box screen while flipping through the frames with z & x keys, that if I try to pull a bounding box from out of the overall workspace into the workspace, the computer crashes. I assume what has happened here is since it is outside the workspace and cannot see past the edges of the workspace the bounding box doesn't know where to try to lock on to a landmark so it just crashes. I've found a work around however. If I swap to the view where I see the yellow box instead of the blue bounding box (i forget the name of this screen and I'm in the middle of a training set now so I can't check,) and then I pull the box within the bounds of the screen, (while still seeing the yellow box, I don't just flip in and out of the screens) then it won't crash. I can then flip back to the bounding box screen and the bounding box will be within the bounds of the workspace so it doesn't crash as it had.

I should mention I did notice that the files which are extracted from a video do have to be off to the side from the original thumbnail image we see. Best I can guess is the reason the dots for the face are off in a black area instead of over top of the image of the face we see is because the alignments file in that instance is pointing the dots to the location of the actual face as it would appear on the screen while we watched it full view. But then saves a snapshot of that face off to the side, just for reference for us, and in part may not even be needed. (Idk if the program does anything useful with it)

Though alignment files should be corresponding to the faces always when you input a folder of images rather than a video, so I'm unsure why for several hundred images I'm getting some sort of artifact from the video process on my images/alignments data sets in the manual tool, like they had come form a video, if they didn't.

A possible way to help everyone I feel, would be to set another 2 tabs in the program along the same space as the "tools" and "convert" tabs. and simply rename the first extraction tab to "extract 1" and then name the two new tabs "extract 2" and "extract 3".

Though as I write this I notice it may also be convenient to have an "align/alignments 1" "...2" for the first two data sets. If memory serves the 3rd data set doesn't go through the alignment extraction process since they aren't used but instead discarded to make room in the convert step for the new faces.

Gosh, I'm sorry this is long, I'm trying to explain things in a way my meaning isn't lost and it's rough through text. I hope on some level I'm helping refine the workflow in a way that is more intuitive and easy to learn for new-comers.

About this section from before that you didn't understand where I said:
"I've gone back through the same data set each about three times and the best I can figure out is, after you sort and clean the file, assume the alignments file is useless and just start at the extraction step again"
What I meant was with each alignment file and set of images I went through the steps (extract, sort, clean, alignment) about three times over and kept running into errors where it seemed to either say there were no frames/faces/alignments were missing seemingly interchangeably. As I mentioned earlier I think this was caused by running the sort command and choosing a new folder then reverting the faces' file names back after the sort. For some reason even though this should work anyway, it didn't. So what I was doing was running the folder I had my images in after the sort process, back through the original extraction process in hopes of making a new alignment file that wasn't missing something. (be it frames/faces/alignments or whatever else it wanted to not hit errors. and I should mention I was using the "remove faces" tool through this process too, it was just seeming to not help matters much.)


Found a NEW problem and this one I don't think I can fix.

When I attempt to load the manual tool I can get it to load, I can edit masks/points whatever I need. up until the 2107th face. After that I can see in the scrolling section more thumbnails with more faces. I can even click to view their points and masks instead of the thumbs, but when I click them they won't load in the workspace. Also, I can only see them on the "all frames" selection. Further, on the initial load in to the manual tool I get a count of all frames of 2107 (which I even have that count of images in the folder I am viewing and using with the alignments.) However, after switching from all frames to frames with faces my count drops to 2107 (correct number of overall frames) and then up to 2407 (which seems to be including the thumbs which I don't know where they come from.) and I cannot edit any from 2107-2407... and I checked all my files with images, the most I've seen are 2541 with 4 files being the alignments and the backed up alignments. So it doesn't make sense to have the number 2407 showing up. From what I can tell when I run the train command it actually uses the additional "ghost" 300 files and it mucks up the entire neural network's attempt at trying to make a model, even though it started with 2107 good files the 300 ghost ones inevitably send it to hell. This is my new struggle and my only solution? Take my file with the lowest number of images and turn it into the source for my new extraction file and start over for side B of my training...

I shouldn't be getting forced at ever problem to rework a whole extraction over from the beginning like this. Something's gotta give. what can I do?

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Workflow question, wasted three days

Post by torzdf »

TacoDance wrote: Tue Jan 05, 2021 10:51 am

and that when I hit "delete" in the manual tool it isn't deleting the image from the folder, or isn't deleting a space for information in the alignments file, idk which)

Not sure what you mean here. The manual tool does not reference the faces folder at all. When you delete a face it just removes it from the alignments file. Generally, after doing my first pass sort and deleting of faces I don't want (prior to using the manual tool), I will clean the alignments file (by using the remove-faces option in the alignments tool). Once done, I will delete the entirety of my faces folder. I don't want it around, as I want to fix stuff with the manual tool, then re-generate the faces from my manual fixes.

TacoDance wrote: Tue Jan 05, 2021 10:51 am

I however, by random guessing at which tool i can use to clean the alignment's file I've fixed this at least once successfully... I basically ran it through just about everything (missing faces, missing frames, missing alignments, leftover faces, whatever seemed like it may fix it, though I was doing this rather blindly since there isn't a good explanation I've seen on what those tools actually do really. So I just went with blind faith in running everything and hoped it's be okay.)

Yeah, don't do that. You most likely just need "remove-faces". The tooltips tell you what each does.

TacoDance wrote: Tue Jan 05, 2021 10:51 am

It seems confusing how in one step a given set of data seems to be called the "faces folder" and in another it will be called the "frames folder"

Specific examples would be useful here, as I haven't had any other reports of confusion. Faces folder and Frames Folder are never the same thing.

TacoDance wrote: Tue Jan 05, 2021 10:51 am

and in another we even see things like "training images" when best I can tell, I want to have every image in my training set unless I have a set of images beyond say 10,000. So I just select extract every N = 1.

Again, not sure what you mean here. Generally extracted faces serve 2 purposes.

1) They are an easier method for cleaning up your data (i.e. when you extract every face, you can use your browser's file manager to go through the faces and delete those you don't want), and then clean your alignments file.
2) They are used for training. Using every single face in a video is not a great plan, as it gives the model far more data than it actually needs.

There is a disconnect in workflow as alignments files serve 2 purposes. This is something I am attempting to address.

TacoDance wrote: Tue Jan 05, 2021 10:51 am

To me calling it a "face folder" or a "frames folder" gets needlessly confusing, and within the guide from step to step what was once faces in one step becomes frames for another so it gets very "wait whut?" to the newcomers.

Again, examples please, because these are very definitely 2 different things,

TacoDance wrote: Tue Jan 05, 2021 10:51 am

Also, if there were a tool for looking not at the outcome of the alignments file (like the manual tool) but instead for seeing the actual important code within and being able to edit it. I may be able to decipher what exactly is going wrong when I use the delete function on a bounding box in the manual tool, or when I add a new bounding box on an image that didn't have one. (As that too was a problem)

Yeah, I'm fairly sure this would not be as useful as you think it would be. You are welcome to script something up to analyse the file though.

TacoDance wrote: Tue Jan 05, 2021 10:51 am

I do have a guess at the solution though for how to take a lot of work out of the user's end when using the manual tool and fix a few potential issues it causes. When we hit the save key the manual tool should do a final check on all the images within the alignment file and be sure all the images are accounted for, if one isn't then it should remove it from the file. Also it should go through and add in any new alignments given to images which had none before. Then once it's gone through and fixed those issues, save all the data we made changes on that already existed.

Not sure I really understand. An alignments file corresponds to a source video/frames folder. There is nothing to check.

Also I have found that some people mentioned crashes with the manual tool. I have info on the cause of those crashes.

For me if the images and the alignments have de-synced on a file (not a video) to where I have several hundred images with correct facial alignments in terms of the form of the face, however they are far off from where the actual face in the image lies, I can go through and correct them with the manual tool as needed, (however idk why they happen, I've had about 1100 out of a data set of 2300+ faces) but I have noticed while correcting them I tend to stay on the bounding box screen while flipping through the frames with z & x keys, that if I try to pull a bounding box from out of the overall workspace into the workspace, the computer crashes. I assume what has happened here is since it is outside the workspace and cannot see past the edges of the workspace the bounding box doesn't know where to try to lock on to a landmark so it just crashes. I've found a work around however. If I swap to the view where I see the yellow box instead of the blue bounding box (i forget the name of this screen and I'm in the middle of a training set now so I can't check,) and then I pull the box within the bounds of the screen, (while still seeing the yellow box, I don't just flip in and out of the screens) then it won't crash. I can then flip back to the bounding box screen and the bounding box will be within the bounds of the workspace so it doesn't crash as it had.

Best to open another thread for this, as I'm most likely to forget this and it will be lost in the rest of this information.

I should mention I did notice that the files which are extracted from a video do have to be off to the side from the original thumbnail image we see.

Again, not sure I understand. An image demonstrating what you mean would help here.

Though alignment files should be corresponding to the faces always when you input a folder of images rather than a video, so I'm unsure why for several hundred images I'm getting some sort of artifact from the video process on my images/alignments data sets in the manual tool, like they had come form a video, if they didn't.

Ahhhhhh. This may explain most of your issues. The manual tool does not work, and never has worked with faces as an input. The input to the alignments tool should always be frames.

About this section from before that you didn't understand where I said:
"I've gone back through the same data set each about three times and the best I can figure out is, after you sort and clean the file, assume the alignments file is useless and just start at the extraction step again"
What I meant was with each alignment file and set of images I went through the steps (extract, sort, clean, alignment) about three times over and kept running into errors where it seemed to either say there were no frames/faces/alignments were missing seemingly interchangeably. As I mentioned earlier I think this was caused by running the sort command and choosing a new folder then reverting the faces' file names back after the sort. For some reason even though this should work anyway, it didn't. So what I was doing was running the folder I had my images in after the sort process, back through the original extraction process in hopes of making a new alignment file that wasn't missing something. (be it frames/faces/alignments or whatever else it wanted to not hit errors. and I should mention I was using the "remove faces" tool through this process too, it was just seeming to not help matters much.)

I never move faces. Always rename, so there may be an issue there. My advise: rename. There is no need to move the files.

My word is final

Locked