alignment tool extract doesnt match alignment

[Tekniklee]

I'm having a problem with the alignment tool extract option. In the past I've had problem with the alignment tool not agreeing with the faces. Specifically, if I made a change to an alignment, and then did something like a remove-faces operation, the changed alignments would no longer agree with the image in the faces folder. For example, if I tweak an alignment that changes the bounding box, the face orientation and position changes accordingly. I'm assuming this is because the alignments tool doesn't write out a revised face image to the face folder when saving changes. Also, I would fix an alignment problem with a face, but then noticed that training would still go squirlly on it because the face in the folder never changed. So after completing my landmark tweaks I got in the habit of using the alignment tool extract function to re-extract all of the faces (after deleting all of the original faces). However, I noticed that the images written out don't always match the alignments info - i.e. the image written out is not identical to the (green) extract bounding box/image in the face strip. In some cases, where the original landmarks were way off for some reason, the face written out is also way off. Like maybe only a forhead or chin showing, or the face is clearly too small for a properly registered face. It's as though the alignment tool extract is using the original extract bounding box info instead of the revised location, except that it seems to be a vertical offset - too high or too low, rather than off to right or left. Any idea what's going on here? Shouldn't the new images extracted by the tool match the image in the revised extract bounding box? Thanks.

EDIT: I've added a visual. The first two images show the alignment frame and face info, and the third shows the image written out using the alignments tool extract funciton. The last two images should be identical.

Example4.jpg (24.8 KiB) Viewed 6943 times

[pfakanator]

Are you using the manual alignment tool and copying alignments from one frame to another?

[Tekniklee]

Yes, it's the manual tool. And if you're referring to the C or V functions, I sometimes use those, but I can't say if these were a result of using that. I always click to "tighten up" the landmarks after doing a copy like that. It shouldn't matter though - what's in the green extract box should match what's extracted.

[pfakanator]

The destination alignment must differ from the one that it was copied from. This usually happens when dealing with a face that is static from one frame to the next. Double check.

[Tekniklee]

Right, and thans for the help. As I said, I always click after a copy to "tighten up" on the new image, and then save. But just in case I missed a save, I just went in and tweaked the alignments in that frame (editor goes red) then saved, then deleted all faces and re-extracted. The bad image remains the same.

Edit: Also, this was not a one-off problem with just this image. There are at least a dozen obviously bad images in this face set. The problem is, I don't know if there are other images that are not so obvious - i.e. the image and alignments don't match, but maybe not so obvious. I use align to landmarks, so I don't know if those cause a training problem because the face is not registering properly with the landmarks. However, I've gone through several other sets I've created in the same way, and I haven't noticed a similar problem. So this may be have been happening all along and I just noticed it because of something the faces were obviously bad, or it's a problem of some sort made obvious with this data set (like a couple dozen whacky extractions of seemingly clear frames I chose to use manual alignment tool to re-detect), or perhaps a recent bug. Just trying to get a handle on why the alignment tool extracts are not matching the green extract bounding box, and then I can trust all of the others.

[pfakanator]

I'm at a bit of a loss. I've seen this before but I don't recall what caused it or what I did to fix it. You can use some of the other alignment jobs to check for any frame/face issues. Set the log to debug before hand. Maybe you can glean some insight from that. You may also want to check if your source video is a CBR made from a VBR. That's given me wonky alignment issues in the past, but that's just a WAG. As you read in the other forum thread, a complete re-write of the manual tool is in development. No small task for one person .

[Tekniklee]

Yes, I'm definately looking forward to the new tool. The original video is a CBR (512kbps), but I don't know if a previous incarnation was VBR. I will try some other things, but after investing severl hours (okay, 2 days) purging and refining landmarks, I would not be very excited to start from scratch to try something like rewriting the mp4 file, or writing frames to a folder. I will try deleting all of the oddball faces, but if the same thing is affecting all images, then that will be a waste of time. Anyway, thanks for your help so far, and let me know if you can think of anything.

[torzdf]

In that grab, it looks like the last face is a different frame from the original (maybe 1 frame out)?

The manual tool has an issue where it sometimes displays all frames "out by 1". This is a bug in openCV. It's possible that you have fixed your alignments so they are now all out by 1....

We can't fix that bug, but a new version of the manual tool is imminent which does not have that particular issue.

Another issue we have seen (and potential for a fix), is that sometimes when re-extracting/using the alignments tool, the face is shifted in the frame. This seems to be down to some misreporting between video dimensions and the actual size of the video. Re-encoding the video (keeping the same frame rate, so alignments file still matches) tends to fix that particular issue.

[Tekniklee]

Okay, well that's something a bit more to work with, and it allowed me to do some investigating. And although you were on the right track with the faces being from nearby frames, they were not "out by one". I went to several suspect aligned faces, then switched to Standard and searched nearby frames for a match. The example I initially provided relatively slow moving, but the others I selected were from frames where I could use blurriness, hair strands and other things to pin down the exact offset. Here are my results:

Face Actual Offset
10421 10418 -3
23021 23018 -3
23056 23053 -4
25271 25268 -3
25401 25398 -3
26016 26012 -4
26151 26147 -4 (orig example, others close so +/-1)
29541 29537 -4
30991 30987 -4

As you can see, they are always off by 3 or 4, and most are precise (i.e. a 3 that is certain or a 4 that is certain), and only a couple (including the original example) "might" by off by 1. But none were off by only 1. So whatever the problem is, it is manifesting as writing out a face that is either 3 or 4 frames before the correct one. So.

This explains a lot, both in why the faces are wrong, and probably why I notices are large number of instances when I was doing the alignments where the original landmarks were correct, but perfectly offset. So going to a new face, the landmarks were precisly aligned, but shifted off by a fraction of an inch. I couldn't figure out how the extract function could even do that. And that would have been either done by the main extract utility, or the manual alignment tool was causing it to be off. But at least we know that the frames numbers are shifted as opposed to a face being shifted in the frame. And I'm also pretty sure it's the video, since I have not noticed previous extracts having this problem, nor have I needed to tweak nearly every frame before.

So I'm going to assume that all of the landmarks in this set are useless for training using Warp to Landmarks, and I may have to go through everything again. If I re-encode the video (same fps), were you saying that the existing alignments may work? Would that use Update Hashes somehow? Do you have a possible recovery procedure that would avoid having to start from scratch? Thanks for your help so far.

[Tekniklee]

Okay, so I have some additional information about this problem. First, there is this on the source file:

VideoInfo.jpg (58.9 KiB) Viewed 6931 times

The first is the Windows file properties, which shows a CBR if 512kbps. The secont is the Adobe Premeire properties and (last line) says "Variable Frame Rate Detected". Don't know exactly what that means - it's not VBR, but I've never heard of a varialble frame rate. But it can't be good in this context, right? Any idea what that's about??

Also, I re-encoded the video to CBR of same frame rate. TIL that modern video editors no longer even have presets for something as gauche as 320x240, so I had to use custom overrides. Things sure have changed over the past few decades. Anyway, I was able to use that in the manual alignment tool with no problem, but all frames that have alignments are are on the bad frames, shifted off by 3 or 4 frames now. I was debating going into each alignment, finding the correct frame, using the C to copy the nearby alignment, and then delete it from the bad frame. But that seemed like more thinking and keystrokes than just starting over. I was going to re-detect and use the original list of faces I culled, but all of those are different now also, being offset by 3 or 4 frames, so I will need to figure those out by their offset frames. A mess, but at least I know more today than yesterday, and after all, isn't that the meaning of life?

Thanks to both of you for your help.

[bryanlyon]

Best way to avoid this may be to extract the video into the frames and go from there. Sadly there are all sorts of frame issues in videos which show up in different contexts. Extracting to PNGs is sometimes the only guarantee, but it's not perfect since recreating that video can be a pain.

[Tekniklee]

Thanks. Since the Premiere properties for the new MP4 doesn't have that "Variable Frame Rate detected" message, and everything seems to be "unoffset" now, I'm going to assume that the re-encoding did the trick. But from now on I'm going to suck every source video into Premiere and look for that VFR warning.