How to determine if Convert is still processing or complete?

Got questions or tips about the Conversion process? This is the place to discuss them.


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Converting faces from your trained model.

If you are having issues with the Convert process not working as you would expect, then you should post in the Convert Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
wader
Posts: 14
Joined: Tue Jun 07, 2022 6:54 pm
Been thanked: 4 times

How to determine if Convert is still processing or complete?

Post by wader »

Hello,
This is my first Convert attempt and my question is if Convert's "Stop" button (shown below) implies that it is still processing the outputs, because it's been four hours since it generated converted frames in the output folder and I've seen no obvious progress since that time.

In Task Manager (Windows 10) faceswap appears to be holding around 8GB of RAM but with almost no CPU usage, while something is holding most of the GPU memory,

I don't want to click the "Stop" button if faceswap is still trying to do something! Maybe it's not done applying Unsharpen to over 38K files? I'm not sure, so hints would be appreciated.

For this project, Convert is set to create a folder of converted .PNG images from the source video using Opencv, with other inputs being my alignments file (from the Extract step, for all frames) and model directory (from the Training step.)

Some related settings are shown below, but I should probably note that:

Opencv = PNG, compress level 3

Scaling = 100%
Sharpen = Unsharpen

convert Scaling from Sharpen plugin 100 percent.png
convert Scaling from Sharpen plugin 100 percent.png (38.21 KiB) Viewed 9218 times
convert Scaling from Sharpen plugin 100 percent - options.png
convert Scaling from Sharpen plugin 100 percent - options.png (28.89 KiB) Viewed 9218 times

Thanks,

  • wader
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: How to determine if Convert is still processing or complete?

Post by torzdf »

If it's hit 100% and all the frames are in the output folder, it's finished. I have never had a problem with convert getting stuck though.

If it's possible to provide the video + alignments file then I can look to see if I can recreate and identify the issue with it not completing. If not, no bother. Stopping the process will be fine.

My word is final

User avatar
wader
Posts: 14
Joined: Tue Jun 07, 2022 6:54 pm
Been thanked: 4 times

Re: How to determine if Convert is still processing or complete?

Post by wader »

Thanks! I clicked the button and the output files were usable for creating a final, converted movie file.

I'll see if I can recreate with a smaller project and send across, if so.

User avatar
wader
Posts: 14
Joined: Tue Jun 07, 2022 6:54 pm
Been thanked: 4 times

Re: How to determine if Convert is still processing or complete?

Post by wader »

torzdf wrote: Thu Jul 07, 2022 11:54 pm

If it's hit 100% and all the frames are in the output folder, it's finished. I have never had a problem with convert getting stuck though.

So, I had a closer look and it seems that Convert is saying 100% but has not actually processed the entire video. That's why the Stop button remains on, apparently.

48258 = the number of total frames in the source video, per the Manual alignment edit tool
32855 = the number of frames with faces, per Manual alignment tool and number of extracted files
38570 = the number of frames that Convert processes before it seems to pause indefinitely (with the STOP button remaining on)

I tried the same Convert step with different combinations of {two GPUs, single GPU, CPU-only, single-threaded} and the processing always stopped after frame 38570.

VERBOSE logging was on, but it didn't show anything unusual (that I could see) before or after I clicked STOP after the processing had hung.

Any ideas as to why the Convert step seems to be hanging after a certain number of frames?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: How to determine if Convert is still processing or complete?

Post by torzdf »

Most likely an issue with the video file.

The process just goes through the video file from first frame to last frame. Again, without the video or alignments file I cannot give any more feedback as this is not an issue I have ever had.

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 177 times
Been thanked: 13 times

Re: How to determine if Convert is still processing or complete?

Post by MaxHunter »

Preface this with I'm still learning.

You know, I had a similar problem recently, and my belief is that the file was too large and too long. The video I was attempting was an experiment and over an hour long. I've had similar experiences with 4k videos, and have had to cut them in half and scale them down. The video I was working on was 720 (?) and I thought there would be no problem, but it hung incomplete several times at roughly the same frame. I then went back and realized the video was over an hour and a half-ish long. I then split it up into 4 separate trims and it worked flawlessly.🤷

User avatar
wader
Posts: 14
Joined: Tue Jun 07, 2022 6:54 pm
Been thanked: 4 times

Re: How to determine if Convert is still processing or complete?

Post by wader »

That's interesting - in this case, I am actually using a 26 minute portion of a longer video and thought that maybe there was some sort of "limit" being reached, but further analysis showed that the Convert process started to encounter errors and finally stopped without explanation:


The original video has 48259 total frames.
There were 32855 frames with alignments (out of the 48259 total).

The Convert process correctly interweaves non-alignment and converted frames up to frame number 38467.
But, loss starts to occur at frame 38468: only 103 of the next 209 frames get output (all of which had alignments).

At frame 38570 the Convert process stops moving forward, but continues running and holding onto resources: it has to be explicitly stopped at this point and the logs don't seem to show why it stopped processing more frames.

Alignment and Training steps had no issues working with the entire video clip length - even the Manual alignment tool recognized all 48259 frames from the original video source.

I'm wondering if something in the source video's format is giving Convert a problem, where it just loses synch or something at some point. Here's the format, if anyone sees a potential problem in that respect:

Metadata:
major_brand : isom
minor_version : 0
compatible_brands: isommp42
encoder : Lavf58.76.100
Duration: 00:26:48.65, start: 0.000000, bitrate: 1357 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv), 1280x720 [SAR 1:1 DAR 16:9], 1232 kb/s, 24.02 fps, 30 tbr, 15360 tbn, 60 tbc (default)
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 120 kb/s (default)

Question
You are giving me the idea to try breaking up this source video and seeing if Convert will work for each. I wonder if that can be done without losing all the manual alignment work done on the full-length clip?

User avatar
wader
Posts: 14
Joined: Tue Jun 07, 2022 6:54 pm
Been thanked: 4 times

Re: How to determine if Convert is still processing or complete?

Post by wader »

There's definitely something wrong in Convert's perceptions of how long the video happens to be.

I copied the prior alignments file and extracted alignment .PNGs to another folder, then deleted the initial 25000 PNGs and removed those frames from the alignments file. It was a guess that maybe fewer alignment frames to convert would help.

Nope, the Convert process still thinks there are only 38570 frames to process in the source video file, despite Extract, Manual Alignment and other tools seeing all 48259 frames.

convert 38570.jpg
convert 38570.jpg (4.95 KiB) Viewed 9164 times

TLDR: It appears that something in Convert's code may not able to find all the frames in the source video, even though other faceswap components are able to do so.

User avatar
wader
Posts: 14
Joined: Tue Jun 07, 2022 6:54 pm
Been thanked: 4 times

Re: How to determine if Convert is still processing or complete?

Post by wader »

I tried [mention]MaxHunter[/mention] 's idea and split up the source video, but it turns out that the Convert step for each section also hit unresolvable pauses in every try. So, Convert may have more issues in dealing with certain videos compared to other faceswap tools which showed no such limitations.

To avoid the problem altogether, I added an extra step of extracting individual PNG frames from the source video using

Tools -> Effmpeg -> Extract

and then performed all Extract, Training and Convert operations using that folder of images when source frames were needed. Convert successfully processed all of the frames using source images, so then I used

Tools -> Effmpeg -> Gen-Vid

to reassemble the converted frames into the same video format as used by the source video. Success!

Think I'm going to work with extracted images instead of videos for now on, it seems a safer path.

[mention]torzdf[/mention] I wonder if it might be useful to add a FAQ item that suggests use of extracted frames - instead of the native source video - if Convert (or other steps relying on the source) can't complete due to unknown issues that may be related to the video format?

Locked