Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by ianstephens »

Just a quick feature request and thought for future releases...

At the moment, we process high def (4K/5K) videos using Faceswap - for both (mainly) extraction and sometimes conversion.

The bottleneck seems to be FFmpeg decoding (or decoding and encoding for convert) the video frames and passing them to the extract/convert process. Most of what we process are in H265 (HEVC) format and relying on the CPU is super-duper-slow.

I know it's a challenge and may take some redesigning as the extract/convert process also uses the GPU and reserves all GPU memory (if allow growth is not enabled) but perhaps there could be some kind of sharing going to enable FFmpeg to use its GPU functionality too. Share and share alike :)

It's painfully slow using the included CPU-bound version of FFmpeg (especially extraction). However, the CPU is nowhere near utilized fully (especially on our machine) so maybe an off branch of this request would be to increase FFmpeg threads - for example, 4+ instances (or user-specified) of FFmpeg grabbing video frames in a set order in order to utilize the CPU to its full potential.

I feel this would greatly improve speeds with both extraction and conversion (at least for our HEVC encoded stuff anyway).

Perhaps we are a fringe case and there isn't much interest in this but thought we'd share the idea.

Thank you in advance.

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by torzdf »

Thanks for the idea.... for the reasons you mention (the complication when competing with GPU resources), as well as the sheer size of my to do list, this isn't something I can see myself implementing any time soon.

I believe ffmpeg is multithreaded by default (unless you pass a parameter limiting threads), but as we have to do linear reads and writes, I'm not sure how much difference that would do anyway.

HEVC is heavy for most CPUs, sadly. My advice (which may or may not be doable for you) would be to convert to H.264 for any media you wish to use, away from Faceswap and live with the larger file sizes, that way you can separate the encoding/decoding bottleneck away from Faceswap

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by ianstephens »

Thanks for that reply and explanation.

We will convert to H264 before we extract/convert for now - it makes sense to do so.

Another quick question (and I'm sure it's possible but I don't know how):

Is there any way to run extraction on a batch of videos? I.e. we may set a 5-hour extraction process to begin in the evening and it would stop at 1 am. It seems a waste of precious time with the machine idling there until we get in in the morning to run the next video extraction.

Is there any way to schedule a batch of videos? They would all have the same extraction parameters.

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by torzdf »

There is, but it means using the cli. One day I would like to add the possibility of batch processing to Faceswap, but see my earlier explanation of the massive to do list ;)

I believe you are using Windows, so this wouldn't work for you (I use Linux), however a quick google around for Windows scripting should net you something similar you can adapt for Windows.

For multiple tasks I tend to do the following (the example below is for applying masks to a series of videos, but it could be adapted for any task).

In this example, I have all of the videos, along with their corresponding alignments file, in the same folder:

Code: Select all

for VID in /deepfake/vids/crop/*.mp4 ; do BNAME=$(basename "$VID") ; FNAME="${BNAME%.*}" ; python tools.py mask -i $VID -a "/deepfake/vids/crop/${FNAME}_alignments.fsa" -M bisenet-fp -p all; done

This scans through all the mp4 files in a folder, strips off the folder location from the filename, then strips the extension, prior to adding the bisenet-fp mask for each source

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by ianstephens »

torzdf wrote: Tue Jul 20, 2021 9:50 am

I believe ffmpeg is multithreaded by default (unless you pass a parameter limiting threads), but as we have to do linear reads and writes, I'm not sure how much difference that would do anyway.

We've noticed it seems ffmpeg (during the extraction) seems to be only using one CPU core to decode the frames presented to the extractor. This seems (at least for us) to be the rate-limiting factor to making things process a lot faster - even with H264.

I don't know if there is a way around this/patch or how the extractor grabs the frames but I'm sure there is a better way (although you guys are the experts) :) .

Example during an extraction process of 4K footage (H264):

thread-example.png
thread-example.png (93.01 KiB) Viewed 14097 times
User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by ianstephens »

Update:

That could also indeed (and may actually be) the Python process - not ffmpeg. Is Python multithreaded? Perhaps not?

User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by torzdf »

Python itself is single threaded, but it can (and we do) use multiprocessing libs.

As to FFMPEG, as we have to read the frames linearly, it may just not be possible to run it more optimally. To be honest, I have not dug into it in that much detail

My word is final

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by ianstephens »

torzdf wrote: Sun Jul 25, 2021 10:15 pm

Python itself is single threaded, but it can (and we do) use multiprocessing libs.

As to FFMPEG, as we have to read the frames linearly, it may just not be possible to run it more optimally. To be honest, I have not dug into it in that much detail

Thanks for the info.

It was just an observation from this end as to what we can see. I think the only way to do it would be to analyze the number of frames per video and then grab them from ffmpeg in a set order using several processes of ffmpeg.

For example the video is analyzed and the frames to be extracted would be split into say 4 sets. Then 4 instances of ffmpeg are fired and passed the split sets frames to grab.

I don't know the technical details of how they are passed to the extraction process so maybe it's a lot of work.

Anyway, just a thought from this end - no expectation for such implementation :)

User avatar
wuthapn
Posts: 5
Joined: Sun Aug 02, 2020 2:05 am
Been thanked: 2 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by wuthapn »

I had this probably about a year ago when doing 4k content. Encoding is ridiculously slow as well as decoding because its using the CPU and no accelerator.

I was able to get it to use the built in nvenc and nvdec by downloading the source and editing some of the files.

Inside the file ./plugins/convert/writer/ffmpeg_defaults.py

edited the section around line 58

Code: Select all

    codec=dict(
        default="nvenc",
        info="Video codec to use:"
             "\n\t libx264: H.264. A widely supported and commonly used codec."
             "\n\t libx265: H.265 / HEVC video encoder application library.",
        datatype=str,
        rounding=None,
        min_max=None,
        choices=["nvenc", "libx264", "libx265"],
        group="codec",
        gui_radio=True,
    ),

Once you do that, edit config/convert.ini [writer.ffmpeg] section to say:

codec = nvenc

I was able to then run a conversion through the console ( windows 10 conda shell ). I haven't tried to do it in via the GUI.

This drastically lowered cpu and memory usage during the conversion process ( 4k video that was 38 minutes long ). For whatever reason when I use libx264 on 4k content it likes to use 4-8GB of ram, but using nvenc it uses 100-200mb. Also, if you're not scaling content, it does not use the cuda cores for encoding unless you use h265 or scaling. If you use the nvenc/nvdec it will not impact the conversion process in anything but a positive way at least in my opinion.

I only had to do this a few times, 4k is quite taxing. Also keep in mind while the encoding is much faster and less resource intensive, the quality might not be as high as libx264 unless you adjust the profile accordingly. I'll take 4 hour conversions any day over 12 hour ones though.

I ended up just not doing much 4k content, even though it does work fine on my 3090 and also my 1080Ti when i used that.

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by ianstephens »

[mention]wuthapn[/mention] - thank you for that - it's most helpful.

I can see the mods are for convert - what did you use for enabling hardware decoding on extraction?

User avatar
wuthapn
Posts: 5
Joined: Sun Aug 02, 2020 2:05 am
Been thanked: 2 times

Re: Feature Request: Make Use of NVIDIA Hardware Video Decoding/Encoding with FFmpeg

Post by wuthapn »

I don't recall, but i think they use the same function in the code. I believe I just used ffmpeg and outputted all the frames manually instead of trying to mess around with the code for the extraction process as it was faster. Sorry it was a while back and a one off. I ended up just using lower resolution content.

Locked