Strange Performance Limitation Observation (Windows)

Getting errors or found a bug when converting faces from a trained model? Post about them here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Convert process. If you want to get tips, or better understand the Convert process, then you should look in the Convert Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Strange Performance Limitation Observation (Windows)

Post by ianstephens »

We are noticing something strange from a performance and hardware utilization standpoint.

In particular, let's talk about the converting process. We are running on Windows.

We have a machine with dual 12 core Xeons (24 physical cores). We also have 32GB RAM along with a GTX 3090 that provides 24GB of VRAM.

No matter what we do, we can't seem to use more than 50-55% of available system RAM during conversion. We have tried to modify/increase "jobs" but it always results in the same.

We first noticed this issue when we tried a 4K video conversion and the conversion process had an OOM error. It was puzzling as there was at least 50% available RAM. We then turned on "allow growth" which resolved the issue.

Our thinking for this error was that with "allow growth" disabled, the conversion process would allocate 100% VRAM and Windows would therefore not release any "shared" system RAM to the conversion process. Allow growth fixed this as the conversion only allocated what it needed and when it needed it and we could run the 4K conversion.

So even with "allow growth" enabled, although we no longer get OOM, we can never use more than 50-55% of our system RAM. No matter what we do.

It's as if the process/libs/python that runs the conversion is never able to allocate any RAM in the "shared" pool.

We'd love to max out the RAM, CPUs, GPUs but we feel as though we're not able to achieve maximum performance and something is limiting/hitting a bottleneck somewhere.

Any thoughts, suggestions, or ideas are greatly appreciated. I think the best thought we had would be to move over to Ubuntu :D

User avatar
ianstephens
Posts: 117
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 12 times
Been thanked: 15 times

Re: Strange Performance Limitation Observation (Windows)

Post by ianstephens »

P.s. allocated a spare SSD we have here and will install it in the machine tomorrow for dual boot with Ubuntu. It's my daughters birthday tomorrow so likely work the the evening - while the world is peaceful and sleeping :)

Will report on Ubuntu performance and if it's still affected by this issue.

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Strange Performance Limitation Observation (Windows)

Post by bryanlyon »

System ram is not really critical importance. OOMs apply to vram and not to system ram.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Strange Performance Limitation Observation (Windows)

Post by torzdf »

FWIW convert is not particularly well optimized.

Either way, It should never use all of your RAM. It's specifically built to try not to use it by only holding onto those frames that it is processing at any given time.

My word is final

Locked