Strange Performance Limitation Observation (Windows)

Getting errors or found a bug when converting faces from a trained model? Post about them here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Convert process. If you want to get tips, or better understand the Convert process, then you should look in the Convert Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Post Reply
User avatar
ianstephens
Posts: 67
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times

Strange Performance Limitation Observation (Windows)

Post by ianstephens »

We are noticing something strange from a performance and hardware utilization standpoint.

In particular, let's talk about the converting process. We are running on Windows.

We have a machine with dual 12 core Xeons (24 physical cores). We also have 32GB RAM along with a GTX 3090 that provides 24GB of VRAM.

No matter what we do, we can't seem to use more than ~50-55% of available system RAM during conversion. We have tried to modify/increase "jobs" but it always results in the same.

We first noticed this issue when we tried a 4K video conversion and the conversion process had an OOM error. It was puzzling as there was at least 50% available RAM. We then turned on "allow growth" which resolved the issue.

Our thinking for this error was that with "allow growth" disabled, the conversion process would allocate 100% VRAM and Windows would therefore not release any "shared" system RAM to the conversion process. Allow growth fixed this as the conversion only allocated what it needed and when it needed it and we could run the 4K conversion.

So even with "allow growth" enabled, although we no longer get OOM, we can never use more than ~50-55% of our system RAM. No matter what we do.

It's as if the process/libs/python that runs the conversion is never able to allocate any RAM in the "shared" pool.

We'd love to max out the RAM, CPUs, GPUs but we feel as though we're not able to achieve maximum performance and something is limiting/hitting a bottleneck somewhere.

Any thoughts, suggestions, or ideas are greatly appreciated. I think the best thought we had would be to move over to Ubuntu :D


User avatar
ianstephens
Posts: 67
Joined: Sun Feb 14, 2021 7:20 pm
Has thanked: 9 times

Re: Strange Performance Limitation Observation (Windows)

Post by ianstephens »

P.s. allocated a spare SSD we have here and will install it in the machine tomorrow for dual boot with Ubuntu. It's my daughters birthday tomorrow so likely work the the evening - while the world is peaceful and sleeping :)

Will report on Ubuntu performance and if it's still affected by this issue.


User avatar
bryanlyon
Site Admin
Posts: 634
Joined: Fri Jul 12, 2019 12:49 am
Answers: 41
Location: San Francisco
Has thanked: 3 times
Been thanked: 161 times
Contact:

Re: Strange Performance Limitation Observation (Windows)

Post by bryanlyon »

System ram is not really critical importance. OOMs apply to vram and not to system ram.


User avatar
torzdf
Posts: 1495
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 51 times
Been thanked: 287 times

Re: Strange Performance Limitation Observation (Windows)

Post by torzdf »

FWIW convert is not particularly well optimized.

Either way, It should never use all of your RAM. It's specifically built to try not to use it by only holding onto those frames that it is processing at any given time.

My word is final


Post Reply