Is my CPU a bottleneck?

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
363LS2GTO
Posts: 30
Joined: Fri Jul 08, 2022 7:06 pm
Has thanked: 1 time
Been thanked: 4 times

Is my CPU a bottleneck?

Post by 363LS2GTO »

Installed the NVIDIA RTX 3050 and the card was utilized first try.

So I can say that although the NVIDIA 1030 is a relatively new architecture, albeit lower end, the program as is does not work with it.

The program is reporting more iterations in 20 minutes with the card vs 12 hours with the processor alone.

Task manager in windows shows that CUDA usage is around 40-60%, and nearly all of the VRAM is being used. I am running the PCIE 4 card on a PCIE 3 slot.

It also shows that system ram is being used about 6 gigs (75% in my case) and the CPU is averaging 90% or greater.

Do these stats look typical? Guess I need a faster CPU now to drop usage below 90 -100%, or is that amount of usage normal? Is it a bottleneck at this point? (Pentium G4400)

Thank you.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: GPU Not being Used

Post by torzdf »

That CPU should be ok.

Stats seem reasonable. I would expect GPU utilization to be higher, but every system is different.

The system CPU/RAM is utilized for data augmentation/feeding the neural network.

Glad you got up and running.


Edit:

Just checked your CPU and it is dual core. You may well be being bottlenecked by the CPU. If you have a way of graphing GPU utilization then you should be able to tell. CPU bottleneck manifests itself as lots of peaks and valleys in GPU usage graph (close to 100% when it is working, close to 0% when it is waiting for data).

My word is final

User avatar
363LS2GTO
Posts: 30
Joined: Fri Jul 08, 2022 7:06 pm
Has thanked: 1 time
Been thanked: 4 times

Re: GPU Not being Used

Post by 363LS2GTO »

Its not a smooth line. I have attached a screen shot from task manager.

Its a socket 1151. Benchmark on the processor is around 2500. Would upgrading to a quad core and doubling the benchmark make much difference, or are we talking more like near current generation AMD / Intel scoring 20,000 or more in benchmark?

Not much money to purchase a used I5 for the existing board if it would help, but not quite sure I am ready for an entire new computer at the moment.

Good news is that temperatures are cool after extended sessions--one of the reasons I purchased this processor in the first place.

Attachments
Screenshot task manager 7.10.22.png
Screenshot task manager 7.10.22.png (87.35 KiB) Viewed 1583 times
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Is my CPU a bottleneck?

Post by torzdf »

Honestly, I couldn't really advise one way or another. I'm not sure how accurately Windows graph represents utilization for AI tasks. I develop Faceswap on a Xeon processor with a considerably lower clock speed than yours, but with considerably more cores.

If you are just looking to upgrade because of Faceswap, I would say, no. It's not worth the upgrade. Personally I would take the fact that you have massively increased your training throughput and consider it a win.

My word is final

User avatar
363LS2GTO
Posts: 30
Joined: Fri Jul 08, 2022 7:06 pm
Has thanked: 1 time
Been thanked: 4 times

Re: Is my CPU a bottleneck?

Post by 363LS2GTO »

Thanks for the advice.

I did want to post that a Windows system process was acting up for a few hours wile running face swap. Face swap usually consumes 90% resources and was at 60-65 Eg/s. (these are for a different model than the stats reported in later posts) Consuming 50% cpu resources dropped Eg/s to 35-40 and CUDA utilization in task manager showed a corresponding drop.

I am happy with the current speed as I have a lot of time to run the program. I may spend $45 on a used I5-6500 quad core as it is an easy drop in solution for my current rig and I could use the headroom for other tasks.

Last edited by 363LS2GTO on Fri Jul 15, 2022 10:05 pm, edited 1 time in total.
User avatar
363LS2GTO
Posts: 30
Joined: Fri Jul 08, 2022 7:06 pm
Has thanked: 1 time
Been thanked: 4 times

Re: Is my CPU a bottleneck?

Post by 363LS2GTO »

Purchased a used I5 6500 that I am going to install tomorrow.

Playing around with windows settings tonight I found that my CPU is a bottleneck but not as much as I originally thought.

First, I enabled my "second graphics." Intel on chip graphics.

This allowed task manager to show both GPUs.

Next, I went to settings, display, graphics.

I was able to choose which programs used which graphics core. I set chrome, fire fox, etc to use the Integrated graphics. No more momentary black screens when an embedded video starts while running faceswap.

There was not much of an improvement, and maybe a slight slowdown at this point in EG/S, even though CUDA usage was improved in the task manager. Not sure about the reason for that outcome.

Next, still under graphics settings, I enabled windows hardware acceleration.

Windows task manager no longer gives me the option of CUDA but the "3D" graph displays near 100% discreet NVIDIA usage. At least 25% more graphics card use averaged over time. I can hear the graphics card fans at a higher setting. There are still peaks and valleys in GPU usage as my CPU is maxing out. However, my CPU is no longer pegged at 100% and varies with time.

My EG/S has improved at least 20% and multitasking has also improved.

I expect further improvements by reducing the CPU bottleneck.

I did not see this information in the forum and thought it might be useful.

I keep reading that you will get better results on linux as windows is a RAM hog. I am sure that this is probably true, but adjusting a few windows settings seems to help a lot.

I noticed that my "dedicated GPU" memory dropped from 7.1 to 6.8 GB after the change. When I am not running my current project on faceswap, the "dedicated GPU memory" is around .3GB so it would seem that a drop in dedicated memory for the same task means that more GPU memory is still available.

I am guessing that now I will have more available GDDR6 VRAM for pixel size / batch size now that windows itself is utilizing DDR4 RAM and integrated graphics for mundane tasks?

User avatar
363LS2GTO
Posts: 30
Joined: Fri Jul 08, 2022 7:06 pm
Has thanked: 1 time
Been thanked: 4 times

Re: Is my CPU a bottleneck?

Post by 363LS2GTO »

Installed the new processor and EG/s bumped up again.

Instead of pegged at 100% the 4 core I5 6500 is running around 60%. I have just enough ram at 8gb and just enough power at 450 watt bronze EVGA. Temps are the same as before the swap-- 50s for the CPU and around 60 for the GPU.

I can now run faceswap in the background and use my computer as normal with office, chrome, and a monitoring program open as I used to.

Runnning DFS-Sae at 80/120 batch size 16 the EG/s have reported:

Pentium G4400
NVIDIA RTX 3050
8GB DDR4
No windows hardware acceleration
EG/s average 44

Enabled integrated graphics (2nd GPU), enabled hardware acceleration, and tweaked graphics card to program assignments.
Eg/s average 54

Installed Core I5 6500 with above settings.
Eg/s average 59

An increase of over 20% just with properly using all of the hardware / software and another 10% plus extra headroom for the $55 I paid for the used chip.

Hope this helps someone.

Attachments
Task Manager 7_15_2022 5_09_15 PM.png
Task Manager 7_15_2022 5_09_15 PM.png (83.2 KiB) Viewed 1543 times
User avatar
363LS2GTO
Posts: 30
Joined: Fri Jul 08, 2022 7:06 pm
Has thanked: 1 time
Been thanked: 4 times

Re: Is my CPU a bottleneck?

Post by 363LS2GTO »

I wanted to follow up with a few other tests that I have done regarding this subject.

One person suggested running the monitor using the main board output and integrated graphics instead of the PCI graphics card. I tried this and can report that I gained 30 it/s and 1 eg/s or about 1% total.

I can also report that this was a hassle and that I see similar results just by turning the monitor off during training. I see similar movement by using other programs while faceswap is running. So bottom line is that windows does a good job managing multiple graphics cards regardless of how the actual cables are output.

On a different note, I wanted to report that I have seen batch size affect the CPU utilization but only in dramatic changes. I have not seen much of an effect in the 6-16 batch size where you are supposed to stay for best results.

Last, for my 3050 series graphics card, enabling mixed precision seems to be a good idea. It allows me to train villain at batch 8 and 25.6 eg/s with around 5,500 it/hour on windows. I did not even bother trying batch size 9 as going from 6 to 8 only netted a 2 eg/s increase and 8 has been stable. I am sure it would be faster on the linux version but this is good enough for me.

I have not used mixed precision enough to know how often problems arise, but it seems there is a net gain to be had utilizing the tensor cores versus any lost training due to having to roll back to the previous snapshot if an error occurs.

Locked