Search found 177 matches

by abigflea
Sat Oct 17, 2020 11:53 pm
Forum: Training Support
Topic: Distributed with Dual 2060 supers
Replies: 46
Views: 16195

Re: Distributed with Dual 2060 supers

If its that, Then yea, get yourself a nice B550 or x570 chipset that screams on Pcie.
Im going to test this real quick, I can force mine to 4x

by abigflea
Sat Oct 17, 2020 10:28 pm
Forum: Training Support
Topic: Distributed with Dual 2060 supers
Replies: 46
Views: 16195

Re: Distributed with Dual 2060 supers

8x doesn't surprise me, but only gen 2 does.

Mine all run at pcie 8x gen 3. ( Or if I just put a single card then the top one will run x16)

Need a extra fancy mb to support gen 3 at pcie x16 on all slots.

by abigflea
Sat Oct 17, 2020 9:00 pm
Forum: Training Support
Topic: Distributed with Dual 2060 supers
Replies: 46
Views: 16195

Re: Distributed with Dual 2060 supers

I would suspect, it should be higher. I usually get about 160% Egs of a single on Linux. Ok, now I'm really curious. Look in GpuZ (Windows) , or NvTop (Linux) and see what speeds that are actually communicating at, over their respective Pcie slots during training. May say pcie x16 gen 2, or pcie x4 ...
by abigflea
Sat Oct 17, 2020 9:13 am
Forum: Training Support
Topic: Distributed with Dual 2060 supers
Replies: 46
Views: 16195

Re: Distributed with Dual 2060 supers

revisiting this post I was wondering if you had it sorted? I have noticed the set up time on distributed takes longer if you are using slower pcie slots. for example, if one card is running 16x and the other is running 4x on the pcie slot. for myself I had rearranged my hardware configuration, and o...
by abigflea
Thu Oct 15, 2020 12:34 am
Forum: Training Discussion
Topic: The realface model
Replies: 3
Views: 4734

Re: The realface model

Realface seems to be pretty dang good.
I've been meaning to do a comparison between the two models and take them to completion I just and I had the time maybe you should run it for a week or two and see what the results are

by abigflea
Wed Oct 14, 2020 8:16 am
Forum: Hardware
Topic: GPU upgrade on a budge: 1070 8GB vs. 1660 6GB
Replies: 9
Views: 11738

Re: GPU upgrade on a budge: 1070 8GB vs. 1660 6GB

System Ram is fine at 8gb, but if running Windows, 16 GB would be plenty of breathing room.

For GPU Ive been looking at this graph to get a general Idea

pic_disp.jpeg
pic_disp.jpeg (71.7 KiB) Viewed 11733 times
by abigflea
Thu Oct 08, 2020 3:26 am
Forum: Training Discussion
Topic: Strange Training Graph
Replies: 4
Views: 2182

Re: Strange Training Graph

I can't imagine what could cause that other than turning on warp to landmarks, and then turning it back off and it jumped back up

by abigflea
Tue Oct 06, 2020 10:27 pm
Forum: Training Support
Topic: Distributed with Dual 2060 supers
Replies: 46
Views: 16195

Re: Distributed with Dual 2060 supers

Ok I really don't know whats going on. I just upgraded, now have 2x 2070 and 2x 1070 The 2070s running distributed are no problem. FYI I had a somewhat unrelated driver issue, that snowballed when I tried to fix it (purge and reinstall) Long story short, fresh install of Ubuntu. They train like a ch...
by abigflea
Fri Oct 02, 2020 7:16 pm
Forum: Training Support
Topic: Distributed with Dual 2060 supers
Replies: 46
Views: 16195

Re: Distributed with Dual 2060 supers

try ONLY gpu1 as well

by abigflea
Fri Oct 02, 2020 6:36 pm
Forum: Training Support
Topic: Distributed with Dual 2060 supers
Replies: 46
Views: 16195

Re: Distributed with Dual 2060 supers

Ok. While im no expert , Lets try a couple ideas if you havent already. First try a different model. try original. If that doesn't work, regenerate your alignments. i swear I had that MainProcess _training_0 multithreading run DEBUG Error in thread (_training_0): OOM when allocating tensor with shap...
by abigflea
Fri Oct 02, 2020 5:38 pm
Forum: Training Support
Topic: Distributed with Dual 2060 supers
Replies: 46
Views: 16195

Re: Distributed with Dual 2060 supers

You have tried with a single GPU?

Tried activating "allow growth"?

Reduce your batch size to something like 2 - 4 to see if that helps?

that is a frustrating snag.

by abigflea
Thu Oct 01, 2020 9:03 am
Forum: Training Support
Topic: Distributed with Dual 2060 supers
Replies: 46
Views: 16195

Re: Distributed with Dual 2060 supers

sometimes it will take 2-3 min for me. Never considered it an issue when its going to be training for 10 days anyway

by abigflea
Tue Sep 29, 2020 7:20 pm
Forum: Hardware
Topic: Epyc 7551 with 2x gtx 1070 is good enough?
Replies: 7
Views: 3463

Re: Epyc 7551 with 2x gtx 1070 is good enough?

I would get the 2070 for a couple reasons.
Per card they are faster.
FS support for mixed precision in the new RTX core architecture , even faster.

they are all capable of the same quality work, the 2070s will be considerably faster.

FYI Your CPU , while very nice, will make no real difference.

by abigflea
Mon Sep 28, 2020 12:15 pm
Forum: Training Discussion
Topic: Min Max Iterations
Replies: 3
Views: 3287

Re: Min Max Iterations

You must be using windows.
Windows doesn't always report usage well.

If you're getting iteration at least every couple seconds it's probably running on GPU.

On CPU it is practically glacial in training speed. Is that is whats happening?

by abigflea
Mon Sep 28, 2020 10:11 am
Forum: Training Discussion
Topic: Min Max Iterations
Replies: 3
Views: 3287

Re: Min Max Iterations

There is always much discussion about this. If you are doing "medium" quality, I personally figure 100k as a time to test convert and see how its doing. Recently I've been hitting 300 - 600K before I feel its "finished" with any arbitrary other settings.. Original is usually done...
by abigflea
Sat Sep 26, 2020 6:45 pm
Forum: Swap Showcase
Topic: Bill Murray as Han Solo
Replies: 3
Views: 19853

Bill Murray as Han Solo

I could do better, still fun as is.

Bill Murray declined the role as Han Solo.

by abigflea
Fri Sep 25, 2020 10:57 pm
Forum: Installation Support
Topic: First run... cannot create cudnn handle
Replies: 3
Views: 2567

Re: First run... cannot create cudnn handle

Allow growth and I see you have a 2070 (me too!)

You are starting with the original Model?

by abigflea
Fri Sep 25, 2020 10:52 pm
Forum: Installation Support
Topic: First run... cannot create cudnn handle
Replies: 3
Views: 2567

Re: First run... cannot create cudnn handle

#1 enable Allow growth.
Let me read through the log if there is something else

by abigflea
Fri Sep 25, 2020 5:39 am
Forum: Training Discussion
Topic: Iterations, batch site and Eg / s
Replies: 6
Views: 5259

Re: Iterations, batch site and Eg / s

Of course, I know that. In the example of my first deepfake there are graphs. My question was different. That's what I meant. If bs = 16 I run 100,000 iterations and with bs = 32 I run 50,000 at the same time, is there any difference? No it's not the same. With a higher BS , the neural network will...