Faceswap Forum

PCIe 2.0 8x should be somewhere near 4 Gb/sec. Faceswap uses that much?

The MSI site says 2x16. Other sites show 1x16 and 1x8. Which would explain why the drop down to 1x8. Well, gonna take out one of the cards and see if the single runs at 16x.

-edit-
Yep, one card shows 16x.

Time to start saving up for a Ryzen.....

Hmmm, you might be on to something. This is showing only 8x PCIe when they are both in use. Specs on my MB show 2x 16...

Now with just GPU1 doing the work....

Still 8x

Also I noticed this the other day.

Distributed with a batch of 14, and only gpu1 with a batch of 7.

Shouldn't the distributed batch of 14 have roughly 2x the EG/s of the single gpu with a batch of 7?

The Analysis tab shows more iterations than the status bar.

Also, the graph crashes or doesn't respond if you change smoothing and his the refresh button.

Due to my snapping off the sata connector to my hdd I'm on a fresh install of Ubuntu 20.04 with the 450 driver also.

My only problem is the DFL-SAE model. (Allow growth checked everywhere it is an option)

All other models seem to work fine.

After further testing it looks like all my problems come from the DFL-SAE model. It will only train on GPU1. Training on GPU0 or Distributed fail.

Training Villain with a batch of 16 right now distributed.

Thanks for all the help.

I notice this A LOT in the verbose logs as it starts to train.

failed to allocate 3.24G (3477464576 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

Without MIXED PRECISION I see it a lot.

With it check just one or two instances. I beginning to think this is where the issue is lying.

Distributed Dlight works too.

Question: Is it normal for it to take 15-25 mins to start training on distributed? I've got a pretty beefy system, just wondering what's normal?

Distributed, DFL-SAE batch 2, FAILS.

Distributed, Villain batch 4, works.

Could it be an issue with SAE and distributed?

Redid alignments.

Distributed, Original, Batch 128 - worked!

Distributed, DFL-H128, batch 32 - worked!

I'll test more, but you might have figured it out. THANKS

Selecting ONLY GPU0 I can get lightweight (batch32), DF128 (10) and original working.

SAE with a batch of 1 fails.

Distributed - DFL-128 with batch of 8 running. This is wierd.

I'll try the alignment thing next.

Allow growth selected. Tried with batches of 2 and 1 still fails.

Attached is the system info.

For any linux users I opened a ticket with Nvidia. Not a faceswap issue. It is clearly a driver problem with Nvidia.

https://forums.developer.nvidia.com/t/2 ... -04/156103

Definitely starting to look like it is a problem with X on linux and probably not a FaceSwap issue. It doesn't like anything to do with gpu0 no matter how I have the cards (1 or 2 installed). Found some others having similar probs, any help appreciated.

The verbose log is too big to attach. It keeps giving

2020-10-01 22:41:27.315577: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.06G (5437426176 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-10-