nnifj wrote: ↑Mon Aug 31, 2020 7:38 pm
If by out of memory you mean it runs out of my computer memory, perhaps. It does seem to work best (best meaning it works for an hour or 2 ) when I restart my computer and it tends to crash when I open up even light applications.
If your computer is crashing after an hour or 2, I personally would start thinking some component is getting hot and unstable.
nnifj wrote: ↑Mon Aug 31, 2020 7:38 pm
A batchsize of 2 is what I'm using as any larger crashes after 30 seconds. Turning it down to 1 didn't seem to help that much. I would have thought my 16gb ram and rtx 2060 would have been able to handle DFLSAE @ 144pixles, growth, mixed precision, multiscale decoder, 512 autoencoder, 42 encoder and 30 decoder,a lot better than this.
DFL uses a lot of VRAM. I have a model with 42Enc, 22 Dec, at 128pix and get a Batch of 12 on my RTX 2070 8GB.
I can run a batch of 16 but get OOM after a few 1000 iterations.
With your 6GB card and those settings, you are not going to get a high batch.
Maybe tone down that decoder back to 21(default).
Big disclaimer here, every model will use slightly different amounts of memory, batch may go up and down a notch.
Windows will take more for itself and knock down your maximum batch size.
I suspect other programs running will use more VRAM, I use mine not connected to any monitor to keep every megabyte free as possible.
If your System Ram is filling up, you have some other program(s) stealing your memory away. FS doesn't use much (Thanks Torzdf). I can run 3 separate instances , all training, and it used about 14GB in Linux .
nnifj wrote: ↑Mon Aug 31, 2020 7:38 pm
In the info boxes, adjusting the encoders state that lower setting can free up vram. Would turning off growth, mixed precision, or multiscale decoder also help free up vram? Also would turning down the learning rate help in any way?
Mixed Precision should increase your available memory so leave that on.
Allow growth shouldn't make much difference with this issue. I have to leave it on because my 2070 won't behave without it.
I haven't really tried turning Multi-scale decoder on/off. It may help a little, try it out.