Search found 8 matches

by JimmyBoy
Fri Jun 12, 2020 7:29 pm
Forum: Training Discussion
Topic: Training speed over time / Training speed on resume
Replies: 7
Views: 6736

Re: Training speed over time / Training speed on resume

Noted, thanks for the advice. Why would the issue essentially go away when changing from batch size of 64 to 8? Does this give any clue, to me it would suggest some kind of memory issue, maybe GFX card is getting clogged up over time, although the memory usage is always at about 7.2GB of 8GB used. M...
by JimmyBoy
Fri Jun 12, 2020 5:57 pm
Forum: Training Discussion
Topic: Training speed over time / Training speed on resume
Replies: 7
Views: 6736

Re: Training speed over time / Training speed on resume

Thought I would follow up with some findings for future reference and anyone reading this post. Using the original model with a batch size of 64, I was finding the EG/s would slow from 120 to 40 - after advice and installing GPU-Z and Afterburner, I noticed my GPU was hitting its thermal limit of 83...
by JimmyBoy
Thu Jun 11, 2020 6:08 pm
Forum: Training Discussion
Topic: At what point do faces start to 'pop' ?
Replies: 2
Views: 5041

At what point do faces start to 'pop' ?

I realise this may be a difficult question to answer as many different variables are involved, but is there any guidance as to when you should see faces start to 'pop', i.e there is defined clarity - eyes are sharp, teeth are defined etc... I'm currently at 300,000 iterations and faces still look so...
by JimmyBoy
Wed Jun 10, 2020 6:53 pm
Forum: Training Discussion
Topic: Training speed over time / Training speed on resume
Replies: 7
Views: 6736

Re: Training speed over time / Training speed on resume

Thanks for pointing this out to me. After some investigating it appears the GPU is hitting its thermal limit, the card appears to have quite a relaxed fan profile as default with only 58% fan speed at 100% gpu usage. Ive modified the fan curve so the card is not hitting its thermal limit, and will a...
by JimmyBoy
Wed Jun 10, 2020 8:30 am
Forum: Training Discussion
Topic: Training speed over time / Training speed on resume
Replies: 7
Views: 6736

Re: Training speed over time / Training speed on resume

Hmmmmm, so this is not normal ?

It's a dell r720 server with a GTX 1070Ti blower, should have sufficient cooling. Will investigate.

by JimmyBoy
Tue Jun 09, 2020 11:58 pm
Forum: Training Discussion
Topic: Training speed over time / Training speed on resume
Replies: 7
Views: 6736

Training speed over time / Training speed on resume

I have a question regarding training speed over time. I believe that it is normal for the training to slow down over time, which is what I am experiencing. When starting off, my EG/s are around 120 EG/s, after about 48 hours they have dropped to 40 EG/s. However... it confuses me that if I stop trai...
by JimmyBoy
Sun Jun 07, 2020 1:20 pm
Forum: Extract Discussion
Topic: Tools > Alignments > Manual not showing faces
Replies: 1
Views: 3114

Tools > Alignments > Manual not showing faces

I am trying to examine the alignments file using the manual tool. The 'Frame' window is showing the image / face, but the faces window is not showing anything. The first time I attempted this it worked showing the faces with landmarks, bounding box etc... but now I get nothing... anybody know what t...
by JimmyBoy
Fri Jun 05, 2020 4:33 pm
Forum: Training Support
Topic: Training for a face vs scene / reusing a model
Replies: 3
Views: 5118

Training for a face vs scene / reusing a model

Hi, Im trying to understand the training process. I believe that you are training for a face and not a scene per se and you want as many facial expressions, angles and different lighting conditions for both face A and B. So, using the ubiquitous cage to trump, lets say I have the following videos of...