Search found 8 matches
- Fri Jun 12, 2020 7:29 pm
- Forum: Training Discussion
- Topic: Training speed over time / Training speed on resume
- Replies: 7
- Views: 6736
Re: Training speed over time / Training speed on resume
Noted, thanks for the advice. Why would the issue essentially go away when changing from batch size of 64 to 8? Does this give any clue, to me it would suggest some kind of memory issue, maybe GFX card is getting clogged up over time, although the memory usage is always at about 7.2GB of 8GB used. M...
- Fri Jun 12, 2020 5:57 pm
- Forum: Training Discussion
- Topic: Training speed over time / Training speed on resume
- Replies: 7
- Views: 6736
Re: Training speed over time / Training speed on resume
Thought I would follow up with some findings for future reference and anyone reading this post. Using the original model with a batch size of 64, I was finding the EG/s would slow from 120 to 40 - after advice and installing GPU-Z and Afterburner, I noticed my GPU was hitting its thermal limit of 83...
- Thu Jun 11, 2020 6:08 pm
- Forum: Training Discussion
- Topic: At what point do faces start to 'pop' ?
- Replies: 2
- Views: 5041
At what point do faces start to 'pop' ?
I realise this may be a difficult question to answer as many different variables are involved, but is there any guidance as to when you should see faces start to 'pop', i.e there is defined clarity - eyes are sharp, teeth are defined etc... I'm currently at 300,000 iterations and faces still look so...
- Wed Jun 10, 2020 6:53 pm
- Forum: Training Discussion
- Topic: Training speed over time / Training speed on resume
- Replies: 7
- Views: 6736
Re: Training speed over time / Training speed on resume
Thanks for pointing this out to me. After some investigating it appears the GPU is hitting its thermal limit, the card appears to have quite a relaxed fan profile as default with only 58% fan speed at 100% gpu usage. Ive modified the fan curve so the card is not hitting its thermal limit, and will a...
- Wed Jun 10, 2020 8:30 am
- Forum: Training Discussion
- Topic: Training speed over time / Training speed on resume
- Replies: 7
- Views: 6736
Re: Training speed over time / Training speed on resume
Hmmmmm, so this is not normal ?
It's a dell r720 server with a GTX 1070Ti blower, should have sufficient cooling. Will investigate.
- Tue Jun 09, 2020 11:58 pm
- Forum: Training Discussion
- Topic: Training speed over time / Training speed on resume
- Replies: 7
- Views: 6736
Training speed over time / Training speed on resume
I have a question regarding training speed over time. I believe that it is normal for the training to slow down over time, which is what I am experiencing. When starting off, my EG/s are around 120 EG/s, after about 48 hours they have dropped to 40 EG/s. However... it confuses me that if I stop trai...
- Sun Jun 07, 2020 1:20 pm
- Forum: Extract Discussion
- Topic: Tools > Alignments > Manual not showing faces
- Replies: 1
- Views: 3114
Tools > Alignments > Manual not showing faces
I am trying to examine the alignments file using the manual tool. The 'Frame' window is showing the image / face, but the faces window is not showing anything. The first time I attempted this it worked showing the faces with landmarks, bounding box etc... but now I get nothing... anybody know what t...
- Fri Jun 05, 2020 4:33 pm
- Forum: Training Support
- Topic: Training for a face vs scene / reusing a model
- Replies: 3
- Views: 5118
Training for a face vs scene / reusing a model
Hi, Im trying to understand the training process. I believe that you are training for a face and not a scene per se and you want as many facial expressions, angles and different lighting conditions for both face A and B. So, using the ubiquitous cage to trump, lets say I have the following videos of...