Noob Training Progress Question

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Balmmaker
Posts: 15
Joined: Wed Nov 17, 2021 1:36 pm
Been thanked: 2 times

Noob Training Progress Question

Post by Balmmaker »

I have been training for 60 hours now on at rtx2080 with 32gb of ram using realface 128k-->256k. Thus far, I have completed 281k iterations the faces appear to be heading in the right direction yet the swapped face is still pretty blurry and so more training is needed.

My questions simply status based: does all that seem right? Is the number of iterations appropriate to the amount of time spent? And finally, about how many iterations might it take before I see significant progress on getting rid of the blur? (I know this can't possibly be defined accurately yet a rough range/ballpark would be nice. Like another couple days of training? And how quickly does the blur disappear? Is this something that progresses at a glacial pace or does the model hit a tipping point and quickly sharpen up?)

Thanks so much.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Noob Training Progress Question

Post by torzdf »

I can't speak specifically for RealFace as it's been a long time since I used it (always had an issue with eye direction), however there are many factors which contribute to blur, not least data.

Also, number of iterations, without including batchsize, is not a good measure. Basically a general rule of thumb is for every doubling of resolution, it will take 4 times longer to train (e.g. if a 64px model takes 12 hours, then a 128px model will take 48 hours)

My word is final

User avatar
Balmmaker
Posts: 15
Joined: Wed Nov 17, 2021 1:36 pm
Been thanked: 2 times

Re: Noob Training Progress Question

Post by Balmmaker »

Ah, well the batch size is 16. The training images are all very sharp and from high quality pictures --so the data is good (i think!). I am now up to 450k iterations and it still looks like a 'blur' filter has been applied to the face, smudging it a bit.

I attached a photo. Just wondering if I am on the right track and just need more training time or if I should be exploring some other corrective measures.

blurry.PNG
blurry.PNG (55.44 KiB) Viewed 1427 times
Locked