recording and graphiing training history

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
korrupt78
Posts: 50
Joined: Wed Jan 29, 2020 1:34 am
Has thanked: 2 times
Been thanked: 1 time

recording and graphiing training history

Post by korrupt78 »

Would there be any value in recording the iteration/loss values (at the 100x sampling rate used by the snapshotting feature for convenience) and graphing it to watch how the X relates to Y from one model to another?

I found a python wrapper for inotify, and am working on a standalone script to do that myself to see if it produces anything interesting, but it'd be more convenient if it was built-in, obviously.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: recording and graphiing training history

Post by torzdf »

I may be misunderstanding, but this is built in.... We store logs in the model folder which can be loaded up in tensorboard.

You can parse these files to extract the data

My word is final

User avatar
korrupt78
Posts: 50
Joined: Wed Jan 29, 2020 1:34 am
Has thanked: 2 times
Been thanked: 1 time

Re: recording and graphiing training history

Post by korrupt78 »

torzdf wrote: Sat Apr 25, 2020 12:28 pm

I may be misunderstanding, but this is built in.... We store logs in the model folder which can be loaded up in tensorboard.

You can parse these files to extract the data

Oh? Is that in the original_logs subdirectory? I haven't explored it because it was opaque (binary) and esoteric compared to original_state.json, which I have looked through.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: recording and graphiing training history

Post by torzdf »

Yes. You need to use tensorflow's summary_iterator to parse it.

Example here:
https://github.com/deepfakes/faceswap/b ... ats.py#L68

You can also load the logs straight up into Tensorboard. You can google around how to do that. It's fairly straightforward.

My word is final

Locked