Page 1 of 1
recording and graphiing training history
Posted: Sat Apr 25, 2020 1:13 am
by korrupt78
Would there be any value in recording the iteration/loss values (at the 100x sampling rate used by the snapshotting feature for convenience) and graphing it to watch how the X relates to Y from one model to another?
I found a python wrapper for inotify, and am working on a standalone script to do that myself to see if it produces anything interesting, but it'd be more convenient if it was built-in, obviously.
Re: recording and graphiing training history
Posted: Sat Apr 25, 2020 12:28 pm
by torzdf
I may be misunderstanding, but this is built in.... We store logs in the model folder which can be loaded up in tensorboard.
You can parse these files to extract the data
Re: recording and graphiing training history
Posted: Sat Apr 25, 2020 1:16 pm
by korrupt78
torzdf wrote: ↑Sat Apr 25, 2020 12:28 pm
I may be misunderstanding, but this is built in.... We store logs in the model folder which can be loaded up in tensorboard.
You can parse these files to extract the data
Oh? Is that in the original_logs subdirectory? I haven't explored it because it was opaque (binary) and esoteric compared to original_state.json, which I have looked through.
Re: recording and graphiing training history
Posted: Sat Apr 25, 2020 1:48 pm
by torzdf
Yes. You need to use tensorflow's summary_iterator to parse it.
Example here:
https://github.com/deepfakes/faceswap/b ... ats.py#L68
You can also load the logs straight up into Tensorboard. You can google around how to do that. It's fairly straightforward.