Would there be any value in recording the iteration/loss values (at the 100x sampling rate used by the snapshotting feature for convenience) and graphing it to watch how the X relates to Y from one model to another?
I found a python wrapper for inotify, and am working on a standalone script to do that myself to see if it produces anything interesting, but it'd be more convenient if it was built-in, obviously.
I may be misunderstanding, but this is built in.... We store logs in the model folder which can be loaded up in tensorboard.
You can parse these files to extract the data
Oh? Is that in the original_logs subdirectory? I haven't explored it because it was opaque (binary) and esoteric compared to original_state.json, which I have looked through.