3 small training abnormalities and questions.

[cosmico]

Question 1
When Training for a long time, I've noticed on the overall summary statistic, it will just stop graphing any data or progress for a certain side. (Usually A). Like its will continue to keep graphing B, but just stop for A

But when I click on this session's summary statistic, it will be normal like this:

Is this a glitch? and if not does it actually mean its no longer training, like it hit its max?

Question 2
I'm occasionally getting training stoppages due to a corrupted image in my training data, and it wont start training again until I delete it. No big deal I just delete it and it starts again, but this can't be a good sign. And it can happen either on A or B's data. Is there something wrong I'm doing? example of what corrupted images usually look like

Question 3
Adjusting the eye and mouth multiplier for loss really helps speed up the eye and mouth training, but it doesn't come at the cost of everything else training slowly does it? If I'm happy with the eye and mouth for every image, but need the nose and other parts to train more, should I turn the eye and mouth multiplier off?
.
.

[torzdf]

1) There are some minor issues with the stats optimizations I implemented (I did not test as thoroughly as perhaps I should have). I am aware of these (albeit, not your particular "stops graphing" issue) and will look to fix. It is merely a display issue, not a training issue.

2) I have not seen the corrupted image issue before, so can't help you there. Sorry.

3) Mouth/eye multipliers are a relatively new add, so (given how long training takes) the best use or ratios have not been thoroughly tested. This question may be better placed in it's own topic so people can share knowledge about what works and what doesn't. But, yes, you can adjust it up or down between training sessions. Ultimately, it is just telling the model that the eyes/mouth are more important to the loss score than the rest of the face. It will still learn the rest of the face, but will bias towards eyes and mouth.

[cosmico]

Interesting, thanks.