I was premature, with your suggested solution training speed decays more slowly but it does decay still.
Look at the Log, its every 1000 its saved, and it starts at 5 minute interval and slowly increases still
09/18/2023 18:23:29 INFO [Saved model] - Average loss since last save: face_a: 0.08979, face_b: 0.26748
09/18/2023 18:23:31 INFO [Preview Updated]
09/18/2023 18:27:57 INFO [Saved model] - Average loss since last save: face_a: 0.11090, face_b: 0.21795
09/18/2023 18:27:59 INFO [Preview Updated]
09/18/2023 18:32:35 INFO [Saved model] - Average loss since last save: face_a: 0.10716, face_b: 0.21045
09/18/2023 18:32:37 INFO [Preview Updated]
09/18/2023 18:38:00 INFO [Saved model] - Average loss since last save: face_a: 0.10275, face_b: 0.20592
09/18/2023 18:38:03 INFO [Preview Updated]
09/18/2023 18:43:45 INFO [Saved model] - Average loss since last save: face_a: 0.10358, face_b: 0.20465
09/18/2023 18:43:47 INFO [Preview Updated]
09/18/2023 18:51:02 INFO [Saved model] - Average loss since last save: face_a: 0.10256, face_b: 0.20235
09/18/2023 18:51:05 INFO [Preview Updated]
09/18/2023 19:00:06 INFO [Saved model] - Average loss since last save: face_a: 0.09863, face_b: 0.20354
09/18/2023 19:00:09 INFO [Preview Updated]
09/18/2023 19:11:28 INFO [Saved model] - Average loss since last save: face_a: 0.10003, face_b: 0.20090
09/18/2023 19:11:32 INFO [Preview Updated]
09/18/2023 19:24:04 INFO [Saved model] - Average loss since last save: face_a: 0.10050, face_b: 0.19862
09/18/2023 19:24:07 INFO [Preview Updated]
09/18/2023 19:37:21 INFO [Saved model] - Average loss since last save: face_a: 0.09910, face_b: 0.19630
09/18/2023 19:37:25 INFO [Preview Updated]