Training halfway then swap? Start over again?

Training your model
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Post Reply
User avatar
maplingstorie
Posts: 4
Joined: Thu Feb 13, 2020 7:42 pm
Has thanked: 2 times

Training halfway then swap? Start over again?

Post by maplingstorie » Thu Feb 13, 2020 7:47 pm

So basically, I have A(video source) and B (swap source) and I'm on the 12k iteration and realised my training was wrong as it should be the other way around. So I basically stopped the training and saved and redid the command but realised my model training started all over again however it resumed from the 12k iteration but the pictures were blurry. Anyway I can resume training? or do I have to start all over again?

I'm using CLI not GUI btw.
by bryanlyon » Thu Feb 13, 2020 7:50 pm
If you accidentally swapped it, I'd suggest restoring from an earlier snapshot and continuing from that instead of continuing with it swapped. You can then tell convert at the end to swap the model and still get your results in the right direction. If you're on Dlight or Realface then you'll actually need to restart since these models work asymmetrically.
Go to full post

User avatar
bryanlyon
Site Admin
Posts: 197
Joined: Fri Jul 12, 2019 12:49 am
Answers: 19
Location: San Francisco
Has thanked: 3 times
Been thanked: 57 times
Contact:

Re: Training halfway then swap? Start over again?

Post by bryanlyon » Thu Feb 13, 2020 7:50 pm

If you accidentally swapped it, I'd suggest restoring from an earlier snapshot and continuing from that instead of continuing with it swapped. You can then tell convert at the end to swap the model and still get your results in the right direction. If you're on Dlight or Realface then you'll actually need to restart since these models work asymmetrically.

User avatar
maplingstorie
Posts: 4
Joined: Thu Feb 13, 2020 7:42 pm
Has thanked: 2 times

Re: Training halfway then swap? Start over again?

Post by maplingstorie » Thu Feb 13, 2020 7:57 pm

bryanlyon wrote:
Thu Feb 13, 2020 7:50 pm
If you accidentally swapped it, I'd suggest restoring from an earlier snapshot and continuing from that instead of continuing with it swapped. You can then tell convert at the end to swap the model and still get your results in the right direction. If you're on Dlight or Realface then you'll actually need to restart since these models work asymmetrically.
thanks. may I know how do I go about restoring from an earlier snapshot?

User avatar
maplingstorie
Posts: 4
Joined: Thu Feb 13, 2020 7:42 pm
Has thanked: 2 times

Re: Training halfway then swap? Start over again?

Post by maplingstorie » Thu Feb 13, 2020 8:06 pm

never mind. I think I've got it. I ran the GUI and used the tool and pointed to my model dir and let it restore my backup and ran back the re-swapped back the A->B . preview is showing and it's back to normal.

If this is what you meant then thanks! or if not, please do guide me. thanks!

edit: could you explain how it knows which one to restore? and if so, when did it backup?

User avatar
bryanlyon
Site Admin
Posts: 197
Joined: Fri Jul 12, 2019 12:49 am
Answers: 19
Location: San Francisco
Has thanked: 3 times
Been thanked: 57 times
Contact:

Re: Training halfway then swap? Start over again?

Post by bryanlyon » Thu Feb 13, 2020 9:23 pm

The training process saves a backup copy of the model each time the loss goes down. The loss wouldn't have caught up from a short train while reversed so wouldn't overwrite the previous best score. There are also snapshots saved separately every (by default) 25000 iterations.

Post Reply