Way to notify training failure?

Discussions about research, Faceswapping and things that don't fit in the other categories here.


Locked
User avatar
DMCMachine
Posts: 2
Joined: Sun Feb 09, 2020 6:57 am

Way to notify training failure?

Post by DMCMachine »

Been running into a problem where during training, Faceswap will sometimes exit out of the process, either because it ran out of VRAM or something else. Problem is, I usually start the process before leaving for work or bed so it can run while I'm not using my computer.

My question is, does anyone know of a way to get Faceswap to notify me remotely. I don't expect to do this within the program natively, but maybe some code/process running alongside Faceswap, that when my computer sees the failure notification in the Faceswap command line and it can send a message to my phone or something.

I know this may be way out there in terms of capability, but I figured I'd ask anyways. Just hate having to either constantly remote in to see if it's going or wake up to find that it quit an hour after I started it and I lost a lot of time.

In the mean time, I'm trying to figure out the pattern to why it shuts down the process because I usually don't have anything else running when I am training to give it as many resources my PC has. I'm pretty novice to this stuff so I am learning by doing (and failing a lot haha).

Thanks in advance!

User avatar
blackomen
Posts: 9
Joined: Fri Feb 21, 2020 2:14 pm
Has thanked: 2 times
Been thanked: 1 time

Re: Way to notify training failure?

Post by blackomen »

Maybe the devs can offer an option of adding IFTTT integration...

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Way to notify training failure?

Post by bryanlyon »

blackomen wrote: Thu Mar 12, 2020 8:30 pm

Maybe the devs can offer an option of adding IFTTT integration...

Not really likely to happen. This would be extremely fiddly and cause issues for some workcases. You're going to do best finding a soltuion on your own. Something like keeping track of Python processes or using "generate" and adding the command to a batch file with a message sent right after the training to alert you. There are lots of options for this honestly pretty niche need that we're unlikely to implement anything (especially anything online which will cause problems for users that use it offline).

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Way to notify training failure?

Post by torzdf »

As Bryan says, we won't be implementing anything like this in Faceswap

Honestly, your best bet is to run a script to monitor the process, which then just kicks off Faceswap again if the process dies.

My word is final

User avatar
djandg
Posts: 43
Joined: Mon Dec 09, 2019 7:00 pm
Has thanked: 4 times
Been thanked: 2 times

Re: Way to notify training failure?

Post by djandg »

You could run Tight VNC service (remote access) on the training machine and the Ripple remote version on your smart phone,. That way you can monitor, restart, and even change parameters such as batch size remotely. Tight VNC/Ripple works best if you have a static IP, just in case of a IP reset caused by power outage or similar.
It won't notify if training stops, you still have to check yourself at timely intervals but at least you can restart if it does.

https://www.tightvnc.com/

Remote Ripple is available for IPhone and Android, search "remote ripple" in the relative stores.

User avatar
DMCMachine
Posts: 2
Joined: Sun Feb 09, 2020 6:57 am

Re: Way to notify training failure?

Post by DMCMachine »

Thanks guys. Makes sense. I figured it was kind of out there but wanted to ask on the off chance. I may look into djang said.

Thanks again!

Locked