Page 1 of 1

Way to notify training failure?

Posted: Tue Mar 10, 2020 5:56 am
by DMCMachine

Been running into a problem where during training, Faceswap will sometimes exit out of the process, either because it ran out of VRAM or something else. Problem is, I usually start the process before leaving for work or bed so it can run while I'm not using my computer.

My question is, does anyone know of a way to get Faceswap to notify me remotely. I don't expect to do this within the program natively, but maybe some code/process running alongside Faceswap, that when my computer sees the failure notification in the Faceswap command line and it can send a message to my phone or something.

I know this may be way out there in terms of capability, but I figured I'd ask anyways. Just hate having to either constantly remote in to see if it's going or wake up to find that it quit an hour after I started it and I lost a lot of time.

In the mean time, I'm trying to figure out the pattern to why it shuts down the process because I usually don't have anything else running when I am training to give it as many resources my PC has. I'm pretty novice to this stuff so I am learning by doing (and failing a lot haha).

Thanks in advance!


Re: Way to notify training failure?

Posted: Thu Mar 12, 2020 8:30 pm
by blackomen

Maybe the devs can offer an option of adding IFTTT integration...


Re: Way to notify training failure?

Posted: Thu Mar 12, 2020 8:33 pm
by bryanlyon
blackomen wrote: Thu Mar 12, 2020 8:30 pm

Maybe the devs can offer an option of adding IFTTT integration...

Not really likely to happen. This would be extremely fiddly and cause issues for some workcases. You're going to do best finding a soltuion on your own. Something like keeping track of Python processes or using "generate" and adding the command to a batch file with a message sent right after the training to alert you. There are lots of options for this honestly pretty niche need that we're unlikely to implement anything (especially anything online which will cause problems for users that use it offline).


Re: Way to notify training failure?

Posted: Fri Mar 13, 2020 11:25 am
by torzdf

As Bryan says, we won't be implementing anything like this in Faceswap

Honestly, your best bet is to run a script to monitor the process, which then just kicks off Faceswap again if the process dies.


Re: Way to notify training failure?

Posted: Sat Mar 14, 2020 12:49 am
by djandg

You could run Tight VNC service (remote access) on the training machine and the Ripple remote version on your smart phone,. That way you can monitor, restart, and even change parameters such as batch size remotely. Tight VNC/Ripple works best if you have a static IP, just in case of a IP reset caused by power outage or similar.
It won't notify if training stops, you still have to check yourself at timely intervals but at least you can restart if it does.

https://www.tightvnc.com/

Remote Ripple is available for IPhone and Android, search "remote ripple" in the relative stores.


Re: Way to notify training failure?

Posted: Tue Mar 24, 2020 5:33 am
by DMCMachine

Thanks guys. Makes sense. I figured it was kind of out there but wanted to ask on the off chance. I may look into djang said.

Thanks again!