This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.
Please mark any answers that fixed your problems so others can find the solutions.
2023-10-15 12:28:59.400559: F tensorflow/core/common_runtime/device/device_event_mgr.cc:221] Unexpected Event status: 1
No idea how to fix it.
This happens randomly while training, sometimes an hour in, sometimes 20 min. However if I can get it to go for longer than 4/5 hours it seems to run fine.
My drivers are up to date, my cuda is up to date...
As the previous message said, this is a driver issue. Nothing to do with Faceswap and nothing we can do to prevent or work around it.
As far as I've been able to find on it though, it's almost 100% an overclocking issue. Chances are the card you have is factory overclocked and it's hitting instability because of it. Clock it back to Nvidia's recommendations and I think you'll see this message disappear completely.
