Page 1 of 1

After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Posted: Wed Jan 22, 2020 3:51 am
by fs2paper
I've checked the FAQ and the closest thing is that it might be a GPU overclock issue. My GPU isn't overclocked but I do believe that my model got corrupted. I've pasted the console log below. Could I assume that the lost number gets so big that it becomes bigger than an int or is this a separate issue? Please let me know if there's any other information required.

Code: Select all


Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\USERNAME\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 1705, in __call__
    return self.func(*args)
  File "C:\Users\USERNAME\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 749, in callit
    func(*args)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 289, in refresh
    self.update_plot(initiate=False)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 122, in update_plot
    self.axes_limits_set(fulldata)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 153, in axes_limits_set
    ymin, ymax = self.axes_data_get_min_max(data)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 170, in axes_data_get_min_max
    ymin = floor(min(ymin)) / 1000
ValueError: cannot convert float NaN to integer

Re: After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Posted: Wed Jan 22, 2020 5:08 am
by bryanlyon
This is a relatively common problem. Use the restore tool to load the backup and continue from the last good result.

Re: After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Posted: Wed Jan 22, 2020 1:57 pm
by djandg
If it keeps happening it could possibly point to an instability in your power supply or wattage of the PSU is not up to the task ?.