After training for 24+ hours, mask turns solid black, and lost A and lost B is nan
Posted: Wed Jan 22, 2020 3:51 am
I've checked the FAQ and the closest thing is that it might be a GPU overclock issue. My GPU isn't overclocked but I do believe that my model got corrupted. I've pasted the console log below. Could I assume that the lost number gets so big that it becomes bigger than an int or is this a separate issue? Please let me know if there's any other information required.
Code: Select all
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\USERNAME\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 1705, in __call__
return self.func(*args)
File "C:\Users\USERNAME\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 749, in callit
func(*args)
File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 289, in refresh
self.update_plot(initiate=False)
File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 122, in update_plot
self.axes_limits_set(fulldata)
File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 153, in axes_limits_set
ymin, ymax = self.axes_data_get_min_max(data)
File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 170, in axes_data_get_min_max
ymin = floor(min(ymin)) / 1000
ValueError: cannot convert float NaN to integer