After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Training your model
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Locked
User avatar
fs2paper
Posts: 1
Joined: Wed Jan 22, 2020 3:31 am

After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Post by fs2paper »

I've checked the FAQ and the closest thing is that it might be a GPU overclock issue. My GPU isn't overclocked but I do believe that my model got corrupted. I've pasted the console log below. Could I assume that the lost number gets so big that it becomes bigger than an int or is this a separate issue? Please let me know if there's any other information required.

Code: Select all


Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\USERNAME\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 1705, in __call__
    return self.func(*args)
  File "C:\Users\USERNAME\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 749, in callit
    func(*args)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 289, in refresh
    self.update_plot(initiate=False)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 122, in update_plot
    self.axes_limits_set(fulldata)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 153, in axes_limits_set
    ymin, ymax = self.axes_data_get_min_max(data)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 170, in axes_data_get_min_max
    ymin = floor(min(ymin)) / 1000
ValueError: cannot convert float NaN to integer

User avatar
bryanlyon
Site Admin
Posts: 419
Joined: Fri Jul 12, 2019 12:49 am
Answers: 33
Location: San Francisco
Has thanked: 3 times
Been thanked: 104 times
Contact:

Re: After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Post by bryanlyon »

This is a relatively common problem. Use the restore tool to load the backup and continue from the last good result.

User avatar
djandg
Posts: 38
Joined: Mon Dec 09, 2019 7:00 pm
Has thanked: 3 times
Been thanked: 2 times

Re: After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Post by djandg »

If it keeps happening it could possibly point to an instability in your power supply or wattage of the PSU is not up to the task ?.

Locked