After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
fs2paper
Posts: 1
Joined: Wed Jan 22, 2020 3:31 am

After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Post by fs2paper »

I've checked the FAQ and the closest thing is that it might be a GPU overclock issue. My GPU isn't overclocked but I do believe that my model got corrupted. I've pasted the console log below. Could I assume that the lost number gets so big that it becomes bigger than an int or is this a separate issue? Please let me know if there's any other information required.

Code: Select all


Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\USERNAME\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 1705, in __call__
    return self.func(*args)
  File "C:\Users\USERNAME\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 749, in callit
    func(*args)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 289, in refresh
    self.update_plot(initiate=False)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 122, in update_plot
    self.axes_limits_set(fulldata)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 153, in axes_limits_set
    ymin, ymax = self.axes_data_get_min_max(data)
  File "C:\Users\USERNAME\faceswap\lib\gui\display_graph.py", line 170, in axes_data_get_min_max
    ymin = floor(min(ymin)) / 1000
ValueError: cannot convert float NaN to integer
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Post by bryanlyon »

This is a relatively common problem. Use the restore tool to load the backup and continue from the last good result.

User avatar
djandg
Posts: 43
Joined: Mon Dec 09, 2019 7:00 pm
Has thanked: 4 times
Been thanked: 2 times

Re: After training for 24+ hours, mask turns solid black, and lost A and lost B is nan

Post by djandg »

If it keeps happening it could possibly point to an instability in your power supply or wattage of the PSU is not up to the task ?.

Locked