Model load error

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
bba20
Posts: 11
Joined: Tue Sep 07, 2021 3:29 pm
Has thanked: 3 times
Been thanked: 2 times

Model load error

Post by bba20 »

I've had this model since earlier this year but when I attempted to load recently I get the following in the display window:

Code: Select all

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\XX\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 1921, in __call__
    return self.func(*args)
  File "C:\Users\XX\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 839, in callit
    func(*args)
  File "C:\Users\XX\faceswap\lib\gui\display_analysis.py", line 193, in <lambda>
    self.after(1000, lambda msg=message: self._set_session_summary(msg))
  File "C:\Users\XX\faceswap\lib\gui\display_analysis.py", line 196, in _set_session_summary
    result = self._thread.get_result()
  File "C:\Users\XX\faceswap\lib\gui\utils\misc.py", line 103, in get_result
    raise self.err[1].with_traceback(self.err[2])
  File "C:\Users\XX\faceswap\lib\gui\utils\misc.py", line 72, in run
    retval = self._target(*self._args, **self._kwargs)
  File "C:\Users\XX\faceswap\lib\gui\display_analysis.py", line 215, in _summarise_data
    return session.full_summary
  File "C:\Users\XX\faceswap\lib\gui\analysis\stats.py", line 74, in full_summary
    return self._summary.get_summary_stats()
  File "C:\Users\XX\faceswap\lib\gui\analysis\stats.py", line 313, in get_summary_stats
    self._get_time_stats()
  File "C:\Users\XX\faceswap\lib\gui\analysis\stats.py", line 340, in _get_time_stats
    self._session.get_timestamps(None)).items()}
  File "C:\Users\XX\faceswap\lib\gui\analysis\stats.py", line 232, in get_timestamps
    retval = self._tb_logs.get_timestamps(session_id=session_id)
  File "C:\Users\XX\faceswap\lib\gui\analysis\event_reader.py", line 609, in get_timestamps
    self._check_cache(idx)
  File "C:\Users\XX\faceswap\lib\gui\analysis\event_reader.py", line 548, in _check_cache
    self._cache_data(session_id)
  File "C:\Users\XX\faceswap\lib\gui\analysis\event_reader.py", line 536, in _cache_data
    parser.cache_events(session_id)
  File "C:\Users\XX\faceswap\lib\gui\analysis\event_reader.py", line 699, in cache_events
    self._cache.cache_data(session_id, data, self._loss_labels, is_live=self._live_data)
  File "C:\Users\XX\faceswap\lib\gui\analysis\event_reader.py", line 269, in cache_data
    timestamps, loss = self._to_numpy(data, is_live)
  File "C:\Users\XX\faceswap\lib\gui\analysis\event_reader.py", line 326, in _to_numpy
    n_times, n_loss = (np.array(times, dtype="float64"), np.array(loss, dtype="float32"))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (16835,) + inhomogeneous part.
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Model load error

Post by torzdf »

Can you zip up the model folder and provide me with a link to download.

I need to recreate this error to be able to fix it. Thanks

My word is final

User avatar
bba20
Posts: 11
Joined: Tue Sep 07, 2021 3:29 pm
Has thanked: 3 times
Been thanked: 2 times

Re: Model load error

Post by bba20 »

Sorry I think I jumped the gun. I just clicked train and it began training again fine. It said I couldn't switch from Full Precision to Mixed Precision on older model file. The last time I trained this file I was using my older videocard and my newer one I turned on mixed precision. I thought the model was corrupted at first when I got the error but it just wasn't showing the session data like usual. Edit: Also I started training again with mixed precision on and it didn't revert to Full like the first time. It's just not showing the session data which I dont care about but the model is fine.

Edit 2: I deleted my old session files and it fixed the error. It shows session data again but it just looks like a brand new session.

Last edited by bba20 on Sat Sep 09, 2023 4:32 pm, edited 3 times in total.
Locked