An error I've never seen...

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 177 times
Been thanked: 13 times

An error I've never seen...

Post by MaxHunter »

I've never seen this before. Should I be worried? The last line is what really concerns me.,

Code: Select all

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\e4978\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 1892, in __call__
    return self.func(*args)
  File "C:\Users\e4978\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 814, in callit
    func(*args)
  File "C:\Users\e4978\faceswap\lib\gui\display_analysis.py", line 190, in <lambda>
    self.after(1000, lambda msg=message: self._set_session_summary(msg))
  File "C:\Users\e4978\faceswap\lib\gui\display_analysis.py", line 196, in _set_session_summary
    result = self._thread.get_result()
  File "C:\Users\e4978\faceswap\lib\gui\utils\misc.py", line 101, in get_result
    raise self.err[1].with_traceback(self.err[2])
  File "C:\Users\e4978\faceswap\lib\gui\utils\misc.py", line 70, in run
    retval = self._target(*self._args, **self._kwargs)
  File "C:\Users\e4978\faceswap\lib\gui\display_analysis.py", line 215, in _summarise_data
    return session.full_summary
  File "C:\Users\e4978\faceswap\lib\gui\analysis\stats.py", line 75, in full_summary
    return self._summary.get_summary_stats()
  File "C:\Users\e4978\faceswap\lib\gui\analysis\stats.py", line 319, in get_summary_stats
    self._get_time_stats()
  File "C:\Users\e4978\faceswap\lib\gui\analysis\stats.py", line 352, in _get_time_stats
    latest = cast(np.ndarray, self._session.get_timestamps(session_id))
  File "C:\Users\e4978\faceswap\lib\gui\analysis\stats.py", line 233, in get_timestamps
    retval = self._tb_logs.get_timestamps(session_id=session_id)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 614, in get_timestamps
    self._check_cache(idx)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 555, in _check_cache
    self._cache_data(session_id)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 541, in _cache_data
    parser.cache_events(session_id)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 707, in cache_events
    self._cache.cache_data(session_id, data, self._loss_labels, is_live=self._live_data)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 279, in cache_data
    self._add_latest_live(session_id, loss, timestamps)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 420, in _add_latest_live
    self._data[session_id].add_live_data(timestamps, loss)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 205, in add_live_data
    old = old.reshape(-1, *data.shape[1:])
ValueError: cannot reshape array of size 538 into shape (4)
Last edited by MaxHunter on Sat Apr 29, 2023 7:53 pm, edited 1 time in total.
User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 177 times
Been thanked: 13 times

Re: An error I've never seen...

Post by MaxHunter »

Okay. So apparently this is affecting my A face from learning. Why? Is it a setting? Would my dimensions setting in Phaze A be the cause? My B face is learning just fine and seems to be placed on my A face in the previews, but is it learning properly if my A Face isn't learning? Hmmm....

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: An error I've never seen...

Post by torzdf »

Ignore it

It's a logging error and just means your live graph won't work.

It only impacts live data. It happens sometimes (due to Tensorboard event reader). I have put many mitigations in place and haven't hit that error for a while. Not sure why you still get it tbh.

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 177 times
Been thanked: 13 times

Re: An error I've never seen...

Post by MaxHunter »

Here's what I'm confused about: My A Faces' loss is stuck in the .60's while my B face continues to learn (currently in the .04's.) I'm honestly more concerned with my B face learning. Even though the A face is stuck, is the B face still learning the A face, or is it just learning it's own face without learning how to replace the A face?

User avatar
Andentze
Posts: 15
Joined: Mon Dec 06, 2021 6:34 pm
Has thanked: 1 time
Been thanked: 21 times

Re: An error I've never seen...

Post by Andentze »

Does the preview still look like its improving? If so, then it's just Tensorboard and/or Tensorflow bugging out and not showing loss correctly, so you can just dismiss this "error" and keep training.

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 177 times
Been thanked: 13 times

Re: An error I've never seen...

Post by MaxHunter »

Thanks! I'll keep going then. It'll be interesting to see what happens if I load these weights when it's finished into another model. 🤔

I thought maybe this had something to do with "FC dimensions" being wrong, as I never mess with that function.

Last edited by MaxHunter on Mon May 01, 2023 7:43 pm, edited 1 time in total.
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: An error I've never seen...

Post by torzdf »

MaxHunter wrote: Mon May 01, 2023 4:11 pm

Here's what I'm confused about: My A Faces' loss is stuck in the .60's while my B face continues to learn (currently in the .04's.) I'm honestly more concerned with my B face learning. Even though the A face is stuck, is the B face still learning the A face, or is it just learning it's own face without learning how to replace the A face?

That's not right. So is the A loss not moving at all? That's not something I've seen before. The loss that is updated in the status bar at the bottom is read from a different place than where the graph/analysis tab gets its data. If the status bar is not updating, then something is very wrong.

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 177 times
Been thanked: 13 times

Re: An error I've never seen...

Post by MaxHunter »

Sorry for the late reply, I burned out my Motherboard somehow so I've been on the downlow.

It's crazy, it just sat and bounced around the .60s and wouldn't go down, or UP, and no picture in the preview. On the other hand the B side was fine and was masking the A side with(seemingly) little difficulty. I loaded weights at the beginning, but I was concerned if the B side loss-rate was going down only based on the loaded weights, and not actually learning.

Like I said, I apparently did something to my board (Ugh, My Asrock Taichi didn't last longer than 6 months!) I'm having to RMA it this week, so I can't give you more details.

Last edited by MaxHunter on Sat May 06, 2023 10:16 pm, edited 2 times in total.
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: An error I've never seen...

Post by torzdf »

MaxHunter wrote: Sat May 06, 2023 10:07 pm

It's crazy, it just sat and bounced around the .60s and wouldn't go down, or UP, and no picture in the preview.

Sounds like model corruption to me on the A side. If the loss is changinf (even if it is not trending anywhere) then the model is doing something. It's just not doing anything useful, sadly.

My word is final

User avatar
MaxHunter
Posts: 193
Joined: Thu May 26, 2022 6:02 am
Has thanked: 177 times
Been thanked: 13 times

Re: An error I've never seen...

Post by MaxHunter »

I just got my board back last night and decided to just start from scratch. Before I was reusing model weights. There must have been something in the model weights, a corruption perhaps, because I haven't had any problems as of yet.

Thanks for your insight. 😉

Locked