Faceswap Forum

I've never seen this before. Should I be worried? The last line is what really concerns me.,

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\e4978\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 1892, in __call__
    return self.func(*args)
  File "C:\Users\e4978\MiniConda3\envs\faceswap\lib\tkinter\__init__.py", line 814, in callit
    func(*args)
  File "C:\Users\e4978\faceswap\lib\gui\display_analysis.py", line 190, in <lambda>
    self.after(1000, lambda msg=message: self._set_session_summary(msg))
  File "C:\Users\e4978\faceswap\lib\gui\display_analysis.py", line 196, in _set_session_summary
    result = self._thread.get_result()
  File "C:\Users\e4978\faceswap\lib\gui\utils\misc.py", line 101, in get_result
    raise self.err[1].with_traceback(self.err[2])
  File "C:\Users\e4978\faceswap\lib\gui\utils\misc.py", line 70, in run
    retval = self._target(*self._args, **self._kwargs)
  File "C:\Users\e4978\faceswap\lib\gui\display_analysis.py", line 215, in _summarise_data
    return session.full_summary
  File "C:\Users\e4978\faceswap\lib\gui\analysis\stats.py", line 75, in full_summary
    return self._summary.get_summary_stats()
  File "C:\Users\e4978\faceswap\lib\gui\analysis\stats.py", line 319, in get_summary_stats
    self._get_time_stats()
  File "C:\Users\e4978\faceswap\lib\gui\analysis\stats.py", line 352, in _get_time_stats
    latest = cast(np.ndarray, self._session.get_timestamps(session_id))
  File "C:\Users\e4978\faceswap\lib\gui\analysis\stats.py", line 233, in get_timestamps
    retval = self._tb_logs.get_timestamps(session_id=session_id)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 614, in get_timestamps
    self._check_cache(idx)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 555, in _check_cache
    self._cache_data(session_id)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 541, in _cache_data
    parser.cache_events(session_id)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 707, in cache_events
    self._cache.cache_data(session_id, data, self._loss_labels, is_live=self._live_data)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 279, in cache_data
    self._add_latest_live(session_id, loss, timestamps)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 420, in _add_latest_live
    self._data[session_id].add_live_data(timestamps, loss)
  File "C:\Users\e4978\faceswap\lib\gui\analysis\event_reader.py", line 205, in add_live_data
    old = old.reshape(-1, *data.shape[1:])
ValueError: cannot reshape array of size 538 into shape (4)

Okay. So apparently this is affecting my A face from learning. Why? Is it a setting? Would my dimensions setting in Phaze A be the cause? My B face is learning just fine and seems to be placed on my A face in the previews, but is it learning properly if my A Face isn't learning? Hmmm....

Ignore it

It's a logging error and just means your live graph won't work.

It only impacts live data. It happens sometimes (due to Tensorboard event reader). I have put many mitigations in place and haven't hit that error for a while. Not sure why you still get it tbh.

Here's what I'm confused about: My A Faces' loss is stuck in the .60's while my B face continues to learn (currently in the .04's.) I'm honestly more concerned with my B face learning. Even though the A face is stuck, is the B face still learning the A face, or is it just learning it's own face without learning how to replace the A face?

Does the preview still look like its improving? If so, then it's just Tensorboard and/or Tensorflow bugging out and not showing loss correctly, so you can just dismiss this "error" and keep training.

Thanks! I'll keep going then. It'll be interesting to see what happens if I load these weights when it's finished into another model.

I thought maybe this had something to do with "FC dimensions" being wrong, as I never mess with that function.

MaxHunter wrote: ↑Mon May 01, 2023 4:11 pm

Here's what I'm confused about: My A Faces' loss is stuck in the .60's while my B face continues to learn (currently in the .04's.) I'm honestly more concerned with my B face learning. Even though the A face is stuck, is the B face still learning the A face, or is it just learning it's own face without learning how to replace the A face?

That's not right. So is the A loss not moving at all? That's not something I've seen before. The loss that is updated in the status bar at the bottom is read from a different place than where the graph/analysis tab gets its data. If the status bar is not updating, then something is very wrong.

Sorry for the late reply, I burned out my Motherboard somehow so I've been on the downlow.

It's crazy, it just sat and bounced around the .60s and wouldn't go down, or UP, and no picture in the preview. On the other hand the B side was fine and was masking the A side with(seemingly) little difficulty. I loaded weights at the beginning, but I was concerned if the B side loss-rate was going down only based on the loaded weights, and not actually learning.

Like I said, I apparently did something to my board (Ugh, My Asrock Taichi didn't last longer than 6 months!) I'm having to RMA it this week, so I can't give you more details.

MaxHunter wrote: ↑Sat May 06, 2023 10:07 pm

It's crazy, it just sat and bounced around the .60s and wouldn't go down, or UP, and no picture in the preview.

Sounds like model corruption to me on the A side. If the loss is changinf (even if it is not trending anywhere) then the model is doing something. It's just not doing anything useful, sadly.

I just got my board back last night and decided to just start from scratch. Before I was reusing model weights. There must have been something in the model weights, a corruption perhaps, because I haven't had any problems as of yet.

Thanks for your insight.