No masks in several columns. Is it normal?

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
tokafondo
Posts: 32
Joined: Mon Dec 16, 2019 1:43 pm
Has thanked: 10 times
Been thanked: 5 times

No masks in several columns. Is it normal?

Post by tokafondo »

_Preview_20210105_154420.jpg
_Preview_20210105_154420.jpg (180.86 KiB) Viewed 3313 times

Model: Realface

Already refreshed and updated masks in both Original and Swap sources:faces and frames.
Masks exported successfully as independent files just to check.
Masks shown in Manual Tool.

Is is normal?

User avatar
tokafondo
Posts: 32
Joined: Mon Dec 16, 2019 1:43 pm
Has thanked: 10 times
Been thanked: 5 times

Re: No masks in several columns. Is it normal?

Post by tokafondo »

UPDATE: Restarted with Unbalanced model and no masks.

This is shown in LOG screen:

Code: Select all

Exception in Tkinter callback
Traceback (most recent call last):
  File "/home/yo/miniconda3/envs/faceswap/lib/python3.8/tkinter/__init__.py", line 1883, in __call__
    return self.func(*args)
  File "/home/yo/miniconda3/envs/faceswap/lib/python3.8/tkinter/__init__.py", line 804, in callit
    func(*args)
  File "/home/yo/faceswap/lib/gui/display_graph.py", line 300, in refresh
    self.calcs = self.thread.get_result()  # Terminate the LongRunningTask object
  File "/home/yo/faceswap/lib/gui/utils.py", line 1151, in get_result
    raise self.err[1].with_traceback(self.err[2])
  File "/home/yo/faceswap/lib/gui/utils.py", line 1122, in run
    retval = self._target(*self._args, **self._kwargs)
  File "/home/yo/faceswap/lib/gui/stats.py", line 862, in refresh
    self._get_raw()
  File "/home/yo/faceswap/lib/gui/stats.py", line 925, in _get_raw
    loss_dict = Session.get_loss(self._session_id)
  File "/home/yo/faceswap/lib/gui/stats.py", line 168, in get_loss
    loss_dict = self._tb_logs.get_loss(session_id=session_id, is_training=self._is_training)
  File "/home/yo/faceswap/lib/gui/stats.py", line 514, in get_loss
    for sess, info in self._from_cache(session_id, is_training).items():
  File "/home/yo/faceswap/lib/gui/stats.py", line 482, in _from_cache
    self._cache_data(session_id)
  File "/home/yo/faceswap/lib/gui/stats.py", line 326, in _cache_data
    iterator = tf.compat.v1.io.tf_record_iterator(self._log_filenames[session_id])
KeyError: 11
User avatar
tokafondo
Posts: 32
Joined: Mon Dec 16, 2019 1:43 pm
Has thanked: 10 times
Been thanked: 5 times

Re: No masks in several columns. Is it normal?

Post by tokafondo »

tokafondo wrote: Tue Jan 05, 2021 4:16 pm

UPDATE: Restarted with Unbalanced model and no masks.

This did not helped at all.

I restarted again with DFL-SAE with the same sources, masks and everything and now the masks are shown in the right columns. No more red squares.

Preview_20210105_163417.jpg
Preview_20210105_163417.jpg (138.31 KiB) Viewed 3309 times
User avatar
tokafondo
Posts: 32
Joined: Mon Dec 16, 2019 1:43 pm
Has thanked: 10 times
Been thanked: 5 times

Re: No masks in several columns. Is it normal?

Post by tokafondo »

tokafondo wrote: Tue Jan 05, 2021 4:37 pm

I restarted again with DFL-SAE with the same sources, masks and everything and now the masks are shown in the right columns. No more red squares.

I was not convinced at all, so I restarted again with the same, original settings that produced that "no mask applied" red squares, but from scratch - using the very same faces, alignments and masks.

And now it works. :shock: :shock: :shock: 8-) 8-) 8-)

Realface 128/192, BS 10, ssim, mae, Extended, PLM On.

Preview_20210105_170033.jpg
Preview_20210105_170033.jpg (101.19 KiB) Viewed 3309 times
User avatar
torzdf
Posts: 2651
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 129 times
Been thanked: 622 times

Re: No masks in several columns. Is it normal?

Post by torzdf »

Glad it works. Not sure what happened there.

My word is final

User avatar
tokafondo
Posts: 32
Joined: Mon Dec 16, 2019 1:43 pm
Has thanked: 10 times
Been thanked: 5 times

Re: No masks in several columns. Is it normal?

Post by tokafondo »

I'm back on this issue. Cannot tell what the heck in this about. Reinstalled everything (faceswap, miniconda3) from scratch. Redone all extractions, alignments and masks from scratch.

And the issue is still again. I'm thinking about a complete OS reinstall because this is not something usual at all.

Also, the error...

Code: Select all

Exception in Tkinter callback
Traceback (most recent call last):
  File "/home/yo/miniconda3/envs/faceswap/lib/python3.8/tkinter/__init__.py", line 1883, in __call__
    return self.func(*args)
  File "/home/yo/miniconda3/envs/faceswap/lib/python3.8/tkinter/__init__.py", line 804, in callit
    func(*args)
  File "/home/yo/faceswap/lib/gui/display_graph.py", line 300, in refresh
    self.calcs = self.thread.get_result()  # Terminate the LongRunningTask object
  File "/home/yo/faceswap/lib/gui/utils.py", line 1151, in get_result
    raise self.err[1].with_traceback(self.err[2])
  File "/home/yo/faceswap/lib/gui/utils.py", line 1122, in run
    retval = self._target(*self._args, **self._kwargs)
  File "/home/yo/faceswap/lib/gui/stats.py", line 862, in refresh
    self._get_raw()
  File "/home/yo/faceswap/lib/gui/stats.py", line 925, in _get_raw
    loss_dict = Session.get_loss(self._session_id)
  File "/home/yo/faceswap/lib/gui/stats.py", line 168, in get_loss
    loss_dict = self._tb_logs.get_loss(session_id=session_id, is_training=self._is_training)
  File "/home/yo/faceswap/lib/gui/stats.py", line 514, in get_loss
    for sess, info in self._from_cache(session_id, is_training).items():
  File "/home/yo/faceswap/lib/gui/stats.py", line 482, in _from_cache
    self._cache_data(session_id)
  File "/home/yo/faceswap/lib/gui/stats.py", line 326, in _cache_data
    iterator = tf.compat.v1.io.tf_record_iterator(self._log_filenames[session_id])
KeyError: 3

...still showing.

And after an almost 600K iterations realface model traning (BS 8), the results are not satisfactory at all, even leaving everything as default.

I'm thinking on giving up on faceswap or deepfajes in general.

Locked