Cannot start training with unbalance

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
pmvkpop@gmail.com
Posts: 2
Joined: Wed Dec 07, 2022 9:14 pm
Been thanked: 1 time

Cannot start training with unbalance

Post by pmvkpop@gmail.com »

I tried to run Unbalance with 512x512 and I got an error message.

it was fine with 320x320

what did I do wrong? :?

Code: Select all

===================================================
12/07/2022 16:07:33 INFO       Starting
12/07/2022 16:07:33 INFO     ===================================================
12/07/2022 16:07:34 INFO     Loading data, this may take a while...
12/07/2022 16:07:34 INFO     Loading Model from Unbalanced plugin...
12/07/2022 16:07:34 INFO     No existing state file found. Generating.
12/07/2022 16:07:34 INFO     Mixed precision compatibility check (mixed_float16): OK\nYour GPU will likely run quickly with dtype policy mixed_float16 as it has compute capability of at least 7.0. Your GPU: NVIDIA GeForce GTX 1660 Ti, compute capability 7.5
12/07/2022 16:07:34 INFO     Enabling Mixed Precision Training.
12/07/2022 16:07:35 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 3, 144)
12/07/2022 16:07:35 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 144, 288)
12/07/2022 16:07:35 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 288, 576)
12/07/2022 16:07:36 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 576, 864)
12/07/2022 16:07:38 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 864, 1152)
12/07/2022 16:07:43 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 512, 1600)
12/07/2022 16:07:46 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 400, 1600)
12/07/2022 16:07:49 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 400, 800)
12/07/2022 16:07:50 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 200, 400)
12/07/2022 16:07:51 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 100, 3)
12/07/2022 16:07:51 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 1600)
12/07/2022 16:07:53 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 400, 1600)
12/07/2022 16:07:54 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 400, 800)
12/07/2022 16:07:55 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 200, 400)
12/07/2022 16:07:55 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 100, 1)
12/07/2022 16:07:55 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 560, 2048)
12/07/2022 16:08:00 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 512)
12/07/2022 16:08:01 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 512)
12/07/2022 16:08:02 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 512, 2048)
12/07/2022 16:08:06 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 512)
12/07/2022 16:08:07 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 512)
12/07/2022 16:08:07 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 512, 1024)
12/07/2022 16:08:10 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 256, 256)
12/07/2022 16:08:10 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 256, 256)
12/07/2022 16:08:10 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 256, 512)
12/07/2022 16:08:11 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 128, 3)
12/07/2022 16:08:11 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 560, 2048)
12/07/2022 16:08:14 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 2048)
12/07/2022 16:08:16 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 1024)
12/07/2022 16:08:18 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 256, 512)
12/07/2022 16:08:18 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 128, 1)
12/07/2022 16:08:18 WARNING  Model was constructed with shape (None, 16, 16, 560) for input KerasTensor(type_spec=TensorSpec(shape=(None, 16, 16, 560), dtype=tf.float32, name='input_3'), name='input_3', description="created by layer 'input_3'"), but it was called on an input with incompatible shape (None, 16, 16, 512).
12/07/2022 16:08:19 CRITICAL Error caught! Exiting...
12/07/2022 16:08:19 ERROR    Caught exception in thread: '_training'
12/07/2022 16:08:23 ERROR    Got Exception on main handler:
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Cannot start training with unbalance

Post by torzdf »

Looks like a bug in the model.

Can you provide the full crash report please (linked at the last line of the output error)

My word is final

User avatar
pmvkpop@gmail.com
Posts: 2
Joined: Wed Dec 07, 2022 9:14 pm
Been thanked: 1 time

Re: Cannot start training with unbalance

Post by pmvkpop@gmail.com »

I sent the crash report through PM.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Cannot start training with unbalance

Post by torzdf »

Ok, the pertinent part of your log is this:

Code: Select all

[model.unbalanced]
input_size: 256
lowmem: False
nodes: 1024
complexity_encoder: 144
complexity_decoder_a: 400
complexity_decoder_b: 560

Change complexity_decoder_b to 512 and it will train. I would look into this more to prevent this from happening, but I'm not a huge fan of the unbalanced model, so am unlikely to make the time.

My word is final

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Cannot start training with unbalance

Post by torzdf »

pmvkpop@gmail.com wrote: Thu Dec 08, 2022 8:32 pm

I sent the crash report through PM.

I found the bug and pushed a fix. This issue should be resolved.

My word is final

Locked