Page 1 of 1

Cannot start training with unbalance

Posted: Wed Dec 07, 2022 9:26 pm
by pmvkpop@gmail.com

I tried to run Unbalance with 512x512 and I got an error message.

it was fine with 320x320

what did I do wrong? :?

Code: Select all

===================================================
12/07/2022 16:07:33 INFO       Starting
12/07/2022 16:07:33 INFO     ===================================================
12/07/2022 16:07:34 INFO     Loading data, this may take a while...
12/07/2022 16:07:34 INFO     Loading Model from Unbalanced plugin...
12/07/2022 16:07:34 INFO     No existing state file found. Generating.
12/07/2022 16:07:34 INFO     Mixed precision compatibility check (mixed_float16): OK\nYour GPU will likely run quickly with dtype policy mixed_float16 as it has compute capability of at least 7.0. Your GPU: NVIDIA GeForce GTX 1660 Ti, compute capability 7.5
12/07/2022 16:07:34 INFO     Enabling Mixed Precision Training.
12/07/2022 16:07:35 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 3, 144)
12/07/2022 16:07:35 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 144, 288)
12/07/2022 16:07:35 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 288, 576)
12/07/2022 16:07:36 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 576, 864)
12/07/2022 16:07:38 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 864, 1152)
12/07/2022 16:07:43 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 512, 1600)
12/07/2022 16:07:46 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 400, 1600)
12/07/2022 16:07:49 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 400, 800)
12/07/2022 16:07:50 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 200, 400)
12/07/2022 16:07:51 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 100, 3)
12/07/2022 16:07:51 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 1600)
12/07/2022 16:07:53 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 400, 1600)
12/07/2022 16:07:54 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 400, 800)
12/07/2022 16:07:55 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 200, 400)
12/07/2022 16:07:55 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 100, 1)
12/07/2022 16:07:55 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 560, 2048)
12/07/2022 16:08:00 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 512)
12/07/2022 16:08:01 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 512)
12/07/2022 16:08:02 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 512, 2048)
12/07/2022 16:08:06 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 512)
12/07/2022 16:08:07 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 512)
12/07/2022 16:08:07 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 512, 1024)
12/07/2022 16:08:10 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 256, 256)
12/07/2022 16:08:10 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 256, 256)
12/07/2022 16:08:10 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 256, 512)
12/07/2022 16:08:11 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 128, 3)
12/07/2022 16:08:11 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 560, 2048)
12/07/2022 16:08:14 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 2048)
12/07/2022 16:08:16 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 512, 1024)
12/07/2022 16:08:18 INFO     Calculating Convolution Aware Initializer for shape: (3, 3, 256, 512)
12/07/2022 16:08:18 INFO     Calculating Convolution Aware Initializer for shape: (5, 5, 128, 1)
12/07/2022 16:08:18 WARNING  Model was constructed with shape (None, 16, 16, 560) for input KerasTensor(type_spec=TensorSpec(shape=(None, 16, 16, 560), dtype=tf.float32, name='input_3'), name='input_3', description="created by layer 'input_3'"), but it was called on an input with incompatible shape (None, 16, 16, 512).
12/07/2022 16:08:19 CRITICAL Error caught! Exiting...
12/07/2022 16:08:19 ERROR    Caught exception in thread: '_training'
12/07/2022 16:08:23 ERROR    Got Exception on main handler:

Re: Cannot start training with unbalance

Posted: Thu Dec 08, 2022 12:09 pm
by torzdf

Looks like a bug in the model.

Can you provide the full crash report please (linked at the last line of the output error)


Re: Cannot start training with unbalance

Posted: Thu Dec 08, 2022 8:32 pm
by pmvkpop@gmail.com

I sent the crash report through PM.


Re: Cannot start training with unbalance

Posted: Sun Dec 18, 2022 3:07 pm
by torzdf

Ok, the pertinent part of your log is this:

Code: Select all

[model.unbalanced]
input_size: 256
lowmem: False
nodes: 1024
complexity_encoder: 144
complexity_decoder_a: 400
complexity_decoder_b: 560

Change complexity_decoder_b to 512 and it will train. I would look into this more to prevent this from happening, but I'm not a huge fan of the unbalanced model, so am unlikely to make the time.


Re: Cannot start training with unbalance

Posted: Fri Jan 13, 2023 11:53 pm
by torzdf
pmvkpop@gmail.com wrote: Thu Dec 08, 2022 8:32 pm

I sent the crash report through PM.

I found the bug and pushed a fix. This issue should be resolved.