New computer build issues

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
stinger85
Posts: 10
Joined: Fri Jan 24, 2020 3:55 am
Has thanked: 3 times

New computer build issues

Post by stinger85 »

I just recently started using faceswap a few months ago on my old computer and I was able to run dfl-sae at about 12 batch size on my old 1050ti card that had 4gb ddr5 vram. This past weekend I build myself a newer system and have been having some issues training with anything other than the original trainer. I upgraded to a 1660ti card so I figured I would be able to do a bit more with the training with the 6gb ddr6 vram. But so far I haven't had much luck. Only thing I can figure is that I switch from using a first generation I7 to a ryzen 5 3600 and maybe using a AMD + Nvidia setup is causing me to have issues? I reinstalled faceswap just to make sure I selected the nvidia gpu settings but that still didn't help at all. Here is my system info and error logs. I tried training at 12 batch with df-sae with memory saving settings enabled same as my last video card.

Code: Select all

01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/residual_32_3_conv2d_0/convolution
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/conv2d_32_2/convolution
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/residual_32_2_leakyrelu_0/LeakyRelu
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/upscale_32_1_conv2d/BiasAdd
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/conv2d_32_3/BiasAdd
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/residual_32_2_conv2d_0/convolution
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/residual_32_3_conv2d_0/BiasAdd
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/upscale_32_1_conv2d/convolution
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/conv2d_32_2/BiasAdd
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/residual_32_3_leakyrelu_1/LeakyRelu
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/residual_32_2_leakyrelu_1/LeakyRelu
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/add_7/add
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/add_8/add
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/upscale_32_1_pixelshuffler/Reshape
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/residual_32_2_leakyrelu_3/LeakyRelu
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/residual_32_3_leakyrelu_3/LeakyRelu
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/upscale_32_1_pixelshuffler/transpose
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/residual_32_2_conv2d_0/BiasAdd
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Copied [<tf.Operation 'decoder_b/residual_32_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/conv2d_32_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_32_2/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_32_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_7/add' type=AddV2>, <tf.Operation 'decoder_b/add_8/add' type=AddV2>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/BiasAdd' type=BiasAdd>] to dict_values([<tf.Operation 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/conv2d_32_2/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/conv2d_32_2/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/add_7/add' type=AddV2>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/conv2d_32_3/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/conv2d_32_3/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/add_8/add' type=AddV2>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/upscale_32_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/upscale_32_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose' type=Transpose>]): []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Rewired %s in place of %s restricted to %s: [['training_1/Adam/decoder_b/residual_128_3_leakyrelu_3/LeakyRelu_sg:0', 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1_sg:0', 'training_1/Adam/decoder_b/upscale_64_1_conv2d/BiasAdd_sg:0', 'training_1/Adam/encoder/conv_16_0_conv2d/BiasAdd_sg:0', 'training_1/Adam/encoder/upscale_8_0_conv2d/BiasAdd_sg:0'], ['decoder_b/residual_128_3_leakyrelu_3/LeakyRelu:0', 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1:0', 'decoder_b/upscale_64_1_conv2d/BiasAdd:0', 'encoder/conv_16_0_conv2d/BiasAdd:0', 'encoder/upscale_8_0_conv2d/BiasAdd:0'], ['training_1/Adam/decoder_b/add_7/add', 'training_1/Adam/decoder_b/add_8/add', 'training_1/Adam/decoder_b/conv2d_32_2/BiasAdd', 'training_1/Adam/decoder_b/conv2d_32_2/convolution', 'training_1/Adam/decoder_b/conv2d_32_3/BiasAdd', 'training_1/Adam/decoder_b/conv2d_32_3/convolution', 'training_1/Adam/decoder_b/residual_32_2_conv2d_0/BiasAdd', 'training_1/Adam/decoder_b/residual_32_2_conv2d_0/convolution', 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_0/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_1/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_3/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_3_conv2d_0/BiasAdd', 'training_1/Adam/decoder_b/residual_32_3_conv2d_0/convolution', 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_0/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_1/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_3/LeakyRelu', 'training_1/Adam/decoder_b/upscale_32_1_conv2d/BiasAdd', 'training_1/Adam/decoder_b/upscale_32_1_conv2d/convolution', 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/Reshape', 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose']]
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Got gradients [None, None, None, None, <tf.Tensor 'training_1/Adam/gradients_3/AddN_1:0' shape=(?, 32, 32, 504) dtype=float32>, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/residual_32_2_conv2d_0/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/residual_32_2_conv2d_0/BiasAdd_grad/BiasAddGrad:0' shape=(504,) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/conv2d_32_2/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/conv2d_32_2/BiasAdd_grad/BiasAddGrad:0' shape=(504,) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/residual_32_3_conv2d_0/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/residual_32_3_conv2d_0/BiasAdd_grad/BiasAddGrad:0' shape=(504,) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/conv2d_32_3/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/conv2d_32_3/BiasAdd_grad/BiasAddGrad:0' shape=(504,) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/upscale_32_1_conv2d/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 1008) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/upscale_32_1_conv2d/BiasAdd_grad/BiasAddGrad:0' shape=(1008,) dtype=float32>, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    for [<tf.Tensor 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    with respect to [<tf.Tensor 'training_1/Adam/encoder/conv_16_0_conv2d/BiasAdd_sg:0' shape=(?, 8, 8, 1008) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/residual_128_3_leakyrelu_3/LeakyRelu_sg:0' shape=(?, 128, 128, 126) dtype=float32>, <tf.Tensor 'training_1/Adam/encoder/upscale_8_0_conv2d/BiasAdd_sg:0' shape=(?, 8, 8, 2048) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/upscale_64_1_conv2d/BiasAdd_sg:0' shape=(?, 64, 64, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1_sg:0' shape=(?, 32, 32, 504) dtype=float32>, <tf.Variable 'conv_128_0_conv2d/kernel:0' shape=(5, 5, 3, 126) dtype=float32_ref>, <tf.Variable 'conv_128_0_conv2d/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv_64_0_conv2d/kernel:0' shape=(5, 5, 126, 252) dtype=float32_ref>, <tf.Variable 'conv_64_0_conv2d/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv_32_0_conv2d/kernel:0' shape=(5, 5, 252, 504) dtype=float32_ref>, <tf.Variable 'conv_32_0_conv2d/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv_16_0_conv2d/kernel:0' shape=(5, 5, 504, 1008) dtype=float32_ref>, <tf.Variable 'conv_16_0_conv2d/bias:0' shape=(1008,) dtype=float32_ref>, <tf.Variable 'dense_1/kernel:0' shape=(64512, 512) dtype=float32_ref>, <tf.Variable 'dense_1/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'dense_2/kernel:0' shape=(512, 32768) dtype=float32_ref>, <tf.Variable 'dense_2/bias:0' shape=(32768,) dtype=float32_ref>, <tf.Variable 'upscale_8_0_conv2d/kernel:0' shape=(3, 3, 512, 2048) dtype=float32_ref>, <tf.Variable 'upscale_8_0_conv2d/bias:0' shape=(2048,) dtype=float32_ref>, <tf.Variable 'upscale_16_1_conv2d/kernel:0' shape=(3, 3, 512, 2016) dtype=float32_ref>, <tf.Variable 'upscale_16_1_conv2d/bias:0' shape=(2016,) dtype=float32_ref>, <tf.Variable 'residual_32_2_conv2d_0/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'residual_32_2_conv2d_0/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv2d_32_2/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'conv2d_32_2/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'residual_32_3_conv2d_0/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'residual_32_3_conv2d_0/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv2d_32_3/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'conv2d_32_3/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'upscale_32_1_conv2d/kernel:0' shape=(3, 3, 504, 1008) dtype=float32_ref>, <tf.Variable 'upscale_32_1_conv2d/bias:0' shape=(1008,) dtype=float32_ref>, <tf.Variable 'residual_64_2_conv2d_0/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'residual_64_2_conv2d_0/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv2d_64_2/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'conv2d_64_2/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'residual_64_3_conv2d_0/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'residual_64_3_conv2d_0/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv2d_64_3/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'conv2d_64_3/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'upscale_64_1_conv2d/kernel:0' shape=(3, 3, 252, 504) dtype=float32_ref>, <tf.Variable 'upscale_64_1_conv2d/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'residual_128_2_conv2d_0/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'residual_128_2_conv2d_0/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv2d_128_2/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'conv2d_128_2/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'residual_128_3_conv2d_0/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'residual_128_3_conv2d_0/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv2d_128_3/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'conv2d_128_3/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'face_out_128_1/kernel:0' shape=(5, 5, 126, 3) dtype=float32_ref>, <tf.Variable 'face_out_128_1/bias:0' shape=(3,) dtype=float32_ref>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    with boundary backprop substitutions [<tf.Tensor 'training_1/Adam/gradients_2/training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/Reshape_1_grad/Reshape:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Processing list [<tf.Tensor 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1:0' shape=(?, 32, 32, 504) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Found 9 ops to copy within [<tf.Operation 'decoder_b/residual_128_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_16_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_128_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_128_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_64_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_64_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_32_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/MatMul' type=MatMul>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_64_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_128_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_128_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/face_out_128/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_1/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_64_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_1/MatMul' type=MatMul>, <tf.Operation 'decoder_b/face_out_128/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_32_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_128_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_64_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'encoder/upscale_8_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_16_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_128_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_64_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_32_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/add_12/add' type=AddV2>, <tf.Operation 'encoder/reshape_1/Shape' type=Shape>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_10/add' type=AddV2>, <tf.Operation 'encoder/conv_16_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_7/add' type=AddV2>, <tf.Operation 'decoder_b/add_9/add' type=AddV2>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_64_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/face_out_128/Sigmoid' type=Sigmoid>, <tf.Operation 'encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_128_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_32_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_11/add' type=AddV2>, <tf.Operation 'decoder_b/add_8/add' type=AddV2>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/flatten_1/Shape' type=Shape>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'loss_1/decoder_b_loss/sub' type=Sub>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/strided_slice' type=StridedSlice>, <tf.Operation 'encoder/flatten_1/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'encoder/flatten_1/strided_slice' type=StridedSlice>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'loss_1/decoder_b_loss/Abs' type=Abs>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/Reshape/shape' type=Pack>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'loss_1/decoder_b_loss/Mean' type=Mean>, <tf.Operation 'encoder/flatten_1/Prod' type=Prod>, <tf.Operation 'encoder/flatten_1/stack' type=Pack>, <tf.Operation 'loss_1/decoder_b_loss/Mean_1' type=Mean>, <tf.Operation 'loss_1/decoder_b_loss/mul' type=Mul>, <tf.Operation 'loss_1/decoder_b_loss/truediv' type=RealDiv>, <tf.Operation 'loss_1/decoder_b_loss/Mean_3' type=Mean>, <tf.Operation 'loss_1/mul' type=Mul>], seed [<tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>], stop_at [<tf.Tensor 'encoder/conv_16_0_conv2d/BiasAdd:0' shape=(?, 8, 8, 1008) dtype=float32>, <tf.Tensor 'decoder_b/residual_128_3_leakyrelu_3/LeakyRelu:0' shape=(?, 128, 128, 126) dtype=float32>, <tf.Tensor 'encoder/upscale_8_0_conv2d/BiasAdd:0' shape=(?, 8, 8, 2048) dtype=float32>, <tf.Tensor 'decoder_b/upscale_64_1_conv2d/BiasAdd:0' shape=(?, 64, 64, 504) dtype=float32>, <tf.Tensor 'decoder_b/upscale_32_1_pixelshuffler/transpose:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    ops_to_copy = [<tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>]: []
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: encoder/upscale_8_0_pixelshuffler/Reshape
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/upscale_16_1_conv2d/convolution
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: encoder/upscale_8_0_leakyrelu/LeakyRelu
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/upscale_16_1_pixelshuffler/Reshape_1
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/upscale_16_1_conv2d/BiasAdd
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: encoder/upscale_8_0_pixelshuffler/Reshape_1
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/upscale_16_1_pixelshuffler/transpose
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: encoder/upscale_8_0_pixelshuffler/transpose
01/23/2020 20:07:59 MainProcess     _training_0     transform       _connect_control_inputs   DEBUG    Connecting control inputs of op: decoder_b/upscale_16_1_pixelshuffler/Reshape
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Copied [<tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>] to dict_values([<tf.Operation 'training_1/Adam/encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>]): []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Rewired %s in place of %s restricted to %s: [['training_1/Adam/decoder_b/residual_128_3_leakyrelu_3/LeakyRelu_sg:0', 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose_sg:0', 'training_1/Adam/decoder_b/upscale_64_1_conv2d/BiasAdd_sg:0', 'training_1/Adam/encoder/conv_16_0_conv2d/BiasAdd_sg:0', 'training_1/Adam/encoder/upscale_8_0_conv2d/BiasAdd_sg:0'], ['decoder_b/residual_128_3_leakyrelu_3/LeakyRelu:0', 'decoder_b/upscale_32_1_pixelshuffler/transpose:0', 'decoder_b/upscale_64_1_conv2d/BiasAdd:0', 'encoder/conv_16_0_conv2d/BiasAdd:0', 'encoder/upscale_8_0_conv2d/BiasAdd:0'], ['training_1/Adam/decoder_b/upscale_16_1_conv2d/BiasAdd', 'training_1/Adam/decoder_b/upscale_16_1_conv2d/convolution', 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape', 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1', 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/transpose', 'training_1/Adam/encoder/upscale_8_0_leakyrelu/LeakyRelu', 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/Reshape', 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/Reshape_1', 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/transpose']]
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Got gradients [None, None, <tf.Tensor 'training_1/Adam/gradients_4/training_1/Adam/encoder/upscale_8_0_leakyrelu/LeakyRelu_grad/LeakyReluGrad:0' shape=(?, 8, 8, 2048) dtype=float32>, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, <tf.Tensor 'training_1/Adam/gradients_4/training_1/Adam/decoder_b/upscale_16_1_conv2d/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 512, 2016) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_4/training_1/Adam/decoder_b/upscale_16_1_conv2d/BiasAdd_grad/BiasAddGrad:0' shape=(2016,) dtype=float32>, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    for [<tf.Tensor 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1:0' shape=(?, 32, 32, 504) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    with respect to [<tf.Tensor 'training_1/Adam/encoder/conv_16_0_conv2d/BiasAdd_sg:0' shape=(?, 8, 8, 1008) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/residual_128_3_leakyrelu_3/LeakyRelu_sg:0' shape=(?, 128, 128, 126) dtype=float32>, <tf.Tensor 'training_1/Adam/encoder/upscale_8_0_conv2d/BiasAdd_sg:0' shape=(?, 8, 8, 2048) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/upscale_64_1_conv2d/BiasAdd_sg:0' shape=(?, 64, 64, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose_sg:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>, <tf.Variable 'conv_128_0_conv2d/kernel:0' shape=(5, 5, 3, 126) dtype=float32_ref>, <tf.Variable 'conv_128_0_conv2d/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv_64_0_conv2d/kernel:0' shape=(5, 5, 126, 252) dtype=float32_ref>, <tf.Variable 'conv_64_0_conv2d/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv_32_0_conv2d/kernel:0' shape=(5, 5, 252, 504) dtype=float32_ref>, <tf.Variable 'conv_32_0_conv2d/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv_16_0_conv2d/kernel:0' shape=(5, 5, 504, 1008) dtype=float32_ref>, <tf.Variable 'conv_16_0_conv2d/bias:0' shape=(1008,) dtype=float32_ref>, <tf.Variable 'dense_1/kernel:0' shape=(64512, 512) dtype=float32_ref>, <tf.Variable 'dense_1/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'dense_2/kernel:0' shape=(512, 32768) dtype=float32_ref>, <tf.Variable 'dense_2/bias:0' shape=(32768,) dtype=float32_ref>, <tf.Variable 'upscale_8_0_conv2d/kernel:0' shape=(3, 3, 512, 2048) dtype=float32_ref>, <tf.Variable 'upscale_8_0_conv2d/bias:0' shape=(2048,) dtype=float32_ref>, <tf.Variable 'upscale_16_1_conv2d/kernel:0' shape=(3, 3, 512, 2016) dtype=float32_ref>, <tf.Variable 'upscale_16_1_conv2d/bias:0' shape=(2016,) dtype=float32_ref>, <tf.Variable 'residual_32_2_conv2d_0/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'residual_32_2_conv2d_0/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv2d_32_2/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'conv2d_32_2/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'residual_32_3_conv2d_0/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'residual_32_3_conv2d_0/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv2d_32_3/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'conv2d_32_3/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'upscale_32_1_conv2d/kernel:0' shape=(3, 3, 504, 1008) dtype=float32_ref>, <tf.Variable 'upscale_32_1_conv2d/bias:0' shape=(1008,) dtype=float32_ref>, <tf.Variable 'residual_64_2_conv2d_0/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'residual_64_2_conv2d_0/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv2d_64_2/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'conv2d_64_2/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'residual_64_3_conv2d_0/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'residual_64_3_conv2d_0/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv2d_64_3/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'conv2d_64_3/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'upscale_64_1_conv2d/kernel:0' shape=(3, 3, 252, 504) dtype=float32_ref>, <tf.Variable 'upscale_64_1_conv2d/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'residual_128_2_conv2d_0/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'residual_128_2_conv2d_0/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv2d_128_2/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'conv2d_128_2/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'residual_128_3_conv2d_0/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'residual_128_3_conv2d_0/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv2d_128_3/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'conv2d_128_3/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'face_out_128_1/kernel:0' shape=(5, 5, 126, 3) dtype=float32_ref>, <tf.Variable 'face_out_128_1/bias:0' shape=(3,) dtype=float32_ref>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    with boundary backprop substitutions [<tf.Tensor 'training_1/Adam/gradients_3/AddN_1:0' shape=(?, 32, 32, 504) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Processing list [<tf.Tensor 'encoder/upscale_8_0_conv2d/BiasAdd:0' shape=(?, 8, 8, 2048) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    Found 16 ops to copy within [<tf.Operation 'decoder_b/residual_128_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_16_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_128_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_128_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_64_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_64_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_32_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/MatMul' type=MatMul>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_64_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_128_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_128_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/face_out_128/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_1/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_64_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_1/MatMul' type=MatMul>, <tf.Operation 'decoder_b/face_out_128/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_32_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_128_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_64_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'encoder/upscale_8_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_16_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_128_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_64_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_32_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/add_12/add' type=AddV2>, <tf.Operation 'encoder/reshape_1/Shape' type=Shape>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_10/add' type=AddV2>, <tf.Operation 'encoder/conv_16_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_7/add' type=AddV2>, <tf.Operation 'decoder_b/add_9/add' type=AddV2>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_64_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/face_out_128/Sigmoid' type=Sigmoid>, <tf.Operation 'encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_128_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_32_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_11/add' type=AddV2>, <tf.Operation 'decoder_b/add_8/add' type=AddV2>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/flatten_1/Shape' type=Shape>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'loss_1/decoder_b_loss/sub' type=Sub>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/strided_slice' type=StridedSlice>, <tf.Operation 'encoder/flatten_1/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'encoder/flatten_1/strided_slice' type=StridedSlice>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'loss_1/decoder_b_loss/Abs' type=Abs>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/Reshape/shape' type=Pack>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'loss_1/decoder_b_loss/Mean' type=Mean>, <tf.Operation 'encoder/flatten_1/Prod' type=Prod>, <tf.Operation 'encoder/flatten_1/stack' type=Pack>, <tf.Operation 'loss_1/decoder_b_loss/Mean_1' type=Mean>, <tf.Operation 'loss_1/decoder_b_loss/mul' type=Mul>, <tf.Operation 'loss_1/decoder_b_loss/truediv' type=RealDiv>, <tf.Operation 'loss_1/decoder_b_loss/Mean_3' type=Mean>, <tf.Operation 'loss_1/mul' type=Mul>], seed [<tf.Operation 'encoder/upscale_8_0_conv2d/BiasAdd' type=BiasAdd>], stop_at [<tf.Tensor 'encoder/conv_16_0_conv2d/BiasAdd:0' shape=(?, 8, 8, 1008) dtype=float32>, <tf.Tensor 'decoder_b/residual_128_3_leakyrelu_3/LeakyRelu:0' shape=(?, 128, 128, 126) dtype=float32>, <tf.Tensor 'decoder_b/upscale_64_1_conv2d/BiasAdd:0' shape=(?, 64, 64, 504) dtype=float32>, <tf.Tensor 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1:0' shape=(?, 32, 32, 504) dtype=float32>, <tf.Tensor 'decoder_b/upscale_32_1_pixelshuffler/transpose:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess     _training_0     memory_saving_gradients debug_print               DEBUG    ops_to_copy = [<tf.Operation 'encoder/dense_1/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/reshape_1/Shape' type=Shape>, <tf.Operation 'encoder/dense_1/MatMul' type=MatMul>, <tf.Operation 'encoder/flatten_1/strided_slice' type=StridedSlice>, <tf.Operation 'encoder/reshape_1/Reshape/shape' type=Pack>, <tf.Operation 'encoder/conv_16_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/flatten_1/Prod' type=Prod>, <tf.Operation 'encoder/flatten_1/Reshape' type=Reshape>, <tf.Operation 'encoder/reshape_1/Reshape' type=Reshape>, <tf.Operation 'encoder/dense_2/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/MatMul' type=MatMul>, <tf.Operation 'encoder/upscale_8_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/flatten_1/Shape' type=Shape>, <tf.Operation 'encoder/flatten_1/stack' type=Pack>, <tf.Operation 'encoder/reshape_1/strided_slice' type=StridedSlice>]: []
Top
User avatar
torzdf
Posts: 2679
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 133 times
Been thanked: 625 times

Re: New computer build issues

Post by torzdf »

That log is just the debug log... Is it getting stuck there?

My word is final

Top
User avatar
stinger85
Posts: 10
Joined: Fri Jan 24, 2020 3:55 am
Has thanked: 3 times

Re: New computer build issues

Post by stinger85 »

Sorry about that this took some time to get posted last night so I didn't get a chance to edit this post until now. The crash log was so long I just added it to a attachment file. But here is my system info maybe someone could tell me if something is wrong here.

============ System Information ============
encoding: cp1252
git_branch: master
git_commits: 04b3f86 setup.py - pin plaidml to 0.6.4
gpu_cuda: No global version found. Check Conda packages for Conda Cuda
gpu_cudnn: No global version found. Check Conda packages for Conda cuDNN
gpu_devices: GPU_0: GeForce GTX 1660 Ti
gpu_devices_active: GPU_0
gpu_driver: 441.87
gpu_vram: GPU_0: 6144MB
os_machine: AMD64
os_platform: Windows-10-10.0.18362-SP0
os_release: 10
py_command: C:\Users\Stinger\faceswap\faceswap.py train -A D:/OldFiles/FS/Rainy/working/Sort_A -B D:/OldFiles/FS/Rainy/working/Align_for_Training_B -m D:/OldFiles/FS/Rainy/working/Modeltest -t dfl-sae -bs 12 -it 1000000 -g 1 -msg -s 100 -ss 25000 -ps 50 -ag -L INFO -gui
py_conda_version: conda 4.8.1
py_implementation: CPython
py_version: 3.7.6
py_virtual_env: True
sys_cores: 12
sys_processor: AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
sys_ram: Total: 32682MB, Available: 21304MB, Used: 11378MB, Free: 21304MB

=============== Pip Packages ===============
absl-py==0.8.1
astor==0.8.0
certifi==2019.11.28
cloudpickle==1.2.2
cycler==0.10.0
cytoolz==0.10.1
dask==2.9.2
decorator==4.4.1
fastcluster==1.1.26
ffmpy==0.2.2
gast==0.2.2
google-pasta==0.1.8
grpcio==1.16.1
h5py==2.9.0
imageio==2.6.1
imageio-ffmpeg==0.3.0
joblib==0.14.1
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
matplotlib==3.1.1
mkl-fft==1.0.15
mkl-random==1.1.0
mkl-service==2.3.0
networkx==2.4
numpy==1.17.4
nvidia-ml-py3==7.352.1
olefile==0.46
opencv-python==4.1.2.30
opt-einsum==3.1.0
pathlib==1.0.1
Pillow==6.2.1
protobuf==3.11.2
psutil==5.6.7
pyparsing==2.4.6
pyreadline==2.1
python-dateutil==2.8.1
pytz==2019.3
PyWavelets==1.1.1
pywin32==227
PyYAML==5.2
scikit-image==0.15.0
scikit-learn==0.22.1
scipy==1.3.2
six==1.13.0
tensorboard==2.0.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1
termcolor==1.1.0
toolz==0.10.0
toposort==1.5
tornado==6.0.3
tqdm==4.41.1
Werkzeug==0.16.0
wincertstore==0.2
wrapt==1.11.2

============== Conda Packages ==============

packages in environment at C:\Users\Stinger\MiniConda3\envs\faceswap:

#

Name Version Build Channel

_tflow_select 2.1.0 gpu
absl-py 0.8.1 py37_0
astor 0.8.0 py37_0
blas 1.0 mkl
ca-certificates 2019.11.27 0
certifi 2019.11.28 py37_0
cloudpickle 1.2.2 py_0
cudatoolkit 10.0.130 0
cudnn 7.6.5 cuda10.0_0
cycler 0.10.0 py37_0
cytoolz 0.10.1 py37he774522_0
dask-core 2.9.2 py_0
decorator 4.4.1 py_0
fastcluster 1.1.26 py37he350917_0 conda-forge
ffmpeg 4.2 h6538335_0 conda-forge
ffmpy 0.2.2 pypi_0 pypi
freetype 2.9.1 ha9979f8_1
gast 0.2.2 py37_0
git 2.23.0 h6bb4b03_0
google-pasta 0.1.8 py_0
grpcio 1.16.1 py37h351948d_1
h5py 2.9.0 py37h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha66f8fd_1
imageio 2.6.1 py37_0
imageio-ffmpeg 0.3.0 py_0 conda-forge
intel-openmp 2019.4 245
joblib 0.14.1 py_0
jpeg 9b hb83a4c4_2
keras 2.2.4 0
keras-applications 1.0.8 py_0
keras-base 2.2.4 py37_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.1.0 py37ha925a31_0
libmklml 2019.0.5 0
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.11.2 h7bd577a_0
libtiff 4.1.0 h56a325e_0
markdown 3.1.1 py37_0
matplotlib 3.1.1 py37hc8f65d3_0
mkl 2019.4 245
mkl-service 2.3.0 py37hb782905_0
mkl_fft 1.0.15 py37h14836fe_0
mkl_random 1.1.0 py37h675688f_0
networkx 2.4 py_0
numpy 1.17.4 py37h4320e6b_0
numpy-base 1.17.4 py37hc3f5095_0
nvidia-ml-py3 7.352.1 pypi_0 pypi
olefile 0.46 py37_0
opencv-python 4.1.2.30 pypi_0 pypi
openssl 1.1.1d he774522_3
opt_einsum 3.1.0 py_0
pathlib 1.0.1 py37_1
pillow 6.2.1 py37hdc69c19_0
pip 19.3.1 py37_0
protobuf 3.11.2 py37h33f27b4_0
psutil 5.6.7 py37he774522_0
pyparsing 2.4.6 py_0
pyqt 5.9.2 py37h6538335_2
pyreadline 2.1 py37_1
python 3.7.6 h60c2a47_2
python-dateutil 2.8.1 py_0
pytz 2019.3 py_0
pywavelets 1.1.1 py37he774522_0
pywin32 227 py37he774522_1
pyyaml 5.2 py37he774522_0
qt 5.9.7 vc14h73c81de_0
scikit-image 0.15.0 py37ha925a31_0
scikit-learn 0.22.1 py37h6288b17_0
scipy 1.3.2 py37h29ff71c_0
setuptools 44.0.0 py37_0
sip 4.19.8 py37h6538335_0
six 1.13.0 py37_0
sqlite 3.30.1 he774522_0
tensorboard 2.0.0 pyhb38c66f_1
tensorflow 1.15.0 gpu_py37hc3743a6_0
tensorflow-base 1.15.0 gpu_py37h1afeea4_0
tensorflow-estimator 1.15.1 pyh2649769_0
tensorflow-gpu 1.15.0 h0d30ee6_0
termcolor 1.1.0 py37_1
tk 8.6.8 hfa6e2cd_0
toolz 0.10.0 py_0
toposort 1.5 py_3 conda-forge
tornado 6.0.3 py37he774522_0
tqdm 4.41.1 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_1
werkzeug 0.16.0 py_0
wheel 0.33.6 py37_0
wincertstore 0.2 py37_0
wrapt 1.11.2 py37he774522_0
xz 5.2.4 h2fa13f4_4
yaml 0.1.7 hc54c509_2
zlib 1.2.11 h62dcd97_3
zstd 1.3.7 h508b16e_0

================= Configs ==================
--------- .faceswap ---------
backend: nvidia

--------- convert.ini ---------

[color.color_transfer]
clip: True
preserve_paper: True

[color.manual_balance]
colorspace: HSV
balance_1: 0.0
balance_2: 0.0
balance_3: 0.0
contrast: 0.0
brightness: 0.0

[color.match_hist]
threshold: 99.0

[mask.box_blend]
type: gaussian
distance: 11.0
radius: 5.0
passes: 1

[mask.mask_blend]
type: normalized
kernel_size: 3
passes: 4
threshold: 4
erosion: 0.0

[scaling.sharpen]
method: unsharp_mask
amount: 150
radius: 0.3
threshold: 5.0

[writer.ffmpeg]
container: mp4
codec: libx264
crf: 23
preset: medium
tune: none
profile: auto
level: auto

[writer.gif]
fps: 25
loop: 0
palettesize: 256
subrectangles: False

[writer.opencv]
format: png
draw_transparent: False
jpg_quality: 75
png_compress_level: 3

[writer.pillow]
format: png
draw_transparent: False
optimize: False
gif_interlace: True
jpg_quality: 75
png_compress_level: 3
tif_compression: tiff_deflate

--------- extract.ini ---------

[global]
allow_growth: False

[align.fan]
batch-size: 12

[detect.cv2_dnn]
confidence: 50

[detect.mtcnn]
minsize: 20
threshold_1: 0.6
threshold_2: 0.7
threshold_3: 0.7
scalefactor: 0.709
batch-size: 8

[detect.s3fd]
confidence: 70
batch-size: 4

[mask.unet_dfl]
batch-size: 8

[mask.vgg_clear]
batch-size: 6

[mask.vgg_obstructed]
batch-size: 2

--------- gui.ini ---------

[global]
fullscreen: False
tab: extract
options_panel_width: 30
console_panel_height: 20
icon_size: 14
font: default
font_size: 9
autosave_last_session: prompt
timeout: 120
auto_load_model_stats: True

--------- train.ini ---------

[global]
coverage: 68.75
mask_type: none
mask_blur_kernel: 3
mask_threshold: 4
learn_mask: False
icnr_init: False
conv_aware_init: False
subpixel_upscaling: False
reflect_padding: False
penalized_mask_loss: True
loss_function: mae
learning_rate: 5e-05

[model.dfl_h128]
lowmem: False

[model.dfl_sae]
input_size: 128
clipnorm: True
architecture: df
autoencoder_dims: 0
encoder_dims: 42
decoder_dims: 21
multiscale_decoder: False

[model.dlight]
features: best
details: good
output_size: 256

[model.original]
lowmem: False

[model.realface]
input_size: 64
output_size: 128
dense_nodes: 1536
complexity_encoder: 128
complexity_decoder: 512

[model.unbalanced]
input_size: 128
lowmem: False
clipnorm: True
nodes: 1024
complexity_encoder: 128
complexity_decoder_a: 384
complexity_decoder_b: 512

[model.villain]
lowmem: False

[trainer.original]
preview_images: 14
zoom_amount: 5
rotation_range: 10
shift_range: 5
flip_chance: 50
color_lightness: 30
color_ab: 8
color_clahe_chance: 50
color_clahe_max_size: 4

Attachments
crash_report.2020.01.23.200817057472.log
(93.26 KiB) Downloaded 177 times
Top
User avatar
torzdf
Posts: 2679
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 133 times
Been thanked: 625 times

Re: New computer build issues

Post by torzdf »

Yes. So ultimately, you are running out of VRAM. Outside of the standard fixes, you can try reducing settings on that model's config.

Otherwise, I would highly recommend dual booting Linux (Xubuntu is a good choice) for Faceswap, as you will have access to approximately 20% more VRAM.

My word is final

Top
User avatar
stinger85
Posts: 10
Joined: Fri Jan 24, 2020 3:55 am
Has thanked: 3 times

Re: New computer build issues

Post by stinger85 »

Darn I was kind of hoping I had something set up wrong. :lol: I just cant believe this new setup can't do what my 6 year old computer could get away with. Although I can train on the original trainer at a lot higher batch setting so I suppose I should be happy about that. I have never used Linux before so not sure if I will do that or not. But you said I should reduce some settings on the model what do you mean? Just make sure the vram saving settings are all checked or something else?

Top
User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:
Contact bryanlyon

Re: New computer build issues

Post by bryanlyon »

Batch size is the first one to try. If that's not enough you can go in to the model settings to reduce quality or you can enable memory options while training.

Ultimately you have 6gb of vram so can use most lighter models without too much trouble but will struggle with the larger models.

Top
User avatar
stinger85
Posts: 10
Joined: Fri Jan 24, 2020 3:55 am
Has thanked: 3 times

Re: New computer build issues

Post by stinger85 »

Alright well thanks for the replys I will have to try playing with some settings see if I can at least get dfaker or dfaker light working.

Top
Locked

Return to “Training Discussion”