I just recently started using faceswap a few months ago on my old computer and I was able to run dfl-sae at about 12 batch size on my old 1050ti card that had 4gb ddr5 vram. This past weekend I build myself a newer system and have been having some issues training with anything other than the original trainer. I upgraded to a 1660ti card so I figured I would be able to do a bit more with the training with the 6gb ddr6 vram. But so far I haven't had much luck. Only thing I can figure is that I switch from using a first generation I7 to a ryzen 5 3600 and maybe using a AMD + Nvidia setup is causing me to have issues? I reinstalled faceswap just to make sure I selected the nvidia gpu settings but that still didn't help at all. Here is my system info and error logs. I tried training at 12 batch with df-sae with memory saving settings enabled same as my last video card.
Code: Select all
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/residual_32_3_conv2d_0/convolution
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/conv2d_32_2/convolution
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/residual_32_2_leakyrelu_0/LeakyRelu
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/upscale_32_1_conv2d/BiasAdd
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/conv2d_32_3/BiasAdd
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/residual_32_2_conv2d_0/convolution
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/residual_32_3_conv2d_0/BiasAdd
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/upscale_32_1_conv2d/convolution
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/conv2d_32_2/BiasAdd
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/residual_32_3_leakyrelu_1/LeakyRelu
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/residual_32_2_leakyrelu_1/LeakyRelu
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/add_7/add
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/add_8/add
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/upscale_32_1_pixelshuffler/Reshape
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/residual_32_2_leakyrelu_3/LeakyRelu
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/residual_32_3_leakyrelu_3/LeakyRelu
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/upscale_32_1_pixelshuffler/transpose
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/residual_32_2_conv2d_0/BiasAdd
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Copied [<tf.Operation 'decoder_b/residual_32_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/conv2d_32_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_32_2/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_32_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_7/add' type=AddV2>, <tf.Operation 'decoder_b/add_8/add' type=AddV2>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/BiasAdd' type=BiasAdd>] to dict_values([<tf.Operation 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/conv2d_32_2/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/conv2d_32_2/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/add_7/add' type=AddV2>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/conv2d_32_3/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/conv2d_32_3/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/add_8/add' type=AddV2>, <tf.Operation 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/decoder_b/upscale_32_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/upscale_32_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose' type=Transpose>]): []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Rewired %s in place of %s restricted to %s: [['training_1/Adam/decoder_b/residual_128_3_leakyrelu_3/LeakyRelu_sg:0', 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1_sg:0', 'training_1/Adam/decoder_b/upscale_64_1_conv2d/BiasAdd_sg:0', 'training_1/Adam/encoder/conv_16_0_conv2d/BiasAdd_sg:0', 'training_1/Adam/encoder/upscale_8_0_conv2d/BiasAdd_sg:0'], ['decoder_b/residual_128_3_leakyrelu_3/LeakyRelu:0', 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1:0', 'decoder_b/upscale_64_1_conv2d/BiasAdd:0', 'encoder/conv_16_0_conv2d/BiasAdd:0', 'encoder/upscale_8_0_conv2d/BiasAdd:0'], ['training_1/Adam/decoder_b/add_7/add', 'training_1/Adam/decoder_b/add_8/add', 'training_1/Adam/decoder_b/conv2d_32_2/BiasAdd', 'training_1/Adam/decoder_b/conv2d_32_2/convolution', 'training_1/Adam/decoder_b/conv2d_32_3/BiasAdd', 'training_1/Adam/decoder_b/conv2d_32_3/convolution', 'training_1/Adam/decoder_b/residual_32_2_conv2d_0/BiasAdd', 'training_1/Adam/decoder_b/residual_32_2_conv2d_0/convolution', 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_0/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_1/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_2_leakyrelu_3/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_3_conv2d_0/BiasAdd', 'training_1/Adam/decoder_b/residual_32_3_conv2d_0/convolution', 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_0/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_1/LeakyRelu', 'training_1/Adam/decoder_b/residual_32_3_leakyrelu_3/LeakyRelu', 'training_1/Adam/decoder_b/upscale_32_1_conv2d/BiasAdd', 'training_1/Adam/decoder_b/upscale_32_1_conv2d/convolution', 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/Reshape', 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose']]
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Got gradients [None, None, None, None, <tf.Tensor 'training_1/Adam/gradients_3/AddN_1:0' shape=(?, 32, 32, 504) dtype=float32>, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/residual_32_2_conv2d_0/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/residual_32_2_conv2d_0/BiasAdd_grad/BiasAddGrad:0' shape=(504,) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/conv2d_32_2/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/conv2d_32_2/BiasAdd_grad/BiasAddGrad:0' shape=(504,) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/residual_32_3_conv2d_0/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/residual_32_3_conv2d_0/BiasAdd_grad/BiasAddGrad:0' shape=(504,) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/conv2d_32_3/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/conv2d_32_3/BiasAdd_grad/BiasAddGrad:0' shape=(504,) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/upscale_32_1_conv2d/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 504, 1008) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_3/training_1/Adam/decoder_b/upscale_32_1_conv2d/BiasAdd_grad/BiasAddGrad:0' shape=(1008,) dtype=float32>, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG for [<tf.Tensor 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG with respect to [<tf.Tensor 'training_1/Adam/encoder/conv_16_0_conv2d/BiasAdd_sg:0' shape=(?, 8, 8, 1008) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/residual_128_3_leakyrelu_3/LeakyRelu_sg:0' shape=(?, 128, 128, 126) dtype=float32>, <tf.Tensor 'training_1/Adam/encoder/upscale_8_0_conv2d/BiasAdd_sg:0' shape=(?, 8, 8, 2048) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/upscale_64_1_conv2d/BiasAdd_sg:0' shape=(?, 64, 64, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1_sg:0' shape=(?, 32, 32, 504) dtype=float32>, <tf.Variable 'conv_128_0_conv2d/kernel:0' shape=(5, 5, 3, 126) dtype=float32_ref>, <tf.Variable 'conv_128_0_conv2d/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv_64_0_conv2d/kernel:0' shape=(5, 5, 126, 252) dtype=float32_ref>, <tf.Variable 'conv_64_0_conv2d/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv_32_0_conv2d/kernel:0' shape=(5, 5, 252, 504) dtype=float32_ref>, <tf.Variable 'conv_32_0_conv2d/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv_16_0_conv2d/kernel:0' shape=(5, 5, 504, 1008) dtype=float32_ref>, <tf.Variable 'conv_16_0_conv2d/bias:0' shape=(1008,) dtype=float32_ref>, <tf.Variable 'dense_1/kernel:0' shape=(64512, 512) dtype=float32_ref>, <tf.Variable 'dense_1/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'dense_2/kernel:0' shape=(512, 32768) dtype=float32_ref>, <tf.Variable 'dense_2/bias:0' shape=(32768,) dtype=float32_ref>, <tf.Variable 'upscale_8_0_conv2d/kernel:0' shape=(3, 3, 512, 2048) dtype=float32_ref>, <tf.Variable 'upscale_8_0_conv2d/bias:0' shape=(2048,) dtype=float32_ref>, <tf.Variable 'upscale_16_1_conv2d/kernel:0' shape=(3, 3, 512, 2016) dtype=float32_ref>, <tf.Variable 'upscale_16_1_conv2d/bias:0' shape=(2016,) dtype=float32_ref>, <tf.Variable 'residual_32_2_conv2d_0/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'residual_32_2_conv2d_0/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv2d_32_2/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'conv2d_32_2/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'residual_32_3_conv2d_0/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'residual_32_3_conv2d_0/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv2d_32_3/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'conv2d_32_3/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'upscale_32_1_conv2d/kernel:0' shape=(3, 3, 504, 1008) dtype=float32_ref>, <tf.Variable 'upscale_32_1_conv2d/bias:0' shape=(1008,) dtype=float32_ref>, <tf.Variable 'residual_64_2_conv2d_0/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'residual_64_2_conv2d_0/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv2d_64_2/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'conv2d_64_2/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'residual_64_3_conv2d_0/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'residual_64_3_conv2d_0/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv2d_64_3/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'conv2d_64_3/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'upscale_64_1_conv2d/kernel:0' shape=(3, 3, 252, 504) dtype=float32_ref>, <tf.Variable 'upscale_64_1_conv2d/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'residual_128_2_conv2d_0/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'residual_128_2_conv2d_0/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv2d_128_2/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'conv2d_128_2/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'residual_128_3_conv2d_0/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'residual_128_3_conv2d_0/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv2d_128_3/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'conv2d_128_3/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'face_out_128_1/kernel:0' shape=(5, 5, 126, 3) dtype=float32_ref>, <tf.Variable 'face_out_128_1/bias:0' shape=(3,) dtype=float32_ref>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG with boundary backprop substitutions [<tf.Tensor 'training_1/Adam/gradients_2/training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/Reshape_1_grad/Reshape:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Processing list [<tf.Tensor 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1:0' shape=(?, 32, 32, 504) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Found 9 ops to copy within [<tf.Operation 'decoder_b/residual_128_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_16_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_128_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_128_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_64_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_64_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_32_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/MatMul' type=MatMul>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_64_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_128_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_128_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/face_out_128/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_1/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_64_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_1/MatMul' type=MatMul>, <tf.Operation 'decoder_b/face_out_128/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_32_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_128_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_64_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'encoder/upscale_8_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_16_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_128_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_64_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_32_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/add_12/add' type=AddV2>, <tf.Operation 'encoder/reshape_1/Shape' type=Shape>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_10/add' type=AddV2>, <tf.Operation 'encoder/conv_16_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_7/add' type=AddV2>, <tf.Operation 'decoder_b/add_9/add' type=AddV2>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_64_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/face_out_128/Sigmoid' type=Sigmoid>, <tf.Operation 'encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_128_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_32_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_11/add' type=AddV2>, <tf.Operation 'decoder_b/add_8/add' type=AddV2>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/flatten_1/Shape' type=Shape>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'loss_1/decoder_b_loss/sub' type=Sub>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/strided_slice' type=StridedSlice>, <tf.Operation 'encoder/flatten_1/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'encoder/flatten_1/strided_slice' type=StridedSlice>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'loss_1/decoder_b_loss/Abs' type=Abs>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/Reshape/shape' type=Pack>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'loss_1/decoder_b_loss/Mean' type=Mean>, <tf.Operation 'encoder/flatten_1/Prod' type=Prod>, <tf.Operation 'encoder/flatten_1/stack' type=Pack>, <tf.Operation 'loss_1/decoder_b_loss/Mean_1' type=Mean>, <tf.Operation 'loss_1/decoder_b_loss/mul' type=Mul>, <tf.Operation 'loss_1/decoder_b_loss/truediv' type=RealDiv>, <tf.Operation 'loss_1/decoder_b_loss/Mean_3' type=Mean>, <tf.Operation 'loss_1/mul' type=Mul>], seed [<tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>], stop_at [<tf.Tensor 'encoder/conv_16_0_conv2d/BiasAdd:0' shape=(?, 8, 8, 1008) dtype=float32>, <tf.Tensor 'decoder_b/residual_128_3_leakyrelu_3/LeakyRelu:0' shape=(?, 128, 128, 126) dtype=float32>, <tf.Tensor 'encoder/upscale_8_0_conv2d/BiasAdd:0' shape=(?, 8, 8, 2048) dtype=float32>, <tf.Tensor 'decoder_b/upscale_64_1_conv2d/BiasAdd:0' shape=(?, 64, 64, 504) dtype=float32>, <tf.Tensor 'decoder_b/upscale_32_1_pixelshuffler/transpose:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG ops_to_copy = [<tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>]: []
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: encoder/upscale_8_0_pixelshuffler/Reshape
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/upscale_16_1_conv2d/convolution
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: encoder/upscale_8_0_leakyrelu/LeakyRelu
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/upscale_16_1_pixelshuffler/Reshape_1
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/upscale_16_1_conv2d/BiasAdd
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: encoder/upscale_8_0_pixelshuffler/Reshape_1
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/upscale_16_1_pixelshuffler/transpose
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: encoder/upscale_8_0_pixelshuffler/transpose
01/23/2020 20:07:59 MainProcess _training_0 transform _connect_control_inputs DEBUG Connecting control inputs of op: decoder_b/upscale_16_1_pixelshuffler/Reshape
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Copied [<tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>] to dict_values([<tf.Operation 'training_1/Adam/encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>]): []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Rewired %s in place of %s restricted to %s: [['training_1/Adam/decoder_b/residual_128_3_leakyrelu_3/LeakyRelu_sg:0', 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose_sg:0', 'training_1/Adam/decoder_b/upscale_64_1_conv2d/BiasAdd_sg:0', 'training_1/Adam/encoder/conv_16_0_conv2d/BiasAdd_sg:0', 'training_1/Adam/encoder/upscale_8_0_conv2d/BiasAdd_sg:0'], ['decoder_b/residual_128_3_leakyrelu_3/LeakyRelu:0', 'decoder_b/upscale_32_1_pixelshuffler/transpose:0', 'decoder_b/upscale_64_1_conv2d/BiasAdd:0', 'encoder/conv_16_0_conv2d/BiasAdd:0', 'encoder/upscale_8_0_conv2d/BiasAdd:0'], ['training_1/Adam/decoder_b/upscale_16_1_conv2d/BiasAdd', 'training_1/Adam/decoder_b/upscale_16_1_conv2d/convolution', 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape', 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1', 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/transpose', 'training_1/Adam/encoder/upscale_8_0_leakyrelu/LeakyRelu', 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/Reshape', 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/Reshape_1', 'training_1/Adam/encoder/upscale_8_0_pixelshuffler/transpose']]
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Got gradients [None, None, <tf.Tensor 'training_1/Adam/gradients_4/training_1/Adam/encoder/upscale_8_0_leakyrelu/LeakyRelu_grad/LeakyReluGrad:0' shape=(?, 8, 8, 2048) dtype=float32>, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, <tf.Tensor 'training_1/Adam/gradients_4/training_1/Adam/decoder_b/upscale_16_1_conv2d/convolution_grad/Conv2DBackpropFilter:0' shape=(3, 3, 512, 2016) dtype=float32>, <tf.Tensor 'training_1/Adam/gradients_4/training_1/Adam/decoder_b/upscale_16_1_conv2d/BiasAdd_grad/BiasAddGrad:0' shape=(2016,) dtype=float32>, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG for [<tf.Tensor 'training_1/Adam/decoder_b/upscale_16_1_pixelshuffler/Reshape_1:0' shape=(?, 32, 32, 504) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG with respect to [<tf.Tensor 'training_1/Adam/encoder/conv_16_0_conv2d/BiasAdd_sg:0' shape=(?, 8, 8, 1008) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/residual_128_3_leakyrelu_3/LeakyRelu_sg:0' shape=(?, 128, 128, 126) dtype=float32>, <tf.Tensor 'training_1/Adam/encoder/upscale_8_0_conv2d/BiasAdd_sg:0' shape=(?, 8, 8, 2048) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/upscale_64_1_conv2d/BiasAdd_sg:0' shape=(?, 64, 64, 504) dtype=float32>, <tf.Tensor 'training_1/Adam/decoder_b/upscale_32_1_pixelshuffler/transpose_sg:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>, <tf.Variable 'conv_128_0_conv2d/kernel:0' shape=(5, 5, 3, 126) dtype=float32_ref>, <tf.Variable 'conv_128_0_conv2d/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv_64_0_conv2d/kernel:0' shape=(5, 5, 126, 252) dtype=float32_ref>, <tf.Variable 'conv_64_0_conv2d/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv_32_0_conv2d/kernel:0' shape=(5, 5, 252, 504) dtype=float32_ref>, <tf.Variable 'conv_32_0_conv2d/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv_16_0_conv2d/kernel:0' shape=(5, 5, 504, 1008) dtype=float32_ref>, <tf.Variable 'conv_16_0_conv2d/bias:0' shape=(1008,) dtype=float32_ref>, <tf.Variable 'dense_1/kernel:0' shape=(64512, 512) dtype=float32_ref>, <tf.Variable 'dense_1/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'dense_2/kernel:0' shape=(512, 32768) dtype=float32_ref>, <tf.Variable 'dense_2/bias:0' shape=(32768,) dtype=float32_ref>, <tf.Variable 'upscale_8_0_conv2d/kernel:0' shape=(3, 3, 512, 2048) dtype=float32_ref>, <tf.Variable 'upscale_8_0_conv2d/bias:0' shape=(2048,) dtype=float32_ref>, <tf.Variable 'upscale_16_1_conv2d/kernel:0' shape=(3, 3, 512, 2016) dtype=float32_ref>, <tf.Variable 'upscale_16_1_conv2d/bias:0' shape=(2016,) dtype=float32_ref>, <tf.Variable 'residual_32_2_conv2d_0/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'residual_32_2_conv2d_0/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv2d_32_2/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'conv2d_32_2/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'residual_32_3_conv2d_0/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'residual_32_3_conv2d_0/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'conv2d_32_3/kernel:0' shape=(3, 3, 504, 504) dtype=float32_ref>, <tf.Variable 'conv2d_32_3/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'upscale_32_1_conv2d/kernel:0' shape=(3, 3, 504, 1008) dtype=float32_ref>, <tf.Variable 'upscale_32_1_conv2d/bias:0' shape=(1008,) dtype=float32_ref>, <tf.Variable 'residual_64_2_conv2d_0/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'residual_64_2_conv2d_0/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv2d_64_2/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'conv2d_64_2/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'residual_64_3_conv2d_0/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'residual_64_3_conv2d_0/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'conv2d_64_3/kernel:0' shape=(3, 3, 252, 252) dtype=float32_ref>, <tf.Variable 'conv2d_64_3/bias:0' shape=(252,) dtype=float32_ref>, <tf.Variable 'upscale_64_1_conv2d/kernel:0' shape=(3, 3, 252, 504) dtype=float32_ref>, <tf.Variable 'upscale_64_1_conv2d/bias:0' shape=(504,) dtype=float32_ref>, <tf.Variable 'residual_128_2_conv2d_0/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'residual_128_2_conv2d_0/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv2d_128_2/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'conv2d_128_2/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'residual_128_3_conv2d_0/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'residual_128_3_conv2d_0/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'conv2d_128_3/kernel:0' shape=(3, 3, 126, 126) dtype=float32_ref>, <tf.Variable 'conv2d_128_3/bias:0' shape=(126,) dtype=float32_ref>, <tf.Variable 'face_out_128_1/kernel:0' shape=(5, 5, 126, 3) dtype=float32_ref>, <tf.Variable 'face_out_128_1/bias:0' shape=(3,) dtype=float32_ref>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG with boundary backprop substitutions [<tf.Tensor 'training_1/Adam/gradients_3/AddN_1:0' shape=(?, 32, 32, 504) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Processing list [<tf.Tensor 'encoder/upscale_8_0_conv2d/BiasAdd:0' shape=(?, 8, 8, 2048) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG Found 16 ops to copy within [<tf.Operation 'decoder_b/residual_128_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_16_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_128_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_128_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_64_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_64_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_32_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_32_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/MatMul' type=MatMul>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_64_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_128_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_128_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_128_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/face_out_128/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_3_conv2d_0/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_1/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_64_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_1/MatMul' type=MatMul>, <tf.Operation 'decoder_b/face_out_128/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/conv_32_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_128_2/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_32_3/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/residual_32_2_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_64_1_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/upscale_16_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/residual_64_3_conv2d_0/convolution' type=Conv2D>, <tf.Operation 'encoder/upscale_8_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'decoder_b/conv2d_64_2/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_16_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/conv2d_128_3/convolution' type=Conv2D>, <tf.Operation 'decoder_b/upscale_64_1_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/conv_32_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'decoder_b/add_12/add' type=AddV2>, <tf.Operation 'encoder/reshape_1/Shape' type=Shape>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_10/add' type=AddV2>, <tf.Operation 'encoder/conv_16_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_7/add' type=AddV2>, <tf.Operation 'decoder_b/add_9/add' type=AddV2>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_1/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_64_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/face_out_128/Sigmoid' type=Sigmoid>, <tf.Operation 'encoder/upscale_8_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_128_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/conv_32_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/add_11/add' type=AddV2>, <tf.Operation 'decoder_b/add_8/add' type=AddV2>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/flatten_1/Shape' type=Shape>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'loss_1/decoder_b_loss/sub' type=Sub>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_3/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/strided_slice' type=StridedSlice>, <tf.Operation 'encoder/flatten_1/Reshape' type=Reshape>, <tf.Operation 'decoder_b/residual_32_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_32_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'encoder/flatten_1/strided_slice' type=StridedSlice>, <tf.Operation 'decoder_b/residual_128_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/upscale_64_1_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'loss_1/decoder_b_loss/Abs' type=Abs>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/transpose' type=Transpose>, <tf.Operation 'decoder_b/residual_64_3_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/reshape_1/Reshape/shape' type=Pack>, <tf.Operation 'decoder_b/residual_128_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_64_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'decoder_b/residual_32_2_leakyrelu_0/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/upscale_8_0_pixelshuffler/Reshape_1' type=Reshape>, <tf.Operation 'loss_1/decoder_b_loss/Mean' type=Mean>, <tf.Operation 'encoder/flatten_1/Prod' type=Prod>, <tf.Operation 'encoder/flatten_1/stack' type=Pack>, <tf.Operation 'loss_1/decoder_b_loss/Mean_1' type=Mean>, <tf.Operation 'loss_1/decoder_b_loss/mul' type=Mul>, <tf.Operation 'loss_1/decoder_b_loss/truediv' type=RealDiv>, <tf.Operation 'loss_1/decoder_b_loss/Mean_3' type=Mean>, <tf.Operation 'loss_1/mul' type=Mul>], seed [<tf.Operation 'encoder/upscale_8_0_conv2d/BiasAdd' type=BiasAdd>], stop_at [<tf.Tensor 'encoder/conv_16_0_conv2d/BiasAdd:0' shape=(?, 8, 8, 1008) dtype=float32>, <tf.Tensor 'decoder_b/residual_128_3_leakyrelu_3/LeakyRelu:0' shape=(?, 128, 128, 126) dtype=float32>, <tf.Tensor 'decoder_b/upscale_64_1_conv2d/BiasAdd:0' shape=(?, 64, 64, 504) dtype=float32>, <tf.Tensor 'decoder_b/upscale_16_1_pixelshuffler/Reshape_1:0' shape=(?, 32, 32, 504) dtype=float32>, <tf.Tensor 'decoder_b/upscale_32_1_pixelshuffler/transpose:0' shape=(?, 32, 2, 32, 2, 252) dtype=float32>]: []
01/23/2020 20:07:59 MainProcess _training_0 memory_saving_gradients debug_print DEBUG ops_to_copy = [<tf.Operation 'encoder/dense_1/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/reshape_1/Shape' type=Shape>, <tf.Operation 'encoder/dense_1/MatMul' type=MatMul>, <tf.Operation 'encoder/flatten_1/strided_slice' type=StridedSlice>, <tf.Operation 'encoder/reshape_1/Reshape/shape' type=Pack>, <tf.Operation 'encoder/conv_16_0_leakyrelu/LeakyRelu' type=LeakyRelu>, <tf.Operation 'encoder/flatten_1/Prod' type=Prod>, <tf.Operation 'encoder/flatten_1/Reshape' type=Reshape>, <tf.Operation 'encoder/reshape_1/Reshape' type=Reshape>, <tf.Operation 'encoder/dense_2/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/upscale_8_0_conv2d/BiasAdd' type=BiasAdd>, <tf.Operation 'encoder/dense_2/MatMul' type=MatMul>, <tf.Operation 'encoder/upscale_8_0_conv2d/convolution' type=Conv2D>, <tf.Operation 'encoder/flatten_1/Shape' type=Shape>, <tf.Operation 'encoder/flatten_1/stack' type=Pack>, <tf.Operation 'encoder/reshape_1/strided_slice' type=StridedSlice>]: []