Re: [Resource] Training Using Google Colab
Posted: Thu Jan 27, 2022 6:58 pm
I couldn't fit everything in just one post, but here's the first part of crash report:
Code: Select all
01/27/2022 11:39:46 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _get_headers DEBUG side: 'a', width: 150
01/27/2022 11:39:46 MainProcess _training_0 _base _get_headers DEBUG height: 33, total_width: 450
01/27/2022 11:39:46 MainProcess _training_0 _base _get_headers DEBUG texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(84, 11), (136, 11), (119, 11)], text_x: [33, 157, 315], text_y: 22
01/27/2022 11:39:46 MainProcess _training_0 _base _get_headers DEBUG header_box.shape: (33, 450, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _to_full_frame DEBUG side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 128, 128, 3), (14, 128, 128, 3)])
01/27/2022 11:39:46 MainProcess _training_0 _base _process_full DEBUG full_size: 384, prediction_size: 128, color: (0, 0, 255)
01/27/2022 11:39:46 MainProcess _training_0 _base _resize_sample DEBUG Resizing sample: (side: 'b', sample.shape: (14, 384, 384, 3), target_size: 150, scale: 0.390625)
01/27/2022 11:39:46 MainProcess _training_0 _base _resize_sample DEBUG Resized sample: (side: 'b' shape: (14, 150, 150, 3))
01/27/2022 11:39:46 MainProcess _training_0 _base _process_full DEBUG Overlayed background. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _compile_masked DEBUG masked shapes: [(14, 128, 128, 3), (14, 128, 128, 3), (14, 128, 128, 3)]
01/27/2022 11:39:46 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _get_headers DEBUG side: 'b', width: 150
01/27/2022 11:39:46 MainProcess _training_0 _base _get_headers DEBUG height: 33, total_width: 450
01/27/2022 11:39:46 MainProcess _training_0 _base _get_headers DEBUG texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(69, 11), (102, 11), (119, 11)], text_x: [40, 174, 315], text_y: 22
01/27/2022 11:39:46 MainProcess _training_0 _base _get_headers DEBUG header_box.shape: (33, 450, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _duplicate_headers DEBUG side: a header.shape: (33, 450, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _duplicate_headers DEBUG side: b header.shape: (33, 450, 3)
01/27/2022 11:39:46 MainProcess _training_0 _base _stack_images DEBUG Stack images
01/27/2022 11:39:46 MainProcess _training_0 _base get_transpose_axes DEBUG Even number of images to stack
01/27/2022 11:39:46 MainProcess _training_0 _base _stack_images DEBUG Stacked images
01/27/2022 11:39:46 MainProcess _training_0 _base show_sample DEBUG Compiled sample
01/27/2022 11:39:46 MainProcess _training_0 _base output_timelapse DEBUG Created time-lapse: '/content/drive/My Drive/colab_files/faceswap/output/timelapse/1643305186.jpg'
01/27/2022 11:39:46 MainProcess _training_0 train _run_training_cycle DEBUG Save Iteration: (iteration: 5000
01/27/2022 11:39:46 MainProcess _training_0 _base _save DEBUG Backing up and saving models
01/27/2022 11:39:46 MainProcess _training_0 _base _get_save_averages DEBUG Getting save averages
01/27/2022 11:39:46 MainProcess _training_0 _base _get_save_averages DEBUG Average losses since last save: [0.010751839741133154, 0.01970715313591063]
01/27/2022 11:39:46 MainProcess _training_0 _base _should_backup DEBUG Should backup: False
01/27/2022 11:39:53 MainProcess _training_0 _base save DEBUG Saving State
01/27/2022 11:39:53 MainProcess _training_0 serializer save DEBUG filename: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json, data type: <class 'dict'>
01/27/2022 11:39:53 MainProcess _training_0 serializer _check_extension DEBUG Original filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json', final filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json'
01/27/2022 11:39:53 MainProcess _training_0 serializer marshal DEBUG data type: <class 'dict'>
01/27/2022 11:39:53 MainProcess _training_0 serializer marshal DEBUG returned data type: <class 'bytes'>
01/27/2022 11:39:53 MainProcess _training_0 _base save DEBUG Saved State
01/27/2022 11:39:53 MainProcess _training_0 _base _save INFO [Saved models] - Average loss since last save: face_a: 0.01075, face_b: 0.01971
01/27/2022 12:17:11 MainProcess _training_0 _base output_timelapse DEBUG Ouputting time-lapse
01/27/2022 12:17:11 MainProcess _training_0 _base output_timelapse DEBUG Getting time-lapse samples
01/27/2022 12:17:11 MainProcess _training_0 _base compile_sample DEBUG Compiling samples: (side: 'a', samples: 14)
01/27/2022 12:17:11 MainProcess _training_0 _base compile_sample DEBUG Compiling samples: (side: 'b', samples: 14)
01/27/2022 12:17:11 MainProcess _training_0 _base output_timelapse DEBUG Got time-lapse samples: {'a': 3, 'b': 3}
01/27/2022 12:17:11 MainProcess _training_0 _base show_sample DEBUG Showing sample
01/27/2022 12:17:11 MainProcess _training_0 _base _get_predictions DEBUG Getting Predictions
01/27/2022 12:17:13 MainProcess _training_0 _base _get_predictions DEBUG Returning predictions: {'a_a': (14, 128, 128, 3), 'b_b': (14, 128, 128, 3), 'a_b': (14, 128, 128, 3), 'b_a': (14, 128, 128, 3)}
01/27/2022 12:17:13 MainProcess _training_0 _base _to_full_frame DEBUG side: 'a', number of sample arrays: 3, prediction.shapes: [(14, 128, 128, 3), (14, 128, 128, 3)])
01/27/2022 12:17:13 MainProcess _training_0 _base _process_full DEBUG full_size: 384, prediction_size: 128, color: (0, 0, 255)
01/27/2022 12:17:13 MainProcess _training_0 _base _resize_sample DEBUG Resizing sample: (side: 'a', sample.shape: (14, 384, 384, 3), target_size: 150, scale: 0.390625)
01/27/2022 12:17:13 MainProcess _training_0 _base _resize_sample DEBUG Resized sample: (side: 'a' shape: (14, 150, 150, 3))
01/27/2022 12:17:13 MainProcess _training_0 _base _process_full DEBUG Overlayed background. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _compile_masked DEBUG masked shapes: [(14, 128, 128, 3), (14, 128, 128, 3), (14, 128, 128, 3)]
01/27/2022 12:17:13 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _get_headers DEBUG side: 'a', width: 150
01/27/2022 12:17:13 MainProcess _training_0 _base _get_headers DEBUG height: 33, total_width: 450
01/27/2022 12:17:13 MainProcess _training_0 _base _get_headers DEBUG texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(84, 11), (136, 11), (119, 11)], text_x: [33, 157, 315], text_y: 22
01/27/2022 12:17:13 MainProcess _training_0 _base _get_headers DEBUG header_box.shape: (33, 450, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _to_full_frame DEBUG side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 128, 128, 3), (14, 128, 128, 3)])
01/27/2022 12:17:13 MainProcess _training_0 _base _process_full DEBUG full_size: 384, prediction_size: 128, color: (0, 0, 255)
01/27/2022 12:17:13 MainProcess _training_0 _base _resize_sample DEBUG Resizing sample: (side: 'b', sample.shape: (14, 384, 384, 3), target_size: 150, scale: 0.390625)
01/27/2022 12:17:13 MainProcess _training_0 _base _resize_sample DEBUG Resized sample: (side: 'b' shape: (14, 150, 150, 3))
01/27/2022 12:17:13 MainProcess _training_0 _base _process_full DEBUG Overlayed background. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _compile_masked DEBUG masked shapes: [(14, 128, 128, 3), (14, 128, 128, 3), (14, 128, 128, 3)]
01/27/2022 12:17:13 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _overlay_foreground DEBUG Overlayed foreground. Shape: (14, 150, 150, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _get_headers DEBUG side: 'b', width: 150
01/27/2022 12:17:13 MainProcess _training_0 _base _get_headers DEBUG height: 33, total_width: 450
01/27/2022 12:17:13 MainProcess _training_0 _base _get_headers DEBUG texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(69, 11), (102, 11), (119, 11)], text_x: [40, 174, 315], text_y: 22
01/27/2022 12:17:13 MainProcess _training_0 _base _get_headers DEBUG header_box.shape: (33, 450, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _duplicate_headers DEBUG side: a header.shape: (33, 450, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _duplicate_headers DEBUG side: b header.shape: (33, 450, 3)
01/27/2022 12:17:13 MainProcess _training_0 _base _stack_images DEBUG Stack images
01/27/2022 12:17:13 MainProcess _training_0 _base get_transpose_axes DEBUG Even number of images to stack
01/27/2022 12:17:13 MainProcess _training_0 _base _stack_images DEBUG Stacked images
01/27/2022 12:17:13 MainProcess _training_0 _base show_sample DEBUG Compiled sample
01/27/2022 12:17:14 MainProcess _training_0 _base output_timelapse DEBUG Created time-lapse: '/content/drive/My Drive/colab_files/faceswap/output/timelapse/1643307433.jpg'
01/27/2022 12:17:14 MainProcess _training_0 train _run_training_cycle DEBUG Save Iteration: (iteration: 6000
01/27/2022 12:17:14 MainProcess _training_0 _base _save DEBUG Backing up and saving models
01/27/2022 12:17:14 MainProcess _training_0 _base _get_save_averages DEBUG Getting save averages
01/27/2022 12:17:14 MainProcess _training_0 _base _get_save_averages DEBUG Average losses since last save: [0.010746411305852235, 0.019705730494111777]
01/27/2022 12:17:14 MainProcess _training_0 _base _should_backup DEBUG Should backup: False
01/27/2022 12:17:20 MainProcess _training_0 _base save DEBUG Saving State
01/27/2022 12:17:20 MainProcess _training_0 serializer save DEBUG filename: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json, data type: <class 'dict'>
01/27/2022 12:17:20 MainProcess _training_0 serializer _check_extension DEBUG Original filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json', final filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json'
01/27/2022 12:17:20 MainProcess _training_0 serializer marshal DEBUG data type: <class 'dict'>
01/27/2022 12:17:20 MainProcess _training_0 serializer marshal DEBUG returned data type: <class 'bytes'>
01/27/2022 12:17:20 MainProcess _training_0 _base save DEBUG Saved State
01/27/2022 12:17:20 MainProcess _training_0 _base _save INFO [Saved models] - Average loss since last save: face_a: 0.01075, face_b: 0.01971
01/27/2022 12:28:18 MainProcess _training_0 multithreading run DEBUG Error in thread (_training_0): 2 root error(s) found.\n (0) FAILED_PRECONDITION: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected\n Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2\n Could not flush events file.\n [[node batch_decoder_b_loss\n (defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)\n]]\n [[GroupCrossDeviceControlEdges_0/NoOp/_91]]\n (1) FAILED_PRECONDITION: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected\n Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2\n Could not flush events file.\n [[node batch_decoder_b_loss\n (defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)\n]]\n0 successful operations.\n0 derived errors ignored. [Op:__inference_train_function_14947]\n\nErrors may have originated from an input operation.\nInput Source operations connected to node batch_decoder_b_loss:\nIn[0] batch_decoder_b_loss/writer: \nIn[1] batch_decoder_b_loss/ReadVariableOp: \nIn[2] batch_decoder_b_loss/tag: \nIn[3] batch_decoder_b_loss/Identity:\n\nOperation defined at: (most recent call last)\n>>> File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap\n>>> self._bootstrap_inner()\n>>> \n>>> File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner\n>>> self.run()\n>>> \n>>> File "/content/faceswap/lib/multithreading.py", line 37, in run\n>>> self._target(*self._args, **self._kwargs)\n>>> \n>>> File "/content/faceswap/scripts/train.py", line 242, in _training\n>>> self._run_training_cycle(model, trainer)\n>>> \n>>> File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle\n>>> trainer.train_one_step(viewer, timelapse)\n>>> \n>>> File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step\n>>> loss = self._model.model.train_on_batch(model_inputs, y=model_targets)\n>>> \n>>> File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch\n>>> logs = self.train_function(iterator)\n>>> \n>>> File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function\n>>> return step_function(self, iterator)\n>>> \n>>> File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function\n>>> write_scalar_summaries(outputs, step=model._train_counter) # pylint: disable=protected-access\n>>> \n>>> File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries\n>>> summary_ops_v2.scalar('batch_' + name, value, step=step)\n>>> \n\nInput Source operations connected to node batch_decoder_b_loss:\nIn[0] batch_decoder_b_loss/writer: \nIn[1] batch_decoder_b_loss/ReadVariableOp: \nIn[2] batch_decoder_b_loss/tag: \nIn[3] batch_decoder_b_loss/Identity:\n\nOperation defined at: (most recent call last)\n>>> File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap\n>>> self._bootstrap_inner()\n>>> \n>>> File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner\n>>> self.run()\n>>> \n>>> File "/content/faceswap/lib/multithreading.py", line 37, in run\n>>> self._target(*self._args, **self._kwargs)\n>>> \n>>> File "/content/faceswap/scripts/train.py", line 242, in _training\n>>> self._run_training_cycle(model, trainer)\n>>> \n>>> File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle\n>>> trainer.train_one_step(viewer, timelapse)\n>>> \n>>> File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step\n>>> loss = self._model.model.train_on_batch(model_inputs, y=model_targets)\n>>> \n>>> File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch\n>>> logs = self.train_function(iterator)\n>>> \n>>> File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function\n>>> return step_function(self, iterator)\n>>> \n>>> File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function\n>>> write_scalar_summaries(outputs, step=model._train_counter) # pylint: disable=protected-access\n>>> \n>>> File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries\n>>> summary_ops_v2.scalar('batch_' + name, value, step=step)\n>>> \n\nFunction call stack:\ntrain_function -> cond_2_true_14925 -> train_function -> cond_2_true_14925\n
01/27/2022 12:28:18 MainProcess MainThread train _monitor DEBUG Thread error detected
01/27/2022 12:28:18 MainProcess MainThread train _monitor DEBUG Closed Monitor
01/27/2022 12:28:18 MainProcess MainThread train _end_thread DEBUG Ending Training thread
01/27/2022 12:28:18 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...
01/27/2022 12:28:18 MainProcess MainThread multithreading join DEBUG Joining Threads: '_training'
01/27/2022 12:28:18 MainProcess MainThread multithreading join DEBUG Joining Thread: '_training_0'
01/27/2022 12:28:18 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training_0'
Traceback (most recent call last):
File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
process.process()
File "/content/faceswap/scripts/train.py", line 190, in process
self._end_thread(thread, err)
File "/content/faceswap/scripts/train.py", line 230, in _end_thread
thread.join()
File "/content/faceswap/lib/multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "/content/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/content/faceswap/scripts/train.py", line 252, in _training
raise err
File "/content/faceswap/scripts/train.py", line 242, in _training
self._run_training_cycle(model, trainer)
File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
logs = self.train_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
(0) FAILED_PRECONDITION: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected
Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2
Could not flush events file.
[[node batch_decoder_b_loss
(defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)
]]
[[GroupCrossDeviceControlEdges_0/NoOp/_91]]
(1) FAILED_PRECONDITION: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected
Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2
Could not flush events file.
[[node batch_decoder_b_loss
(defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)
]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14947]
Errors may have originated from an input operation.
Input Source operations connected to node batch_decoder_b_loss:
In[0] batch_decoder_b_loss/writer:
In[1] batch_decoder_b_loss/ReadVariableOp:
In[2] batch_decoder_b_loss/tag:
In[3] batch_decoder_b_loss/Identity:
Operation defined at: (most recent call last)
File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/content/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/content/faceswap/scripts/train.py", line 242, in _training
self._run_training_cycle(model, trainer)
File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
logs = self.train_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function
return step_function(self, iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function
write_scalar_summaries(outputs, step=model._train_counter) # pylint: disable=protected-access
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries
summary_ops_v2.scalar('batch_' + name, value, step=step)
Input Source operations connected to node batch_decoder_b_loss:
In[0] batch_decoder_b_loss/writer:
In[1] batch_decoder_b_loss/ReadVariableOp:
In[2] batch_decoder_b_loss/tag:
In[3] batch_decoder_b_loss/Identity:
Operation defined at: (most recent call last)
File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/content/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/content/faceswap/scripts/train.py", line 242, in _training
self._run_training_cycle(model, trainer)
File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
logs = self.train_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function
return step_function(self, iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function
write_scalar_summaries(outputs, step=model._train_counter) # pylint: disable=protected-access
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries
summary_ops_v2.scalar('batch_' + name, value, step=step)
Function call stack:
train_function -> cond_2_true_14925 -> train_function -> cond_2_true_14925
============ System Information ============
encoding: UTF-8
git_branch: Not Found
git_commits: Not Found
gpu_cuda: 11.1
gpu_cudnn: No global version found
gpu_devices: GPU_0: Tesla K80
gpu_devices_active: GPU_0
gpu_driver: 460.32.03
gpu_vram: GPU_0: 11441MB
os_machine: x86_64
os_platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
os_release: 5.4.144+
py_command: faceswap/faceswap.py train -A face_a -B face_b -m /content/drive/My Drive/colab_files/faceswap/models/JdKQry -t villain -bs 8 -it 250000 -s 1000 -ss 50000 -tia face_a -tib face_b -to /content/drive/My Drive/colab_files/faceswap/output/timelapse -nf
py_conda_version: N/A
py_implementation: CPython
py_version: 3.7.12
py_virtual_env: False
sys_cores: 2
sys_processor: x86_64
sys_ram: Total: 12991MB, Available: 10007MB, Used: 3928MB, Free: 967MB
=============== Pip Packages ===============
absl-py==1.0.0
alabaster==0.7.12
albumentations==0.1.12
altair==4.2.0
appdirs==1.4.4
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arviz==0.11.4
astor==0.8.1
astropy==4.3.1
astunparse==1.6.3
atari-py==0.2.9
atomicwrites==1.4.0
attrs==21.4.0
audioread==2.1.9
autograd==1.3
Babel==2.9.1
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==4.1.0
blis==0.4.1
bokeh==2.3.3
Bottleneck==1.3.2
branca==0.4.2
bs4==0.0.1
CacheControl==0.12.10
cached-property==1.5.2
cachetools==4.2.4
catalogue==1.0.0
certifi==2021.10.8
cffi==1.15.0
cftime==1.5.2
chardet==3.0.4
charset-normalizer==2.0.10
click==7.1.2
cloudpickle==1.3.0
cmake==3.12.0
cmdstanpy==0.9.5
colorcet==3.0.0
colorlover==0.3.0
community==1.0.0b1
contextlib2==0.5.5
convertdate==2.4.0
coverage==3.7.1
coveralls==0.5
crcmod==1.7
cufflinks==0.17.3
cupy-cuda111==9.4.0
cvxopt==1.2.7
cvxpy==1.0.31
cycler==0.11.0
cymem==2.0.6
Cython==0.29.26
daft==0.0.4
dask==2.12.0
datascience==0.10.6
debugpy==1.0.0
decorator==4.4.2
defusedxml==0.7.1
descartes==1.1.0
dill==0.3.4
distributed==1.25.3
dlib @ file:///dlib-19.18.0-cp37-cp37m-linux_x86_64.whl
dm-tree==0.1.6
docopt==0.6.2
docutils==0.17.1
dopamine-rl==1.0.5
earthengine-api==0.1.295
easydict==1.9
ecos==2.0.10
editdistance==0.5.3
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.5/en_core_web_sm-2.2.5.tar.gz
entrypoints==0.3
ephem==4.1.3
et-xmlfile==1.1.0
fa2==0.3.5
fastai==1.0.61
fastcluster==1.2.4
fastdtw==0.3.4
fastprogress==1.0.0
fastrlock==0.8
fbprophet==0.7.1
feather-format==0.4.1
ffmpy==0.2.3
filelock==3.4.2
firebase-admin==4.4.0
fix-yahoo-finance==0.0.22
Flask==1.1.4
flatbuffers==2.0
folium==0.8.3
future==0.16.0
gast==0.4.0
GDAL==2.2.2
gdown==3.6.4
gensim==3.6.0
geographiclib==1.52
geopy==1.17.0
gin-config==0.5.0
glob2==0.7
google==2.0.3
google-api-core==1.26.3
google-api-python-client==1.12.10
google-auth==1.35.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.6
google-cloud-bigquery==1.21.0
google-cloud-bigquery-storage==1.1.0
google-cloud-core==1.0.3
google-cloud-datastore==1.8.0
google-cloud-firestore==1.7.0
google-cloud-language==1.2.0
google-cloud-storage==1.18.1
google-cloud-translate==1.5.0
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz
google-pasta==0.2.0
google-resumable-media==0.4.1
googleapis-common-protos==1.54.0
googledrivedownloader==0.4
GPUtil==1.4.0
graphviz==0.10.1
greenlet==1.1.2
grpcio==1.43.0
gspread==3.4.2
gspread-dataframe==3.0.8
gym==0.17.3
h5py==3.1.0
HeapDict==1.0.1
hijri-converter==2.2.2
holidays==0.10.5.2
holoviews==1.14.7
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.17.4
httplib2shim==0.0.3
humanize==0.5.1
hyperopt==0.1.2
ideep4py==2.0.0.post3
idna==2.10
imageio==2.14.1
imageio-ffmpeg==0.4.5
imagesize==1.3.0
imbalanced-learn==0.8.1
imblearn==0.0
imgaug==0.2.9
importlib-metadata==4.10.1
importlib-resources==5.4.0
imutils==0.5.4
inflect==2.1.0
iniconfig==1.1.1
intel-openmp==2022.0.2
intervaltree==2.1.0
ipykernel==4.10.1
ipython==5.5.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.6.5
itsdangerous==1.1.0
jax==0.2.25
jaxlib @ https://storage.googleapis.com/jax-releases/cuda111/jaxlib-0.1.71+cuda111-cp37-none-manylinux2010_x86_64.whl
jdcal==1.4.1
jedi==0.18.1
jieba==0.42.1
Jinja2==2.11.3
joblib==1.1.0
jpeg4py==0.1.4
jsonschema==4.3.3
jupyter==1.0.0
jupyter-client==5.3.5
jupyter-console==5.2.0
jupyter-core==4.9.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.2
kaggle==1.5.12
kapre==0.3.7
keras==2.7.0
Keras-Preprocessing==1.1.2
keras-vis==0.4.1
kiwisolver==1.3.2
korean-lunar-calendar==0.2.1
libclang==12.0.0
librosa==0.8.1
lightgbm==2.2.3
llvmlite==0.34.0
lmdb==0.99
LunarCalendar==0.0.9
lxml==4.2.6
Markdown==3.3.6
MarkupSafe==2.0.1
matplotlib==3.2.2
matplotlib-inline==0.1.3
matplotlib-venn==0.11.6
missingno==0.5.0
mistune==0.8.4
mizani==0.6.0
mkl==2019.0
mlxtend==0.14.0
more-itertools==8.12.0
moviepy==0.2.3.5
mpmath==1.2.1
msgpack==1.0.3
multiprocess==0.70.12.2
multitasking==0.0.10
murmurhash==1.0.6
music21==5.5.0
natsort==5.5.0
nbclient==0.5.10
nbconvert==5.6.1
nbformat==5.1.3
nest-asyncio==1.5.4
netCDF4==1.5.8
networkx==2.6.3
nibabel==3.0.2
nltk==3.2.5
notebook==5.3.1
numba==0.51.2
numexpr==2.8.1
numpy==1.19.5
nvidia-ml-py==11.495.46
nvidia-ml-py3==7.352.0
oauth2client==4.1.3
oauthlib==3.1.1
okgrade==0.4.3
opencv-contrib-python==4.1.2.30
opencv-python==4.5.5.62
openpyxl==2.5.9
opt-einsum==3.3.0
osqp==0.6.2.post0
packaging==21.3
palettable==3.3.0
pandas==1.1.5
pandas-datareader==0.9.0
pandas-gbq==0.13.3
pandas-profiling==1.4.1
pandocfilters==1.5.0
panel==0.12.1
param==1.12.0
parso==0.8.3
pathlib==1.0.1
patsy==0.5.2
pep517==0.12.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.0.0
pip-tools==6.2.0
plac==1.1.3
plotly==5.5.0
plotnine==0.6.0
pluggy==0.7.1
pooch==1.6.0
portpicker==1.3.9
prefetch-generator==1.0.1
preshed==3.0.6
prettytable==3.0.0
progressbar2==3.38.0
prometheus-client==0.12.0
promise==2.3
prompt-toolkit==1.0.18
protobuf==3.17.3
psutil==5.9.0
psycopg2==2.7.6.1
ptyprocess==0.7.0
py==1.11.0
pyarrow==3.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.4
pycparser==2.21
pyct==0.4.8
pydata-google-auth==1.3.0
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyerfa==2.0.0.1
pyglet==1.5.0
Pygments==2.6.1
pygobject==3.26.1
pymc3==3.11.4
PyMeeus==0.5.11
pymongo==4.0.1
pymystem3==0.2.0
PyOpenGL==3.1.5
pyparsing==3.0.7
pyrsistent==0.18.1
pysndfile==1.3.8
PySocks==1.7.1
pystan==2.19.1.1
pytest==3.6.4
python-apt==0.0.0
python-chess==0.23.11
python-dateutil==2.8.2
python-louvain==0.15
python-slugify==5.0.2
python-utils==3.1.0
pytz==2018.9
pyviz-comms==2.1.0
PyWavelets==1.2.0
PyYAML==3.13
pyzmq==22.3.0
qdldl==0.1.5.post0
qtconsole==5.2.2
QtPy==2.0.0
regex==2019.12.20
requests==2.23.0
requests-oauthlib==1.3.0
resampy==0.2.2
rpy2==3.4.5
rsa==4.8
scikit-image==0.18.3
scikit-learn==1.0.2
scipy==1.4.1
screen-resolution-extra==0.0.0
scs==3.1.0
seaborn==0.11.2
semver==2.13.0
Send2Trash==1.8.0
setuptools-git==1.2
Shapely==1.8.0
simplegeneric==0.8.1
six==1.15.0
sklearn==0.0
sklearn-pandas==1.8.0
smart-open==5.2.1
snowballstemmer==2.2.0
sortedcontainers==2.4.0
SoundFile==0.10.3.post1
spacy==2.2.4
Sphinx==1.8.6
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib-websupport==1.2.4
SQLAlchemy==1.4.31
sqlparse==0.4.2
srsly==1.0.5
statsmodels==0.10.2
sympy==1.7.1
tables==3.4.4
tabulate==0.8.9
tblib==1.7.0
tenacity==8.0.1
tensorboard==2.7.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow @ file:///tensorflow-2.7.0-cp37-cp37m-linux_x86_64.whl
tensorflow-datasets==4.0.1
tensorflow-estimator==2.7.0
tensorflow-gcs-config==2.7.0
tensorflow-hub==0.12.0
tensorflow-io-gcs-filesystem==0.23.1
tensorflow-metadata==1.6.0
tensorflow-probability==0.15.0
termcolor==1.1.0
terminado==0.12.1
testpath==0.5.0
text-unidecode==1.3
textblob==0.15.3
Theano-PyMC==1.1.2
thinc==7.4.0
threadpoolctl==3.0.0
tifffile==2021.11.2
toml==0.10.2
tomli==2.0.0
toolz==0.11.2
torch @ https://download.pytorch.org/whl/cu111/torch-1.10.0%2Bcu111-cp37-cp37m-linux_x86_64.whl
torchaudio @ https://download.pytorch.org/whl/cu111/torchaudio-0.10.0%2Bcu111-cp37-cp37m-linux_x86_64.whl
torchsummary==1.5.1
torchtext==0.11.0
torchvision @ https://download.pytorch.org/whl/cu111/torchvision-0.11.1%2Bcu111-cp37-cp37m-linux_x86_64.whl
tornado==5.1.1
tqdm==4.62.3
traitlets==5.1.1
tweepy==3.10.0
typeguard==2.7.1
typing-extensions==3.10.0.2
tzlocal==1.5.1
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.9.0
wasabi==0.9.0
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.2
wordcloud==1.5.0
wrapt==1.13.3
xarray==0.18.2
xgboost==0.90
xkit==0.0.0
xlrd==1.1.0
xlwt==1.3.0
yellowbrick==1.3.post1
zict==2.0.0
zipp==3.7.0