Seems Collab now runs python 3.7. I can now install and run version 2 of faceswap.
[Outdated] Training Using Google Colab
Read the FAQs and search the forum before posting a new topic.
NB: The Devs do not directly support using Cloud based services, but you can find community support here.
Please mark any answers that fixed your problems so others can find the solutions.
Re: [Resource] Training Using Google Colab
At default model autosaves every 250 iterations. I don't know the specifics of this Notebook though as I don't use it.
My word is final
Re: [Resource] Training Using Google Colab
I see, it actually makes 360 iteration every 10 minutes. So for my issue, it saves every 360 iteration. Thanks anyway!
And by the way, I can share "the latest" perfectly working, Google Colab.
- foundmyway89
- Posts: 2
- Joined: Wed May 26, 2021 8:16 pm
Re: [Resource] Training Using Google Colab
Please do! I've been trying to get mine working for a week now...
I checked your github and it didn't look like there was a link yet.
Thanks!
Re: [Resource] Training Using Google Colab
foundmyway89 wrote: ↑Thu Jun 03, 2021 12:32 amPlease do! I've been trying to get mine working for a week now...
The notebook from this post should work fine viewtopic.php?p=5136#p5136
- nightking1102
- Posts: 2
- Joined: Sat Jul 10, 2021 2:27 pm
Re: [Resource] Training Using Google Colab
awelmisin wrote: ↑Wed May 26, 2021 11:33 pmhttps://github.com/awelmisin/faceswap-google-colab
can find the latest working version here, i am also taking a youtube video about this.
there is nothing here , I can not find the notebook
- nightking1102
- Posts: 2
- Joined: Sat Jul 10, 2021 2:27 pm
Re: [Resource] Training Using Google Colab
I got the following error when run training:
Questions and feedback: https://faceswap.dev/forum
faceswap.py train: error: the following arguments are required: -A/--input-A, -B/--input-B
- XxXArTiuSXxX
- Posts: 1
- Joined: Fri Oct 01, 2021 11:45 am
Re: [Resource] Training Using Google Colab
Thank you all for the amazing work.
I am catching an error in the final training phase:
Code: Select all
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
category=CustomMaskWarning)
10/01/2021 17:17:38 CRITICAL Error caught! Exiting...
10/01/2021 17:17:38 ERROR Caught exception in thread: '_training_0'
10/01/2021 17:17:39 ERROR Got Exception on main handler:
Traceback (most recent call last):
File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
process.process()
File "/content/faceswap/scripts/train.py", line 190, in process
self._end_thread(thread, err)
File "/content/faceswap/scripts/train.py", line 230, in _end_thread
thread.join()
File "/content/faceswap/lib/multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "/content/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/content/faceswap/scripts/train.py", line 252, in _training
raise err
File "/content/faceswap/scripts/train.py", line 242, in _training
self._run_training_cycle(model, trainer)
File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1854, in train_on_batch
logs = self.train_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
result = self._call(*args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _call
return self._stateless_fn(*args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 3040, in __call__
filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1964, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 596, in call
ctx=ctx)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
[[gradient_tape/LossWrapper_1/DSSIMObjective_2/Reshape_4/_34]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14821]
Function call stack:
train_function -> train_function
Any help is greatly appreciated, thanks again.
Re: [Resource] Training Using Google Colab
You should post the full crash report, but on the face of it, this looks like "out of memory". Lower your batchsize/use a lighter model.
My word is final
- y2k_netizen
- Posts: 4
- Joined: Wed Nov 10, 2021 3:51 am
- Has thanked: 10 times
Re: [Resource] Training Using Google Colab
XxXArTiuSXxX wrote: ↑Fri Oct 01, 2021 11:51 amThank you all for the amazing work.
I am catching an error in the final training phase:
Code: Select all
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) 10/01/2021 17:17:38 CRITICAL Error caught! Exiting... 10/01/2021 17:17:38 ERROR Caught exception in thread: '_training_0' 10/01/2021 17:17:39 ERROR Got Exception on main handler: Traceback (most recent call last): File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script process.process() File "/content/faceswap/scripts/train.py", line 190, in process self._end_thread(thread, err) File "/content/faceswap/scripts/train.py", line 230, in _end_thread thread.join() File "/content/faceswap/lib/multithreading.py", line 121, in join raise thread.err[1].with_traceback(thread.err[2]) File "/content/faceswap/lib/multithreading.py", line 37, in run self._target(*self._args, **self._kwargs) File "/content/faceswap/scripts/train.py", line 252, in _training raise err File "/content/faceswap/scripts/train.py", line 242, in _training self._run_training_cycle(model, trainer) File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle trainer.train_one_step(viewer, timelapse) File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step loss = self._model.model.train_on_batch(model_inputs, y=model_targets) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1854, in train_on_batch logs = self.train_function(iterator) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 885, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _call return self._stateless_fn(*args, **kwds) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 3040, in __call__ filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1964, in _call_flat ctx, args, cancellation_manager=cancellation_manager)) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 596, in call ctx=ctx) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]] [[gradient_tape/LossWrapper_1/DSSIMObjective_2/Reshape_4/_34]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]] 0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_14821] Function call stack: train_function -> train_function
Any help is greatly appreciated, thanks again.
I was getting same error with Training when started using Google Colab in Oct 2021. I managed to resolve it by replacing content of "Install Tensorflow" cell
From:
#@title Install Tensorflow
!pip install -r faceswap/requirements_nvidia.txt
To:
#@title Install Tensorflow
!pip install -r faceswap/_requirements_base.txt
It was working fine for a month. However since yesterday started getting below error,
11/10/2021 09:05:26 ERROR The maximum supported Tensorflow is version 2.6 but you have version 2.7 installed. Please downgrade Tensorflow.
Any help will be appreciated.
- SteveBuscemi
- Posts: 2
- Joined: Mon Nov 29, 2021 1:16 am
- Been thanked: 1 time
Re: [Resource] Training Using Google Colab
y2k_netizen wrote: ↑Wed Nov 10, 2021 3:59 am11/10/2021 09:05:26 ERROR The maximum supported Tensorflow is version 2.6 but you have version 2.7 installed. Please downgrade Tensorflow.
I changed the install requirements cell to
Code: Select all
!pip install -r faceswap/_requirements_base.txt
!pip install tensorflow==2.6
to downgrade tensorflow, but now I'm hitting this error when I try to train:
Code: Select all
!python3 faceswap/faceswap.py train -A face_a/ -B face_b/ -m '/content/drive/My Drive/colab_files/faceswap/models/dtt/dttmodel' -t dfl-h128 -bs 18 -it 1000000 -s 250 -ss 25000 -ps 200 -wl
Setting Faceswap backend to NVIDIA
11/28/2021 17:14:20 INFO Log level set to: INFO
11/28/2021 17:14:25 ERROR Got Exception on main handler:
Traceback (most recent call last):
File "/content/faceswap/lib/cli/launcher.py", line 180, in execute_script
script = self._import_script()
File "/content/faceswap/lib/cli/launcher.py", line 46, in _import_script
module = import_module(mod)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/content/faceswap/scripts/train.py", line 13, in <module>
from lib.image import read_image_meta
File "/content/faceswap/lib/image.py", line 35, in <module>
class FfmpegReader(imageio.plugins.ffmpeg.FfmpegFormat.Reader):
AttributeError: module 'imageio.plugins' has no attribute 'ffmpeg'
11/28/2021 17:14:25 CRITICAL An unexpected crash has occurred. Crash report written to '/content/faceswap/crash_report.2021.11.28.171423693772.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
- y2k_netizen
- Posts: 4
- Joined: Wed Nov 10, 2021 3:51 am
- Has thanked: 10 times
Re: [Resource] Training Using Google Colab
SteveBuscemi wrote: ↑Mon Nov 29, 2021 1:42 amy2k_netizen wrote: ↑Wed Nov 10, 2021 3:59 am11/10/2021 09:05:26 ERROR The maximum supported Tensorflow is version 2.6 but you have version 2.7 installed. Please downgrade Tensorflow.
I changed the install requirements cell to
Code: Select all
!pip install -r faceswap/_requirements_base.txt !pip install tensorflow==2.6
to downgrade tensorflow, but now I'm hitting this error when I try to train:
Code: Select all
!python3 faceswap/faceswap.py train -A face_a/ -B face_b/ -m '/content/drive/My Drive/colab_files/faceswap/models/dtt/dttmodel' -t dfl-h128 -bs 18 -it 1000000 -s 250 -ss 25000 -ps 200 -wl Setting Faceswap backend to NVIDIA 11/28/2021 17:14:20 INFO Log level set to: INFO 11/28/2021 17:14:25 ERROR Got Exception on main handler: Traceback (most recent call last): File "/content/faceswap/lib/cli/launcher.py", line 180, in execute_script script = self._import_script() File "/content/faceswap/lib/cli/launcher.py", line 46, in _import_script module = import_module(mod) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/content/faceswap/scripts/train.py", line 13, in <module> from lib.image import read_image_meta File "/content/faceswap/lib/image.py", line 35, in <module> class FfmpegReader(imageio.plugins.ffmpeg.FfmpegFormat.Reader): AttributeError: module 'imageio.plugins' has no attribute 'ffmpeg' 11/28/2021 17:14:25 CRITICAL An unexpected crash has occurred. Crash report written to '/content/faceswap/crash_report.2021.11.28.171423693772.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Running on Colab is getting difficult day by day as Google keeps on changing many things. Above error is due to latest imageio plugin which appears to be not yet supported by Faceswap. Very crude way of getting it worked again on Colab is as follows,
- Use !pip install -r faceswap/_requirements_base.txt instead of !pip install -r faceswap/requirements_nvidia.txt
- Dowgrade imageio to 2.10.5 by running !pip install 'imageio==2.10.5'
- On left hand side "Files" icon of colab notebook navigate to faceswap > lib > cli folder and download file launcher.py
- Search and modify below entry from 2.6 to 2.7
Code: Select all
FaceswapError If Tensorflow is not found, or is not between versions 2.2 and 2.6 """ min_ver = 2.2 [b]max_ver = 2.7[/b]
- Replace this modified file in faceswap > lib > cli folder of colab notebook
Colab doesnt like downgrading pre-installed 2.7 Tensorflow but appears to be backward compatible with 2.6. So Tensorflow shouldnt be downgraded.
Hope no more functionality breaks in near future
- SteveBuscemi
- Posts: 2
- Joined: Mon Nov 29, 2021 1:16 am
- Been thanked: 1 time
Re: [Resource] Training Using Google Colab
Thanks. Also, you can simplify editing launcher.py with the console, or by adding this to a cell after installing faceswap:
Code: Select all
!sed -i 's/2.6/2.7/g' /content/faceswap/lib/cli/launcher.py
Re: [Resource] Training Using Google Colab
Hey guys I got everything to run but the last part also
First time configuration. Please select the required backend
1: AMD, 2: CPU, 3: NVIDIA: 3
Faceswap config written to: /content/faceswap/config/.faceswap
Setting Faceswap backend to NVIDIA
usage: faceswap.py [-h] {extract,train,convert,gui} ...
positional arguments:
{extract,train,convert,gui}
extract Extract the faces from pictures or a video
train Train a model for the two faces A and B
convert Convert source pictures or video to a new one with the
face swapped
gui Launch the Faceswap Graphical User Interface
optional arguments:
-h, --help show this help message and exit
faceswap.py: error: unrecognized arguments: -ala face_a/alignments.fsa -alb face_b/alignments.fsa 1
Re: [Resource] Training Using Google Colab
y2k_netizen wrote: ↑Wed Dec 01, 2021 1:41 pmRunning on Colab is getting difficult day by day as Google keeps on changing many things. Above error is due to latest imageio plugin which appears to be not yet supported by Faceswap. Very crude way of getting it worked again on Colab is as follows,
- Use !pip install -r faceswap/_requirements_base.txt instead of !pip install -r faceswap/requirements_nvidia.txt
- Dowgrade imageio to 2.10.5 by running !pip install 'imageio==2.10.5'
- On left hand side "Files" icon of colab notebook navigate to faceswap > lib > cli folder and download file launcher.py
- Search and modify below entry from 2.6 to 2.7
Code: Select all
FaceswapError If Tensorflow is not found, or is not between versions 2.2 and 2.6 """ min_ver = 2.2 [b]max_ver = 2.7[/b]
- Replace this modified file in faceswap > lib > cli folder of colab notebook
Colab doesnt like downgrading pre-installed 2.7 Tensorflow but appears to be backward compatible with 2.6. So Tensorflow shouldnt be downgraded.
Hope no more functionality breaks in near future
Well, it happened again. I'm now getting this error message:
Code: Select all
Setting Faceswap backend to NVIDIA
Traceback (most recent call last):
File "faceswap/faceswap.py", line 6, in <module>
from lib.cli import args as cli_args
File "/content/faceswap/lib/cli/args.py", line 22, in <module>
_GPUS = GPUStats().cli_devices
File "/content/faceswap/lib/gpu_stats.py", line 78, in __init__
self._driver = self._get_driver()
File "/content/faceswap/lib/gpu_stats.py", line 273, in _get_driver
driver = pynvml.nvmlSystemGetDriverVersion().decode("utf-8")
AttributeError: 'str' object has no attribute 'decode'
Being a total n00b, the only thing I could think of was uninstalling Tensorflow and CUDA and then reinstalling:
Code: Select all
!pip uninstall tensorflow -y
!apt-get --purge remove cuda nvidia* libnvidia-*
!dpkg -l | grep cuda- | awk '{print $2}' | xargs -n1 dpkg --purge
!apt-get remove cuda-*
!apt autoremove
!apt-get -y install cuda = 11.1.0
!apt-get update
!pip install tensorflow-gpu==2.6.0
!pip install tensorflow==2.6.0
And then , since it was asking for imageio, I simply did:
Code: Select all
!pip install imageio-ffmpeg
Aaand... of course it didn't work. Now the training starts, but it's obviously not using the GPU. So, I'm stuck here.
- y2k_netizen
- Posts: 4
- Joined: Wed Nov 10, 2021 3:51 am
- Has thanked: 10 times
Re: [Resource] Training Using Google Colab
Looks like developers have fixed this,
https://github.com/deepfakes/faceswap/c ... 469319ebd3
Only 2 changes required in original colab notebook posted at the start of this thread
- !pip install -r faceswap/_requirements_base.txt instead of !pip install -r faceswap/requirements_nvidia.txt
- Add and run cell >>> !sed -i 's/2.6/2.7/g' /content/faceswap/lib/cli/launcher.py
Thanks developers!!!
Re: [Resource] Training Using Google Colab
You're right, it works! I actually had done some messy modifications along the way to get it to work (without knowing what I was doing), and they had been working. But as of today, the best way to go is just use the original notebook with the two modifications you mention.
I'm also throwing the !echo "{\"backend\": \"nvidia\"}" > faceswap/config/.faceswap thing in there, so that I'm not asked to run the training in GPU every time.
You're awesome, guys!
Re: [Resource] Training Using Google Colab
Oops... this just happened after days of using Colab happily and not messing with anything else:
Code: Select all
[12:28:15] [#220292] Loss A: 0.01102, Loss B: 0.0174001/27/2022 12:28:18 CRITICAL Error caught! Exiting...
01/27/2022 12:28:18 ERROR Caught exception in thread: '_training_0'
01/27/2022 12:28:20 ERROR Got Exception on main handler:
Traceback (most recent call last):
File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
process.process()
File "/content/faceswap/scripts/train.py", line 190, in process
self._end_thread(thread, err)
File "/content/faceswap/scripts/train.py", line 230, in _end_thread
thread.join()
File "/content/faceswap/lib/multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "/content/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/content/faceswap/scripts/train.py", line 252, in _training
raise err
File "/content/faceswap/scripts/train.py", line 242, in _training
self._run_training_cycle(model, trainer)
File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
logs = self.train_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
(0) FAILED_PRECONDITION: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected
Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2
Could not flush events file.
[[node batch_decoder_b_loss
(defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)
]]
[[GroupCrossDeviceControlEdges_0/NoOp/_91]]
(1) FAILED_PRECONDITION: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected
Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2
Could not flush events file.
[[node batch_decoder_b_loss
(defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)
]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14947]
Errors may have originated from an input operation.
Input Source operations connected to node batch_decoder_b_loss:
In[0] batch_decoder_b_loss/writer:
In[1] batch_decoder_b_loss/ReadVariableOp:
In[2] batch_decoder_b_loss/tag:
In[3] batch_decoder_b_loss/Identity:
Operation defined at: (most recent call last)
File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/content/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/content/faceswap/scripts/train.py", line 242, in _training
self._run_training_cycle(model, trainer)
File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
logs = self.train_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function
return step_function(self, iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function
write_scalar_summaries(outputs, step=model._train_counter) # pylint: disable=protected-access
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries
summary_ops_v2.scalar('batch_' + name, value, step=step)
Input Source operations connected to node batch_decoder_b_loss:
In[0] batch_decoder_b_loss/writer:
In[1] batch_decoder_b_loss/ReadVariableOp:
In[2] batch_decoder_b_loss/tag:
In[3] batch_decoder_b_loss/Identity:
Operation defined at: (most recent call last)
File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/content/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/content/faceswap/scripts/train.py", line 242, in _training
self._run_training_cycle(model, trainer)
File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
logs = self.train_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function
return step_function(self, iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function
write_scalar_summaries(outputs, step=model._train_counter) # pylint: disable=protected-access
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries
summary_ops_v2.scalar('batch_' + name, value, step=step)
Function call stack:
train_function -> cond_2_true_14925 -> train_function -> cond_2_true_14925
01/27/2022 12:28:20 CRITICAL An unexpected crash has occurred. Crash report written to '/content/faceswap/crash_report.2022.01.27.122818466581.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting