[Outdated] Training Using Google Colab

Want to use Faceswap in The Cloud? This is not directly supported by the Devs, but you may find community support here


Forum rules

Read the FAQs and search the forum before posting a new topic.

NB: The Devs do not directly support using Cloud based services, but you can find community support here.

Please mark any answers that fixed your problems so others can find the solutions.

User avatar
Korben
Posts: 26
Joined: Wed Aug 19, 2020 3:17 pm
Has thanked: 2 times
Been thanked: 3 times

Re: [Resource] Training Using Google Colab

Post by Korben »

Seems Collab now runs python 3.7. I can now install and run version 2 of faceswap.


Tags:
User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: [Resource] Training Using Google Colab

Post by torzdf »

awelmisin wrote: Wed May 26, 2021 11:33 pm

another question of mine is, does it autosave? or do i need to place something next " -ss '{save_model_every}' \" code?

At default model autosaves every 250 iterations. I don't know the specifics of this Notebook though as I don't use it.

My word is final

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: [Resource] Training Using Google Colab

Post by torzdf »

Korben wrote: Thu May 27, 2021 1:57 am

Seems Collab now runs python 3.7. I can now install and run version 2 of faceswap.

This is excellent news :)

My word is final

User avatar
awelmisin
Posts: 3
Joined: Wed May 26, 2021 1:32 pm

Re: [Resource] Training Using Google Colab

Post by awelmisin »

torzdf wrote: Thu May 27, 2021 10:39 am
awelmisin wrote: Wed May 26, 2021 11:33 pm

another question of mine is, does it autosave? or do i need to place something next " -ss '{save_model_every}' \" code?

At default model autosaves every 250 iterations. I don't know the specifics of this Notebook though as I don't use it.

I see, it actually makes 360 iteration every 10 minutes. So for my issue, it saves every 360 iteration. Thanks anyway!

And by the way, I can share "the latest" perfectly working, Google Colab.

User avatar
foundmyway89
Posts: 2
Joined: Wed May 26, 2021 8:16 pm

Re: [Resource] Training Using Google Colab

Post by foundmyway89 »

awelmisin wrote: Wed May 26, 2021 11:33 pm

And by the way, I can share "the latest" perfectly working, Google Colab.

Please do! I've been trying to get mine working for a week now...

I checked your github and it didn't look like there was a link yet.

Thanks!

User avatar
sp13
Posts: 15
Joined: Sat Apr 10, 2021 12:20 am
Has thanked: 3 times
Been thanked: 4 times

Re: [Resource] Training Using Google Colab

Post by sp13 »

foundmyway89 wrote: Thu Jun 03, 2021 12:32 am

Please do! I've been trying to get mine working for a week now...

The notebook from this post should work fine viewtopic.php?p=5136#p5136

User avatar
nightking1102
Posts: 2
Joined: Sat Jul 10, 2021 2:27 pm

Re: [Resource] Training Using Google Colab

Post by nightking1102 »

awelmisin wrote: Wed May 26, 2021 11:33 pm

https://github.com/awelmisin/faceswap-google-colab

can find the latest working version here, i am also taking a youtube video about this.

there is nothing here , I can not find the notebook

User avatar
nightking1102
Posts: 2
Joined: Sat Jul 10, 2021 2:27 pm

Re: [Resource] Training Using Google Colab

Post by nightking1102 »

I got the following error when run training:

Questions and feedback: https://faceswap.dev/forum
faceswap.py train: error: the following arguments are required: -A/--input-A, -B/--input-B

User avatar
XxXArTiuSXxX
Posts: 1
Joined: Fri Oct 01, 2021 11:45 am

Re: [Resource] Training Using Google Colab

Post by XxXArTiuSXxX »

Thank you all for the amazing work.

I am catching an error in the final training phase:

Code: Select all

/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  category=CustomMaskWarning)
10/01/2021 17:17:38 CRITICAL Error caught! Exiting...
10/01/2021 17:17:38 ERROR    Caught exception in thread: '_training_0'
10/01/2021 17:17:39 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
    process.process()
  File "/content/faceswap/scripts/train.py", line 190, in process
    self._end_thread(thread, err)
  File "/content/faceswap/scripts/train.py", line 230, in _end_thread
    thread.join()
  File "/content/faceswap/lib/multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/content/faceswap/scripts/train.py", line 252, in _training
    raise err
  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1854, in train_on_batch
    logs = self.train_function(iterator)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 3040, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1964, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
	 [[gradient_tape/LossWrapper_1/DSSIMObjective_2/Reshape_4/_34]]
  (1) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14821]

Function call stack:
train_function -> train_function

Any help is greatly appreciated, thanks again.

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: [Resource] Training Using Google Colab

Post by torzdf »

You should post the full crash report, but on the face of it, this looks like "out of memory". Lower your batchsize/use a lighter model.

My word is final

User avatar
y2k_netizen
Posts: 4
Joined: Wed Nov 10, 2021 3:51 am
Has thanked: 10 times

Re: [Resource] Training Using Google Colab

Post by y2k_netizen »

XxXArTiuSXxX wrote: Fri Oct 01, 2021 11:51 am

Thank you all for the amazing work.

I am catching an error in the final training phase:

Code: Select all

/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  category=CustomMaskWarning)
10/01/2021 17:17:38 CRITICAL Error caught! Exiting...
10/01/2021 17:17:38 ERROR    Caught exception in thread: '_training_0'
10/01/2021 17:17:39 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
    process.process()
  File "/content/faceswap/scripts/train.py", line 190, in process
    self._end_thread(thread, err)
  File "/content/faceswap/scripts/train.py", line 230, in _end_thread
    thread.join()
  File "/content/faceswap/lib/multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/content/faceswap/scripts/train.py", line 252, in _training
    raise err
  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1854, in train_on_batch
    logs = self.train_function(iterator)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 3040, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1964, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
	 [[gradient_tape/LossWrapper_1/DSSIMObjective_2/Reshape_4/_34]]
  (1) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14821]

Function call stack:
train_function -> train_function

Any help is greatly appreciated, thanks again.

I was getting same error with Training when started using Google Colab in Oct 2021. I managed to resolve it by replacing content of "Install Tensorflow" cell

From:

#@title Install Tensorflow

!pip install -r faceswap/requirements_nvidia.txt

To:

#@title Install Tensorflow

!pip install -r faceswap/_requirements_base.txt

It was working fine for a month. However since yesterday started getting below error,

11/10/2021 09:05:26 ERROR The maximum supported Tensorflow is version 2.6 but you have version 2.7 installed. Please downgrade Tensorflow.

Any help will be appreciated.

User avatar
SteveBuscemi
Posts: 2
Joined: Mon Nov 29, 2021 1:16 am
Been thanked: 1 time

Re: [Resource] Training Using Google Colab

Post by SteveBuscemi »

y2k_netizen wrote: Wed Nov 10, 2021 3:59 am

11/10/2021 09:05:26 ERROR The maximum supported Tensorflow is version 2.6 but you have version 2.7 installed. Please downgrade Tensorflow.

I changed the install requirements cell to

Code: Select all

!pip install -r faceswap/_requirements_base.txt
!pip install tensorflow==2.6

to downgrade tensorflow, but now I'm hitting this error when I try to train:

Code: Select all

!python3 faceswap/faceswap.py train -A face_a/ -B face_b/ -m '/content/drive/My Drive/colab_files/faceswap/models/dtt/dttmodel' -t dfl-h128 -bs 18 -it 1000000 -s 250 -ss 25000 -ps 200 -wl

Setting Faceswap backend to NVIDIA
11/28/2021 17:14:20 INFO     Log level set to: INFO
11/28/2021 17:14:25 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 180, in execute_script
    script = self._import_script()
  File "/content/faceswap/lib/cli/launcher.py", line 46, in _import_script
    module = import_module(mod)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/content/faceswap/scripts/train.py", line 13, in <module>
    from lib.image import read_image_meta
  File "/content/faceswap/lib/image.py", line 35, in <module>
    class FfmpegReader(imageio.plugins.ffmpeg.FfmpegFormat.Reader):
AttributeError: module 'imageio.plugins' has no attribute 'ffmpeg'
11/28/2021 17:14:25 CRITICAL An unexpected crash has occurred. Crash report written to '/content/faceswap/crash_report.2021.11.28.171423693772.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

User avatar
y2k_netizen
Posts: 4
Joined: Wed Nov 10, 2021 3:51 am
Has thanked: 10 times

Re: [Resource] Training Using Google Colab

Post by y2k_netizen »

SteveBuscemi wrote: Mon Nov 29, 2021 1:42 am
y2k_netizen wrote: Wed Nov 10, 2021 3:59 am

11/10/2021 09:05:26 ERROR The maximum supported Tensorflow is version 2.6 but you have version 2.7 installed. Please downgrade Tensorflow.

I changed the install requirements cell to

Code: Select all

!pip install -r faceswap/_requirements_base.txt
!pip install tensorflow==2.6

to downgrade tensorflow, but now I'm hitting this error when I try to train:

Code: Select all

!python3 faceswap/faceswap.py train -A face_a/ -B face_b/ -m '/content/drive/My Drive/colab_files/faceswap/models/dtt/dttmodel' -t dfl-h128 -bs 18 -it 1000000 -s 250 -ss 25000 -ps 200 -wl

Setting Faceswap backend to NVIDIA
11/28/2021 17:14:20 INFO     Log level set to: INFO
11/28/2021 17:14:25 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 180, in execute_script
    script = self._import_script()
  File "/content/faceswap/lib/cli/launcher.py", line 46, in _import_script
    module = import_module(mod)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/content/faceswap/scripts/train.py", line 13, in <module>
    from lib.image import read_image_meta
  File "/content/faceswap/lib/image.py", line 35, in <module>
    class FfmpegReader(imageio.plugins.ffmpeg.FfmpegFormat.Reader):
AttributeError: module 'imageio.plugins' has no attribute 'ffmpeg'
11/28/2021 17:14:25 CRITICAL An unexpected crash has occurred. Crash report written to '/content/faceswap/crash_report.2021.11.28.171423693772.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

Running on Colab is getting difficult day by day as Google keeps on changing many things. Above error is due to latest imageio plugin which appears to be not yet supported by Faceswap. Very crude way of getting it worked again on Colab is as follows,

  • Use !pip install -r faceswap/_requirements_base.txt instead of !pip install -r faceswap/requirements_nvidia.txt
  • Dowgrade imageio to 2.10.5 by running !pip install 'imageio==2.10.5'
  • On left hand side "Files" icon of colab notebook navigate to faceswap > lib > cli folder and download file launcher.py
  • Search and modify below entry from 2.6 to 2.7

    Code: Select all

    FaceswapError
        If Tensorflow is not found, or is not between versions 2.2 and 2.6
    """
    min_ver = 2.2
    [b]max_ver = 2.7[/b]
  • Replace this modified file in faceswap > lib > cli folder of colab notebook

Colab doesnt like downgrading pre-installed 2.7 Tensorflow but appears to be backward compatible with 2.6. So Tensorflow shouldnt be downgraded.

Hope no more functionality breaks in near future :x :|

User avatar
SteveBuscemi
Posts: 2
Joined: Mon Nov 29, 2021 1:16 am
Been thanked: 1 time

Re: [Resource] Training Using Google Colab

Post by SteveBuscemi »

Thanks. Also, you can simplify editing launcher.py with the console, or by adding this to a cell after installing faceswap:

Code: Select all

!sed -i 's/2.6/2.7/g' /content/faceswap/lib/cli/launcher.py
User avatar
wearaface
Posts: 1
Joined: Sat May 15, 2021 9:33 pm

Re: [Resource] Training Using Google Colab

Post by wearaface »

Hey guys I got everything to run but the last part also

First time configuration. Please select the required backend
1: AMD, 2: CPU, 3: NVIDIA: 3
Faceswap config written to: /content/faceswap/config/.faceswap
Setting Faceswap backend to NVIDIA
usage: faceswap.py [-h] {extract,train,convert,gui} ...

positional arguments:
{extract,train,convert,gui}
extract Extract the faces from pictures or a video
train Train a model for the two faces A and B
convert Convert source pictures or video to a new one with the
face swapped
gui Launch the Faceswap Graphical User Interface

optional arguments:
-h, --help show this help message and exit
faceswap.py: error: unrecognized arguments: -ala face_a/alignments.fsa -alb face_b/alignments.fsa 1

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 155
Has thanked: 128 times
Been thanked: 614 times

Re: [Resource] Training Using Google Colab

Post by torzdf »

wearaface wrote: Thu Dec 09, 2021 10:44 pm

faceswap.py: error: unrecognized arguments: -ala face_a/alignments.fsa -alb face_b/alignments.fsa 1

search.php?keywords=unrecognized+arguments%3A+ala

My word is final

User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Re: [Resource] Training Using Google Colab

Post by aolvera27 »

y2k_netizen wrote: Wed Dec 01, 2021 1:41 pm

Running on Colab is getting difficult day by day as Google keeps on changing many things. Above error is due to latest imageio plugin which appears to be not yet supported by Faceswap. Very crude way of getting it worked again on Colab is as follows,

  • Use !pip install -r faceswap/_requirements_base.txt instead of !pip install -r faceswap/requirements_nvidia.txt
  • Dowgrade imageio to 2.10.5 by running !pip install 'imageio==2.10.5'
  • On left hand side "Files" icon of colab notebook navigate to faceswap > lib > cli folder and download file launcher.py
  • Search and modify below entry from 2.6 to 2.7

    Code: Select all

    FaceswapError
        If Tensorflow is not found, or is not between versions 2.2 and 2.6
    """
    min_ver = 2.2
    [b]max_ver = 2.7[/b]
  • Replace this modified file in faceswap > lib > cli folder of colab notebook

Colab doesnt like downgrading pre-installed 2.7 Tensorflow but appears to be backward compatible with 2.6. So Tensorflow shouldnt be downgraded.

Hope no more functionality breaks in near future :x :|

Well, it happened again. I'm now getting this error message:

Code: Select all

Setting Faceswap backend to NVIDIA
Traceback (most recent call last):
  File "faceswap/faceswap.py", line 6, in <module>
    from lib.cli import args as cli_args
  File "/content/faceswap/lib/cli/args.py", line 22, in <module>
    _GPUS = GPUStats().cli_devices
  File "/content/faceswap/lib/gpu_stats.py", line 78, in __init__
    self._driver = self._get_driver()
  File "/content/faceswap/lib/gpu_stats.py", line 273, in _get_driver
    driver = pynvml.nvmlSystemGetDriverVersion().decode("utf-8")
AttributeError: 'str' object has no attribute 'decode'

Being a total n00b, the only thing I could think of was uninstalling Tensorflow and CUDA and then reinstalling:

Code: Select all

!pip uninstall tensorflow -y
!apt-get --purge remove cuda nvidia* libnvidia-*
!dpkg -l | grep cuda- | awk '{print $2}' | xargs -n1 dpkg --purge
!apt-get remove cuda-*
!apt autoremove
!apt-get -y install cuda = 11.1.0
!apt-get update
!pip install tensorflow-gpu==2.6.0
!pip install tensorflow==2.6.0

And then , since it was asking for imageio, I simply did:

Code: Select all

!pip install imageio-ffmpeg

Aaand... of course it didn't work. Now the training starts, but it's obviously not using the GPU. So, I'm stuck here.

User avatar
y2k_netizen
Posts: 4
Joined: Wed Nov 10, 2021 3:51 am
Has thanked: 10 times

Re: [Resource] Training Using Google Colab

Post by y2k_netizen »

Looks like developers have fixed this,

https://github.com/deepfakes/faceswap/c ... 469319ebd3

Only 2 changes required in original colab notebook posted at the start of this thread

  • !pip install -r faceswap/_requirements_base.txt instead of !pip install -r faceswap/requirements_nvidia.txt
  • Add and run cell >>> !sed -i 's/2.6/2.7/g' /content/faceswap/lib/cli/launcher.py

Thanks developers!!!

User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Re: [Resource] Training Using Google Colab

Post by aolvera27 »

You're right, it works! I actually had done some messy modifications along the way to get it to work (without knowing what I was doing), and they had been working. But as of today, the best way to go is just use the original notebook with the two modifications you mention.

I'm also throwing the !echo "{\"backend\": \"nvidia\"}" > faceswap/config/.faceswap thing in there, so that I'm not asked to run the training in GPU every time.

You're awesome, guys!

User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Re: [Resource] Training Using Google Colab

Post by aolvera27 »

Oops... this just happened after days of using Colab happily and not messing with anything else:

Code: Select all

[12:28:15] [#220292] Loss A: 0.01102, Loss B: 0.0174001/27/2022 12:28:18 CRITICAL Error caught! Exiting...
01/27/2022 12:28:18 ERROR    Caught exception in thread: '_training_0'
01/27/2022 12:28:20 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
    process.process()
  File "/content/faceswap/scripts/train.py", line 190, in process
    self._end_thread(thread, err)
  File "/content/faceswap/scripts/train.py", line 230, in _end_thread
    thread.join()
  File "/content/faceswap/lib/multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/content/faceswap/scripts/train.py", line 252, in _training
    raise err
  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
    logs = self.train_function(iterator)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
  (0) FAILED_PRECONDITION:  /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected
	Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2
	Could not flush events file.
	 [[node batch_decoder_b_loss
 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)
]]
	 [[GroupCrossDeviceControlEdges_0/NoOp/_91]]
  (1) FAILED_PRECONDITION:  /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2; Transport endpoint is not connected
	Failed to flush 11 events to /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_logs/session_61/train/events.out.tfevents.1643293867.66309e05e96b.401.0.v2
	Could not flush events file.
	 [[node batch_decoder_b_loss
 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:2912)
]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14947]

Errors may have originated from an input operation.
Input Source operations connected to node batch_decoder_b_loss:
In[0] batch_decoder_b_loss/writer:	
In[1] batch_decoder_b_loss/ReadVariableOp:	
In[2] batch_decoder_b_loss/tag:	
In[3] batch_decoder_b_loss/Identity:

Operation defined at: (most recent call last)
  File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()

  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()

  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)

  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)

  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)

  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
    logs = self.train_function(iterator)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function
    return step_function(self, iterator)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function
    write_scalar_summaries(outputs, step=model._train_counter)  # pylint: disable=protected-access

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries
    summary_ops_v2.scalar('batch_' + name, value, step=step)


Input Source operations connected to node batch_decoder_b_loss:
In[0] batch_decoder_b_loss/writer:	
In[1] batch_decoder_b_loss/ReadVariableOp:	
In[2] batch_decoder_b_loss/tag:	
In[3] batch_decoder_b_loss/Identity:

Operation defined at: (most recent call last)
  File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()

  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()

  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)

  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)

  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)

  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1848, in train_on_batch
    logs = self.train_function(iterator)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 859, in train_function
    return step_function(self, iterator)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 852, in step_function
    write_scalar_summaries(outputs, step=model._train_counter)  # pylint: disable=protected-access

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2912, in write_scalar_summaries
    summary_ops_v2.scalar('batch_' + name, value, step=step)


Function call stack:
train_function -> cond_2_true_14925 -> train_function -> cond_2_true_14925

01/27/2022 12:28:20 CRITICAL An unexpected crash has occurred. Crash report written to '/content/faceswap/crash_report.2022.01.27.122818466581.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting
Locked