[Resource] Training Using Google Colab

Want to use Faceswap in The Cloud? This is not directly supported by the Devs, but you may find community support here


Forum rules

Read the FAQs and search the forum before posting a new topic.

NB: The Devs do not directly support using Cloud based services, but you can find community support here.

Please mark any answers that fixed your problems so others can find the solutions.

User avatar
Korben
Posts: 21
Joined: Wed Aug 19, 2020 3:17 pm
Has thanked: 2 times
Been thanked: 3 times

Re: [Resource] Training Using Google Colab

Post by Korben »

Seems Collab now runs python 3.7. I can now install and run version 2 of faceswap.


Tags:

User avatar
torzdf
Posts: 1557
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 55 times
Been thanked: 293 times

Re: [Resource] Training Using Google Colab

Post by torzdf »

awelmisin wrote: Wed May 26, 2021 11:33 pm

another question of mine is, does it autosave? or do i need to place something next " -ss '{save_model_every}' \" code?

At default model autosaves every 250 iterations. I don't know the specifics of this Notebook though as I don't use it.

My word is final


User avatar
torzdf
Posts: 1557
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 55 times
Been thanked: 293 times

Re: [Resource] Training Using Google Colab

Post by torzdf »

Korben wrote: Thu May 27, 2021 1:57 am

Seems Collab now runs python 3.7. I can now install and run version 2 of faceswap.

This is excellent news :)

My word is final


User avatar
awelmisin
Posts: 3
Joined: Wed May 26, 2021 1:32 pm

Re: [Resource] Training Using Google Colab

Post by awelmisin »

torzdf wrote: Thu May 27, 2021 10:39 am
awelmisin wrote: Wed May 26, 2021 11:33 pm

another question of mine is, does it autosave? or do i need to place something next " -ss '{save_model_every}' \" code?

At default model autosaves every 250 iterations. I don't know the specifics of this Notebook though as I don't use it.

I see, it actually makes 360 iteration every 10 minutes. So for my issue, it saves every 360 iteration. Thanks anyway!

And by the way, I can share "the latest" perfectly working, Google Colab.


User avatar
foundmyway89
Posts: 2
Joined: Wed May 26, 2021 8:16 pm

Re: [Resource] Training Using Google Colab

Post by foundmyway89 »

awelmisin wrote: Wed May 26, 2021 11:33 pm

And by the way, I can share "the latest" perfectly working, Google Colab.

Please do! I've been trying to get mine working for a week now...

I checked your github and it didn't look like there was a link yet.

Thanks!


User avatar
sp13
Posts: 14
Joined: Sat Apr 10, 2021 12:20 am
Has thanked: 3 times
Been thanked: 4 times

Re: [Resource] Training Using Google Colab

Post by sp13 »

foundmyway89 wrote: Thu Jun 03, 2021 12:32 am

Please do! I've been trying to get mine working for a week now...

The notebook from this post should work fine viewtopic.php?p=5136#p5136


User avatar
nightking1102
Posts: 2
Joined: Sat Jul 10, 2021 2:27 pm

Re: [Resource] Training Using Google Colab

Post by nightking1102 »

awelmisin wrote: Wed May 26, 2021 11:33 pm

https://github.com/awelmisin/faceswap-google-colab

can find the latest working version here, i am also taking a youtube video about this.

there is nothing here , I can not find the notebook


User avatar
nightking1102
Posts: 2
Joined: Sat Jul 10, 2021 2:27 pm

Re: [Resource] Training Using Google Colab

Post by nightking1102 »

I got the following error when run training:

Questions and feedback: https://faceswap.dev/forum
faceswap.py train: error: the following arguments are required: -A/--input-A, -B/--input-B


User avatar
XxXArTiuSXxX
Posts: 1
Joined: Fri Oct 01, 2021 11:45 am

Re: [Resource] Training Using Google Colab

Post by XxXArTiuSXxX »

Thank you all for the amazing work.

I am catching an error in the final training phase:

Code: Select all

/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  category=CustomMaskWarning)
10/01/2021 17:17:38 CRITICAL Error caught! Exiting...
10/01/2021 17:17:38 ERROR    Caught exception in thread: '_training_0'
10/01/2021 17:17:39 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
    process.process()
  File "/content/faceswap/scripts/train.py", line 190, in process
    self._end_thread(thread, err)
  File "/content/faceswap/scripts/train.py", line 230, in _end_thread
    thread.join()
  File "/content/faceswap/lib/multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/content/faceswap/scripts/train.py", line 252, in _training
    raise err
  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1854, in train_on_batch
    logs = self.train_function(iterator)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 3040, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1964, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
	 [[gradient_tape/LossWrapper_1/DSSIMObjective_2/Reshape_4/_34]]
  (1) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14821]

Function call stack:
train_function -> train_function

Any help is greatly appreciated, thanks again.


User avatar
torzdf
Posts: 1557
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 55 times
Been thanked: 293 times

Re: [Resource] Training Using Google Colab

Post by torzdf »

You should post the full crash report, but on the face of it, this looks like "out of memory". Lower your batchsize/use a lighter model.

My word is final


User avatar
y2k_netizen
Posts: 1
Joined: Wed Nov 10, 2021 3:51 am

Re: [Resource] Training Using Google Colab

Post by y2k_netizen »

XxXArTiuSXxX wrote: Fri Oct 01, 2021 11:51 am

Thank you all for the amazing work.

I am catching an error in the final training phase:

Code: Select all

/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  category=CustomMaskWarning)
10/01/2021 17:17:38 CRITICAL Error caught! Exiting...
10/01/2021 17:17:38 ERROR    Caught exception in thread: '_training_0'
10/01/2021 17:17:39 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
    process.process()
  File "/content/faceswap/scripts/train.py", line 190, in process
    self._end_thread(thread, err)
  File "/content/faceswap/scripts/train.py", line 230, in _end_thread
    thread.join()
  File "/content/faceswap/lib/multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/content/faceswap/scripts/train.py", line 252, in _training
    raise err
  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/content/faceswap/plugins/train/trainer/_base.py", line 193, in train_one_step
    loss = self._model.model.train_on_batch(model_inputs, y=model_targets)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1854, in train_on_batch
    logs = self.train_function(iterator)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 3040, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1964, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
	 [[gradient_tape/LossWrapper_1/DSSIMObjective_2/Reshape_4/_34]]
  (1) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node villain/encoder/conv_128_0_conv2d/Conv2D_1 (defined at content/faceswap/plugins/train/trainer/_base.py:193) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_14821]

Function call stack:
train_function -> train_function

Any help is greatly appreciated, thanks again.

I was getting same error with Training when started using Google Colab in Oct 2021. I managed to resolve it by replacing content of "Install Tensorflow" cell

From:

#@title Install Tensorflow

!pip install -r faceswap/requirements_nvidia.txt

To:

#@title Install Tensorflow

!pip install -r faceswap/_requirements_base.txt

It was working fine for a month. However since yesterday started getting below error,

11/10/2021 09:05:26 ERROR The maximum supported Tensorflow is version 2.6 but you have version 2.7 installed. Please downgrade Tensorflow.

Any help will be appreciated.


Post Reply