Crash when training on colab but converting locally

Want to use Faceswap in The Cloud? This is not directly supported by the Devs, but you may find community support here


Forum rules

Read the FAQs and search the forum before posting a new topic.

NB: The Devs do not directly support using Cloud based services, but you can find community support here.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
UEW44
Posts: 1
Joined: Wed Dec 09, 2020 5:28 pm

Crash when training on colab but converting locally

Post by UEW44 »

Hey all,
Not really a question as much as an FYI in case anyone out there runs into this problem when training on Google Colab.
I'd been training a model for a few days on Colab and downloaded to convert on a local machine when I got the following crash:

Code: Select all

...
01/25/2021 23:20:21 MainProcess     MainThread                     _base           __init__                       DEBUG    Initialized ModelBase (Model)
01/25/2021 23:20:21 MainProcess     MainThread                     _base           strategy_scope                 DEBUG    Using strategy scope: <tensorflow.python.distribute.distribute_lib._DefaultDistributionContext object at 0x000001AC86C14610>
01/25/2021 23:20:21 MainProcess     MainThread                     _base           _load                          DEBUG    Loading model: A:\ML\colab\colab_models\Original_model\original.h5
Traceback (most recent call last):
  File "C:\Users\me\faceswap\lib\cli\launcher.py", line 181, in execute_script
    process = script(arguments)
  File "C:\Users\me\faceswap\tools\preview\preview.py", line 62, in __init__
    self._samples = Samples(arguments, 5, self._display, self._lock, trigger_patch)
  File "C:\Users\me\faceswap\tools\preview\preview.py", line 203, in __init__
    self._predictor = Predict(queue_manager.get_queue("preview_predict_in"),
  File "C:\Users\me\faceswap\scripts\convert.py", line 661, in __init__
    self._model = self._load_model()
  File "C:\Users\me\faceswap\scripts\convert.py", line 746, in _load_model
    model.build()
  File "C:\Users\me\faceswap\plugins\train\model\_base.py", line 258, in build
    model = self._io._load()  # pylint:disable=protected-access
  File "C:\Users\me\faceswap\plugins\train\model\_base.py", line 519, in _load
    model = load_model(self._filename, compile=False)
  File "C:\Users\me\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\saving\save.py", line 184, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
  File "C:\Users\me\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py", line 177, in load_model_from_hdf5
    model = model_config_lib.model_from_config(model_config,
  File "C:\Users\me\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\saving\model_config.py", line 55, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "C:\Users\me\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 105, in deserialize
    return deserialize_keras_object(
  File "C:\Users\me\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 361, in deserialize_keras_object
    (cls, cls_config) = class_and_config_for_serialized_keras_object(
  File "C:\Users\me\MiniConda3\envs\faceswap\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 321, in class_and_config_for_serialized_keras_object
    raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
ValueError: Unknown layer: Functional

Something about an unknown layer. I got an unknown layer: Functional, but this could be different for others depending on what versions your have in the next part.

Anyway, the root of the issue is that colab was using a different tensorflow-gpu version than my local install. Colab was using tensorflow-gpu 2.4.0, whereas the locally installed version used 2.2.1.
Not entirely sure why these are different tbh, I used the latest faceswap in both with the default installation(maybe the linux and windows conda install behaves differently?), but they were.

The solution is fairly simple, just upgrade one or downgrade the other, but make sure they match. I was able to overwrite my local conda env that faceswap uses, and it works just fine now.

Hope this saves someone in the future some headache :)

Locked