Location of “forward pass” and/or Where to implement a Custom Training Loop

[FancyButConfusedTF]

Sorry if this is covered somewhere else, I have not been able to find anything that addresses this in the FAQs.

I’ve been digging through the code and looking for the actual forward pass of any of the model combinations and have not had any luck. Is there a generic config or something that I’ve missed that actually defines the flow of model inputs and outputs? i.e. spells out face_image2 = self.decoder(self.bottleneck(sec.encoder(face_image1)))?

I was hoping to drop my own custom architectures in to this (and, if successful, share them) but I can’t seem to find a forward pass in a syntax I am familiar with. I usually work with lower level tensorflow custom loops/gradient_tape oriented trainings, and these higher level keras abstractions (i.e. model.train_on_batch(..)) have me scratching my head.

[bryanlyon]

You don't have to hook into the passes at all to add your own architecture. You can add new models by following the plugin architecture here: https://github.com/deepfakes/faceswap/t ... rain/model or add a new training paradigm following the plugins here: https://github.com/deepfakes/faceswap/b ... n/trainer/

[torzdf]

I would add to this...

We would very much welcome new models. We are not too fussy as long as it ultimately works

I would add the following for creating models:

• I would recommend looking at the Original (and maybe IAE) model layouts, as they will be the easiest to understand how the models get defined and hang together. The Original model is heavily commented to help you see what is required (if you want to see a more insanely complex version, then look at the Phaze-A model.... then run away). The Original Model is also documented here: https://faceswap.readthedocs.io/en/late ... uild_model
• Also look at the 'defaults.py` files, as these show how you can define user configuration options for a model.
• Whilst the training loop is structured as a plugin (as per @bryanlyon's link), there is only 1 training loop at present, so that will be harder to follow, and would require some work to properly split it out to separate plugins. If you are looking to develop a new training loop, then you would be best served communicating with me, so that we can work on the best way forward to split this out.

[FancyButConfusedTF]

Thanks for the prompt reply. I think I need to do some more reading first to make sure I’m not missing anything .

There seems to be an implicit flow that I’m not quite seeing just yet. Not that there’s any actual faceswap reason to do something like this, but from a coding perspective, it doesn’t seem like it would be easy to construct a flow that consists of, say, an encoder with 5 output tensors, 2 of which plug into the bottleneck->decoder, 2 of which connect to a secondary decoder (that we have for some reason in this example) , and the final is returned to be used in a custom loss.

I’ll do some more reading and get back to you, this is a fantastic repository with so many loss function implementations and features, just lots of moving parts in here. thanks again

[torzdf]

Thanks for the prompt reply. I think I need to do some more reading first to make sure I’m not missing anything .

There seems to be an implicit flow that I’m not quite seeing just yet. Not that there’s any actual faceswap reason to do something like this, but from a coding perspective, it doesn’t seem like it would be easy to construct a flow that consists of, say, an encoder with 5 output tensors, 2 of which plug into the bottleneck->decoder, 2 of which connect to a secondary decoder (that we have for some reason in this example) , and the final is returned to be used in a custom loss.

I’ll do some more reading and get back to you, thanks again

Your example should all work with the current training loop, with the exception of the custom loss.

The current training loop expects 2 inputs (into the encoder(s)) and 2 outputs (from the decoder(s)). What you do, and how you route the flow between these is entirely up to you.

To be able to have custom loss on additional outputs, we would need to look at creating a new training plugin for that.

The current execution of the training loop is done here:

https://github.com/deepfakes/faceswap/b ... se.py#L206

[FancyButConfusedTF]

Ah I chose a foolish example but i think I got an answer to what I was meaning to ask (the spirit of the question is for a single forward pass from Identity_1->identity_2: change 5 to any random integer >5, add 4 more decoders for some reason that are only used in training, add 2 discriminators from stylegan etc etc. Using these random additional outputs in the loss will be difficult as it is written).

Although, I’m realizing now that one could override Kmodel.train_on_batch to do all sorts of crazy stuff, without breaking the structure of the rest of the code so long as the inputs/outputs match up with what is currently passed in at line 206 (is this what you mean by training plug-in?). An example of this type of thing is shown here with the ConditionalGAN class https://keras.io/examples/generative/conditional_gan/ (granted they override/implement model.train_step(..), but the idea is the same).

I will give this a shot tomorrow and let you know if I have any success.

Thank you again

[torzdf]

Please do.

The training 'plugin' is not really a plugin per-se.... It is just structured in a way that I can split out for new plugins etc. in future. None of the work has actually been done, because I wouldn't like to guess at what would/would not be useful, but the architecture is built in a manner that this can be done. (This probably sounds a little vague, but in short, we will be able to do what you want, but it may take a bit of work). This was basically done with the view of adding GAN code, but my experiments at the time were unsatisfactory.

If you do dig into this some more, feel free to join us on our Discord server where it will probably be easier to get quicker feedback/guidance.