[Guide] A Rough Guide to Converting

Post by **torzdf** » Tue Oct 20, 2020 12:28 am

NB: This guide was correct at time of writing, but things do change. I will try to keep it updated.

Last edited: 28 September 2023

Introduction

If you've landed on this page and you don't know how to extract a face set or train a model, then stop right now and go read the Extract Guide and then the Training Guide.

So, you've got this far? The hard part is over. We've got a trained model and we're ready to create a swap. Converting is by far the least involved part of the process, but there are still some settings and final tweaks we need to do to get a good result.

As in the other guides, I will be using the GUI, but all the commands here are also available in the cli, You can execute python faceswap.py convert -h to get a list of the commands, and python tools.py -h to get a list of the available tools.

Settings

Before heading to the convert tab, we need to do a little preparation work and lock in some settings which look good.

Alignments

For creating our final swap, we're going to need a source video (or folder of frames) and the trained model. There is one other thing that we need, and that is an alignments file for the source video. This is not the same alignments file that was used for training, more, it is an alignments file that is created specifically for the video we want to swap on to. The purpose of the alignments file here is to tell Faceswap which is the target face that we should be swapping and where it exists within the frame. If you have not already created an alignments file for your source video and cleaned it up, then you should do so now. Head over to the Extract Guide for instructions on how to create an alignments file for converting.

Preview Tool

Faceswap provides a Preview Tool where you can preview the results of the swap and adjust settings to better blend the face with the frame.

Head over to the Tools tab and then the Preview sub-tab:

prev_tab.png (13.5 KiB) Viewed 395289 times
There aren't many options to configure here, but they are explained below:

prev_opts.png (27.26 KiB) Viewed 395289 times
- Input Dir - Select the video icon for a source video, or the image icon for a source folder of frames and navigate to the source that you want to swap.
- Alignments File - If the alignments file is in the default location that you extracted from, then you can leave this blank, otherwise, click the icon and navigate to the .fsa alignment file that corresponds to the video you entered in the previous field.
- Model Dir - This should be the folder that you specified during the training phase. It will contain a <model_name>.h5 file and a <model_name>_state.json file.
- Swap Model - This is here in case you have trained your model the wrong way around (i.e. when you try to convert, but the subject's face remains unchanged). You can enable this option to switch the model direction, and make a mental note to train the correct way around next time.
Hit the Preview button to launch the preview tool.
The process will load the model, source video and alignments file and then display a window similar to below:

prev_overview.jpg (103.95 KiB) Viewed 395275 times

A basic overview of the layout is described below, but we will drill into the options in more detail next.
- Top row of images - the faces in the original frame
- Bottom row of images - the swapped face placed into the original frame
- Bottom left option panel - These relate to Command Line Options (that is options you set in the main GUI for convert). Anything you set here, you should take a note of, as you will need to replicate these options when you run the convert.
- Bottom right option panel - These relate to Configuration Options. These are the settings you access from the Settings > Edit Convert Settings... dialog in the GUI (or ./config/convert.ini in the cli). Any settings you change (and save) here will be remembered for the convert process.
- Update Samples - This button will refresh the images shown in the top frames, pulling in a new selection of faces from the source video file. Be patient when pressing this button. Depending on your setup this may take a few seconds, as the faces need to be passed through the model to generate new swaps.
- Utility Buttons - There are two sets of utility buttons. Those on the left hand side (under the Command Line Options) impact the full configuration (i.e. the options that correspond to every tab in the right hand panel). The buttons on the right hand side (under the Configuration Options) just impact the tab that you currently have selected. Either way, these buttons allow you to reset the options their saved values or their default values, and allows you to save the configuration.
Command Line Options
These options relate to settings that you will configure in the main Faceswap GUI when running your convert later. Anything you change here, you will want to take a note of, as you will need to use the same settings when running the convert task.

prev_cli_choices.jpg (22.17 KiB) Viewed 108588 times
- Color - This is the type of color matching to use between the original face and the swapped face. Ideally you would leave this at "None" but depending on the model used and the quality of data, then you may want to use one of the built in color matching methods. Some of the methods have settings which can be adjusted in the right hand panel (but I will cover that later). It is worth noting that there is no "right" option to select here. Different swaps will require different color matching techniques. The current methods are:
  - None - Don't perform any color matching.
  - Avg Color - Takes an average of the colors between the original face and the swap face and applies this to the swap face.
  - Color Transfer - A color matching algorithm developed by Reinhard et al., 2001. This method has a couple of options in the right hand pane.
  - Manual Balance - Allows you to manually alter the color/contrast/brightness of the swapped face. This method has settings available in the right hand pane.
  - Match Hist - Attempts to match the histogram on the swap face from the histogram in the source face. This method has settings available in the right hand pane.
  - Seamless Clone - This method is good for single images, but terrible for videos, No matter how good the result may look, if you are creating a video, do not use this method.
- Mask Type - This is the mask type you want to use for converting. The available masks and their descriptions are available in the Extract Guide. If you want to use a mask that is not available within the alignments file, then you can use the Mask Tool to generate additional masks. Masks have settings in the right hand pane to adjust how to blend the swapped face onto the original frame.
- Face Scale - Sometimes the swapped faces size needs to be adjusted a bit to better fit the original face. The swap face can be adjusted to be larger (dragged to the right) or smaller (drag to the left).
Configuration Options
These options, in the right hand panel, allow you to configure the settings for each individual plugin that will be run during the convert process. When you make changes here, the saved values will be updated in your convert.ini (accessible from Settings > Edit Convert Settings... in the GUI). This means that once you have got the settings locked in correctly here, they will automatically be picked up for the convert process.

prev_cnf.png (7.32 KiB) Viewed 395274 times

The plugins are broken down into 3 main categories, and each of these categories may contain settings for one or more plugins. I will provide an overview of the main categories and plugins available, but the actual settings you will need to use will vary on a case by case basis, so it's up to you to evaluate the results and choose the best settings. All of the settings have tooltips which explain what each option does.
- Color
  The settings shown here relate to the Color pull down list from the previous Command Line Options section. E.G, if you have selected "Color-Transfer" as your color matching method in the left-hand pane, then you would select the "Color Transfer" tab in the right-hand pane to configure that plugin's options.
  - Color Transfer - Exposes a couple of options to turn off/on for this plugin. Turning these on or off will have different effects depending on the source.
  - Manual Balance - Allows you to manually adjust the color/brightness/contrast for the final swap, if you have selected this plugin.
    
    prev_col1.png (9.12 KiB) Viewed 395274 times
    
    The Color Balance section allows you to choose a color space to work in, and then adjust the balance for that color space. The Brightness Contrast section allows you to adjust the brightness and contrast.
  - Match Hist - Only one parameter to edit for this plugin, which allows you to set the Threshold at which color matching takes place. This helps cut out "extreme" colors.
- Mask
  The settings here correspond to how the mask and extract box blend into the background image. If no blending is applied, then there will be a very clear boundary between the swapped face and the original frame.
  As a tip, if you change the Color plugin type to "manual-balance" and set the brightness to -100% you can see very clearly where the mask edges are, and the impact that changing the settings have:
  - Mask Blend
    Edit the settings to blend the edges of the face mask into the original Frame. Use the tool tips to get an understanding of which each setting does, and adjust to get your swap blended seamlessly.
    
    prev_msk2.png (7.54 KiB) Viewed 395273 times
- Scaling
  The final settings that can be adjusted is "scaling", or more specifically sharpening. Quite often the swap is likely to have to be upscaled to fit into the final frame. Here you will find options to artificially sharpen the image. It is a matter of personal preference whether sharpening yields good results, and these settings will always nearly have to vary on a case by case basis:
  
  prev_shr1.png (6.36 KiB) Viewed 395273 times
  
  The best quality sharpening option is "Unsharp Mask" followed by "Gaussian" and finally "Box". Use whichever works best for you or set to "None" to disable. You can google these sharpening methods to get a better idea of what the parameters do if the tooltips do not give you enough information.

Once you are happy that everything looks good, make sure your settings are saved, make a note of what mask type and color plugin you have chosen and close the Preview Tool

All the settings that we set above are available in the Preview Tool. These can be ignored, as we have already locked them in. However, if you are running from the command line, so don't have access to the tools, you can edit these options in the file faceswap/config/convert.ini. The options are exactly the same as shown in the Preview Tool section, so you can refer to that for help. The writer options we discuss next are also in the same file.

Writer plugin
There is one section within the Settings Menu we are still interested in, and that's the plugins relating to the "Writer". The Writer plugin dictates how your final frames are generated, be it a video, a series of images or an animated GIF. We will choose which Writer we use in the next section, but will set the options here.
- Select `Settings > Configure Settings..." (or select the convert settings icon on the Task Bar to jump straight to the correct section)
  
  wri_men.png (11.19 KiB) Viewed 394827 times
- Select the "Writer" node:
  
  wri_tab.jpg (58.11 KiB) Viewed 103945 times
- I won't go into all the individual settings as they are varied, so hover for tooltips, but an overview of the available writers:
  - Ffmpeg - Output the swap to a video file
  - Gif - Create an animated GIF from the swap
  - OpenCV - Output the swap to a series of images.
  - Patch - Output the swap as raw face patches. This writer only saves the output from the model (and mask), it does not do any compositing into the final frame. The transformation matrix (and ROI points) are provided in the image header and, optionally, as a json file. If you do not understand how to use this data to composite the face patch into the final frame, then this writer is not for you.
  - Pillow - Output the swap to a series of images (more format support than OpenCV, but slower)
- Some top level things you should probably note.
  - H265 encoding using the FFMPEG plugin will slow down convert considerably
  - You can output the mask to the alpha channel of a 4 channel PNG or TIFF using the OpenCV or Pillow writers and selecting the draw-transparent option. This is useful if you want to load the swap up into an external application for post processing.
- Once you have configured your plugin as you please click Ok to save your settings and exit the configuration page.

Convert

We're on the home straight now! We've got everything ready, and we just need to input the correct information into the convert tab and let it generate our swap. As we have everything configured correctly, this is straightforward.

Select the Convert tab in the main GUI:

con_tab.png (12.8 KiB) Viewed 395269 times
Data - The first section we will configure is Data:

con_data.png (7.54 KiB) Viewed 395269 times
- Input Dir - This is the video or folder of images than you wish to swap the faces on to. Select the corresponding video or images icon, depending on your source, and navigate to the correct location.
- Output DIr - This is where the output will be saved. A new folder will be created at this location if it does not already exist.
- Alignments - This is the alignments file that relates to the frames that you will be swapping faces onto. If it is at the default location (with the source), then you can leave this blank and the file will automatically get picked up.
- Reference Video - This option is only required if your input is a folder full of images and the output uses the FFMPEG writer. If that is the case, then this should be a path to the original video that the folder full of images was extracted from.
- Model Dir - This should be the folder that you specified during the training phase. It will contain a <model_name>.h5 file and a <model_name>_state.json file.
Plugins - Next we'll configure the plugins:

con_plg.png (8.04 KiB) Viewed 395267 times
- Color Adjustment - Select the color adjustment that you chose when using the Preview Tool
- Mask Type - Select the mask type you chose when using the Preview Tool
- Writer - Select the writer you wish to use (see the previous section for a breakdown on writers)
Frame Processing - I don't tend to play with any of these, but an explanation follows:

con_fra.png (2.58 KiB) Viewed 395267 times
- Output Scale - This will scale the final output. For example, a 720p video at 100% scale (default) will be output as a 720p video. At 50% scale, it will output as a 360p video and at 200% scale it will output as a 1440p video. Nothing fancy is done here, it just scales the output.
- Frame Ranges - You can specify a range of frames to be converted. For example, if you have a video that is 2000 frames long, but you only want to swap on the faces from frame 500 - 1500, then you would enter 500-1500
- Keep Unchanged - This option is only relevant if "Frame Ranges" has been selected. Using the example above, with this option enabled you will receive 2000 frames out, but only frames 500-1500 will be swapped. With this option disabled you will only receive 1000 frames out (the frames that you selected for swapping).
Face Processing - There is only one useful item in this section and that is Face Scale. You shouldn't need to use any of the other items in this section. If you find you need to use them, then you are just covering for bad workflow. However, the options exist, so I will describe hem (and why you shouldn't be using them).

con_face.jpg (21.22 KiB) Viewed 108586 times
- Face Scale - This can be very useful when you find that the original face and the swapped face do not match up closely enough. The swapped face can be adjusted to be slightly larger or slightly smaller to get a better swap. The impact of this can be seen in the Preview Tool. Drag the slider to the right to increase the face size. Drag it to the left to decrease the face size.
- Input Aligned Dir - This allows you to point Faceswap at a folder of known faces from the video you are swapping on to. Only faces that exist within the alignments file AND in this folder will be swapped. If you are having to use this folder, then it is because you have not cleaned your alignments file properly. A clean alignments file is paramount for getting a good swap, so... don't use this option. Clean your alignments file.
- Filter/nFilter, Ref Threshold - These options are for filtering in/out unwanted faces. Don't use it. You should have already cleansed your alignments file, so this should not be necessary. You will always be more effective at filtering out faces than a machine is, and it doesn't take long. Enabling these options also severely slows down the convert process.
Settings - Finally we have the settings for convert:

con_set.png (3.12 KiB) Viewed 395266 times
- Jobs - Generally you shouldn't need to touch this. The default setting will use as many parallel threads as are available. However, on some systems, convert can run out of RAM, in which case you may want to set this to "1" or a low number.
- Trainer - The model type used for to train your model. Generally you don't need to fill this in, as the trainer will be read from the model folder. However, if you have multiple models in the same folder (bad practice and will be unsupported in future) then you can specify the trainer for the model you want to convert here.
- Distributed - Whether to run inference on multiple GPUs. Probably makes no difference, as the speed bottleneck is on the CPU not the GPU at the convert stage
- On The Fly - Never use this option. Always generate an alignments file for your convert. However, if for some reason you don't want to do that, and you want to get a terrible swap that converts slowly, then by all means enable this option.
- Swap Model - When people start out, they often train the model the wrong way around (i.e. have the A and B folders wrong when training). If you find your swap just has a lower quality version of the original face on it, then you have trained the wrong way around. Enable this option and remember to train the right way around next time.
- Single Process - See "Jobs", does the same thing as setting "Jobs" to "1"

Now that we have all our settings locked in, we can go ahead and hit the Convert button and wait for our final swap to be generated.

bodyhammer32 · Post by **bodyhammer32** » Fri Nov 27, 2020 7:00 pm

This is not the same alignments file that was used for training, more, it is an alignments file that is created specifically for the video we want to swap on to. The purpose of the alignments file here is to tell Faceswap which is the target face that we should be swapping and where it exists within the frame. If you have not already created an alignments file for your source video and cleaned it up, then you should do so now. Head over to the Extract Guide for instructions on how to create an alignments file for converting.

I am failing to see where this information is located on the Extraction Guide. I see on this page, the referenced file is original_video_alignments.fsa, but I do not see this file created anywhere in these tutorials. Further, if I do use my original alignments file created from the Extract Guide, when I go to Tools->Preview and set my InputDir to my original mp4 video, my aligments to my FSA that I created from your Extract Guide and the Model Dir with the trainer set in the training guide, I get

Code: Select all

11/27/2020 13:59:31 INFO     Reading alignments from: 'D:\workflow\input1_alignments.fsa'
11/27/2020 13:59:31 ERROR    No faces were found in any of the frames passed in. Make sure you are passing in a frames source rather than extracted faces, and that you have provided the correct alignments file.

Process exited.

So, how to I make this "original_video_alignments.fsa" file?

Post by **torzdf** » Sun Nov 29, 2020 12:58 am

The alignments file is stored next to the video when you extract from it.

Roudrunner_404 · Post by **Roudrunner_404** » Thu Jul 08, 2021 1:37 pm

Ok so. I might not be that smart but. Must I swap face in the video I used for training?

Post by **torzdf** » Mon Jul 12, 2021 12:15 am

Roudrunner_404 wrote: ↑Thu Jul 08, 2021 1:37 pm
Ok so. I might not be that smart but. Must I swap face in the video I used for training?

No. It's best to follow the guides to understand how it all works (it's easier than me just re-explaining it all here)

Roudrunner_404 · Post by **Roudrunner_404** » Mon Jul 12, 2021 2:18 pm

So I must extract it and then put it convert?

Croos666 · Post by **Croos666** » Mon May 16, 2022 6:10 pm

When using the Preview tool, do the "Dir" and "Alignments" entries refer to Face A or Face B?

Post by **torzdf** » Mon May 16, 2022 9:42 pm

They refer to the video that you plan to swap faces onto and the associated alignments file.

unkempt · Post by **unkempt** » Wed Dec 28, 2022 2:11 pm

I have a folder of PNG files as a result of these instructions. Does it not create the video? How do I create a video from here? Just use a video editor program?

EDIT: Apologies. I see the "writer" option I missed which lets me select FFmpeg - direct to video.

MadEls · Post by **MadEls** » Tue Aug 27, 2024 7:59 pm

Hi, thank you for the guide!
I have started converting but it appears stuck.
I read "Loading Scaling from Sharpen plugin...".
The status bar displays: "Converting | Elapsed: 00:00 Remaining: 00:00 | 13,21it/s | 6/6 | 100%.
Have you ever encountered this situation? And do you know how to solve it?
Thank you!

Post by **torzdf** » Tue Oct 01, 2024 3:17 pm

This is likely a bug in the code somewhere.

Sadly, it will be hard to find without the original data.

If you can provide: The video that you are converting + the alignments file, I can look into this (You are welcome to send me a link via private message)

Faceswap Forum