[Guide] A Rough Guide to Converting

Converting source frames into swapped frames
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Post Reply
User avatar
torzdf
Posts: 1011
Joined: Fri Jul 12, 2019 12:53 am
Answers: 128
Has thanked: 28 times
Been thanked: 193 times

[Guide] A Rough Guide to Converting

Post by torzdf »

NB: This guide was correct at time of writing, but things do change. I will try to keep it updated.

Contents


Introduction


If you've landed on this page and you don't know how to extract a face set or train a model, then stop right now and go read the Extract Guide and then the Training Guide.

So, you've got this far? The hard part is over. We've got a trained model and we're ready to create a swap. Converting is by far the least involved part of the process, but there are still some settings and final tweaks we need to do to get a good result.

As in the other guides, I will be using the GUI, but all the commands here are also available in the cli, You can execute python faceswap.py convert -h to get a list of the commands, and python tools.py -h to get a list of the available tools.

Settings


Before heading to the convert tab, we need to do a little preparation work and lock in some settings which look good.

Alignments

For creating our final swap, we're going to need a source video (or folder of frames) and the trained model. There is one other thing that we need, and that is an alignments file for the source video. This is not the same alignments file that was used for training, more, it is an alignments file that is created specifically for the video we want to swap on to. The purpose of the alignments file here is to tell Faceswap which is the target face that we should be swapping and where it exists within the frame. If you have not already created an alignments file for your source video and cleaned it up, then you should do so now. Head over to the Extract Guide for instructions on how to create an alignments file for converting.

Preview Tool

Faceswap provides a Preview Tool where you can preview the results of the swap and adjust settings to better blend the face with the frame.

  • Head over to the Tools tab and then the Preview sub-tab:

    prev_tab.png
    prev_tab.png (13.5 KiB) Viewed 9775 times
  • There aren't many options to configure here, but they are explained below:

    prev_opts.png
    prev_opts.png (27.26 KiB) Viewed 9775 times
    • Input Dir - Select the video icon for a source video, or the image icon for a source folder of frames and navigate to the source that you want to swap.
    • Alignments File - If the alignments file is in the default location that you extracted from, then you can leave this blank, otherwise, click the icon and navigate to the .fsa alignment file that corresponds to the video you entered in the previous field.
    • Model Dir - This should be the folder that you specified during the training phase. It will contain a <model_name>.h5 file and a <model_name>_state.json file.
    • Swap Model - This is here in case you have trained your model the wrong way around (i.e. when you try to convert, but the subject's face remains unchanged). You can enable this option to switch the model direction, and make a mental note to train the correct way around next time.
  • Hit the Preview button to launch the preview tool.
    The process will load the model, source video and alignments file and then display a window similar to below:

    prev_overview.jpg
    prev_overview.jpg (103.95 KiB) Viewed 9761 times

    A basic overview of the layout is described below, but we will drill into the options in more detail next.

    • Top row of images - the faces in the original frame
    • Bottom row of images - the swapped face placed into the original frame
    • Bottom left option panel - These relate to Command Line Options (that is options you set in the main GUI for convert). Anything you set here, you should take a note of, as you will need to replicate these options when you run the convert.
    • Bottom right option panel - These relate to Configuration Options. These are the settings you access from the Settings > Edit Convert Settings... dialog in the GUI (or ./config/convert.ini in the cli). Any settings you change (and save) here will be remembered for the convert process.
    • Update Samples - This button will refresh the images shown in the top frames, pulling in a new selection of faces from the source video file. Be patient when pressing this button. Depending on your setup this may take a few seconds, as the faces need to be passed through the model to generate new swaps.
    • Utility Buttons - There are two sets of utility buttons. Those on the left hand side (under the Command Line Options) impact the full configuration (i.e. the options that correspond to every tab in the right hand panel). The buttons on the right hand side (under the Configuration Options) just impact the tab that you currently have selected. Either way, these buttons allow you to reset the options their saved values or their default values, and allows you to save the configuration.
  • Command Line Options
    These options relate to settings that you will configure in the main Faceswap GUI when running your convert later. Anything you change here, you will want to take a note of, as you will need to use the same settings when running the convert task.

    prev_cli.png
    prev_cli.png (9.11 KiB) Viewed 9761 times
    • Color - This is the type of color matching to use between the original face and the swapped face. Ideally you would leave this at "None" but depending on the model used and the quality of data, then you may want to use one of the built in color matching methods. Some of the methods have settings which can be adjusted in the right hand panel (but I will cover that later). It is worth noting that there is no "right" option to select here. Different swaps will require different color matching techniques. The current methods are:
      • None - Don't perform any color matching.
      • Avg Color - Takes an average of the colors between the original face and the swap face and applies this to the swap face.
      • Color Transfer - A color matching algorithm developed by Reinhard et al., 2001. This method has a couple of options in the right hand pane.
      • Manual Balance - Allows you to manually alter the color/contrast/brightness of the swapped face. This method has settings available in the right hand pane.
      • Match Hist - Attempts to match the histogram on the swap face from the histogram in the source face. This method has settings available in the right hand pane.
      • Seamless Clone - This method is good for single images, but terrible for videos, No matter how good the result may look, if you are creating a video, do not use this method.
    • Mask Type - This is the mask type you want to use for converting. The available masks and their descriptions are available in the Extract Guide. If you want to use a mask that is not available within the alignments file, then you can use the Mask Tool to generate additional masks. Masks have settings in the right hand pane to adjust how to blend the swapped face onto the original frame.
  • Configuration Options
    These options, in the right hand panel, allow you to configure the settings for each individual plugin that will be run during the convert process. When you make changes here, the saved values will be updated in your convert.ini (accessible from Settings > Edit Convert Settings... in the GUI). This means that once you have got the settings locked in correctly here, they will automatically be picked up for the convert process.

    prev_cnf.png
    prev_cnf.png (7.32 KiB) Viewed 9760 times

    The plugins are broken down into 3 main categories, and each of these categories may contain settings for one or more plugins. I will provide an overview of the main categories and plugins available, but the actual settings you will need to use will vary on a case by case basis, so it's up to you to evaluate the results and choose the best settings. All of the settings have tooltips which explain what each option does.

    • Color
      The settings shown here relate to the Color pull down list from the previous Command Line Options section. E.G, if you have selected "Color-Transfer" as your color matching method in the left-hand pane, then you would select the "Color Transfer" tab in the right-hand pane to configure that plugin's options.

      • Color Transfer - Exposes a couple of options to turn off/on for this plugin. Turning these on or off will have different effects depending on the source.
      • Manual Balance - Allows you to manually adjust the color/brightness/contrast for the final swap, if you have selected this plugin.
        prev_col1.png
        prev_col1.png (9.12 KiB) Viewed 9760 times
        The Color Balance section allows you to choose a color space to work in, and then adjust the balance for that color space. The Brightness Contrast section allows you to adjust the brightness and contrast.
      • Match Hist - Only one parameter to edit for this plugin, which allows you to set the Threshold at which color matching takes place. This helps cut out "extreme" colors.
    • Mask
      The settings here correspond to how the mask and extract box blend into the background image. If no blending is applied, then there will be a very clear boundary between the swapped face and the original frame. There are 2 "masks" to edit here. The Box Mask which is the square box that contains the face that is being swapped, and the Face Mask which is the mask that defines the edges of the face.
      Depending on what coverage you trained at will dictate how much of each mask will be used. Using the example posted in the Extract Guide:

      coverage.jpg
      coverage.jpg (76.39 KiB) Viewed 9759 times

      If you trained at 100%, then it is entirely possible that the extract box contains the entire mask, so editing the box blend settings will do nothing. However, at low coverage settings (e.g. 67.5%) it is entirely conceivable that most of the extract box sits within the face mask area, in which case the face blend settings will not have as much of an effect, but both will need to be adjusted.
      As a tip, if you change the Color plugin type to "manual-balance" and set the brightness to -100% you can see very clearly where the box/mask edges are, and the impact that changing the settings have:

      • Box Blend
        Edit the settings to blend the edges of the extract box into the original Frame. Use the tool tips to get an understanding of which each setting does, and adjust to get your swap blended seamlessly.

        prev_msk1.png
        prev_msk1.png (6.57 KiB) Viewed 9759 times
      • Mask Blend
        Edit the settings to blend the edges of the face mask into the original Frame. Use the tool tips to get an understanding of which each setting does, and adjust to get your swap blended seamlessly.

        prev_msk2.png
        prev_msk2.png (7.54 KiB) Viewed 9759 times
    • Scaling
      The final settings that can be adjusted is "scaling", or more specifically sharpening. Quite often the swap is likely to have to be upscaled to fit into the final frame. Here you will find options to artificially sharpen the image. It is a matter of personal preference whether sharpening yields good results, and these settings will always nearly have to vary on a case by case basis:

      prev_shr1.png
      prev_shr1.png (6.36 KiB) Viewed 9759 times

      The best quality sharpening option is "Unsharp Mask" followed by "Gaussian" and finally "Box". Use whichever works best for you or set to "None" to disable. You can google these sharpening methods to get a better idea of what the parameters do if the tooltips do not give you enough information.

Once you are happy that everything looks good, make sure your settings are saved, make a note of what mask type and color plugin you have chosen and close the Preview Tool

Settings Menu

All the settings that we set above are available in the Preview Tool. These can be ignored, as we have already locked them in. However, if you are running from the command line, so don't have access to the tools, you can edit these options in the file faceswap/config/convert.ini. The options are exactly the same as shown in the Preview Tool section, so you can refer to that for help. The writer options we discuss next are also in the same file.

  • Writer plugin
    There is one section within the Settings Menu we are still interested in, and that's the plugins relating to the "Writer". The Writer plugin dictates how your final frames are generated, be it a video, a series of images or an animated GIF. We will choose which Writer we use in the next section, but will set the options here.
    • Select `Settings > Configure Settings..." (or select the convert settings icon on the Task Bar to jump straight to the correct section)
      wri_men.png
      wri_men.png (11.19 KiB) Viewed 9313 times
    • Select the "Writer" node:
      wri_tab.png
      wri_tab.png (19.38 KiB) Viewed 9312 times
    • I won't go into all the individual settings as they are varied, so hover for tooltips, but an overview of the available writers:
      • Ffmpeg - Output the swap to a video file
      • Gif - Create an animated GIF from the swap
      • OpenCV - Output the swap to a series of images.
      • Pillow - Output the swap to a series of images (more format support than OpenCV, but slower)
    • Some top level things you should probably note.
      • H265 encoding using the FFMPEG plugin will slow down convert considerably
      • You can output the mask to the alpha channel of a 4 channel PNG or TIFF using the OpenCV or Pillow writers and selecting the draw-transparent option. This is useful if you want to load the swap up into an external application for post processing.
    • Once you have configured your plugin as you please click Ok to save your settings and exit the configuration page.

Convert


We're on the home straight now! We've got everything ready, and we just need to input the correct information into the convert tab and let it generate our swap. As we have everything configured correctly, this is straightforward.

  • Select the Convert tab in the main GUI:
    con_tab.png
    con_tab.png (12.8 KiB) Viewed 9755 times
  • Data - The first section we will configure is Data:
    con_data.png
    con_data.png (7.54 KiB) Viewed 9755 times
    • Input Dir - This is the video or folder of images than you wish to swap the faces on to. Select the corresponding video or images icon, depending on your source, and navigate to the correct location.
    • Output DIr - This is where the output will be saved. A new folder will be created at this location if it does not already exist.
    • Alignments - This is the alignments file that relates to the frames that you will be swapping faces onto. If it is at the default location (with the source), then you can leave this blank and the file will automatically get picked up.
    • Reference Video - This option is only required if your input is a folder full of images and the output uses the FFMPEG writer. If that is the case, then this should be a path to the original video that the folder full of images was extracted from.
    • Model Dir - This should be the folder that you specified during the training phase. It will contain a <model_name>.h5 file and a <model_name>_state.json file.
  • Plugins - Next we'll configure the plugins:
    con_plg.png
    con_plg.png (8.04 KiB) Viewed 9753 times
    • Color Adjustment - Select the color adjustment that you chose when using the Preview Tool
    • Mask Type - Select the mask type you chose when using the Preview Tool
    • Writer - Select the writer you wish to use (see the previous section for a breakdown on writers)
  • Frame Processing - I don't tend to play with any of these, but an explanation follows:
    con_fra.png
    con_fra.png (2.58 KiB) Viewed 9753 times
    • Output Scale - This will scale the final output. For example, a 720p video at 100% scale (default) will be output as a 720p video. At 50% scale, it will output as a 360p video and at 200% scale it will output as a 1440p video. Nothing fancy is done here, it just scales the output.
    • Frame Ranges - You can specify a range of frames to be converted. For example, if you have a video that is 2000 frames long, but you only want to swap on the faces from frame 500 - 1500, then you would enter 500-1500
    • Keep Unchanged - This option is only relevant if "Frame Ranges" has been selected. Using the example above, with this option enabled you will receive 2000 frames out, but only frames 500-1500 will be swapped. With this option disabled you will only receive 1000 frames out (the frames that you selected for swapping).
  • Face Processing - Don't use this section. Seriously, if you have to use it, then you are just covering for bad workflow. However, the options exist, so I will describe hem (and why you shouldn't be using them).
    con_fac.png
    con_fac.png (4.32 KiB) Viewed 9752 times
    • Input Aligned Dir - This allows you to point Faceswap at a folder of known faces from the video you are swapping on to. Only faces that exist within the alignments file AND in this folder will be swapped. If you are having to use this folder, then it is because you have not cleaned your alignments file properly. A clean alignments file is paramount for getting a good swap, so... don't use this option. Clean your alignments file.
    • Filter/nFilter, Ref Threshold - These options are for filtering in/out unwanted faces. Don't use it. You should have already cleansed your alignments file, so this should not be necessary. You will always be more effective at filtering out faces than a machine is, and it doesn't take long. Enabling these options also severely slows down the convert process.
  • Settings - Finally we have the settings for convert:
    con_set.png
    con_set.png (3.12 KiB) Viewed 9752 times
    • Jobs - Generally you shouldn't need to touch this. The default setting will use as many parallel threads as are available. However, on some systems, convert can run out of RAM, in which case you may want to set this to "1" or a low number.
    • Trainer - The model type used for to train your model. Generally you don't need to fill this in, as the trainer will be read from the model folder. However, if you have multiple models in the same folder (bad practice and will be unsupported in future) then you can specify the trainer for the model you want to convert here.
    • Distributed - Whether to run inference on multiple GPUs. Probably makes no difference, as the speed bottleneck is on the CPU not the GPU at the convert stage
    • On The Fly - Never use this option. Always generate an alignments file for your convert. However, if for some reason you don't want to do that, and you want to get a terrible swap that converts slowly, then by all means enable this option.
    • Swap Model - When people start out, they often train the model the wrong way around (i.e. have the A and B folders wrong when training). If you find your swap just has a lower quality version of the original face on it, then you have trained the wrong way around. Enable this option and remember to train the right way around next time.
    • Single Process - See "Jobs", does the same thing as setting "Jobs" to "1"

Now that we have all our settings locked in, we can go ahead and hit the Convert button and wait for our final swap to be generated.

My word is final


Tags:

User avatar
bodyhammer32
Posts: 1
Joined: Fri Nov 27, 2020 6:49 pm

Re: [Guide] A Rough Guide to Converting

Post by bodyhammer32 »

This is not the same alignments file that was used for training, more, it is an alignments file that is created specifically for the video we want to swap on to. The purpose of the alignments file here is to tell Faceswap which is the target face that we should be swapping and where it exists within the frame. If you have not already created an alignments file for your source video and cleaned it up, then you should do so now. Head over to the Extract Guide for instructions on how to create an alignments file for converting.

I am failing to see where this information is located on the Extraction Guide. I see on this page, the referenced file is original_video_alignments.fsa, but I do not see this file created anywhere in these tutorials. Further, if I do use my original alignments file created from the Extract Guide, when I go to Tools->Preview and set my InputDir to my original mp4 video, my aligments to my FSA that I created from your Extract Guide and the Model Dir with the trainer set in the training guide, I get

Code: Select all

11/27/2020 13:59:31 INFO     Reading alignments from: 'D:\workflow\input1_alignments.fsa'
11/27/2020 13:59:31 ERROR    No faces were found in any of the frames passed in. Make sure you are passing in a frames source rather than extracted faces, and that you have provided the correct alignments file.

Process exited.

So, how to I make this "original_video_alignments.fsa" file?


User avatar
torzdf
Posts: 1011
Joined: Fri Jul 12, 2019 12:53 am
Answers: 128
Has thanked: 28 times
Been thanked: 193 times

Re: [Guide] A Rough Guide to Converting

Post by torzdf »

The alignments file is stored next to the video when you extract from it.

My word is final


Post Reply