Convert stalls

Getting errors or found a bug when converting faces from a trained model? Post about them here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Convert process. If you want to get tips, or better understand the Convert process, then you should look in the Convert Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Post Reply
User avatar
Pos994
Posts: 5
Joined: Mon Nov 13, 2023 4:33 pm

Convert stalls

Post by Pos994 »

Hi,

I am getting the following error every time i press on convert button:

Code: Select all

2023-11-13 17:42:44.046628: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8902
2023-11-13 17:42:44.108460: W tensorflow/stream_executor/gpu/asm_compiler.cc:80] Couldn't get ptxas version string: INTERNAL: Couldn't invoke ptxas.exe --version
2023-11-13 17:42:44.111706: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] INTERNAL: Failed to launch ptxas
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.

I have followed the steps to reinstall the entire faceswap (app.php/faqpage#f1r1) but I always get the same error.

Windows 10, 16GB Ram, Nvidia 2080 8GB (drivers updated)

by torzdf » Thu Nov 16, 2023 11:20 pm

The issue is that Conda-Forge has updated its version of ffmpeg to a version that faceswap does not currently support.

I am looking at a longer term solution, but in the meantime, the following should get Convert up and running:

In Windows open up an Anaconda Prompt:

  • Start > Anaconda Prompt
  • Enter the following lines into the Anaconda Prompt:

    Code: Select all

    conda activate faceswap
conda install -c conda-forge "ffmpeg<5.0" -y

Close the Anaconda prompt and relaunch faceswap.

Go to full post
Last edited by torzdf on Thu Nov 16, 2023 11:33 am, edited 2 times in total.
Top
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Couldnt get ptxas version string

Post by torzdf »

Ignore it. It's not an error, it is a warning, and it is not important.

My word is final

Top
User avatar
Pos994
Posts: 5
Joined: Mon Nov 13, 2023 4:33 pm

Re: Couldnt get ptxas version string

Post by Pos994 »

torzdf wrote: ↑Mon Nov 13, 2023 7:17 pm

Ignore it. It's not an error, it is a warning, and it is not important.

But i cant see previews, the info text remains in "Waiting for preview..." for 1 hour, and the CPU/GPU use is always 1-3%, the input video is 55MB aprox, correctly encoded with H.264. The same if i select "None" on Color adjustment and Mask Type. In TRACE or DEBUG mode shows:

Code: Select all

Setting Faceswap backend to NVIDIA
11/13/2023 20:47:42 INFO     Log level set to: DEBUG
11/13/2023 20:47:45 VERBOSE  Alignments filepath: 'C:\FaceProjects\Piloto 3\A_alignments.fsa'
11/13/2023 20:47:45 INFO     Reading alignments from: 'C:\FaceProjects\Piloto 3\A_alignments.fsa'
11/13/2023 20:47:45 VERBOSE  Aligned directory not specified. All faces listed in the alignments file will be converted
11/13/2023 20:47:45 INFO     Loading Model from Original plugin...
11/13/2023 20:47:45 VERBOSE  Loading config: 'C:\Users\ADMIN-PRO\faceswap\config\train.ini'
11/13/2023 20:47:45 VERBOSE  Loading config: 'C:\Users\ADMIN-PRO\faceswap\config\train.ini'
11/13/2023 20:47:45 INFO     Using configuration saved in state file
2023-11-13 20:47:45.225202: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-13 20:47:45.312350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6076 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5
11/13/2023 20:47:46 INFO     Loaded model from disk: 'C:\FaceProjects\Piloto 3\ModelAB\original.h5'
11/13/2023 20:47:46 VERBOSE  Model: "encoder"
11/13/2023 20:47:46 VERBOSE  ____________________________________________________________________________________________________
11/13/2023 20:47:46 VERBOSE   Layer (type)                                Output Shape                            Param #
11/13/2023 20:47:46 VERBOSE  ====================================================================================================
11/13/2023 20:47:46 VERBOSE   input_1 (InputLayer)                        [(None, 64, 64, 3)]                     0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   conv_128_0_conv2d (Conv2D)                  (None, 32, 32, 128)                     9728
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   conv_128_0_leakyrelu (LeakyReLU)            (None, 32, 32, 128)                     0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   conv_256_0_conv2d (Conv2D)                  (None, 16, 16, 256)                     819456
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   conv_256_0_leakyrelu (LeakyReLU)            (None, 16, 16, 256)                     0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   conv_512_0_conv2d (Conv2D)                  (None, 8, 8, 512)                       3277312
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   conv_512_0_leakyrelu (LeakyReLU)            (None, 8, 8, 512)                       0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   conv_1024_0_conv2d (Conv2D)                 (None, 4, 4, 1024)                      13108224
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   conv_1024_0_leakyrelu (LeakyReLU)           (None, 4, 4, 1024)                      0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   flatten (Flatten)                           (None, 16384)                           0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   dense (Dense)                               (None, 1024)                            16778240
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   dense_1 (Dense)                             (None, 16384)                           16793600
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   reshape (Reshape)                           (None, 4, 4, 1024)                      0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_512_0_conv2d_conv2d (Conv2D)        (None, 4, 4, 2048)                      18876416
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_512_0_conv2d_leakyrelu (LeakyReLU)  (None, 4, 4, 2048)                      0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_512_0_pixelshuffler (PixelShuffler)  (None, 8, 8, 512)                      0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE  ====================================================================================================
11/13/2023 20:47:46 VERBOSE  Total params: 69,662,976
11/13/2023 20:47:46 VERBOSE  Trainable params: 69,662,976
11/13/2023 20:47:46 VERBOSE  Non-trainable params: 0
11/13/2023 20:47:46 VERBOSE  ____________________________________________________________________________________________________
11/13/2023 20:47:46 VERBOSE  Model: "decoder_b"
11/13/2023 20:47:46 VERBOSE  ____________________________________________________________________________________________________
11/13/2023 20:47:46 VERBOSE   Layer (type)                                Output Shape                            Param #
11/13/2023 20:47:46 VERBOSE  ====================================================================================================
11/13/2023 20:47:46 VERBOSE   input_3 (InputLayer)                        [(None, 8, 8, 512)]                     0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_256_1_conv2d_conv2d (Conv2D)        (None, 8, 8, 1024)                      4719616
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_256_1_conv2d_leakyrelu (LeakyReLU)  (None, 8, 8, 1024)                      0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_256_1_pixelshuffler (PixelShuffler)  (None, 16, 16, 256)                    0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_128_1_conv2d_conv2d (Conv2D)        (None, 16, 16, 512)                     1180160
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_128_1_conv2d_leakyrelu (LeakyReLU)  (None, 16, 16, 512)                     0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_128_1_pixelshuffler (PixelShuffler)  (None, 32, 32, 128)                    0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_64_1_conv2d_conv2d (Conv2D)         (None, 32, 32, 256)                     295168
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_64_1_conv2d_leakyrelu (LeakyReLU)   (None, 32, 32, 256)                     0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   upscale_64_1_pixelshuffler (PixelShuffler)  (None, 64, 64, 64)                      0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   face_out_b_0_conv2d (Conv2D)                (None, 64, 64, 3)                       4803
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   face_out_b_0 (Activation)                   (None, 64, 64, 3)                       0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE  ====================================================================================================
11/13/2023 20:47:46 VERBOSE  Total params: 6,199,747
11/13/2023 20:47:46 VERBOSE  Trainable params: 6,199,747
11/13/2023 20:47:46 VERBOSE  Non-trainable params: 0
11/13/2023 20:47:46 VERBOSE  ____________________________________________________________________________________________________
11/13/2023 20:47:46 VERBOSE  Model: "original_inference"
11/13/2023 20:47:46 VERBOSE  ____________________________________________________________________________________________________
11/13/2023 20:47:46 VERBOSE   Layer (type)                                Output Shape                            Param #
11/13/2023 20:47:46 VERBOSE  ====================================================================================================
11/13/2023 20:47:46 VERBOSE   face_in_a (InputLayer)                      [(None, 64, 64, 3)]                     0
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   encoder (Functional)                        (None, 8, 8, 512)                       69662976
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE   decoder_b (Functional)                      (None, 64, 64, 3)                       6199747
11/13/2023 20:47:46 VERBOSE
11/13/2023 20:47:46 VERBOSE  ====================================================================================================
11/13/2023 20:47:46 VERBOSE  Total params: 75,862,723
11/13/2023 20:47:46 VERBOSE  Trainable params: 75,862,723
11/13/2023 20:47:46 VERBOSE  Non-trainable params: 0
11/13/2023 20:47:46 VERBOSE  ____________________________________________________________________________________________________
11/13/2023 20:47:46 INFO     Loading Writer from Opencv plugin...
11/13/2023 20:47:46 VERBOSE  Loading config: 'C:\Users\ADMIN-PRO\faceswap\config\convert.ini'

Converting: 0it [00:00, ?it/s]
Converting: 0it [00:00, ?it/s]
11/13/2023 20:47:46 INFO     Loading Mask from Mask_Blend plugin...
11/13/2023 20:47:46 VERBOSE  Loading config: 'C:\Users\ADMIN-PRO\faceswap\config\convert.ini'
11/13/2023 20:47:46 INFO     Loading Color from Match_Hist plugin...
11/13/2023 20:47:46 VERBOSE  Loading config: 'C:\Users\ADMIN-PRO\faceswap\config\convert.ini'
11/13/2023 20:47:46 INFO     Loading Scaling from Sharpen plugin...
11/13/2023 20:47:46 VERBOSE  Loading config: 'C:\Users\ADMIN-PRO\faceswap\config\convert.ini'
2023-11-13 20:47:46.800989: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8902
2023-11-13 20:47:46.867242: W tensorflow/stream_executor/gpu/asm_compiler.cc:80] Couldn't get ptxas version string: INTERNAL: Couldn't invoke ptxas.exe --version
2023-11-13 20:47:46.870347: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] INTERNAL: Failed to launch ptxas
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.

Thank you for your reply torzdf!

Top
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Couldnt get ptxas version string

Post by torzdf »

Ok, that isn't the error (although I appreciate why you would think it may be). That message is standard on Windows installations.

I would suspect there is an issue with your video file which is causing the process to hang. Try running convert with a random video/images to see if that also hangs. If it does not, then the issue is with the video file, and it may require re-encoding.

My word is final

Top
User avatar
Pos994
Posts: 5
Joined: Mon Nov 13, 2023 4:33 pm

Re: Couldnt get ptxas version string

Post by Pos994 »

I have tried different models and different videos downloaded from different sites (youtube downloads and ripped myself with handbrake), the extraction and the train processes works fine (i see the previews and looks ok) but the Convert process gets stuck on "INFO Loading Scaling from Sharpen plugin.." with no previews.

Top
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Couldnt get ptxas version string

Post by torzdf »

Does the preview tool work?

My word is final

Top
User avatar
Pos994
Posts: 5
Joined: Mon Nov 13, 2023 4:33 pm

Re: Couldnt get ptxas version string

Post by Pos994 »

Tools, Preview

Code: Select all

Setting Faceswap backend to NVIDIA
11/14/2023 09:06:51 INFO     Log level set to: DEBUG
11/14/2023 09:06:54 VERBOSE  Loading config: 'C:\Users\ADMIN-PRO\faceswap\config\convert.ini'
11/14/2023 09:06:54 INFO     Input Video: C:\FaceProjects\Piloto 4\A.mp4
11/14/2023 09:06:54 VERBOSE  Alignments filepath: 'C:\FaceProjects\Piloto 4\A_alignments.fsa'
11/14/2023 09:06:54 INFO     Reading alignments from: 'C:\FaceProjects\Piloto 4\A_alignments.fsa'
11/14/2023 09:06:54 ERROR    No faces were found in any of the frames passed in. Make sure you are passing in a frames source rather than extracted faces, and that you have provided the correct alignments file.
Process exited.

The same error happens with other models, different people, other project, etc

Last edited by Pos994 on Tue Nov 14, 2023 8:09 am, edited 1 time in total.
Top
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Couldnt get ptxas version string

Post by torzdf »

Ok, there appears to be an issue with your alignments file (no faces have been stored)

Firstly load up the manual tool with that video/alignments file combination and see if you can identify the issue..

If you cannot, then I would need a copy of the video + alignments file to investigate further.

My word is final

Top
User avatar
Pos994
Posts: 5
Joined: Mon Nov 13, 2023 4:33 pm

Re: Couldnt get ptxas version string

Post by Pos994 »

I have tried starting faceswap in cpu mode (not nvidia) and now the convert process works. All with the same project files and same people/faces/videos. I would like to be able to solve the problem in nvidia mode :cry:

torzdf wrote: ↑Tue Nov 14, 2023 11:46 am

Ok, there appears to be an issue with your alignments file (no faces have been stored)

Firstly load up the manual tool with that video/alignments file combination and see if you can identify the issue..

If you cannot, then I would need a copy of the video + alignments file to investigate further.

I can see the video A.mp4, I press play and it detects all the faces correctly frame per frame, but it is the first time I use this tool-preview, I don't know if I can tell you if I see any errors.

Last edited by Pos994 on Tue Nov 14, 2023 1:37 pm, edited 1 time in total.
Top
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Couldnt get ptxas version string

Post by torzdf »

Ok, this is a very confusing issue

  • Convert getting stuck indicates a potential issue with the input video
  • The preview tool (which uses much of the same code as convert) indicates that there are no faces in your alignments file
  • The manual tool indicates that there are faces in your alignments file
  • Using CPU everything works

Basically, it is not possible to square the circle based on this information. It is all quite contradictory!

So 3 things will be helpful here:

  1. Please provide me with the output of Help > Output System Information from the Faceswap GUI
  2. Please run convert with "Trace" logging enabled. Let it run to the point it gets stuck and wait about 30 seconds before killing the process. Then please provide the faceswap.log file that is generated in your faceswap folder (it is likely to be large, and will probably require zipping up and hosting somewhere)
  3. Please zip up your model folder, your source video and your alignments file and provide to me to see if I can recreate the issue

You are welcome to PM me with this data. I have no interest in what it actually contains, I am just looking to see if I can identify the issue here.

My word is final

Top
User avatar
mark gi 2000
Posts: 2
Joined: Wed Nov 15, 2023 8:34 pm

Conversion does not proceed

Post by mark gi 2000 »

When I run the convert, I get stuck at 100% with the following message:
11/16/2023 05:33:11 INFO Loading Mask from Mask_Blend plugin...
11/16/2023 05:33:11 INFO Loading Color from Avg_Color plugin...
11/16/2023 05:33:11 INFO Loading Scaling from Sharpen plugin...
2023-11-16 05:33:12.778306: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8902

Nothing seems in progress and no files have been generated in the output folder. What are the possible causes and solutions?

Note:
Extract and Train processes have been finished correctly.
I have changed some configures concerning the convert, but the same results were obtained.
The green progress bar in the lower right corner of the app screen also does not appear.

Top
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Couldnt get ptxas version string

Post by torzdf »

Please provide the output of: `Help> Output Sytsem Information as per: app.php/rules#rule-4b

Please also press the "Generate" button rather than the "Convert" button and paste the output here.

My word is final

Top
User avatar
mark gi 2000
Posts: 2
Joined: Wed Nov 15, 2023 8:34 pm

Conversion does not proceed

Post by mark gi 2000 »

When I run the convert, I get stuck at 100% with the following message:
11/16/2023 05:33:11 INFO Loading Mask from Mask_Blend plugin...
11/16/2023 05:33:11 INFO Loading Color from Avg_Color plugin...
11/16/2023 05:33:11 INFO Loading Scaling from Sharpen plugin...
2023-11-16 05:33:12.778306: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8902

Nothing seems in progress and no files have been generated in the output folder. What are the possible causes and solutions?

Note:
Extract and Train processes have been finished correctly.
I have changed some configures concerning the convert, but the same results were obtained.
The green progress bar in the lower right corner of the app screen also does not appear.

Top
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Convert stalls

Post by torzdf »

Ok, I have now had several reports on this.

There hasn't been a code update in this area for a while, so it is likely to be an issue with one of the libraries we use. I will investigate and see if I can get a fix out.

My word is final

Top
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: Convert stalls

Post by torzdf »

The issue is that Conda-Forge has updated its version of ffmpeg to a version that faceswap does not currently support.

I am looking at a longer term solution, but in the meantime, the following should get Convert up and running:

In Windows open up an Anaconda Prompt:

  • Start > Anaconda Prompt
  • Enter the following lines into the Anaconda Prompt:

    Code: Select all

    conda activate faceswap
conda install -c conda-forge "ffmpeg<5.0" -y

Close the Anaconda prompt and relaunch faceswap.

Last edited by torzdf on Thu Nov 16, 2023 11:21 pm, edited 1 time in total.

My word is final

Top
Post Reply

Return to “Convert Support”