Mask problem

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
lorenzosale
Posts: 25
Joined: Thu May 28, 2020 11:58 am
Answers: 1
Has thanked: 12 times

Mask problem

Post by lorenzosale »

Hello,

I have extracted the faces of a video using Unet DLF mask
I deleted false positive, and ran the "remove faces" tool
I manually fixed the alignments et re extracted the faces

When i try to run the training, i have this error message :

(i tried to run the mask tool on the alignments but it says that 0 faces were updated)

Can someone help please ? Thanks a lot :)

Code: Select all

02/23/2021 16:02:25 CRITICAL Error caught! Exiting...
02/23/2021 16:02:25 ERROR    Caught exception in thread: '_training_0'
02/23/2021 16:02:25 ERROR    You have selected the Mask Type 'unet-dfl' in your training configuration options but at least one face does not have this mask type stored for it.
02/23/2021 16:02:25 ERROR    You should select a mask type that exists within your face data, or generate the required masks with the Mask Tool.
02/23/2021 16:02:25 ERROR    2 faces in side A do not contain the mask 'unet-dfl'. Available masks: {'unet-dfl', 'extended', 'components'}
02/23/2021 16:02:25 ERROR    0 faces in side B do not contain the mask 'unet-dfl'. Available masks: {'unet-dfl', 'extended', 'components'}
by 230V » Mon Mar 01, 2021 9:50 pm

correct me if i am wrong; i remember the function to get a list of the files with missing masks. it would be easy to remove the missing files instead of updating the whole 2k files and extract them.

Go to full post
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Mask problem

Post by torzdf »

remove-faces only updates/updated the alignments file, not the output face images.

I have pushed an update today which also updates the headers in the face set.

However, for the set you have already run "remove-faces" for, you should delete your faces folder and regenerate your faceset from your cleansed alignments file. It should fix your issue (assuming that you have updated masks in the alignments file).

My word is final

User avatar
lorenzosale
Posts: 25
Joined: Thu May 28, 2020 11:58 am
Answers: 1
Has thanked: 12 times

Re: Mask problem

Post by lorenzosale »

Thanks a lot I will try it :)

User avatar
lorenzosale
Posts: 25
Joined: Thu May 28, 2020 11:58 am
Answers: 1
Has thanked: 12 times

Re: Mask problem

Post by lorenzosale »

Hello again,

Sorry but i have the same issue

I have started all again :

Re-extracted destination video and source video with unet dfl mask

For both videos, I have :
Sorted faces
Deleted bad faces
Ran the "remove faces" tool
Modified the landmarks
re-extracted the faces using alignments tab-> extract

I tried to ran the mask tool on both, on all the frames but it says : "No masks were updated"

I really dont understand what am i doing wrong, I already trained bunch of models and it never happened.

When i try to train the model, it says :

Code: Select all

02/26/2021 15:38:16 CRITICAL Error caught! Exiting...
02/26/2021 15:38:16 ERROR    Caught exception in thread: '_training_0'
02/26/2021 15:38:16 ERROR    You have selected the Mask Type 'unet-dfl' in your training configuration options but at least one face does not have this mask type stored for it.
02/26/2021 15:38:16 ERROR    You should select a mask type that exists within your face data, or generate the required masks with the Mask Tool.
02/26/2021 15:38:16 ERROR    1 faces in side A do not contain the mask 'unet-dfl'. Available masks: {'components', 'unet-dfl', 'extended'}
02/26/2021 15:38:16 ERROR    5 faces in side B do not contain the mask 'unet-dfl'. Available masks: {'components', 'unet-dfl', 'extended'}

I really dont understand because it says that 1 face do not contain unet dfl but available masks : unet dfl !!

Same for side B

Can you help me again please ?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Mask problem

Post by torzdf »

You can try forcing the regeneration of all masks by specifying process all.

Otherwise, I'm not sure what the issue is here, as I have no outstanding reports of issues in this area.

My word is final

User avatar
230V
Posts: 20
Joined: Sun Jan 19, 2020 6:20 pm
Answers: 1
Has thanked: 7 times
Been thanked: 2 times

Re: Mask problem

Post by 230V »

how to understand this error?

Code: Select all

ERROR    0 faces in side A do not contain the mask 'extended'. Available masks: {'components', 'extended'}
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Mask problem

Post by torzdf »

Ok. That is weird. For one thing, extended is auto-generated.

Can you provide the crash report please.

My word is final

User avatar
230V
Posts: 20
Joined: Sun Jan 19, 2020 6:20 pm
Answers: 1
Has thanked: 7 times
Been thanked: 2 times

Re: Mask problem

Post by 230V »

Code: Select all

Loading...
Setting Faceswap backend to AMD
02/28/2021 07:53:43 INFO     Log level set to: VERBOSE
02/28/2021 07:53:43 INFO     Setting up for PlaidML
02/28/2021 07:53:43 VERBOSE  Setting Keras Backend to PlaidML
02/28/2021 07:53:43 INFO     Using GPU(s): ['Advanced Micro Devices, Inc. - Ellesmere (experimental)', 'Advanced Micro Devices, Inc. - Ellesmere (supported)']
02/28/2021 07:53:43 INFO     Successfully set up for PlaidML
2021-02-28 07:53:43.158263: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2021-02-28 07:53:43.158412: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
02/28/2021 07:53:44 INFO     Model A Directory: D:\sf\x8\fc
02/28/2021 07:53:44 INFO     Model B Directory: D:\sf\fc\fc_train
02/28/2021 07:53:44 INFO     Training data directory: D:\sf\x8\model
02/28/2021 07:53:44 INFO     ===================================================
02/28/2021 07:53:44 INFO       Starting
02/28/2021 07:53:44 INFO       Press 'Stop' to save and quit
02/28/2021 07:53:44 INFO     ===================================================
02/28/2021 07:53:45 INFO     Loading data, this may take a while...
02/28/2021 07:53:45 INFO     Loading Model from Original plugin...
Using plaidml.keras.backend backend.
02/28/2021 07:53:45 VERBOSE  Loading config: 'C:\Users\faceswap_amd\config\train.ini'
02/28/2021 07:53:45 VERBOSE  Loading config: 'C:\Users\faceswap_amd\config\train.ini'
02/28/2021 07:53:45 INFO     No existing state file found. Generating.
02/28/2021 07:53:45 INFO     Opening device "opencl_amd_ellesmere.0"
02/28/2021 07:53:46 VERBOSE  Using Adam optimizer
02/28/2021 07:53:46 VERBOSE  __________________________________________________________________________________________________
02/28/2021 07:53:46 VERBOSE  Layer (type)                    Output Shape         Param #     Connected to
02/28/2021 07:53:46 VERBOSE  ==================================================================================================
02/28/2021 07:53:46 VERBOSE  face_in_a (InputLayer)          (None, 64, 64, 3)    0
02/28/2021 07:53:46 VERBOSE  __________________________________________________________________________________________________
02/28/2021 07:53:46 VERBOSE  face_in_b (InputLayer)          (None, 64, 64, 3)    0
02/28/2021 07:53:46 VERBOSE  __________________________________________________________________________________________________
02/28/2021 07:53:46 VERBOSE  encoder (Model)                 (None, 8, 8, 512)    69662976    face_in_a[0][0]
02/28/2021 07:53:46 VERBOSE                                                                   face_in_b[0][0]
02/28/2021 07:53:46 VERBOSE  __________________________________________________________________________________________________
02/28/2021 07:53:46 VERBOSE  decoder_a (Model)               (None, 64, 64, 3)    6199747     encoder[1][0]
02/28/2021 07:53:46 VERBOSE  __________________________________________________________________________________________________
02/28/2021 07:53:46 VERBOSE  decoder_b (Model)               (None, 64, 64, 3)    6199747     encoder[2][0]
02/28/2021 07:53:46 VERBOSE  ==================================================================================================
02/28/2021 07:53:46 VERBOSE  Total params: 82,062,470
02/28/2021 07:53:46 VERBOSE  Trainable params: 82,062,470
02/28/2021 07:53:46 VERBOSE  Non-trainable params: 0
02/28/2021 07:53:46 VERBOSE  __________________________________________________________________________________________________
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  Layer (type)                 Output Shape              Param #
02/28/2021 07:53:46 VERBOSE  =================================================================
02/28/2021 07:53:46 VERBOSE  input_1 (InputLayer)         (None, 64, 64, 3)         0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  conv_128_0_conv2d (Conv2D)   (None, 32, 32, 128)       9728
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  conv_128_0_leakyrelu (LeakyR (None, 32, 32, 128)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  conv_256_0_conv2d (Conv2D)   (None, 16, 16, 256)       819456
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  conv_256_0_leakyrelu (LeakyR (None, 16, 16, 256)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  conv_512_0_conv2d (Conv2D)   (None, 8, 8, 512)         3277312
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  conv_512_0_leakyrelu (LeakyR (None, 8, 8, 512)         0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  conv_1024_0_conv2d (Conv2D)  (None, 4, 4, 1024)        13108224
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  conv_1024_0_leakyrelu (Leaky (None, 4, 4, 1024)        0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  flatten_1 (Flatten)          (None, 16384)             0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  dense_1 (Dense)              (None, 1024)              16778240
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  dense_2 (Dense)              (None, 16384)             16793600
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  reshape_1 (Reshape)          (None, 4, 4, 1024)        0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_512_0_conv2d_conv2d  (None, 4, 4, 2048)        18876416
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_512_0_conv2d_leakyre (None, 4, 4, 2048)        0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_512_0_pixelshuffler  (None, 8, 8, 512)         0
02/28/2021 07:53:46 VERBOSE  =================================================================
02/28/2021 07:53:46 VERBOSE  Total params: 69,662,976
02/28/2021 07:53:46 VERBOSE  Trainable params: 69,662,976
02/28/2021 07:53:46 VERBOSE  Non-trainable params: 0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  Layer (type)                 Output Shape              Param #
02/28/2021 07:53:46 VERBOSE  =================================================================
02/28/2021 07:53:46 VERBOSE  input_2 (InputLayer)         (None, 8, 8, 512)         0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_256_0_conv2d_conv2d  (None, 8, 8, 1024)        4719616
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_256_0_conv2d_leakyre (None, 8, 8, 1024)        0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_256_0_pixelshuffler  (None, 16, 16, 256)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_128_0_conv2d_conv2d  (None, 16, 16, 512)       1180160
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_128_0_conv2d_leakyre (None, 16, 16, 512)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_128_0_pixelshuffler  (None, 32, 32, 128)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_64_0_conv2d_conv2d ( (None, 32, 32, 256)       295168
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_64_0_conv2d_leakyrel (None, 32, 32, 256)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_64_0_pixelshuffler ( (None, 64, 64, 64)        0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  face_out_a_conv2d (Conv2D)   (None, 64, 64, 3)         4803
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  face_out_a (Activation)      (None, 64, 64, 3)         0
02/28/2021 07:53:46 VERBOSE  =================================================================
02/28/2021 07:53:46 VERBOSE  Total params: 6,199,747
02/28/2021 07:53:46 VERBOSE  Trainable params: 6,199,747
02/28/2021 07:53:46 VERBOSE  Non-trainable params: 0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  Layer (type)                 Output Shape              Param #
02/28/2021 07:53:46 VERBOSE  =================================================================
02/28/2021 07:53:46 VERBOSE  input_3 (InputLayer)         (None, 8, 8, 512)         0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________

02/28/2021 07:53:46 VERBOSE  upscale_256_1_conv2d_conv2d  (None, 8, 8, 1024)        4719616
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_256_1_conv2d_leakyre (None, 8, 8, 1024)        0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_256_1_pixelshuffler  (None, 16, 16, 256)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_128_1_conv2d_conv2d  (None, 16, 16, 512)       1180160
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_128_1_conv2d_leakyre (None, 16, 16, 512)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_128_1_pixelshuffler  (None, 32, 32, 128)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_64_1_conv2d_conv2d ( (None, 32, 32, 256)       295168
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_64_1_conv2d_leakyrel (None, 32, 32, 256)       0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  upscale_64_1_pixelshuffler ( (None, 64, 64, 64)        0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  face_out_b_conv2d (Conv2D)   (None, 64, 64, 3)         4803
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 VERBOSE  face_out_b (Activation)      (None, 64, 64, 3)         0
02/28/2021 07:53:46 VERBOSE  =================================================================
02/28/2021 07:53:46 VERBOSE  Total params: 6,199,747
02/28/2021 07:53:46 VERBOSE  Trainable params: 6,199,747
02/28/2021 07:53:46 VERBOSE  Non-trainable params: 0
02/28/2021 07:53:46 VERBOSE  _________________________________________________________________
02/28/2021 07:53:46 INFO     Loading Trainer from Original plugin...
02/28/2021 07:53:46 VERBOSE  Loading config: 'C:\Users\faceswap_amd\config\train.ini'
Reading training images (A):   0%|          | 0/1738 [00:00<?, ?it/s]
Reading training images (A):   0%|          | 1/1738 [00:00<03:21,  8.63it/s]
Reading training images (A):  11%|#         | 189/1738 [00:00<02:05, 12.30it/s]
Reading training images (A):  19%|#9        | 337/1738 [00:00<01:20, 17.51it/s]
Reading training images (A):  29%|##8       | 496/1738 [00:00<00:49, 24.89it/s]
Reading training images (A):  37%|###6      | 642/1738 [00:00<00:31, 35.18it/s]
Reading training images (A):  44%|####4     | 773/1738 [00:00<00:19, 49.69it/s]
Reading training images (A):  53%|#####3    | 928/1738 [00:00<00:11, 70.02it/s]
Reading training images (A):  63%|######2   | 1088/1738 [00:00<00:06, 98.19it/s]
Reading training images (A):  71%|#######1  | 1242/1738 [00:00<00:03, 136.52it/s]
Reading training images (A):  80%|########  | 1399/1738 [00:01<00:01, 188.01it/s]
Reading training images (A):  89%|########8 | 1545/1738 [00:01<00:00, 247.59it/s]
Reading training images (A):  98%|#########7| 1696/1738 [00:01<00:00, 327.14it/s]


Reading training images (B):   0%|          | 0/2389 [00:00<?, ?it/s]
Reading training images (B):   6%|5         | 133/2389 [00:00<00:01, 1152.62it/s]
Reading training images (B):  12%|#2        | 291/2389 [00:00<00:01, 1253.63it/s]
Reading training images (B):  19%|#8        | 444/2389 [00:00<00:01, 1324.55it/s]
Reading training images (B):  23%|##2       | 540/2389 [00:00<00:01, 1122.78it/s]
Reading training images (B):  28%|##8       | 675/2389 [00:00<00:01, 1181.67it/s]
Reading training images (B):  34%|###4      | 822/2389 [00:00<00:01, 1254.71it/s]
Reading training images (B):  40%|####      | 966/2389 [00:00<00:01, 1304.16it/s]
Reading training images (B):  47%|####7     | 1125/2389 [00:00<00:00, 1377.57it/s]
Reading training images (B):  54%|#####3    | 1290/2389 [00:00<00:00, 1448.36it/s]
Reading training images (B):  60%|######    | 1435/2389 [00:01<00:00, 1199.71it/s]
Reading training images (B):  67%|######6   | 1589/2389 [00:01<00:00, 1284.05it/s]
Reading training images (B):  73%|#######2  | 1734/2389 [00:01<00:00, 1328.74it/s]
Reading training images (B):  78%|#######8  | 1875/2389 [00:01<00:00, 1351.10it/s]
Reading training images (B):  85%|########4 | 2028/2389 [00:01<00:00, 1399.21it/s]
Reading training images (B):  91%|#########1| 2179/2389 [00:01<00:00, 1429.64it/s]
Reading training images (B):  97%|#########7| 2325/2389 [00:01<00:00, 1437.50it/s]
02/28/2021 07:53:50 CRITICAL Error caught! Exiting...
02/28/2021 07:53:50 ERROR    Caught exception in thread: '_training_0'
02/28/2021 07:53:50 ERROR    You have selected the Mask Type 'extended' in your training configuration options but at least one face does not have this mask type stored for it.
02/28/2021 07:53:50 ERROR    You should select a mask type that exists within your face data, or generate the required masks with the Mask Tool.
02/28/2021 07:53:50 ERROR    0 faces in side A do not contain the mask 'extended'. Available masks: {'extended', 'components'}
02/28/2021 07:53:50 ERROR    149 faces in side B do not contain the mask 'extended'. Available masks: {'extended', 'vgg-obstructed', 'components'}

Process exited.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Mask problem

Post by torzdf »

That means there are missing Extended masks on your B side, but your A side is ok. It was meant to inform you that the A side has no issue, but I realize how that might be confusing, so I will remove the output line if there are no problems with the mask.

My word is final

User avatar
230V
Posts: 20
Joined: Sun Jan 19, 2020 6:20 pm
Answers: 1
Has thanked: 7 times
Been thanked: 2 times

Re: Mask problem

Post by 230V »

correct me if i am wrong; i remember the function to get a list of the files with missing masks. it would be easy to remove the missing files instead of updating the whole 2k files and extract them.

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Mask problem

Post by bryanlyon »

Updating your files to add masks is quite easy and does not require re-extracting. You can simply use the mask tool to add the missing masks.

User avatar
lorenzosale
Posts: 25
Joined: Thu May 28, 2020 11:58 am
Answers: 1
Has thanked: 12 times

Re: Mask problem

Post by lorenzosale »

Btw, i know by reading the tutorial that i must merge the alignments files etc.. but it works like this so im afraid to get another error doing this.

No longer needed, so it looks like it's all good now.

Locked