Not enough RAM for sorting - how to split dataset?

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
congo
Posts: 16
Joined: Mon Dec 16, 2019 3:09 pm
Has thanked: 7 times
Been thanked: 1 time

Not enough RAM for sorting - how to split dataset?

Post by congo »

Hi,

while trying to sort my extracted faces, I got the error 'Not enough RAM available to sort faces...'. The extract guide recommends splitting up the data. I split up the image files into 5 batches and put them into different directories, sorting each of them wasn't a problem. However, fixing the alignment file did only work for the first directory. When I tried to fix the alignments for the second directory, the alignment file wasn't changed. Probably all faces that aren't in the first directory were removed from the alignment file. I cannot simply copy the faces of all 5 dirs back into one dir before fixing the alignment file, since the image files have the same names and would overwrite each other (file names starting at the same number). I don't know if renaming the files before merging the directories will help. Is there a recomended way to do this?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Not enough RAM for sorting - how to split dataset?

Post by bryanlyon »

Renaming them is the correct way. You can simply add a number to the beginning of each set or you can use the alignment tool to rename them back to their original file names. Either way, only clean the alignments file AFTER you have rejoined the files into a single folder.

User avatar
congo
Posts: 16
Joined: Mon Dec 16, 2019 3:09 pm
Has thanked: 7 times
Been thanked: 1 time

Re: Not enough RAM for sorting - how to split dataset?

Post by congo »

Thanks, that did the trick. Probably this info should be included in the extract guide.
Just being curious: How does the tool know which (renamed) face image belongs to which alignment in the alignment file?

User avatar
torzdf
Posts: 2667
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Not enough RAM for sorting - how to split dataset?

Post by torzdf »

Magic!

My word is final

User avatar
torzdf
Posts: 2667
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Not enough RAM for sorting - how to split dataset?

Post by torzdf »

In all seriousness.... There has been an update to sort tool to better utilize ram, so you may want to check for updates.

We store the sha1 hash of the face, so we can track the images without filenames. This is fine, as long as you don't alter the extracted faces at all, otherwise it all comes crashing down.

My word is final

Locked