Page 1 of 1

Not enough RAM for sorting - how to split dataset?

Posted: Mon Dec 16, 2019 3:23 pm
by congo

Hi,

while trying to sort my extracted faces, I got the error 'Not enough RAM available to sort faces...'. The extract guide recommends splitting up the data. I split up the image files into 5 batches and put them into different directories, sorting each of them wasn't a problem. However, fixing the alignment file did only work for the first directory. When I tried to fix the alignments for the second directory, the alignment file wasn't changed. Probably all faces that aren't in the first directory were removed from the alignment file. I cannot simply copy the faces of all 5 dirs back into one dir before fixing the alignment file, since the image files have the same names and would overwrite each other (file names starting at the same number). I don't know if renaming the files before merging the directories will help. Is there a recomended way to do this?


Re: Not enough RAM for sorting - how to split dataset?

Posted: Mon Dec 16, 2019 3:47 pm
by bryanlyon

Renaming them is the correct way. You can simply add a number to the beginning of each set or you can use the alignment tool to rename them back to their original file names. Either way, only clean the alignments file AFTER you have rejoined the files into a single folder.


Re: Not enough RAM for sorting - how to split dataset?

Posted: Tue Dec 17, 2019 2:21 pm
by congo

Thanks, that did the trick. Probably this info should be included in the extract guide.
Just being curious: How does the tool know which (renamed) face image belongs to which alignment in the alignment file?


Re: Not enough RAM for sorting - how to split dataset?

Posted: Tue Dec 17, 2019 2:23 pm
by torzdf

Magic!


Re: Not enough RAM for sorting - how to split dataset?

Posted: Tue Dec 17, 2019 2:26 pm
by torzdf

In all seriousness.... There has been an update to sort tool to better utilize ram, so you may want to check for updates.

We store the sha1 hash of the face, so we can track the images without filenames. This is fine, as long as you don't alter the extracted faces at all, otherwise it all comes crashing down.