Questions regarding Color Spaces

Want to know about the Faceswap's Face Extraction process? Got tips, ideas or just want to learn about how it all works? Then this is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved for Extracting and preparing face sets for training a model in Faceswap.

If you have found a bug are having issues with the Extraction process not working, then you should post in the Extract Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
mat426
Posts: 4
Joined: Fri May 12, 2023 3:22 am
Has thanked: 3 times

Questions regarding Color Spaces

Post by mat426 »

In the newly published book it states that the color spaces of the source material should be linear ones and matching. It also states that if the content is in Rec.709 (or sRGB, which is the same i guess?) it should be kept that way.

Yet from what i searched up Rec.709/sRGB isn't a linear color space, so should there be made extra changes to the input material and if so, what would be a good way to do it? In Da Vinci Resolve I tried to change it, but it would end up in a total mess that looks really wierd.

So currently I extracted material from youtube video sources, one side is 1080p resolution, the other side 2160p including .jpg image sources from the web.
Both have the same Rec709 color space as far as the checkup-tool (from https://mediaarea.net/de/MediaInfo) says, yet the higher resolution side takes longer to train and is less detailed in the previews. Would it make sense to scale down the 2160p to 1080p? or is the problem color space related?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Questions regarding Color Spaces

Post by bryanlyon »

Rec709 is a linearizable color space. It's not exactly linear, but it's not HDR (where color values change somewhat randomly according to human perception). Most important is the consistency of the colorspace. If a person just standing in a room's face values keep changing that's hard for the model to learn. Rec709 data is perfectly acceptable to faceswap.

Youtube you do have to be careful of. There are videos on Youtube in HDR and most Youtube downloaders don't know how to handle HDR so just change the file to say it's not HDR (Rec709) while the data is still dynamic. This is the worst kind of data for training a model as it's completely dynamic and the values can change significantly even in mostly static scenes. Some people try color-grading or color mapping the HDR away, and that HELPS to recover the original non-dynamic data, but it's not perfect and you still have less than perfect data.

In the end, I'm betting either your 1080p data is just "better" (as in more variety and/or not butchered HDR) and so it works better in the training process.

User avatar
mat426
Posts: 4
Joined: Fri May 12, 2023 3:22 am
Has thanked: 3 times

Re: Questions regarding Color Spaces

Post by mat426 »

Thank you.

If the videos are originally recorded in 4k with HDR, does it get preserved when I download it in a lower res or scale it down afterwards?
And what would be a way to tell if a videofile is HDR in the first place?

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: Questions regarding Color Spaces

Post by bryanlyon »

Hdr is hard to see with your eyes to be honest. YouTube generally just crushes hdr uploads down in a poor attempt to make hdr into standard range. But honestly it's not the worst. It's unlikely that you're running into hdr content on YouTube without it being labeled as such.

I might have to create some kind of hdr detector. I'll throw it in my to-do list but it's not going to be an immediate thing.

User avatar
mat426
Posts: 4
Joined: Fri May 12, 2023 3:22 am
Has thanked: 3 times

Re: Questions regarding Color Spaces

Post by mat426 »

An update on that would be really cool.

Thanks for your help!

Locked