More Accurate Align Tool?

Discussions about research, Faceswapping and things that don't fit in the other categories here.


Locked
User avatar
pike
Posts: 2
Joined: Sun Feb 05, 2023 4:13 am
Has thanked: 2 times
Been thanked: 1 time

More Accurate Align Tool?

Post by pike »

The current align tools are not accurate for profile faces,manual editing of landmarks takes a lot of time.
I'm not a professional in this field, but I found Google face mesh seems to be very powerful.
Is there any possibility to integrate google face mesh or some similar more powerful tools in the future?
Thank you for developing this project anyway. :) :)

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 215 times
Contact:

Re: More Accurate Align Tool?

Post by bryanlyon »

Google's face mesh is unfortunately not publicly available. It's only usable through API calls on (some) Android devices.

User avatar
torzdf
Posts: 2636
Joined: Fri Jul 12, 2019 12:53 am
Answers: 156
Has thanked: 128 times
Been thanked: 614 times

Re: More Accurate Align Tool?

Post by torzdf »

Ok, alignment is a constant bugbear, as manual processing of faces is by far the most labour intensive and tedious part of the process, so anything to help with that would be an improvement.

I am constantly keeping an eye out for better methodologies, although we do have some limitations; our entire process is built around 68 point 2D landmark points, and anything which does not give us data in this way is not useable in Faceswap without a significant code rewrite (ref: @bryanlyon reply above)

However, the real issue here is the weights for the FAN model. They were trained on publicly available datasets. And whilst these sets are fine, they are not best suited for real world application use. Really we need to train up some new weights for our usecase. Why don't we do this? Time and the laborious amount of work to create an extensive data set to train the model on. It is on my to do list, but there are many, many items on my to do list.

An example of how much of an impact training on data targetted at our use case can be is in the bisenet-fp mask. The original weights are still available in Faceswap, but these were trained on the publicly available CelebA-HQ dataset. We also have the 'faceswap' weights for the same mask. This was trained on data, compiled and annotated by myself (over many, many months). The performance difference between the original weights and our custom trained weights is night and day.

My word is final

User avatar
pike
Posts: 2
Joined: Sun Feb 05, 2023 4:13 am
Has thanked: 2 times
Been thanked: 1 time

Re: More Accurate Align Tool?

Post by pike »

torzdf wrote: Sun Feb 05, 2023 3:14 pm

Ok, alignment is a constant bugbear, as manual processing of faces is by far the most labour intensive and tedious part of the process, so anything to help with that would be an improvement.

I am constantly keeping an eye out for better methodologies, although we do have some limitations; our entire process is built around 68 point 2D landmark points, and anything which does not give us data in this way is not useable in Faceswap without a significant code rewrite (ref: @bryanlyon reply above)

However, the real issue here is the weights for the FAN model. They were trained on publicly available datasets. And whilst these sets are fine, they are not best suited for real world application use. Really we need to train up some new weights for our usecase. Why don't we do this? Time and the laborious amount of work to create an extensive data set to train the model on. It is on my to do list, but there are many, many items on my to do list.

An example of how much of an impact training on data targetted at our use case can be is in the bisenet-fp mask. The original weights are still available in Faceswap, but these were trained on the publicly available CelebA-HQ dataset. We also have the 'faceswap' weights for the same mask. This was trained on data, compiled and annotated by myself (over many, many months). The performance difference between the original weights and our custom trained weights is night and day.

Good to hear that this project is still getting a boost. thanks for your work.

Locked