I hope i can explain this well. I’m doing a job where i have to make one talking head mimic the exact movement and lipsync to another talking head. So the end result is two talking heads identical but one of them is completely fake in a different environment.
The scenario is a web chat but the person being spoken to is not the real person. This all had to be done like this as there was a real actor and multiple takes of the web chat.
My approach has been to use Unreal face capture with a phone on set then a digital scan of the heads and to make a CGI head. ( I have help on this so it isn’t just me doing such wizardry). Am i doing this in a very long winded complex way or is there an AI approach or bit of software that people know of that will allow me to upload the two talking heads and allow one to mimic the other i guess a deepfake but the whole head. The tough bit is it isn’t just the lipsync it’s the whole head. I would love to know if anyone has done something like this before and if they have what is the best approach? I always feel i’m terrible at explaining things but in short 2 talking people on webcams in completely locations but there both identical. Thaaaank you all.
DeepFaceLab has whole head replacement, but results are very dependent on the nature and qualityof the videos you input. Hair is rarely good so transferring between two bald persons is the ideal scenario Expect quite a bit of compositing to clean stuff up.
Whole head might become challenging due to the fact of hair style and overall shape @Marcus_M. Face replacement on the other hand can be done more easily I think. I worked on a BMW commercial where the face was replaced with a deepfake. I did the “quite a bit of compositing” part as @vkastrup wrote
I can give you the contact for the deepfake/AI artist. Hit me up if you need it.
Thanks all. I feel there is a solution in here for sure that might be better than my initial approach so i’m going to dig deeper. Sinan if i fail then i’d love the deepfake chaps deets. I’ll let you know Many thanks
Only lip-sync and facial gestures? No hair replacement?
Do you need to rotate the head? If so, how much rotation?
How much movement in the shot? Or static?
How far how close from camera?
Is the footage exr linear 32bits or just a mov prores444?
There are several things to take in consideration in order to know what approach you should take.
I’ve done “deepfake” for de-aging, faceswaping, lip-sync to change dialogs, facial expressions to change the drama…and all of these has a different ways to do it.
If you just need to replace facial expressions and lipsync then DeepFaceLab is not the way to go, it is no designed to do this, not to mention it takes ages and need a lot of compute power.
Have you ever used ComfyUI? there are several ways to do this, you may need some training on the footage but may not.
Only lip-sync and facial gestures? No hair replacement?
No It’s not just face replacement. Think of it as two different actors in two completely different locations but both doing exactly the same sync and movement.
Do you need to rotate the head? If so, how much rotation?
No head rotation really, a little on the left and right but no head turns just straight talking to camera.
How much movement in the shot? Or static? Static camera on both actors as they are talking into webcams.
How far how close from camera?
The fake people will be on computer screens within a shot, lots of data eye candy etc so a little forgiving. This is why i was building as CGI heads. Wire frame models etc to help sell …
Is the footage exr linear 32bits or just a mov prores444? Final main plate is EXR but the actor plates are iphone for one and quicktime DNXHD
I did do a little test using the motion vectors of the hero plate actor and locking the duplicate actors face on. It actually was not to bad but not great and would need a tonne of clean up which is cool.
Expect quite a bit of compositing to clean stuff up.