VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresence [SIGGRAPH Asia 2024] 

We introduce VOODOO XP: a 3D-aware one-shot head reenactment method that can generate highly expressive facial expressions driven by an input video from a single 2D portrait. Our approach is real-time, view-consistent, and can be instantly used without calibration or fine-tuning. We demonstrate our solution in a monocular video setting and an end-to-end VR telepresence system for two-way communication. Compared to 2D head reenactment methods, 3D-aware approaches aim to preserve the identity of the subject and ensure view-consistent facial geometry for novel camera poses, which makes them suitable for immersive applications. While various facial disentanglement techniques have been introduced, cutting-edge 3D-aware neural reenactment techniques still lack expressiveness and fail to reproduce complex and fine-scale facial expressions. We present a novel cross-reenactment architecture that directly transfers the driver’s facial expressions to transformer blocks of the input source’s 3D lifting module. We show that highly effective disentanglement is possible using a new multi-stage self-supervision approach. It relies on a coarse-to-fine training strategy, which is combined with explicit face neutralization and 3D lifted frontalization during its initial training stage. We further integrate our novel head reenactment solution into an accessible high-fidelity VR telepresence system, where any person can instantly build a personalized neural head avatar from any photo and bring it to life using the headset. Furthermore, our proposed method demonstrates state-of-the-art expressiveness and likeness preservation on diverse subjects and capture conditions.
Video: • VOODOO XP: Expressive ...
Project URL: mbzuai-metaver...
Paper: arxiv.org/abs/...
#mbzuai #ethz #vinai #pinscreen #facialreenactment #voodoovr #volumetric #portrait #reenactment #oneshot #headreenactment #computervision #neuralradiancefield #transformer #avatar #singleview #cvpr #disentanglement #virtualreality #telepresence #virtualreality #digitalhuman #virtualhuman #oculusquest #siggraphasia #2024