Ruobing Zheng
Paper download is intended for registered attendees only, and is
subjected to the IEEE Copyright Policy. Any other use is strongly forbidden.
Papers from this author
A Neural Lip-Sync Framework for Synthesizing Photorealistic Virtual News Anchors
Ruobing Zheng, Zhou Zhu, Bo Song, Ji Changjiang
Auto-TLDR; Lip-sync: Synthesis of a Virtual News Anchor for Low-Delayed Applications
Abstract Slides Poster Similar
Lip sync has emerged as a promising technique to generate mouth movements from audio signals. However, synthesizing a high-resolution and photorealistic virtual news anchor with current methods is still challenging. The lack of natural appearance, visual consistency, and processing efficiency is the main issue. In this paper, we present a novel lip-sync framework specially designed for producing a virtual news anchor for a target person. A pair of Temporal Convolutional Networks are used to learn the seq-to-seq mapping from audio signals to mouth movements, followed by a neural rendering model that translates the intermediate face representation to the high-quality appearance. This fully-trainable framework avoids several time-consuming steps in traditional graphics-based methods, meeting the requirements of many low-delay applications. Experiments show that our method has advantages over modern neural-based methods in both visual appearance and processing efficiency.