Kavli Affiliate: Yi Zhou
| First 5 Authors: Haiyang Liu, Zhan Xu, Fa-Ting Hong, Hsin-Ping Huang, Yi Zhou
| Summary:
We present Video Motion Graphs, a system designed to generate realistic human
motion videos. Using a reference video and conditional signals such as music or
motion tags, the system synthesizes new videos by first retrieving video clips
with gestures matching the conditions and then generating interpolation frames
to seamlessly connect clip boundaries. The core of our approach is HMInterp, a
robust Video Frame Interpolation (VFI) model that enables seamless
interpolation of discontinuous frames, even for complex motion scenarios like
dancing. HMInterp i) employs a dual-branch interpolation approach, combining a
Motion Diffusion Model for human skeleton motion interpolation with a
diffusion-based video frame interpolation model for final frame generation. ii)
adopts condition progressive training to effectively leverage identity strong
and weak conditions, such as images and pose. These designs ensure both high
video texture quality and accurate motion trajectory. Results show that our
Video Motion Graphs outperforms existing generative- and retrieval-based
methods for multi-modal conditioned human motion video generation. Project page
can be found at https://h-liu1997.github.io/Video-Motion-Graphs/
| Search Query: ArXiv Query: search_query=au:”Yi Zhou”&id_list=&start=0&max_results=3