Kavli Affiliate: Wei Gao
| First 5 Authors: Qiyao Xue, Xiangyu Yin, Boyuan Yang, Wei Gao,
| Summary:
Text-to-video (T2V) generation has been recently enabled by transformer-based
diffusion models, but current T2V models lack capabilities in adhering to the
real-world common knowledge and physical rules, due to their limited
understanding of physical realism and deficiency in temporal modeling. Existing
solutions are either data-driven or require extra model inputs, but cannot be
generalizable to out-of-distribution domains. In this paper, we present PhyT2V,
a new data-independent T2V technique that expands the current T2V model’s
capability of video generation to out-of-distribution domains, by enabling
chain-of-thought and step-back reasoning in T2V prompting. Our experiments show
that PhyT2V improves existing T2V models’ adherence to real-world physical
rules by 2.3x, and achieves 35% improvement compared to T2V prompt enhancers.
The source codes are available at: https://github.com/pittisl/PhyT2V.
| Search Query: ArXiv Query: search_query=au:”Wei Gao”&id_list=&start=0&max_results=3