Kavli Affiliate: Wei Gao
| First 5 Authors: Linxuan Xin, Zheng Zhang, Jinfu Wei, Wei Gao, Duan Gao
| Summary:
Prior material creation methods had limitations in producing diverse results
mainly because reconstruction-based methods relied on real-world measurements
and generation-based methods were trained on relatively small material
datasets. To address these challenges, we propose DreamPBR, a novel
diffusion-based generative framework designed to create spatially-varying
appearance properties guided by text and multi-modal controls, providing high
controllability and diversity in material generation. Key to achieving diverse
and high-quality PBR material generation lies in integrating the capabilities
of recent large-scale vision-language models trained on billions of text-image
pairs, along with material priors derived from hundreds of PBR material
samples. We utilize a novel material Latent Diffusion Model (LDM) to establish
the mapping between albedo maps and the corresponding latent space. The latent
representation is then decoded into full SVBRDF parameter maps using a
rendering-aware PBR decoder. Our method supports tileable generation through
convolution with circular padding. Furthermore, we introduce a multi-modal
guidance module, which includes pixel-aligned guidance, style image guidance,
and 3D shape guidance, to enhance the control capabilities of the material LDM.
We demonstrate the effectiveness of DreamPBR in material creation, showcasing
its versatility and user-friendliness on a wide range of controllable
generation and editing applications.
| Search Query: ArXiv Query: search_query=au:”Wei Gao”&id_list=&start=0&max_results=3