Kavli Affiliate: Yi Zhou
| First 5 Authors: Desai Xie, Jiahao Li, Hao Tan, Xin Sun, Zhixin Shu
| Summary:
Recent advancements in the text-to-3D task leverage finetuned text-to-image
diffusion models to generate multi-view images, followed by NeRF
reconstruction. Yet, existing supervised finetuned (SFT) diffusion models still
suffer from multi-view inconsistency and the resulting NeRF artifacts. Although
training longer with SFT improves consistency, it also causes distribution
shift, which reduces diversity and realistic details. We argue that the SFT of
multi-view diffusion models resembles the instruction finetuning stage of the
LLM alignment pipeline and can benefit from RL finetuning (RLFT) methods.
Essentially, RLFT methods optimize models beyond their SFT data distribution by
using their own outputs, effectively mitigating distribution shift. To this
end, we introduce Carve3D, a RLFT method coupled with the Multi-view
Reconstruction Consistency (MRC) metric, to improve the consistency of
multi-view diffusion models. To compute MRC on a set of multi-view images, we
compare them with their corresponding renderings of the reconstructed NeRF at
the same viewpoints. We validate the robustness of MRC with extensive
experiments conducted under controlled inconsistency levels. We enhance the
base RLFT algorithm to stabilize the training process, reduce distribution
shift, and identify scaling laws. Through qualitative and quantitative
experiments, along with a user study, we demonstrate Carve3D’s improved
multi-view consistency, the resulting superior NeRF reconstruction quality, and
minimal distribution shift compared to longer SFT. Project webpage:
https://desaixie.github.io/carve-3d.
| Search Query: ArXiv Query: search_query=au:”Yi Zhou”&id_list=&start=0&max_results=3