Kavli Affiliate: Ke Wang
| First 5 Authors: Weikang Shi, Weikang Shi, , ,
| Summary:
While Large Language Models (LLMs) have excelled in textual reasoning, they
struggle with mathematical domains like geometry that intrinsically rely on
visual aids. Existing approaches to Visual Chain-of-Thought (VCoT) are often
limited by rigid external tools or fail to generate the high-fidelity,
strategically-timed diagrams necessary for complex problem-solving. To bridge
this gap, we introduce MathCanvas, a comprehensive framework designed to endow
unified Large Multimodal Models (LMMs) with intrinsic VCoT capabilities for
mathematics. Our approach consists of two phases. First, a Visual Manipulation
stage pre-trains the model on a novel 15.2M-pair corpus, comprising 10M
caption-to-diagram pairs (MathCanvas-Imagen) and 5.2M step-by-step editing
trajectories (MathCanvas-Edit), to master diagram generation and editing.
Second, a Strategic Visual-Aided Reasoning stage fine-tunes the model on
MathCanvas-Instruct, a new 219K-example dataset of interleaved visual-textual
reasoning paths, teaching it when and how to leverage visual aids. To
facilitate rigorous evaluation, we introduce MathCanvas-Bench, a challenging
benchmark with 3K problems that require models to produce interleaved
visual-textual solutions. Our model, BAGEL-Canvas, trained under this
framework, achieves an 86% relative improvement over strong LMM baselines on
MathCanvas-Bench, demonstrating excellent generalization to other public math
benchmarks. Our work provides a complete toolkit-framework, datasets, and
benchmark-to unlock complex, human-like visual-aided reasoning in LMMs. Project
Page: https://mathcanvas.github.io/
| Search Query: ArXiv Query: search_query=au:”Ke Wang”&id_list=&start=0&max_results=3