Kavli Affiliate: Avi Shporer

| First 5 Authors: Iddo Drori, Sunny Tran, Roman Wang, Newman Cheng, Kevin Liu

| Summary:

We demonstrate that a neural network pre-trained on text and fine-tuned on

code solves Mathematics problems by program synthesis. We turn questions into

programming tasks, automatically generate programs, and then execute them,

perfectly solving university-level problems from MIT’s large Mathematics

courses (Single Variable Calculus 18.01, Multivariable Calculus 18.02,

Differential Equations 18.03, Introduction to Probability and Statistics 18.05,

Linear Algebra 18.06, and Mathematics for Computer Science 6.042), Columbia

University’s COMS3251 Computational Linear Algebra course, as well as questions

from a MATH dataset (on Prealgebra, Algebra, Counting and Probability, Number

Theory, and Precalculus), the latest benchmark of advanced mathematics problems

specifically designed to assess mathematical reasoning. We explore prompt

generation methods that enable Transformers to generate question solving

programs for these subjects, including solutions with plots. We generate

correct answers for a random sample of questions in each topic. We quantify the

gap between the original and transformed questions and perform a survey to

evaluate the quality and difficulty of generated questions. This is the first

work to automatically solve, grade, and generate university-level Mathematics

course questions at scale. This represents a milestone for higher education.

| Search Query: ArXiv Query: search_query=au:”Avi Shporer”&id_list=&start=0&max_results=10