Solving, Explaining, and Generating University STEM Course Questions at Scale

Solving, Explaining, and Generating University STEM Course Questions at Scale (via Zoom)

Abstract: We automatically solve, explain, and generate university-level course problems from thirty STEM courses (at MIT, Harvard, and Columbia) for the first time. We curate a new dataset of course questions and answers across a dozen departments: Aeronautics and Astronautics, Chemical Engineering, Chemistry, Computer Science, Economics, Electrical Engineering, Materials Science, Mathematics, Mechanical Engineering, Nuclear Science, Physics, and Statistics. In order to test the quality of our machine generated questions, we generate new questions and use them in a Columbia University course, and perform A/B tests demonstrating that these machine generated questions are indistinguishable from human-written questions and that machine generated explanations are as useful as human-written explanations, again for the first time.

Our approach consists of five steps:

1. Given course questions, turn them into programming tasks;
2. Automatically generate programs from the programming tasks using a Transformer model, OpenAI Codex, pre-trained on text and fine-tuned on code;
3. Execute the programs to obtain and evaluate the answers; 
4. Automatically explain the correct solutions using Codex;
5. Automatically generate new questions that are qualitatively indistinguishable from human-written questions.

This work is a significant step forward in applying machine learning for education, automating a considerable part of the work involved in teaching. 

Our approach allows personalization of questions based on difficulty level and student backgrounds, and scales up to a broad range of courses across the schools of engineering and science.

Bio: Professor Iddo Drori is a Lecturer at MIT EECS and adjunct Associate Professor at Columbia University in Computer Science. His new book, "The Science of Deep Learning", is being published with Cambridge University Press. He was a visiting Associate Professor at Cornell University in Operations Research and Information Engineering, a research scientist and adjunct Professor at NYU Center for Data Science, Courant Institute, and NYU Tandon. He holds a PhD in Computer Science and was a postdoctoral research fellow at Stanford University in Statistics. His main research is in machine learning, AI, and computer vision, with 70 publications with over 5,000 citations. He has won multiple competitions in computer vision conferences and received multiple best paper awards in machine learning conferences.