Students Learned Twice As Much With AI Tutor Than Typical Lectures

We may be on the verge of a revolutionary breakthrough in education where specifically designed generative AI tutors are as essential as laptops and smartboards.

According to a new study from a team of Harvard University researchers, genAI models trained on best teaching practices may usher in a wave of adaptive bots catering to students’ individualized pace and preferred style of learning.

AI Tutor Engineered With Best Teaching Practices

Gregory Kestin, PhD, lecturer on Physics, associate director of Science Education in the Faculty of Arts and Sciences and Kelly Miller, PhD, senior lecturer on Applied Physics in the School of Engineering and Applied Science are lead co-authors of the study.

In an email exchange with both, Kestin explained that the study was driven to explore the gap that exists between current in-class teaching methods and more personalized learning strategies — which have proven benefits for student performance.

“Given the lack of empirical evidence of using an AI tutor that behaved like a seasoned instructor (e.g. with well-structured lessons), this research was necessary to test the viability of AI tutors in enhancing learning outcomes and to push the boundaries of current educational models,” he wrote.

Past attempts at evaluating the use of AI within educational settings had mixed results. The current study attributed those shortcomings to the underdesign of the genAI models used in those earlier trials. To avoid similar issues, Kestin’s team carefully engineered its AI tutor using the following best teaching tactics:

• proactively engaging the student in the learning process
• managing information overload
• supporting and promoting a growth mindset
• moving from basic to complex concepts, while preparing for future units
• giving the student timely, specific and accurate feedback and information
• while enabling the learner to set their own pace

“A wealth of education research shows that active learning — pedagogies involving instructional activities that engage students beyond passive listening and memorizing — is significantly more effective than traditional lectures,” wrote Miller.

“If we wish to learn while using AI tutors, we need to be actively engaged. I thought that a new framework — beyond general chat bots like ChatGPT — was needed to make the lessons and the AI tutor to adhere to well-established research-based pedagogical principles,” Kestin added.

AI Tutored Students Learned More In Less Time

The study enrolled 194 undergraduate Harvard physics students split into two groups and was conducted over a two week period. Each participant experienced a different teaching method each week. In week one, the first group had the AI tutor lesson from their dorm, while the second group attended a typical instructor-led lecture on campus. The following week, the content and conditions were flipped.

The findings showed that the AI tutored students learned more than twice as much as when they engaged with the same content during the lecture.

“Another unexpected finding was the magnitude of learning gains achieved in a shorter time with AI tutoring. Students learned more effectively without spending additional time on task. Interestingly, there was no correlation between the amount of time spent and performance on the test of learning, indicating that personalized pacing was a key driver of success,” Kestin wrote.

AI Tutors Are Resources Not Replacements For Teachers

The researcher went on to explain that he believes these results are largely due to the personalized on-demand design of the AI tutor, giving students greater control over the learning experience.

“This allowed them to ask as many questions as they wished and address their personal points of confusion in a short period of time. Self-pacing meant that students could spend more time on concepts they found challenging and move quickly through material they understood, leading to more efficient learning,” he added.

Kestin stressed that this research is not a call to replace real-world teachers and in-classroom learning with AI tutors. While more research is necessary and underway, he said AI tutors could be a beneficial resource to advance learning and lighten the load of educators.

“Teachers can also be more informed about their students’ understanding before class even begins. With AI-generated summaries of students’ pre-class interactions with AI tutors, educators can know where each student stands and what gaps in understanding they might have, and they can tailor their instruction accordingly,” Kestin stated.