The internet is plastered with AI clickbait promising you 10x productivity gains if you use the right ChatGPT prompts. Is there any truth to the hype? Researchers assigned 194 people in a Harvard physics class to two groups. One group went to in-person classes. The other group practiced their learning at home with questions generated by an AI tutor. Then the groups switched, to evaluate whether learning with AI or in-class produced better outcomes. The study found that students learned “significantly” more at home with an AI tutor than they did in class, even when they spent less time learning. People spent a median of 49 minutes with the AI tutor (less than the 60-min class), yet still outpaced classroom learning. From the study authors: "Providing timely feedback that targets the specific needs of individual students (vi) and self-pacing (vii) are difficult to achieve and impossible to maintain in a typical classroom. We believe that the increased learning from structured AI tutoring is largely due to its ability to offer personalized feedback on demand—just as one-on-one tutoring from a (human) expert is superior to classroom instruction15." Some notes / clarifications for the video: 1. 233 people were enrolled in the class but only 194 were ultimately eligible for the study 2. This was a crossover study where the "AI group" and "in class" groups switched such that the outcomes with AI tutoring could be compared to those with in class feedback. 3. The researchers controlled AI prompts such that the generated questions were the same as those given to the in-class group (The difference between the groups was AI administered feedback vs. peer and teacher feedback.)
From Big Think
