In 1924, Sidney Pressey, a professor at Ohio State University, invented an educational machine. The mechanical device, about the size of a portable typewriter, allowed students to press one of four keys to answer questions selected by an expert instructor. In the latest version, candy was distributed for correct answers.
Education optimists were fascinated, and Pressey promised that the technology would accelerate student learning. However, the machine was a commercial failure.
Exactly 100 years later, similar programs including i-Ready, DreamBox, Khan Academy, and IXL were shaking up American classrooms. They are driven by clever algorithms rather than the power of your fingers. While none offer a candy dispenser as a reward, some do provide an animation or video explaining what the student did wrong. The pandemic craze for teaching children about computers has spurred a surge in adoption of such programs.
Do they work? In August 2022, three researchers from Khan Academy, a popular math practice website, published the results of a large-scale study of students in 99 districts. For students who used the program as recommended, we found an effect size of 0.26 standard deviations (SD) (equivalent to several months of additional training).
Although it didn't benefit from Sal Khan's satin voiceover, a 2016 Harvard study on a competing math platform, DreamBox, found an effect size of 0.20 SD for students who used the program as recommended. A 2019 study of a similar program, i-Ready, reported an effect size of 0.22 SD in math for students who used the recommended program. And in 2023, IXL, another online math program, reported an effect size of 0.14 SD for students who used the program as designed.
These profits and the many other achievements reported each year are impressive. Because the use of these tools is so widespread, you could be forgiven for asking why American students aren't making impressive gains in math achievement. “I was impressed by the fact that educational technology had no effect on humans,” says MIT neuroscientist John Gabrieli. . . result.” He was talking about reading, but he could equally have been referring to math, another big area where educational technology is widely used but where achievement growth has not occurred.
There is a clue in the absurd phrase ‘students who used the referral program’. How many students are there? do Use this program as recommended. At least 30 minutes a week for Khan Academy? If answers are reported, they are usually buried in footnotes. In the case of Khan Studies, it is 4.7% of students. The proportion of students using other prescribed products was similarly low.
Imagine a doctor prescribing a sophisticated new drug to 100 patients, only to discover that 95 of them did not take the medication as prescribed. This is precisely the context in which many online math interventions in K-12 education occur today. It's a solution for the 5%. The remaining 95% get minimal benefit.
Worse, some studies report that the 5% that do check the results are biased toward higher-income, higher-achieving students. A 2022 study of Zearn, another math learning platform in Washington, D.C., found that in public schools, students who used the program the most were more likely to be white or Asian, from higher-income areas of the city, and more likely to be considered for college. We found that it was less. danger. (Other studies, including the Khan Academy, do not show any particular patterns across student groups.) While it is welcome that any group of students gains learning benefits, only 5% of learners achieve strong results from these programs. % might also give you the following result: You can achieve the same powerful results with any exercise program, including paper and pencil exercises. At a minimum, district leaders who adopt online learning programs to close math equity gaps should recognize that the gap may widen.
It's not at all clear that the program vendor is at fault, any more than it is blaming pharmaceutical companies for the lack of results in patients who didn't take their medications. In fact, vendors point to data showing that students who use their programs more perform higher. But it's a correlation. Hilary Yamtich, a fourth-grade math teacher at a school in Oakland, California, says she has conducted her own research, and she notes that “students who are more motivated to learn are more likely to choose Khan.”
There may be reasons other than motivation why some students use these programs to such varying degrees. One possibility is that some teachers are more committed to program implementation than others. For example, if you choose the program yourself. And Sarah Johnson of Teaching Lab, a nonprofit focused on teacher coaching, says, “Programs that are carefully integrated into the curriculum rather than viewed as complementary to programs are likely to see more consistent usage patterns.” A Harvard study on DreamBox found that changes in student use were driven more by “teacher and school-level practices” than by “student preferences.”
The second theory focuses on student behavior. Perhaps some students use the program at home while others do not. The pandemic has forced many school districts to address gaps in technology access, but not all students have parents who are pushing them to do their homework or have time to help them with it.
As Yamtich said, other students may simply be more motivated to do well in math or more eager to follow their teacher's instructions. Another study by Zearn found that high-usage students were more likely to believe their math skills could be improved. Researchers call this attitude a “growth mindset.” The researchers concluded that although using their program led to better ways of thinking, the causal arrow could equally well point in the opposite direction. In other words, students with a growth mindset invest more time to improve.
Third, says Stacy Marple, a researcher at WestEd who has studied several online programs, the programs may have unintentionally been designed to be better suited to high achievers. She Marple tells the story of how she observed an online program in her seventh-grade classroom that asked her students to “separate variables into equations using the principles of equality and inverse operations.” The student clicked on each possible answer in turn. “You know what? [the question] “Are you asking this?” Marple asked the student. “Well, not really.” she answered. (Your correspondent is equally unconvinced.) The program did not provide a way for students to find the meaning of words like “quarantine.”
In other classrooms, students were provided with video explanations of concepts they were struggling with. However, few students watched the video. This is because watching the video was considered a “hint” in the program and deducted points from your score. As a result, students may end up having to repeat problems set from the beginning. Instead, what is intended to help students makes them feel like birds in a box, which is what they most want to avoid.
Whatever the reason for low usage (and it is likely a combination of all of these), schools should assume that the impact of their online learning programs will be limited unless they take steps to ensure that the students who need them most get the recommended doses. . This is especially important because, as Ken Koedinger, an educational technology expert at Carnegie Mellon University, points out, there is compelling evidence that the amount of practice a student has has a direct impact on learning. For example, one initiative in Texas recognized this, providing grants only to districts that submitted a plan for achieving fidelity to their learning programs and a method for tracking it. At a minimum, school districts should get into the habit of looking at usage data from these platforms.
Since schools pay for these programs, it would be fair for taxpayers to ask whether their money is being wasted. This may result in schools contracting with suppliers and paying only for the time they use. “Better yet, pay only for improved student achievement,” says Raymond Pierce, president of the Southern Education Foundation, a nonprofit founded to improve educational access. This certainly seems to focus the minds of supplier executives, who can expect to be disappointed with current performance.
School districts may have been lulled into a false sense of security by research reports published by vendors. Federal regulations governing the use of Title I funds by low-performing schools require that they purchase only interventions that have evidence of effectiveness for a sample of at least 300 students. But it says nothing about what percentage of the 300 students should represent the student body. The pressing question is not whether the tool is effective, but for whom. 100 years have passed since Fresh, and we still don't know.