What Two New Studies Reveal About Learning Recovery

Two new studies shed light on the promise—and challenges—of learning recovery strategies that schools adopted in the wake of the pandemic.

The key theme from the research is as much a truism as it is a prescription: Getting the implementation right really matters. And it’s also difficult.

One working paper, from researchers at the American Institutes for Research, Harvard University, and the assessment provider NWEA, analyzed eight districts’ summer 2022 learning programs. Researchers found that the programs had a small but positive effect on student math scores, and no effect on reading.

Another working paper, from researchers at Vanderbilt University and Stanford University, examined tutoring programs—specifically, what conditions led to successful implementation. They identified some commonalities: Programs that functioned well had the support of school leaders and a structured process for recruiting tutors and managing logistics.

Together, the two studies underscore that academic recovery strategies require sustained, coordinated effort, said Susanna Loeb, a professor in the Graduate School of Education at Stanford, and one of the authors on the tutoring study.

“Anything that you’re going to do in schools, whether it is high-impact tutoring or a summer program, in order to have the kinds of impact that we need to get students engaged in school and learning at a faster pace, … you’re going to need a lot of effort into implementation,” Loeb said.

Giving students many and varied opportunities for extra learning is important, too, said Emily Morton, a researcher at AIR’s National Center for Analysis of Longitudinal Data in Education Research and one of the authors on the summer learning study.

The summer programs, which included time for academic instruction and enrichment activities, helped students make “marginal progress”—progress that is meaningful, she said.

“This is some of the very first good news we’re getting on any COVID recovery,” Morton said. Still, she added, “layering supports is going to be essential. Students can’t just get summer school.”

Districts have used federal COVID relief funds to support summer programs, tutoring initiatives, and other academic recovery plans—money that runs out in September 2024.

But maintaining multiple options for students requires funding and resources, Morton said. “To be able to do that, they’re going to need support from states and even from the federal government.”

Summer programs show small gains in math

The summer school study focused on summer learning programs in eight school districts. All of these districts served higher percentages of Black and Hispanic students and higher percentages of students eligible for free- and reduced-price lunch than the national average.

The researchers compared students who were eligible to attend summer school and did attend to students who were eligible but did not attend. They controlled for students’ characteristics, including race and free- and reduced-price lunch status, and their spring 2022 achievement levels.

Students who attended summer programs scored higher on an assessment of their math skills than students who did not. The researchers found a small, but positive, effect of summer learning on math achievement—0.03 of a standard deviation, which they estimated would mitigate about 2-3 percent of the total learning loss experienced in those districts. Most of this improvement was driven by upper elementary students.

The researchers found no effect for summer school in reading.

The programs did help students get extra learning time—the researchers found that each day of summer school produced academic gains roughly equivalent to each day of a regular school year. But these gains only made a tiny dent in academic recovery writ large.

Programs in the study were between 15-20 days long. On average, only 25 percent of targeted students participated. “There’s an opportunity here to scale summer school more widely, and potentially to make it longer,” said Morton.

These kinds of changes would bring programs more in line with summer learning best practices, said Jodi Grant, the executive director of the Afterschool Alliance, an advocacy organization. Grant was not involved in the study.

“Even with intense support, it’s really hard to move the needle over a short period of time,” she said.

Grant also cited research-backed recommendations that summer learning should include opportunities for enrichment activities—not only academics. Doing so can make summer programs more attractive; they shouldn’t have to feel like a “punishment,” she said.

Conditions for effective tutoring programs

Designing an effective program is the first step. Implementing that program presents a whole new set of challenges.

This piece of the puzzle—implementation—is the focus on the working paper on tutoring.

“We know this intervention is really, really effective,” said Loeb. “But what hasn’t gotten nearly as much attention is that it’s not that easy to implement.”

Researchers reviewed 33 articles published since 2000. They found strong implementation relied in large part on the buy-in of principals and school leaders.

“Education leaders served as gatekeepers of student and staff time, school space, and data/documentation on students’ skills and instructional needs,” the researchers wrote.

Having a dedicated and informed program manager was important, too, Loeb said. “You really need someone at the district level who is kind of organizing the approach, making sure that it aligns with what we can see from the results are the most promising practices,” she said.

Program managers need to be compensated for the work they’re doing, she added. That money could come from new funding, or districts could reprioritize funds. “But you definitely need somebody who has paid time to do it, and who is really responsible [for] it,” she said. “You can’t just add to people’s work without taking something away.”

Meeting conditions like these has been challenging for school districts.

The National Center for Education Statistics’ School Pulse Survey, an ongoing study of how schools are changing since the pandemic, polled school leaders in December 2022.

About a third of school leaders said they didn’t have the capacity to provide high-dosage tutoring to students who needed it. Two out of five schools said they couldn’t find time in their regular schedule for tutoring.

Still, Loeb said, better implementation is “not undoable.”

“Because it’s not easy,” she said, “we should be putting effort into thinking about the implementation process and really working on doing the implementation well.”