Learning Gains During the Pandemic: Math Program Shows Promise for Minnesota Students

November 9, 2022

According to the latest results from the National Assessment of Educational Progress, known as the Nation’s Report Card, learning loss from the pandemic hit the hardest in math and particularly among underserved populations. But in some places, among some groups, students’ math learning improved, even amid the pandemic. One example is in Mankato Area Public Schools in Minnesota.

Before anyone knew the COVID-19 pandemic was coming, WestEd was contracted by Art of Problem Solving (AoPS) to conduct an independent evaluation in Mankato of AoPS’s Beast Academy, a comic-based math program used with elementary and middle school-aged students. Despite adjusting the evaluation plan to accommodate the pandemic’s dramatic changes to schooling, researchers conducted a rigorous study involving more than a thousand students in treatment and comparison groups. They found that participation in Beast Academy had a significant, positive impact on students’ math achievement. Additionally, program use was associated with more positive attitudes toward math.

The study’s findings are summarized in the Evaluation of Beast Academy report. To learn more, WestEd Senior Editor Noel White asked the study’s director, Juan Carlos Bojorquez, and its lead analyst, Thomas Torre Gibney, questions about Beast Academy and the evaluation.

What is the Beast Academy math program? 

Juan Carlos Bojorquez: Beast Academy is designed to be a self-paced program that can be used as a supplement or for enrichment. It is one of several programs that Art of Problem Solving offers. Beast Academy is a comic-based, math-focused program for elementary and middle school students. The comic-based element leverages comics and characters to engage students in mastery, critical thinking, problem-solving, games, and puzzle activities while they learn math.

Although it’s usually used for enrichment, there is some indication of Mankato teachers incorporating Beast Academy into their regular curriculum. It can be used as an online or book-based program, and both modes cover the same concepts and follow the same order of units.

Your report describes Beast Academy as focusing on mastery rather than memorization. Can you say more? 

Thomas Torre Gibney: The curriculum is not about getting students to memorize math procedures and formulas. Instead, it provides engaging, rigorous tasks to build problem-solving skills. It focuses on the how and why behind math rather than memorizing what to do. As part of developing mastery, students delve deeper into math and develop proficiencies before moving on to other topics. They move at their own pace and build on what they’ve learned.

Q: What was the focus of your evaluation? 

JCB: Our evaluation looked at Beast Academy in Mankato Area Public Schools, a district in Minnesota that serves about 8,500 students. There, Beast Academy is primarily used with two groups of students: “Cluster” students, the district’s designation for traditional “gifted and talented” students, and “Rising Scholars,” the district’s designation for high-ability learners from populations that have been historically marginalized from programming for academically talented students. 

TTG: AoPS sought an external evaluator to study the efficacy of its programming. They wanted to know if it was working and, if so, for whom and under what conditions. The original design, pre-COVID, also included an implementation study with a heavier formative component. With these goals in mind while adjusting to the pandemic, we successfully conducted an impact study of math performance outcomes by matching students with a record of Beast Academy use with their peers who had similar starting levels of math performance but no exposure to the program.

We supplemented this analysis with a correlational analysis of student attitudes toward learning math, based on responses from a student survey we administered in partnership with the district. The originally planned implementation study component would have included looking at how teachers used the program, including its use online versus in-person with books. Unfortunately, anything related to our plans for an implementation study was set aside due to COVID.

Yes, so how did the pandemic change your study? 

TTG: Trying to conduct a research study during a pandemic was challenging. COVID-19 disrupted the entire education system across the country and took its toll on student learning. Most relevant to our study, it changed the mode by which students learned—moving to online or hybrid learning—and how teachers engaged students in that learning.

In our original design, we planned to gather implementation data to understand how teachers used Beast Academy. But given the new and shifting environment during those first two years of the pandemic, we—in collaboration with the district and AoPS—decided that collecting teacher use data was no longer feasible.

JCB: The district was nonetheless a great partner and critical to our shift to using and incorporating student-level data on the number of lessons attempted and total time spent doing math-related activities on the Beast Academy Online platform. Although not in our original design, these data were a valuable addition. Losing the implementation data that could reveal how teachers used the program in their classrooms was a complicating factor. But, in the end, the student use data we could access (time on the program and lessons attempted) provided valuable student exposure data. What was clear from the data was that students across the district continued to use Beast Academy. We also needed to pivot from using the state math assessment for our outcome data to using the district’s benchmark assessment, the NWEA MAP assessment.

Ultimately, our analysis indicated that despite the pandemic, its disruption to schooling and student learning, and changes to the district use of the program, students who used the program had higher achievement scores and more positive attitudes toward math than those students who didn’t use the program.

In retrospect, is there anything you would like to have done differently? 

JCB: Hindsight is always 20/20. As much as we wanted to avoid burdening teachers with additional data collection about their use of the program, those data would have been very valuable. Even though it would not have informed how teachers in the district use Beast Academy under “typical” circumstances, we would have gained insight into how they used it during a pandemic.

A pandemic is a once-in-a-lifetime event, but, in some ways, those two years resembled what low-income, disadvantaged students face all too often: living in stress-filled environments. Rising Scholar students in Mankato included some of these students, and the program had its largest impact on Rising Scholars. Insight into how Beast Academy was used during the pandemic to reach these historically marginalized students could be very powerful.

Q: Did any results of the evaluation surprise you? 

TTG: Simply finding an impact on student achievement during those first trying years of the pandemic was surprising. Teachers and students were experiencing so much stress, change, and unpredictability that we were somewhat surprised to see that Beast Academy students performed so well in math compared to their peers who did not use the program. Beast Academy students recorded larger gains and achieved at higher levels than their peers at the conclusion of the study period.

JCB: Learning loss during the pandemic has been well documented. NCES [the National Center for Education Statistics] recently released results for its NAEP [National Assessment of Educational Progress] long-term assessment for 9-year-olds. These data revealed a decrease in student scores in math; the first decrease on the long-term assessment since the 1970s. Decreases were present for students from different regions of the country, different student groups, males and females. The impact of the pandemic was felt by all students. Our study used a different assessment and student population than NAEP, of course, but the NAEP results underscore just how challenging learning conditions were during the period covered by the Beast Academy evaluation. The findings in Mankato are all the more interesting and promising in light of the national trend in math learning revealed by these NAEP results.

According to your study, both Rising Scholar and Cluster students improved more than the comparison students, but why do you think the Rising Scholars improved somewhat more, on average, than Cluster students? 

JCB: That is an interesting finding that requires further study to unpack. I think these being high-ability, historically marginalized students—who in one sense outperformed the traditionally identified “gifted” students—carries significant implications related to access and equity. This is where we see hope and potential for this programming to help address achievement gaps, particularly for underserved students.

TTG: It also underscores the importance of studying how educational interventions are delivered, in addition to examining whether they result in better outcomes. Future research on Beast Academy could help unearth how teachers of Rising Scholar students use the program to address the specific learning needs of this population, compared to how it may be used in a traditional gifted and talented classroom, for example. The Beast Academy program may have specific features that resonate with Rising Scholar students in culturally responsive or inclusive ways. These are the kinds of questions that a well-designed implementation study could really dig into. ♦

Thomas Torre Gibney is a Senior Research Associate and Project Director at WestEd. He collaborates with schools, districts, states, and other partners to help identify what works in their programs. His research focuses on policies intended to improve equitable access to educational opportunities and reduce inequalities in schools.

Senior Research Associate and Project Director Juan Carlos Bojorquez has worked on numerous evaluations at WestEd, including NEARStar, “6 to 6” Extended School Day, Beckman Scholars, Families and Communities Together, California Arts Council, National Endowment for the Arts, and Public Broadcasting Service.

Resources to Support Learning Acceleration

WestEd works to address the most pressing challenges affecting teaching and learning. Our trusted expertise can help guide learning acceleration and improve outcomes for all.

Connecting Research & Practice

WestEd operates at the intersection of research, policy, and practice, bringing the methods of systematic inquiry to bear on real-world problems. Contact us to learn more about our research work.