June 24, 2022
In 2021, WestEd published Anti-Racist Evaluation Strategies: A Guide for Evaluation Teams to help evaluation teams and others employ strategies for more inclusive, equitable, and ultimately effective evaluations.
The guide is divided into five sections, corresponding to the main stages of evaluation projects, and offers two to four strategies for teams to use at each stage. Three overarching themes run throughout: engaging in anti-racist self-reflection and learning; forming collaborative and equitable partnerships; and considering cultural, historical, and political contexts.
More than a dozen WestEd staff co-authored the guide, which also incorporated input from numerous others. For this Insights post, WestEd Senior Editor Noel White sought to learn more about what was behind the guide by talking with three of its authors: Rose Owens-West, Equity Project Director; Mary Rauner, Senior Research Associate; and Sylvia Kwon, Senior Research Associate.
Q: What led you and colleagues to develop this guide?
Rose Owens-West: In a multi-race, multilingual, multicultural society such as ours, there is a need to engage multiple voices and perspectives in all areas of our work. There is also a need to improve our practice to avoid causing additional harm to communities that are already marginalized or disenfranchised. We created the guide to help people examine and expand whose voices and perspectives influence evaluations.
It was also really important to have the specific focus on anti-racism, because the short- and long-term policy and program decisions based on evaluation findings have had distinct racist implications and have impacted communities of color. Those communities have frequently been the focus of evaluation studies, but the outcomes of the studies often have resulted in decisions that harmed rather than helped those communities because of implicit, institutional, and structural racism.
Q: Who did you have in mind for an audience?
Mary Rauner: Initially we conceived of the guide as being for evaluators and evaluation teams. We wanted to create something concise and focused, so we decided on one particular area of WestEd’s work and expertise—evaluation. As we developed the guide, though, we found that most of the guidance is relevant to a whole range of the work we engage in—research, technical assistance, design. We’re hopeful that people working on many different kinds of projects will find the guide helpful for deepening a commitment to equity and anti-racism.
We also want to emphasize that this guide isn’t meant to be relevant only to projects that focus explicitly on equity or anti-racism. Rather, it is about the practices, processes, and methods that we employ in all kinds of work, to help us promote diversity and inclusion in all that we do.
Q: The guide emphasizes ways to incorporate onto the evaluation team—or at least partner with—representatives of the groups most affected by what is being evaluated. Why is representation such a key point?
Sylvia Kwon: The importance of representation is closely connected to and illustrative of all three of the guide’s overarching themes: anti-racist self-reflection and learning; collaborative and equitable partnerships; and cultural, historical, and political contexts.
ROW: There is a growing demand to change how research and evaluation fields have focused on communities of color and low-income and other marginalized communities as targets of investigation, with conclusions and actions that have negatively impacted those communities.
Q: Can you say more about the context or contributions that went into developing this guide?
MR: While there is a growing body of research on anti-racist research and evaluation, I was unable to locate a straightforward guide that would prompt me to ask the right questions at the right time and help me strive to conduct anti-racist research and evaluation. I’ve found that WestEd staff members have a strong commitment to anti-racism and what that means for the content and methods in our work. In conversations with colleagues across WestEd, we decided to develop the document that we were looking for. The philanthropic sector has also been a leader in this field, and our own colleagues at WestEd’s Justice and Prevention Research Center wrote a brief for the Annie E. Casey Foundation on equitable evaluation that was a primary resource for our guide. In fact, that brief’s lead author was also a central team member in developing our guide.
Q: There are so many different kinds of people who have typically been excluded from the evaluation process, and so much variety within any group—did that variety affect how you chose to focus the guidance you provided?
ROW: I don’t think variety of groups is an issue. Let the people who are the focus or who will be most impacted by the evaluation contribute to understanding, defining, and resolving the issues. At least there should be an attempt for representation. Historically, representation has not been a principle guiding evaluations, and we are trying to make it one. Variation should not be used as an excuse to avoid representation, shared ownership, or equity. Fundamentally, we have to get away from thinking that “we” know what is better for “them” than they do!
Q: For the strategies that the guide recommends, rather than fleshing out each strategy with examples or specific steps to follow, it offers a series of questions associated with each strategy. Can you say more about this approach to providing guidance?
SK: When we were developing this guide, we wanted to avoid including checklists, directions, or specific approaches. We saw the guide as an invitation for teams to participate in discussion based on the question prompts for each evaluation stage. Through this process, we hope the questions can help teams be transparent about the choices they make and be reflective about the role of power, culture, and history in their work.
Q: The guide mentions evaluation budgets and funders a couple times. Can you say more about the role of funders in ensuring that evaluations are anti-racist?
MR: Engaging in anti-racist work takes time. Staff—individually and as a team—need time to understand personal, community, and societal assumptions about racial groups and the historical, political, and cultural contexts of the communities being served. Developing and strengthening partnerships with program staff, participants, and community members can also be time-consuming. There are monetary costs associated with all of these endeavors, with actively including partners in all stages of the work and ideally compensating them for their time. It is important for funders to provide financial support for these efforts to help evaluation teams incorporate the themes addressed in the guide.
Q: How have you used the guide in your own work?
MR: I keep it by my side when designing evaluations and writing proposals because it provides guidance for all of the project stages. During transition points on existing projects—for example, when preparing to interpret our findings—I find it helpful to revisit the relevant section of the guide to keep us on track or to rethink how we’re doing the work. Did we allocate time to reflect on the way that our biases and assumptions can influence our interpretation? Are we engaging our partners in the process?
ROW: In a number of our technical assistance projects, we try to build representative teams that we are sending out to deliver services. In data gathering, we ensure that diverse perspectives and voices, including student voices, inform our understanding of the issues, the goals, and the solutions.
Q: How would you say this guide is unique or might best be distinguished from similar kinds of guides that other organizations have developed?
SK: I think WestEd’s guide is more accessible and actionable than others. It was important to base the guide on evaluation or research stages to provide a structure for readers that resonated with their work. We wanted anyone to be able to pick it up and use it immediately to have difficult but important conversations related to the overarching themes.
ROW: Given the breadth of the field, and the number of studies that are produced every year, there are relatively few resources addressing the issues that are addressed in the anti-racist evaluation guide. We hope it will serve as a much-needed contribution to the resources available for evaluators and other researchers.