WestEd will be present at Evaluation 2019, the American Evaluation Association’s Annual Conference, November 11–November 16, at the Minneapolis Convention Center and the Hilton Minneapolis, in Minneapolis, Minnesota.
With this year’s theme, “Paths to the Future of EVALUATION: Contribution, Leadership, Renewal,” this conference will explore the following themes:
- How do we appreciate what we’ve contributed to the world through evaluation, and build on these successes?
- How do we show up and conduct rigorous evaluation in ways that are relevant to key issues of our time, such as climate change disasters, forced migration, gender-based violence, and shifts in resources due to economic globalization?
- How do we welcome the “new evaluators” into the profession and listen to new voices?
Evaluators, researchers, instructors, and students of evaluation are invited to join WestEd in considering the evaluation profession’s path forward. Please stop by our table in the exhibition hall (BOOTH # 18) to learn about WestEd’s evaluation-related services, research, resources, and career opportunities.
We encourage attendees to share their insights using the conference hashtag, #Eval19, and invite you to join us at the following events and presentations:
Wednesday, November 13
Poster Session: Connected Youth Initiative: A Collaborative Evaluation Study of Well-Being in Rural Nebraska
Time & Location: 6:30pm–8:30pm / Poster Room 171
Presenters & Authors: Claire M. Hoff (Nebraska Children and Families Foundation), Catherine Humphries Brown (Nebraska Children and Families Foundation), and Trevor Fronius (WestEd)
This poster session will summarize an external evaluation of the Connected Youth Initiative (CYI), which engages older youth in Nebraska who have been involved in the child welfare/juvenile justice systems, and/or who are homeless, and helps them realize positive outcomes across several domains through individual coaching, youth leadership, emergency support funds, and financial literacy training, which are all set within a community-based collaborative system. The evaluation focused on six community collaboratives in rural areas of Nebraska implementing CYI with support from the Social Innovation Fund, a program of the Corporation for National and Community Service funded 2010–2016. Overall, this project demonstrates ways to ensure evidence-building evaluation approaches remain relevant and useful to multiple audiences with varying purposes.
Poster Session: Incorporating Qualitative Data Collection and Analysis Techniques in the Context of a Randomized Control Trial: An Example from the Evaluation of Capturing Kid’s Hearts
Time & Location: 6:30pm–8:30pm / Poster Room 125
Presenters & Authors: Breauna Spencer, Ashley Boal (WestEd), and Tom Hanson (WestEd)
Randomized controlled trials (RCTs) typically focus on examining quantitative data and oftentimes limit the use of qualitative data to explore the implementation and satisfaction within the treatment group exclusively. By omitting outcome-focused qualitative data collection across both the treatment and control groups, researchers may miss opportunities to further explore outcomes of interest, and triangulate and contextualize quantitative findings. This presentation explores the value of qualitative data collection in the context of an RCT using an evaluation of Capturing Kid’s Hearts, a schoolwide program to improve school climate and culture in South Carolina, as a useful example. A content analysis of student focus group and principal interview data was conducted by evaluators blind to the treatment condition using Atlas.ti. The treatment and control sites were then compared to determine whether experiences differed in schools that did and did not receive the intervention. Findings were triangulated with quantitative impact findings.
Poster Session: Using What We Have: Using Online Tools to Facilitate Participation in a Confidential, Sensitive Project
Time & Location: 6:30pm–8:30pm / Poster Room 75
Presenters & Authors: Guadalupe-Samuel Ruiz Jimenez (WestEd and San Francisco State University), Kerry Kriener-Althen (WestEd), Emily Newton (WestEd), and Zsofia Tallai (WestEd)
As a nonprofit education research agency, it is frequently necessary to leverage available tools to piece together efficient, cost effective systems that meet the demands of our clients and do not place undue burden on participants. For a recent project with limited staff resources and a compressed timeline, we faced the challenge of organizing and sharing a lot of dense, sensitive information with workgroup participants in an online environment in a confidential and restricted way. We needed a methodology that would efficiently and effectively guide participants through a series of complex tasks that required consideration of information from numerous sources. We created an online closed-loop environment using multiple platforms from the existing business tools available through our employer. This method proved to be an effective way for efficiently guiding participants through a complex series of tasks, preparing participants for the video workgroup call, and compiling recommendations across participants.
Thursday, November 14
Session: Evaluating School Climate and Family Partnerships
Time & Location: 1:45pm–2:30pm / Room CC 200 I
Presenters & Authors: Sara Miller (WestEd) and Kerry Kriener-Althen (WestEd)
Presentation: Leave No Parent Behind: Effective Strategies for Surveying Diverse Parent Populations
This presentation summarizes an approach for surveying parents about their participation in a parent engagement program. The parent population had low incomes, were primarily Spanish speaking, and had low literacy. We needed a methodology that would yield sufficient response from a broad range of parents participating in the program, many of whom would not typically respond to a mailed or emailed survey. We successfully implemented a multi-tier strategy that involved creating the survey to be completed online or via a mobile device, such as a phone or tablet; emailing or texting the survey link to parents, enlisting support from preschool teachers and program staff to notify parents about the survey, and providing onsite survey support at the program sites.
Session: Novel Approaches and Dangers
Time & Location: 3:45pm–4:45pm / Room CC 101 C
Presenters & Authors: Janie Kimball (WestEd), Mirella Almaraz (WestEd), Zsofia Tallai (WestEd), Missy Baiocchi (WestEd), and Kerry Kriener-Althen (WestEd)
Presentation: Improving Survey Response Using Technology
A key challenge with receiving formative feedback from the field is the ability to receive information in a timely way, to minimize the amount of manual intervention necessary, and to improve upon our existing response rates. Receiving our surveys via the mail meant long delays due to how bulk mail is processed, significant manual processing efforts, and a much lower response rate than desired. In an effort to improve our survey response rate and reduce manual intervention, we effectively utilized multiple existing business tools provided by our nonprofit company. We effectively utilized Survey Monkey and Smartsheet to implement an online survey integrated with our participant database in Smartsheet. The results of our new process was an immediate increase in response rate, an increase in efficiency, a reduction in errors, and a reduction in labor and postage costs.
Friday, November 15
Roundtable Session: The STEMworks Design Principles Rubric – A Holistic Approach to Evaluation and Accountability
Time & Location: 10:30am–11:15am / Room CC 200 B
Presenters & Authors: Danielle Oberbeck (WestEd), Andrew Grillo-Hill (WestEd), Marina Varfolomeeva (WestEd), and Daniel Brenner (WestEd)
Accountability. Effectiveness. Impact. These key words all figure strongly in the evaluation landscape, but how can we measure the strength of a STEM education program beyond the traditional look at its academic outcomes? The STEMworks Design Principles Rubric forms the basis of a holistic evaluation that focuses not only on best practices for STEM education program delivery, but also the health and development of the programs’ business practices, strategic planning, sustainability, and support of their staff. In this roundtable session, we will highlight key issues addressed by the rubric, and invite attendees to discuss additional approaches.
Session: Methods and Means of Improving College Access
Time & Location: 5:45pm–6:30pm / Hilton Symphony II
Presenters & Authors: Marycruz Diaz (WestEd), Staci Wendt (WestEd), and Arena Lam (WestEd)
Presentation: Does College Affordability Advising to Low-Income Students Help Them Afford and Attend College?: Results of an RCT Study
WestEd conducted a rigorous evaluation of the uAspire Afford program, in partnership with uAspire and the San Francisco Unified School District, focused on both program implementation and the program’s impact on high school students. uAspire’s Afford program offered workshops, individual advising, and two-way texting on college affordability to high school seniors to better prepare students to consider college within their financial reach, navigate the financial aid process, and find an affordable path to college. The implementation evaluation studied the program’s implementation and the factors that challenged implementation. The impact evaluation assessed the impact of the program in increasing student understanding of and access to financial aid.
Saturday, November 16
Session: Measuring Quality of Interventions
Time & Location: 9:15am–10am / Room CC 200 I
Session Chair: Sara Miller (WestEd)
Session Presentations:
Measuring Quality of Delivery in School Interventions: What to Measure and When?
The Use of a Student Rubric to Measure Students’ 21st Century Skills