Survey Development for Assessing Student Computing Career Intentions: Technical Report
This report provides technical documentation for psychometricians and researchers interested in developing brief, culturally sensitive computing career intentions surveys.
Nonformal computing education programs and learning opportunities conducted outside of school time provide a pathway for students, especially those from historically underrepresented groups, to enter computing careers. These programs typically evaluate their effectiveness through surveys that measure students’ attitudes about computing. Measuring students’ career intentions is more challenging, and surveys are typically long and impractical.
In 2022, WestEd used seven research-based constructs to develop a brief, culturally sensitive computing career intentions survey that measures individual, situational, and societal factors. WestEd then developed and conducted preliminary validation testing of the survey through a series of cognitive interviews and a small-scale field test with 50 respondents. The findings were reported in a 2023 paper presented at the 19th ACM Conference on International Computing Education Research (ICER).
This technical report is a companion to the 2023 ICER paper. The ICER paper describes the background and motivation for the survey design and the initial survey design and validation through interviews with leaders and participants of informal computer science education programs. This technical report contains additional unpublished details about the field testing of the survey.
Key Insights
This report contains the results of two studies:
- a large-scale field test to more robustly establish the instrument’s reliability and test for differential item functioning (DIF)
- a test of the survey’s test–retest reliability with a subsample of participants from the large-scale field test
Evaluation of the Silicon Valley Research Practice Partnership for Computational Thinking and Positive IdentityÂ
In 2016, former President Barack Obama endorsed expanded funding for computer science education in the nation’s secondary schools. State agencies, industry leaders, and educators embraced the challenge eagerly, knowing the future health of the U.S. economy would rely on a workforce that is equipped with skills in computer science and computational thinking. Despite the need, inequities exist between who is receiving high-quality computer science instruction and who is not.
In California, the Silicon Valley Research Practice Partnership for Computational Thinking and Positive Identity for Computer Science (SVRPP) felt that cultural stereotypes were prohibitive in encouraging underrepresented female students to pursue a career in computer science or engineering. The SVRPP received a Computer Science for All grant from the National Science Foundation for April 2021 through March 2024. This 3-year grant provided the opportunity for the SVRPP to form a research and practice partnership in order to increase computational thinking and positive identity for Latina students in grades 4 and 5.
This report covers the impact of the SVRPP during the 2022–23 school year and the first semester of the 2023–24 school year:
- Computational thinking: The SVRPP had a statistically significant impact on Latina students’ skill and understanding of abstraction in grades 4 and 5 during both school years.
- Positive identity: During the 2023–24 school year, Latina students showed a statistically significant growth in their perceptions of culturally responsive pedagogy and their identification with math. During the 2022–23 school year, no statistically significant growth was observed in Latina students’ ratings in these two areas.
- Assessment performance: Latina students taught by SVRPP teachers scored higher on the Smarter Balanced Assessment Consortium standards-based math assessment than did Latina students taught by non-SVRPP teachers. This difference, while not statistically significant, shows a positive trend in academic achievement that may be influenced by enriched educational support.
- Teachers’ knowledge: During the 2022–23 school year, teachers’ culturally responsive pedagogy and computational thinking knowledge improved from the start of their participation in the SVRPP to after their participation in the SVRPP for one school year. The difference was statistically significant. During the 2023–24 school year, a similar trend was observed, though not statistically significant.
Understanding Your Child’s Smarter Balanced Test Results and Supporting Learning in Grades 3-5
This webinar is designed to help parents of students, grades 3-5, gain a deeper understanding of California’s new statewide testing system, what the test scores mean, and how to support their child’s learning.
Who Will Benefit
- Parents of students in grades 3-5
What You Will Learn
- How computer adaptive testing works to find a child’s ability level
- How to support your child’s success on the new tests
- How teachers are changing instruction to help students deepen subject matter understanding, think more critically, and apply what they have learned in new ways
Other Webinars in This Series
Understanding Your Child’s Smarter Balanced Test Results and Supporting Learning in Grades 6-12
This webinar is designed to help parents of students, grades 6-12, gain a deeper understanding of California’s new statewide testing system, what the test scores mean, and how to support your child’s learning.
Who Will Benefit
- Parents of students in grades 6-12
What You Will Learn
- How computer adaptive testing works to find a child’s ability level
- How to support your child’s success on the new tests
- How teachers are changing instruction to help students deepen subject matter understanding, think more critically, and apply what they have learned in new ways
Other Webinars in This Series
The Achievement Progress of English Learner Students in Arizona
With states increasingly committed to better understanding and supporting the academic progress of their English learner students, this REL West study offers insights into how these students progressed in the state of Arizona over six school years.
This study followed three cohorts of English learner students in kindergarten, grade 3, and grade 6 from 2006/07 through 2011/12 to assess their English proficiency and track their academic progress in English language arts and math content knowledge.
Key Findings
- English learner students in lower grades did better than students in higher grades on subject matter tests
- English learner students who were eligible for special education services generally had the lowest passing rates on all tests
- English learner students who started the study at lower English language proficiency (ELP) levels generally had lower passing rates on all tests than students at the higher ELP levels
- English learner students who were eligible for the school lunch program scored lower on all tests than their peers who were not eligible
- Male English learner students had lower passing rates than their female peers on the English language proficiency and English language arts tests, but scored about the same — mostly slightly higher and sometimes slightly lower — on the math test
The Beaverton School District Arts for Learning (A4L) Lessons Project, An Investing in Innovation (i3) Development Grant: Student Impact Findings from Years 1, 2, and 3
WestEd’s Evaluation Research Program recently completed a study that evaluated the impact of the Arts for Learning (A4L) Lessons supplemental literacy curriculum on the literacy and life skills of elementary students.
A4L Lessons are designed to integrate learning science with the creativity and discipline of the arts to improve student achievement in reading and writing, as well as life skills.
Over a three-year period, researchers assessed the impact of A4L Lessons using a cluster-randomized trial in the Beaverton School District in Oregon. Researchers randomly assigned 32 elementary schools to receive the A4L intervention or the status-quo control condition with approximately 5,700 students in grades 3, 4, and 5 in 16 intervention schools receiving the A4L Lessons.
Researchers then compared achievement on the Oregon Assessment of Knowledge and Skills (OAKS) Reading/Literature test (i.e., the state reading test) for the two groups of students. Researchers also compared achievement on the Comprehensive Cross Unit (CCU) Assessments (i.e., writing tests designed specifically to measure the impact of the A4L Lessons on literacy and life skills) for a subset of six intervention and six control schools.
Key Findings
- There were no statistically significant impacts of the A4L Lessons intervention on students’ achievement on the OAKS Reading/Literature test.
- The differences between the treatment and control students on the OAKS Reading/Literature test were very small after one, two, and three years of program participation.
- Only treatment students in grade 4 consistently scored reliably higher than control students on the CCU Assessment, indicating a positive impact of A4L on student literacy and life skills; the effect sizes indexing the differences ranged from 0.30 to 0.36 across study years.
Although the study did not show A4L improved students’ scores on standardized tests, the curriculum holds promise for improving literacy and life skills as assessed by the CCU Assessments. Further research is needed to determine whether the findings would replicate in other school districts or different learning contexts.
The Achievement Progress of English Learner Students in Nevada
With states increasingly committed to better understanding and supporting the academic progress of their English learner students, this REL West study offers insights into how these students progressed in the state of Nevada over six school years.
The study followed cohorts of English learner students — kindergarten, grades 3, and grades 6 — in Nevada’s two largest school districts, Clark County School District and Washoe County School District, from 2006/07 through 2011/12. Researchers assessed English learners’ progress in English proficiency and reading and math content knowledge.
Key Findings
- English learner students in lower grades did better than students in higher grades on subject matter tests
- English learner students who were eligible for special education services generally had the lowest passing rates on all tests
- English learner students who started the study at lower English language proficiency (ELP) levels generally had lower passing rates on all tests than students at the higher ELP levels
- English learner students who were eligible for the school lunch program scored lower on all tests than their peers who were not eligible
- Male English learner students had lower passing rates than their female peers on the English language proficiency and English language arts tests, but scored about the same — mostly slightly higher and sometimes slightly lower — on the math test
The Achievement Progress of English Learner Students
This document provides an overview of three REL West reports that examine the academic achievement of English learner students in kindergarten, grade 3, and grade 6 from 2006/07 through 2011/12 in Arizona, Nevada, and Utah.
The reports focus specifically on students’ progress in English fluency and in mastering core academic content in English language arts and math.
Key Findings
- English learner students in lower grades did better than students in higher grades on subject matter tests
- English learner students who were eligible for special education services generally had the lowest passing rates on all tests
- English learner students who started the study at lower English language proficiency (ELP) levels generally had lower passing rates on all tests than students at the higher ELP levels
- English learner students who were eligible for the school lunch program scored lower on all tests than their peers who were not eligible
- Male English learner students had lower passing rates than their female peers on the English language proficiency and English language arts tests, but scored about the same — mostly slightly higher and sometimes slightly lower — on the math test
The Achievement Progress of English Learner Students in Utah
With states increasingly committed to better understanding and supporting the academic progress of their English learner students, this REL West study offers insights into how these students progressed in the state of Utah from 2006/07 through 2011/12.
This study followed cohorts of English learner students — kindergarten, grades 3, and grades 6 — to assess their progress in English proficiency and English language arts and math content knowledge.
Key Findings
- English learner students in lower grades did better than students in higher grades on subject matter tests
- English learner students who were eligible for special education services generally had the lowest passing rates on all tests
- English learner students who started the study at lower English language proficiency (ELP) levels generally had lower passing rates on all tests than students at the higher ELP levels
- English learner students who were eligible for the school lunch program scored lower on all tests than their peers who were not eligible
- Male English learner students had lower passing rates than their female peers on the English language proficiency and English language arts tests, but scored about the same — mostly slightly higher and sometimes slightly lower — on the math test
Making the Most of the Governor’s Budget Proposal: Recommendations for Aligning Reporting Requirements From the LCFF Test Kitchen
In the January 2019–20 State Budget Proposal, California Governor Gavin Newsom introduced a K–12 education spending plan that intends to make significant investments in the Local Control Funding Formula (LCFF), special education, state preschool, and partial relief for pension liabilities.
In addition, it includes a significant investment to plan and develop a longitudinal data system that would link student data from early education providers and K–12 systems all the way through institutions of higher education, employers, and health and human services agencies. The stated intent of the data system is to improve coordination across multiple, disparate data systems, improve collaboration between agencies, and collect relevant data on public education and its impact on workforce capacity.
Drawing from the experiences of the LCFF Test Kitchen — a joint project of the California Collaborative for Educational Excellence, Pivot Learning, and WestEd designed to foster innovation in local school districts as they implement LCFF — this brief offers four recommendations for designing a single, web-based reporting platform for K–12 education agencies through a process that values end users and fosters innovation. They include:
- Articulate the goals and desired outcomes of a single web-based reporting platform that align to reporting structures
- Engage end users throughout the development process
- Foster competition to generate an innovative, single web-based reporting platform design
- Create structures and supports to build trust between school districts and their communities
About the LCFF Test Kitchen
The LCFF Test Kitchen is a joint project of the California Collaborative on District Reform, the California Collaborative for Educational Excellence, Pivot Learning, and WestEd. Visit the LCFF Test Kitchen website for more information.