The Cultural Validity in Assessment (CVA) project investigated how cultural background influences the ways in which students interpret science and mathematics assessment items and the cognitive activities they use in completing those items.
To carry out our investigation, we administered science and mathematics test items to a sample of students from 10 cultural groups and geographical areas in the United States.
The test items were used in the past in large-scale testing (specifically, the National Assessment of Educational Progress) and were deemed psychometrically sound.
In addition, we worked individually with a subsample of students for each cultural group and asked them to “think aloud” as they engaged in responding to the items. After completing each item, we asked students to explain how they interpreted it and how they related its content to their personal experience.
We examined the mean score differences between cultural groups for each item, the inferred cognitive activity of the students as they take the tests, and the students’ socio-cultural activity (participation, context, and system of values, beliefs, and epistemologies).
Based on our analysis of the data, we determined: (1) whether students from different cultural groups exhibit different patterns by which they understand the science and mathematics exercises; (2) how culture influences the inferred cognitive activity elicited by those exercises; and (3) whether those differences accounted for performance score differences among cultural groups.
Our goal was to contribute to equitable testing in science and mathematics by offering a new perspective for addressing cultural diversity in testing. We determined whether procedures for test development and evaluation should pay more attention to culture and whether the notion of cultural validity should be considered in assessment development and testing practices.