Despite the recent emergence of test optional and/or test flexible programs, wherein students do not need to, or can decide whether they want to submit their standardized test scores such as SAT/ACT for admission consideration (Belasco, Rosinger, & Hearn, 2014; Syverson, 2007), these standardized admission tests continue to play an important role in college choice, access, and admission decisions. These tests are also typically used as inputs for indices that predict students’ likelihood of continuation and eventual graduation. In addition, states with generous merit-based grants and scholarships (such as Tennessee and Georgia) use these tests as an important component of eligibility to receive this financial award. Taken together then, these facts indicate that standardized test scores do not only impact access to selective institutions, but also play a prominent role in financial-aid eligibility, hence impacting the academic prospects of hundreds of thousands of high school graduates every year.
Standardized admissions tests originally intended to provide a “‘common currency’ that allows admissions officers … to place students on the same footing” (Garvey, 1981, p. 1) when applying to college, regardless of demographic characteristics such as race, ethnicity, or socioeconomic status. Nonetheless, there have been concerns of bias that result in unexplained differences in standardized test results among test-takers who have similar ability levels and only differ by socio-demographic grounds (Gierl & Khaliq, 2001; Kruse, 2016). These unexplained differences are commonly referred to as cultural bias (Kruse, 2016) and have been attributed to problems in the design and interpretation of tests (Brown, Reynolds, & Whitaker, 1999; Kruse, 2016).
Dr. Manuel S González CanchéIn the design part, the prevalence of designers of White, well-educated, middle/upper middle backgrounds arguably led to the development of questions and or tasks that reflected ethnic and racial attributes of similar test-takers’ cosmovision rather than those of non-White and less privileged backgrounds who have been traditionally referred to as underserved test-takers (e.g., minoritized, first-generation, and underrepresented in higher education students). This flaw in the design-side translated into underserved test-takers being more likely to attain lower scores than their White and/or more privileged counterparts given racial and cultural pre-dispositions that went over and above differences in scholastic aptitudes. In the interpretation phase, reasoning skill tasks, and or analogies addressed by underserved test-takers yielded lower scores given the evaluators’ (and or designers serving as evaluators) cultural and/or racial pre-conceptions or cosmovisions that, once more, had no relation with underserved test-takers scholastic abilities. This negative loop or vicious circle explains that “concerns of equity and social justice for the students taking these tests have risen” (Kruse, 2016, p. 23) in past decades.
These concerns and criticisms motivated research aiming to surpass such potential biases (Freedle, 2003). For example, in 2005 the College Board decided to redesign the SAT to focus more on students’ academic preparation than on reasoning skills and removed analogies and quantitative-comparison problems (Epstein, 2009) that could be affected by these cultural differences. Notably, these changes led to today’s standing of these tests as being considered robust to these problems, which incidentally also led to their increased weight and presence in access and merit aid-eligibility, as evidenced by their continued use as “unbiased” independent predictors of scholastic ability, aptitude, and eventual academic success.
Have We Transcended Cultural Bias in Standardized Testing?
The framework of concentrated advantages or disadvantages based on individuals’ location (Elijah, 1990; Jargowsky & Tursi, 2015; Pacione, 1997) argues that individuals’ prospects for mobility are affected by their places of origin given the divergent forms of access (or lack of access) to resources. Although this framework is frequently employed in disciplines and fields of study, such as sociology, urban studies, geopolitics, it has yet to be critically utilized to assess the extent to which variation in standardized scores occurs as a function of test-takers’ proximal location. From this view, methodological perspectives such as spatial analyses employed in conjunction with this conceptual framework of concentrated advantages/disadvantages may prove useful to assess and problematize factors directly affecting the social reproduction of inequality given participants’ location.
What is Gained by Looking in the Right Space?
We should care about spatial dependence of standardized admission scores because if such scores are not robust to these dependence problems, they would not only be capturing scholastic aptitude but also test-takers’ common exposure to spatially contextualized situations that impact their greater or lower likelihood of success in these tests. For example, using these conceptual lenses, students are assumed to systemically and systematically be affected by their attendance in the same school districts and schools, exposure to similar levels of localized crime, violence, unemployment, and poverty rates.
In sum, standardized test scores co-variation is worrisome for these scores are assumed to have become robust to cultural biases, but location or geographical bias, and its relationship with cultural bias have been grossly overlooked in the literature. From this perspective then, it follows that given the concentration of inequality, these standardized scores could also be capturing racial and cultural biases taking place in spaces overwhelmingly occupied by minoritized students who face similar levels of crime, unemployment, and poverty.
Hypotheses Testing
Relying on one cohort of first-time 2006 test-takers (N=4,632) who enrolled at a Research I university several analytic samples were created for sensitivity and robustness checks. The goal of each analytic sample was to identify test-takers who lived nearby and address whether their outcomes co-varied given this proximity (See Figures 1 to 4).
All identification strategies consistently indicated the presence of spatial dependence with a more marked problem in the Mathematics than in the Verbal subject Area. An important caveat of the analyses is the age of the data, analyses with more recent estimates are recommended.
Conclusion
By looking in the right space, it then becomes clear that to the extent that location captures cultural traits, the problem of cultural bias in testing is far from being over but it is masked through spatial dependence, hence resulting in geographical bias. As such any use of these standardized scores as predictors of students’ performance will continue to yield biased estimates. Their variation is not only capturing students’ scholastic aptitudes but also the effect of their localized contexts. In short, their use in admissions decisions and financial-aid eligibility will continue to fuel social inequality.
Unfortunately, there is evidence indicating that test optional and test flexible policies are not enough to make student bodies more diverse. This letter does not offer solutions, but rather highlights a problem that is not solved yet.
Dr. Manuel S. González Canché is an associate professor at the Higher Education Division of the University of Pennsylvania’s Graduate School of Education. He relies on analysis of big and geo-referenced data, geographical network analysis, and quasi-experimental design to study factors affecting low-income and “at-risk” students’ likelihood of success with the purpose of identifying academic and socioeconomic gap-closing strategies.
Figure 1. 1-mile radius
Figure 2. 5-mile radius
Figure 3. Closest neighbor
Figure 4. Five closest neighborsReferences
Belasco, A.S., Rosinger, K.O. & Hearn, J.C. (2014). The test-optional movement at America’s selective liberal arts colleges: A boon for equity or something else? Educational Evaluation and Policy Analysis, 20(10), 1-18.
Elijah, A. (1990). Streetwise: Race, class, and change in an urban community. Chicago: University of Chicago.
Epstein, J. (2009). Behind the SAT-optional movement: Context and controversy. Journal of College Admission, 9-19.
Freedle, R.O. (2003). Correcting the SAT’s ethnic and social-class bias: A method for reestimating SAT scores. Harvard Educational Review, 73, 1-43.
Garvey, J.E. (1981). Testing for college admissions: Trends and issues. Arlington, VA: Educational Research Service, Inc.
Gierl, M. J., & Khaliq, S. N. (2001). Identifying sources of differential item and bundle functioning on translated achievement tests: A confirmatory analysis. Journal of Educational Measurement, 38(2), 164-187.
Jargowsky, P. A., & Tursi, N. O. (2015). Concentrated disadvantage. In J. D. Wright (Ed.), International encyclopedia of the social and behavioral sciences (second edition) (Second
Edition ed., p. 525 – 530). Oxford: Elsevier. Retrieved from https://www.sciencedirect.com/science/article/pii/
Kruse, A. J. (2016). Cultural bias in testing: A review of literature and implications for music education. Update: Applications of Research in Music Education, 35(1), 23-31.
Pacione, M. (1997). The geography of educational disadvantage in Glasgow. Applied Geography, 17(3), 169–192.
Syverson, S. (2007). The role of standardized tests in college admissions: Test-optional admissions. New Directions for Student Services, 118, 55-70.
