Earlier this year at a gathering with members of the press, the presidents of a handful of top research universities were discussing their commitment to diversity. As if with one voice they said standardized test scores play only a small part in their admissions process. Their institutions, they pronounced, are virtually disinterested in SAT and ACT scores.

The high-powered buzz of insistence ended abruptly, however when the presidents were asked whether their institutions issue press releases whenever their average freshman SAT scores go up.

The embarrassed silence illustrated the central paradox surrounding standardized test scores -- that even though higher education professionals understand their limitations, especially as an aggregate measure of groups of students, they also know that the general public is impressed by high test scores and considers them of paramount importance.

Last week, the College Board released its annual report on student performance on the SAT. As the public and higher education institutions now clamor to interpret and market the findings, it is useful to examine what testing experts have to say about what these scores really mean and the context in which they should be used.

The Power of Numbers

"America believes three is bigger than two. They believe these numbers," is the way Joan Snowden, a former director of a policy center at the Educational Testing Service (ETS), describes the public's understanding of test scores.

Examples abound of the public's reliance on SAT and ACT scores as a measure of academic achievement. Colleges consistently publish average freshman test scores, which high school students then use to determine where they should apply. Real estate agents and parents use the high school-by-high school listings of scores, published by local newspapers, as guides to the "best" schools and neighborhoods. And the National Collegiate Athletics Association (NCAA), which governs intercollegiate sports competition, uses SAT and ACT scores as a way to decide who can play: A student who receives a score of 17 on the ACT, for example, has to sit out a year before playing intercollegiate sports.

Even a magazine like The New Republic, which prides itself on puncturing conventional wisdom, published an article recently with the following assertion: "Institutions have long relied on standardized tests because such tests, for all their faults, tend to be highly reliable in their estimation of how well a particular applicant will actually perform in college or on the job."

The testing agencies themselves have much more modest claims.

The College Board, which administers the SAT, LSAT, MCAT, and other standardized tests, claims only that the SAT correlates with first-year grades a little less than half the time (42 percent). In other words, students at a particular school who scored highest on the SAT wind up with the highest grades in the first year of college a little less than half the time. High school grades correlate with first-year college grades a little better (48 percent). When combined, the SAT and high school grades can be used to predict freshman year grades a little more than half the time (55 percent).

That's it. The College Board does not claim anything more. It has nothing to say about later college performance -- not even about the likelihood of the student to graduate.

"I am proposing research on that," says Gretchen Rigole, a researcher at The College Board who did the latest study on what is called the "predictability" of the SAT -- or how well it predicts first-year grades.

"[ETS's] claim has always been correlation with first-year grades," says Snowden. "What's shocking is when you realize how narrow that [claim] is."

ACT -- the college entrance exam taken primarily in the Midwest, South, and West -- makes a similar claim.

"It's meant to measure how the student will do in the first year of college," says Kelley Hayden, ACT spokesman. And, he says, it does so about half the time -- like the SAT.

The claims of predictability narrow even further when subsets of students are studied. For example, women tend to get slightly higher freshman grades than their scores would indicate when compared to men.

And, Rigole says, "Consistent findings for both Latinos and African Americans show that both grades and SAT scores overpredict."

In other words, African Americans and Latinos who score 1200 on their SATs tend to have lower freshman grades than White students with the same score.

"When you look at students who are clones of to each other in every way [college preparation, GPA, test scores, and family income], once they sit in the classroom together there is a difference in grade performance," says Dr. Michael T. Nettles, executive director of the Frederick D. Patterson Research Institute of The College Fund/UNCF, and a former ETS researcher. "That suggests to us that there is something happening once they get to campus in the freshman year."

SAT and ACT scores are reliable predictors of college performance, according to Nettles, but only when students are compared to others within the same racial group.

"There are problems in the college performance and grading processes that need to be examined more so than the test itself. This may have implications for how [institutions] treat the test[s] when they admit students," he says.

There is a lot of anecdotal and some research evidence that indicates that African American and Latino students need more time than White students to adjust to college. But once they "get on their feet," as Rigole puts it, they perform equally or better. Rigole calls this the "late bloomer theory," and has proposed research on that subject as well -- but so far The College Board has yet to do it.

"What I believe is that [SATs and ACTs] fail to take into consideration two important things," says Dr. John Gardner of the University of South Carolina. "First is [the student's] motivation and second is the university's ability to intervene, to teach students, and to motivate students." Gardner heads a freshman-year program that, he says, works with students whose test scores and grades mean they are predicted by the university to have low grades and high drop-out rates. And yet, he says, the students do very well. "What I've learned, working with African American students in South Carolina, is their high degree of motivation."

These are the kinds of discrepancies and limitations that have brought standardized test scores under attack for decades from educational reformers who say two things: first, that SAT and ACT are not good measures of academic aptitude and are biased against women, African Americans, Hispanic Americans and children from poor or uneducated homes; and second, that the tests are used inappropriately, compounding the unfairness.

The most vigorous of the groups opposing the widespread use of standardized test scores is FairTest, which is funded by the Ford Foundation, the Joyce Foundation, the Rockefeller Family Fund and others. People connected with it include: Dr. Deborah Meier of the Annenberg Institute for School Reform, Chuck Stone of the University of North Carolina Chapel Hill, Dr. Asa Hilliard of Georgia State University, and Dr. Howard Gardner of the Harvard Graduate School of Education.

Despite such backing, however, FairTest has had relatively little success in denting the public perception that standardized test scores are valid measures of success.

Dismantling Affirmative Action

Perhaps the most profound example of how the widespread acceptance of standardized scores is affecting educational policy is the way affirmative action is being dismantled because of them.

In Hopwood v. The State of Texas, four White students claimed that they deserved admittance to the University of Texas at Austin law school more than Hispanic students who had been admitted. Their case rested on their slightly higher "Texas Index" scores, a combination of college grade point averages and scores on the LSAT, the law school version of the SAT. A panel of the Fifth U.S. Circuit Court of Appeals agreed with them last year and dismantled the university's affirmative action program on that basis.

In an odd twist, a different panel of the same court ruled this spring in Ayers v. Fordice that Mississippi's use of ACT scores as the basis for scholarships was discriminatory against African Americans. But the ACT is still used as a basis for admissions in that state. Only those students with minimum scores can get into the prestigious University of Mississippi, for example, which eliminates many of Mississippi's African American students who score consistently lower on ACT tests than White students.

In California, where a referendum ended higher education's affirmative action programs, disparate SAT scores are widely cited as proof that Black students are less qualified than their white and Asian counterparts and thus are less deserving of places in the prestigious University of California system.

These are uses of the SAT and ACT scores that almost no one in higher education defends, including the testing services themselves. They say -- repeatedly and in many different forums -- that the test scores should never be viewed in isolation from other information about students.

This is a particularly important point for those whose scores are low -- and, despite dramatic gains in the last decade, the average scores of African American, Hispanic and Native American students who take the test continue to lag behind the average scores of White and Asian students who take the test. This is also true of the average score of poor students as opposed to wealthy students.

Both The College Board and ACT say that when students take rigorous college preparatory curriculum, they have better test scores. This is true for all ethnicities and income groups.

"The correlation is very high," says Rigole. "If you take more courses, you'll do better."

The problem, they say, is that too many poor, African American, and Latino students do not have access to those courses. Too often, they are shunted away from college preparatory classes into vocational education or other non-college preparatory programs.

"What bothers me is the implication that [the discrepancies] are the test's fault," says The College Board's spokesman, Fred Moreno, "completely ignoring the fact that so many kids don't take the courses that will help them do well. They don't take calculus and physics and all the classes that help them with reasoning skills."

Moreno continues: "You've got to start taking academic classes beginning in the ninth grade -- which really means starting earlier."

Despite the fact that many newspapers and magazines try to portray the SAT and ACT as measures of the educational progress of the nation, they were not designed to do that. As great grandchildren of the IQ tests developed in the first part of the century, the ACT and SAT are simply devices to help colleges sort through the piles of applications.

"Admissions is a sorting process," Rigole says. "If [colleges] have more applicants than places, they have to sort somehow."

But it seems that this is where FairTest and other critics of standardized test scores have succeeded, at least in part. Even as the public perception of standardized test scores as objective measures of achievement appears to have crystallized, FairTest and other critics of standardized test scores have succeeded in convincing many higher education officials to lessen their reliance on test scores in the admissions process -- witness the protestations of the university presidents noted at the beginning of this article.

Curriculum, SCUGA and Test Scores

"Admission officers look at any measure with skepticism," says Roger Swanson, a former admissions director at Arizona State University and California Polytechnic State University-San Luis Obispo. Swanson is now the associate executive director of the American Association of Collegiate Registrars and Admissions Officers (AACRAO). When he was working in admissions, he recalls that the officers would rate every application on a scale in which SAT and ACT test scores were a fairly small part. The curriculum the student had taken -- whether it was a rigorous college preparatory curriculum, for example -- held much more weight.

The University of Michigan -- a large, selective public university -- appears to agree with Swanson. University spokeswoman Julie Peterson says that each of the 20,000 applications the university receives to fill its 5,200 freshman spaces is scored on a complicated scale by academic performance -- with grades weighing twice as much as test scores -- and by something the admissions people named SCUGA, an acronym for school, curriculum, unusual, geographic, and alumni. The scale weighs elements such as whether the student's high school had a rigorous curriculum; whether the student took the most rigorous curriculum offered by the school; any unusual characteristics such as leadership, community service, or overcoming difficult circumstances; whether the student can offer some geographic diversity to the student body; and whether the student is a child of an alumnus.

The University of Michigan admissions office has done no predictability studies of how well the SAT and ACT predict success in college because, Peterson says, "So much national research has been done that shows they are not good predictors that the admissions officers don't give them much weight."

AACRAO's Swanson says that the "one thing that tends to increase the use of test scores are those students with less traditional schooling -- home-schooled students, for example, or students from school systems with performance-based assessment."

This is an ironic twist for people and organizations like FairTest, which have long pushed for more "authentic" performance-based assessments such as that adopted in Wisconsin -- where students leave high school with thick folders filled with projects, papers, and long, written assessments by themselves, their teachers and their parents. Colleges don't always know what to do with all that information and, Swanson says, they sometimes look to SAT and ACT scores in those cases.

If that's the result, says FairTest's Rooney, "That puts us in a bind. That kind of undoes the benefits of using performance assessment at the high school."

But that gets at the reason SATs and ACTs -- despite all their limitations -- are still used, even if sparingly, by admissions offices. They give a number.

Burnie Bond, of the American Federation of Teachers, which has long pushed for national standards, puts it this way: "It's not that they're so good, it's that there's nothing else to use that is reliably constant -- not reliably good, but reliably constant."

As FairTest's Rooney says, "It's not as if we had a national curriculum or national standards that we could test against."

RELATED ARTICLE: The SAT and the ACT claim to be:

(A) measures of academic achievement by students

(B) predictors of whether students will graduate from college

(C) reliable predictors of college performance, race notwithstanding

(D) measures of academic rigor of local school systems

(E) none of the above

RELATED ARTICLE: Earning Points With AP

In contrast to the jaundiced view admissions officers have about the SAT and ACT, they are very impressed by Advanced Placement courses and exams.

Even if students do not take Advanced Placement tests, if they take the courses in high school, "That's a real positive sign," says Roger Swanson, the associate executive director of the American Association of Collegiate Registrars and Admissions Officers (AACRAO). "Most admissions people think that is special."

When he was an admissions officer, Swanson says, "We would give an extra grade point for AP." So, for example, if a student got a 3.5 in an AP class, the admissions office would give the student a 4.5.

The Advanced Placement exams, which are administered by The College Board, are different from the ACT and SAT in that they test mastery of Advanced Placement courses which have very specific curriculum and standards. In contrast, the ACT and SAT pride themselves on not being tied to any specific curricula but being tests of reasoning abilities.

The fact that advanced placement classes are so well regarded increases the importance of the fact that few African American students take them.

Wade Curry, head of Advanced Placement at The College Board, says that most school systems offer the classes. But. he said, the schools that don't offer them fall into four categories: small religious schools, small rural schools; schools in Wyoming, the Dakotas, and a few other states; and, most important for African Americans and Latinos, large urban schools -- particularly if they feed into academic magnet schools. Magnet high schools -- such as Whitney Young Jr. in Chicago and A. Philip Randolph in Harlem -- draw off the most academically inclined students and produce fairly large numbers of AP scholars. But the schools they draw from then tend not to offer the courses, leaving those students behind.

Curry said that as he has gone through the data for this year's freshman profile, he hats found that African American students who do well on the AP tests tend to be either in the urban magnets or in predominantly White, suburban school systems where there are between five and twenty African American students who take the courses.

"That's where achievement seems to be centered" Curry said.

For example, he said, at Quince Orchard High School in suburban Montgomery County, Maryland, 82 percent of the African American students who took AP exams received a three or better (on a scale where five is the highest possible score). This contrasts with a national average of 65 percent.

[ILLUSTRATIONS OMITTED]

RELATED ARTICLE: College Board Develops New SATII Biology

NEW YORK -- Because of changes in high school biology curricula and the emphasis on applied reasoning skills, the College Board has developed a new Advanced Placement test -- now called SAT II -- in biology.

"The new exam responds to the continually expanding biology curriculum in American high schools -- particularly the dual emphasis on ecological and molecular approaches." said Donald M. Stewart, president of the College Board. "Although we have offered a biology exam for several years, the new test reflects shifts in the teaching of biology, with greater emphasis on fundamental concepts rather than just collections of facts."

The new exam, SAT II: Biology E/M, allows students to choose an ecological emphasis or a molecular emphasis in the same test, giving them the opportunity to take the test for which they feel better prepared. The test, which took 18 months of research and development contains eighty questions -- sixty of which are common to both forms of the test, and twenty that emphasize either ecological or molecular biology.

"This new test will put greater focus on scientific reasoning and less on memorization of facts," said J. Jose Bonner, chair of the test's development committee. "The questions will allow students to analyze and interpret data from hypothetical experiments, and they represent a whole new effort to measure how students will apply the subject matter to concrete situations."