ETS Research Focuses Computer Testing

WASHINGTON —— Educational Testing Service officials announced earlier this month that the organization is launching a three-year,

multimillion dollar research project to improve the quality of its tests based on data gleaned from the use of computer adaptive tests.
 Since 1993, ETS, the nation's largest private, nonprofit organization devoted to educational measurement and research, has administered graduate school admissions exams and other tests through a format known as computer adaptive testing.
 "CAT, as we call it, uses computer technology to adapt the difficulty of test questions to the performance of each test taker, allowing more reliable testing with fewer test questions," explains ETS president Dr. Nancy Cole.
 ETS officials say the information collected from assessing students who take computer adaptive tests can provide researchers with valuable insight into the test-taking experience of individuals, which contrasts to the group testing model underlying traditional pencil-and-paper standardized tests, such as the SAT.
"Instead of dealing with an undifferentiated group of test takers, we can now examine the individual test taker and explore new ways to make the test as effective as possible for each," Cole says.
     "Now, for the first time, we see the possibility for developing our measurement models and describing test performance at the individual level. This is a great challenge, but it suggests the possibility for some real breakthroughs in the science of educational measurement," says Dr. Drew Gitomer, ETS vice president of research.
The project's goal, she says, is for the research to enable ETS to improve the quality of the computer adaptive tests as well as standardized ones. She estimates that $6 million to $8 million will be spent on the initiative. Currently, ETS offers computer adaptive testing with the Graduate Record Exam, the Test of English as a Foreign Language, the Graduate Management Admissions Test and the National Council Licensure Examination. The SAT, developed and administered by ETS under the supervision of The College Board, is strictly a pencil-and-paper standardized test.
Robert Schaeffer, public education director at FairTest  — a Cambridge, Mass.-based education advocacy organization that has long been critical of the use of standardized tests in higher education admissions — says the research initiative is long overdue. He adds that FairTest has previously criticized computer adaptive testing because its disadvantages outweigh the benefits it provides to test takers.
"Clearly there are advantages to the computer-based tests. They provide convenience in allowing people to take them year-round and they provide instant test scores," Schaeffer says. However, one of the biggest problems with CATs are that they do not allow test takers to skip forward and backward over questions, a strategy commonly practiced by takers of pencil-and-paper tests. Another problem is the issue of equity among groups who have experience with computers compared to those who have lesser or no experience with computers, Schaeffer says.
FairTest initially criticized ETS for computer adaptive testing for launching it without conducting adequate research.
"They marched it into the market, and they did it for one reason. The bottom line is that [ETS] feared they would beaten in the market by others," Schaeffer says.
ETS officials say that researching individual performance represents a conceptual leap for the future of testing. Cole says ETS research has previously determined that computer skill is not a factor in taking a CAT. She adds "CAT typically decreased majority-minority gaps in scores."