The Inevitable Corruption of Indicators and Educators Through High-Stakes Testing by

How does this table illustrate Campbell’s law? It does so indirectly by showing that accountability systems yield different results and under such circumstances we should expect that educators will pick the system that makes them look the best. That should not be surprising. Thus, through these articles, we see a defense of local evaluation systems and attacks upon the federal system, which seems to rely upon a harsher indicator. But it is not clear to us that either evaluation system is noticeably better than the other. Good evaluations of schools are conducted over considerable lengths of time, and they rely upon multiple indicators of success. Neither federal nor local evaluation systems usually meet these criteria. And neither system of evaluating achievement would stand up to close scrutiny whether they are value-added or static models of student learning because our psychometrics are not up to the requirements that politicians place on our research community. When there exists no consensus about which indicators might be most appropriate to use in the evaluation of schools, Campbell’s law suggests that there might be more support for the indicators that make schools and districts look the best, rather than engagement in a search to get a better indicator system. That should be guarded against.

Table 8 presents stories of how the meaning of proficiency in various communities changes over time. In high-stakes testing, to pass and fail students, there has to be a point on a scale where one decides that students above or below that point have learned or not learned what is expected. Students either are or are not proficient; they either have earned the right to be passed to the next grade, or not; they either are worthy of graduation, or not. While this sounds like a reasonable problem, and that a reasonable solution might follow, it turns out that this is an enormously complex problem, and it has not yet been solved by some of the brightest people in the field of testing.⁸⁵ Gene Glass wrote on standard setting for tests almost 30 years ago. Standard setting means that you have determined some absolute standard of performance, above which someone is competent and below which they are not. An example might be that a score of 130 or more on a particular IQ test allows you into a program for gifted children, but a score of 129 or less keeps you out. Glass says:

I am confident that the only sensible interpretations of data from assessment programs will be based solely on whether the rate of performance goes up or down. Interpretations and decisions based on absolute levels of performance….will be largely meaningless, since these absolute levels vary unaccountably with exercise content and difficulty, since judges will disagree wildly on the question of what consequences ought to ensue from the same absolute level of performance, and since there is no way to relate absolute levels of performance on exercises to success on the job, at higher levels of schooling, or in life. Setting performance standards on tests….by known methods is a waste of time or worse.⁸⁶

Choosing cut scores on tests, that is, determining the absolute level at which bad performance turns miraculously into good performance, is simply impossible to do. Worse than wasting time trying to do it, is fooling people into thinking it can done. Stated as clearly as possible: The choice of the cut point for high-stakes achievement tests is arbitrary. It is a political decision, not a scientific one. By political we mean that cut scores are determined by the acceptable failure rates in particular communities. Typically, this means that wealthy white students, whose parents have political capital, pass at high rates, while poor minority students, whose parents have no such capital, fail at high rates. Cut scores for high-stakes tests in most states appears to be about choosing an acceptable level of casualties!

A

Article 2 and others in this table make the point that cut scores generally (but not always) go down. This is so a politically acceptable solution to the cut score problem can be found and more students can pass. Less frequently the reverse is true, where too many students appear to be passing, so a tests’ cut score is raised (see Article 9). The more typical danger in all this dancing around a cut score for a high-stakes test is that after the politics plays out, the tests might resemble minimum competency tests. Such tests might ultimately allow almost all students to pass, thus diminishing the assumed benefit of high-stakes tests, namely, to motivate students and teachers to work harder. Because of politics there is not only pressure to lower cut scores, there is also a tendency for the test content to decrease in difficulty over the years, or for the scores to be allowed to drift upward as teachers and students learn what is on the test and teach to it more directly. In all three cases passing rates go up and the indictor no longer means what it once did. It becomes corrupted.

Articles 4 to 11, all concerning the state of New York, show that New Yorkers cannot agree at all on whether their tests are too hard or too easy. Moreover, they do not know what to do about it. They do not know what to do because there is no way to get unanimity among those concerned with the choice of a cut score. Political agreements must be reached. The technical skills of the psychometricians who made the test are not useful in making these kinds of decisions, as was discovered when state designations of proficiency in reading were evaluated against federal designations of proficiency. In this report for the Carnegie Corporation of New York, by Rand Corporation of Santa Monica, incredible variations between the federal and states’ visions of proficient were revealed.⁸⁷ For example, Texas claimed that 85 percent of its students were proficient in reading at the fourth grade, using its own tests. But the federal government claims that only 27 percent of Texas’ students are proficient in reading using the National Assessment for Educational Progress (NAEP) test. Mississippi claimed 87 percent proficiency in reading, but the NAEP data suggested that only 18 percent should be declared proficient. Only a few states showed any close relationship between its reading proficiency designations and those derived from the federal government’s assessments. But in not a single case among the 40 states in the study was there a lower rate of proficiency than claimed by the federal government. States always had more students proficient in reading than suggested by their NAEP scores. Which exams might be used for judging schools, districts and states? No one knows because no one knows where to place a cut score on any examination. Moreover, there is considerable evidence that the NAEP Governing Board has deliberately set its cut scores to make it very difficult to reach proficiency. The National Academy of Sciences panel looking into standard setting on NAEP said, “NAEP's current achievement level setting procedures remain fundamentally flawed. The judgment tasks are difficult and confusing; raters’ judgments of different item types are internally inconsistent; appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results.”⁸⁸ But because the states’ systems of setting standards are faulty too, no one knows how to judge proficiency on their tests in a way that is convincing!

Cut scores are only one problem in determining who is declared proficient. Another problem is associated with the content of the tests, as made clear in Article 20. Arizona has its own example of this problem. From interviews with teachers of young children in Arizona, Annapurna Ganesh learned that there is an item on the SAT9 test about a blizzard.⁸⁹ Flagstaff students who experience winter might have gotten this item right, but Phoenix students apparently did not. Ganesh also heard about another problematic item. This one showed a picture of a bus and asked where it was going. Two of the three choices were “school” and “grocery store,” with the former choice being the correct one. But in this poor neighborhood, students rode buses to grocery stores and had no need of school buses because they lived close to the school. Needless to say, most students got the item wrong. What is troubling is that fixed cut scores, arbitrarily chosen, could result in some students being judged proficient and others being failed due to these ordinary and minor forms of bias that are inherent in all tests. It may not be possible to ever rid tests of all forms of bias, but it is possible to adopt more flexible criteria for deciding who is or is not proficient.

H