The Inevitable Corruption of Indicators and Educators Through High-Stakes Testing by

In this item, none of the answers is correct. The student is expected to use the Pythagorean Theorem (Hypotenuse² = Side1² + Side2²). So, (6m)² = (5m)² + (EF)². To maintain a right triangle, the only correct answer is (11)^1/2 meters, one that is cumbersome in real life, and so requires rounding off to an acceptable level of accuracy. Depending on the convention for rounding, a reasonable height could be 3 meters (if the convention is rounding to the nearest meter), 3.3 meters (if the convention is rounding to the nearest decimeter), 3.32 meters (if the convention is rounding to the nearest centimeter), and so on.

The answer marked as correct, 3.3 meters is actually about 1.2 centimeters off (about 1/2 inch). Any carpenter worth his or her salt would not make an error of 1/2 inch given a tape measure that is precise to 1/32 inch.

Moreover, as a male, I cringe at the thought of a bike competition that requires riders to jump off 3.3 meter heights (between 10 and 11 feet, ouch!). Or if the rider is to ride down the ramp, a slope of 66 percent (33.5 degrees) is steep enough to scare the bejeebers out of me.

Lastly, a 6 meter board? Come on! When was the last time you found a board of 20 feet at Home Depot? In short, the context within which the problem is embedded shows a lack of the everyday sense for numbers that is required in the elementary standards for Arizona children.

If the released items are a representative sample, then this analysis indicated that over 1/4 of the State’s mathematics assessment provide incorrect data to the state department of education, school districts, parents and children anxious to graduate.

But things did not get much better over time. The Arizona State Department of Education just released 18 more items from the spring 2004 administration of the AIMS test. Now only a sixth of these items appear to be wrong or misleading.¹⁰⁵ But if 1 out of 6 items on the AIMS test is mathematically flawed, it could mean that up to 17 percent of a student’s responses are marked incorrect when they should not be. For many of Arizona’s students, just a few poorly written problems of this sort can cause them to fail the test and not graduate from high school.

Badly constructed tests are common. Here is the Editorial Board of the Orlando Sentinel weighing in:¹⁰⁶

As a Sentinel editorial revealed last week, third-graders are likely to be tested on material written at a seventh-grade level or higher. Sentences in the state's sample test were too long-winded and complex for the average young reader.

A look at the state Department of Education's Web site shows that fourth-graders fare no better. The average fourth-grader is likely to encounter FCAT essays that are better suited to much older readers. What's more, one essay contains a spelling error.

An essay about silver ants from a sample fourth-grade test is written at the seventh-grade level, according to the well-known Flesch-Kincaid readability index. The index measures readability based on the length of words and sentences.

Another essay from a previous fourth-grade FCAT also was written at the seventh-grade level, according to the index. The state posts only select items from FCATs given several years ago.

The latter essay, about a butterfly farm, contains challenging words, such as chrysalis and hydrangea and hibiscus. It also refers to gulf “frittilary” butterflies and asks a question about the “frittilaries” in the essay. That word is so tough that it is frequently misspelled -- including on this test. The correct spelling, according to Webster's New World Dictionary and scientific dictionaries and encyclopedias, is “fritillary.”

A report by the National Board on Educational Testing and Public Policy has identified many more examples of these and other types of errors—only a few of which we report in Table 10.¹⁰⁷ Releasing a test with so many bad items means that test companies are spending too little on item review panels and on item field testing. But then, their incentive is money, and as is so often the case when money is the major objective, quality suffers. States are also at fault too. States usually have commercial companies bid to get these contracts and usually pick the cheapest of those bids as the contractor. Because of this the state contributes to the process of having too many bad items on too many high-stakes tests.

T
_____________________________________'>_____________________________________
Pearson’s placement of profits before quality denied thousands of students the opportunity to participate in the once-in-a-lifetime tradition of walking across the stage at graduation. These errors change lives forever. _____________________________________

here are also many examples of scoring errors. As a consequence of being under pressure to get scores back to states, scoring companies oftentimes rush through the scoring process which increases the possibility of making scoring errors. In Minnesota about 8,000 students were unjustly denied a high school diploma because it was determined that they “failed” the test when in fact they did not. Subsequently, many of these students who were “wrongly” failed participated in a suit against the scoring company (NCS Pearson), which they won. The judge in the case was severe, writing “a scathing opinion” that said the company “continually short-staffed the relatively unprofitable Minnesota project….”¹⁰⁸ Compensation did not change the fact that Pearson’s placement of profits before quality denied thousands of students the opportunity to participate in the once-in-a-lifetime tradition of walking across the stage at graduation. These errors change lives forever.

The stories of simple human error and avoidable error are almost endless. In New York, 2,400 students in grades 3, 5, 6 and 7 were asked to retake a test (after they were mistakenly given the wrong practice exam), using a test booklet with a key that didn’t match the answer booklet. “At one school in Brooklyn, teachers clarified the issue by writing on the blackboard that ‘A=E,’ ‘B=F,’ ‘C=G,’ and ‘D=H.’”(Article 20, in Table 10). Were students given extra credit for accurately decoding the adult-created problem with the answer key? Probably not.

There are also examples of reporting errors. These kinds of errors have all sorts of consequences. To students, it is likely a humiliating—and confusing—experience to first be told you failed a test only to be later told you passed. How does a student recover? What is she to make of the meaningful of tests in the first place if test “performance” and its consequences can hinge on one item? To teachers and schools, being wrongly labeled “failing” is demoralizing and difficult to recover from as illustrated in Nebraska (Article 9). In Pennsylvania, it didn’t happen once, but twice that school’s publicly released report cards contained erroneous information, including bad achievement data that affected school level ratings. There are literally hundreds of examples of all three of these types of errors.¹⁰⁹

The US Government Accounting Office (GAO), as part of its investigation into the functioning of NCLB has looked at this problem too.¹¹⁰ One of their major findings was that the problems of unreliable tests and test scoring are common. The report notes:

Concern about the quality and reliability of student data was the most frequently cited impediment to implementing student proficiency requirements….For example, officials in California indicated that they could not obtain racial and ethnic data—used to track the progress of designated student groups—of comparable quality from their school districts. Officials in Illinois reported that about 300 of its 1,055 districts had problems with data accuracy, resulting in those schools’ appealing their progress results to the state. Similarly, officials in Indiana acknowledged data problems but said addressing them would be challenging. Inaccurate data may result in states incorrectly identifying schools as not meting annual goals and incorrectly trigger provisions for school choice and supplemental services.

W

There is one other little noted consequence of pushing commercial publishers into doing tests cheaply. It is under the radar for news reporters, and thus often escapes notice. This problem is related to the “dumbing down” of the test over time. But in this case it is not politics that drives such decisions, as we argued above, it is costs—the costs of making and scoring items that can more closely relate to the constructs in which we have interest. Richard Rothstein describes this problem:¹¹¹

Consider a typical elementary school reading standard, common in many states, that expects children to be able to identify both the main idea and the supporting details in a passage. There is nothing wrong with such a standard. If state tests actually assessed it, there would be nothing wrong with teachers “teaching to the test.” But in actuality, students are more likely to find questions on state tests that simply require identification of details, not the main idea. For example, a passage about Christopher Columbus might ask pupils to identify his ships' names without asking if they understood that, by sailing west, he planned to confirm that the world was spherical. In math, a typical middle-school geometry standard expects students to be able to measure various figures and shapes, like triangles, squares, prisms and cones. Again, that is an appropriate standard, and teachers should prepare students for a test that assessed it. But, in actuality, students are more likely to find questions on state tests that ask only for measurement of the simpler forms, like triangles and squares. It is not unusual to find states claiming that they have “aligned” such tests with their high standards when they have done nothing of the kind.

In this narrowing of vision about what students should know and be able to do as we go from standards to items, we see three things at play. First is the inherent difficulty of getting complex items to be sufficiently reliable for large-scale assessments. This is not at all a simple problem to solve. Second is the managerial problem of getting complex items created and scored in a timely manner. A testing program where new and complex items are required annually is enormously difficult and expensive to manage. And finally, there is the problem of costs for scoring complex and authentic tasks. The latter two problems are managerial, to be solved by increasing both development time and money, should we ever have the will to do so. But that is not likely.

W