Exclusive, Anonymous Book Excerpt: An Insider Reveals the "Drunks, Dingbats, and Dilettantes Who Write & Score America's Standardized Tests"

I am highly critical of the current use of standardized tests in American public schools. While testing certainly deserves a place in the world, I believe that it has been pumped up to such a high-pressure level for students, teachers, and administrators, that its counterproductive, unintended consequences outweigh its benefits.

And that's assuming that the tests are created and scored fairly according to quality standards.

I was recently contacted by an author who has worked for some of the country's largest testing companies. This person has written an important and revealing manuscript about fifteen years spent behind the scenes of testing. The just-finished insider memoir is titled Off-Task: The Strange & True Story of the Drunks, Dingbats, and Dilettantes Who Write & Score America's Standardized Tests.

I've read part of it, and it's a jaw-on-the-table page-turner that couldn't be more relevant at this moment. Below, check out an exclusive excerpt.

I did not write the following vignettes; they are excerpts from a nonfiction manuscript by an anonymous author.

Enjoy.

Random Scenes from a Career In Standardized Testing

Hired to score standardized tests, a temporary employee stares at her computer screen while sucking on a bulbous lollipop. Her supervisor approaches. "Charlotte," he says. "What score are you giving when a student answers 'because he studied for six years'?"

The woman thinks deeply, licking her lollipop. "I think that's a 2," she says.

"What's your scoring rubric say?" the supervisor asks.

The woman puts the lollipop in her mouth and picks up her rubric. She scans it until a look of wonder flashes across her face.

Talking with the lollipop still in her mouth, a big bulge sticking out of one cheek, she mumbles "it says here that 'because he studied for six years' gets a 3".

"Yes," the supervisor agrees. "So do you think you can go ahead and score it that way? Do you think you can at least score the responses correctly that are actually written on the rubric?"

The woman is smart enough to know she is being reprimanded. She takes the lollipop out of her mouth, a string of spittle trailing the candy from her lips. "Whoops," she says. "Sorry."

***

A supervisor is reviewing the work his team has done and he calls one of his scorers up to his computer. "Look at this response," the supervisor says. "Why'd you give it a 2?"

"Let me see," the scorer says, perusing the student answer on-screen. "Yes, I remember this one. I wasn't sure what that word in the middle was so I decided to give it a 2."

The supervisor raises an eyebrow. "You didn't know what the response says but you thought you'd score it anyways?"

The scorer blushes. "Yes," he says. "2 seemed like a good default score."

"Great," the supervisor says sarcastically. "Keep up the good work."

***

Two professional scorers are huddled over a computer, arguing about a word written on-screen. Both of the scorers are male, but English is not the native tongue of either man. The scorer whose first language is Korean keeps repeating the word "joining, joining, joining." He says the disputed word in the student response is "joining." The other scorer, whose first language is Arabic, scoffs at that. "Coming," he says. "The word is coming."

"Joining," the Korean-speaker says, "this response gets 1 point."

"Coming," the Arabic-speaker says, "this response gets 2 points."

The group's supervisor sidles over to the screen and looks over the two men's shoulders. He is from the Midwest and speaks only English. He rolls his eyes, disagreeing with both of his scorers.

"The word is 'moving,'" the supervisor says. "The response gets 3 points."

"3?" the Korean-speaker says. "Oh, sorry."

"Very sorry," the Arabic-speaker says.

***

A supervisor stands pensively in front of his team, who are eagerly awaiting his decision about how to deal with an unusual student response. As the supervisor thinks, alternating looks of bemusement and annoyance flash across his face.

"What the hell," he says. "I guess we do have to accept an answer that shows such understanding."

As his scorers write notes on their rubrics, the supervisor continues. "So for this item only we will accept thoughtful responses."

He laughs.

***

A supervisor stands in front of his team, bemoaning their ability to score consistently. "What's the problem?" the supervisor asks. "This item isn't very hard."

One scorer raises his hand. "I'm having a little trouble with the training," he says. "I have 25% hearing loss in both ears."

Another scorer chimes in. "And I have short-term memory loss," he adds. "Every time you tweak the scoring rules, I can't remember what you said."

The supervisor stares at the scorers. Neither is kidding.

***

At a table beside a hotel pool, two men and a woman drink celebratory beers in honor of the successful completion of a four-week scoring project. They are tipsy from the drinks and desperate to put an end to their month on the road. They hope to never again score another student response.

The hotel manager brings a phone out to their table. "Call for you," he says to the woman, whose face blanches as she is told by the home office that they've found another hundred tests.

"Fine," the woman at the table says. "Just read them to me over the phone." The woman turns the phone's speaker on, and she and the two men listen as the student responses are read in a squeaky voice by a secretary two-thousand miles away.

One of the men waves to the bartender to silently bring another round, and with drinks but not rubrics in their hands the two men and the woman score each student response. When the voice on the phone goes silent after reading each answer, the woman looks at the number of fingers the men hold in the air before announcing her decision.

"Three," the woman says into the phone. "Give that a three."

For one hour and two rounds of drinks this goes on. The tests all get scored.

***

A scoring director is reviewing the statistics of a recently completed project. She stares intently at the information for an item that did poorly. The notes provided by the supervisor in charge of scoring the item conclude as such: "Poor reliability statistics result from employee idiocy."

***

A manager for a test-scoring company addresses the supervisors in charge of the scoring of a project. The project is not producing the results expected, to the dismay of the test-scoring company and its client, a state department of education. The manager has been trying to calm the concerned employees, but she is losing patience.

"I don't care if the scores are right," the manager snaps. "The state wants lower scores and we'll give them lower scores."

***

A psychometrician, newly-hired, addresses his colleagues for the first time. "Tell me what you want the statistics to say," he explains, "and I'll make them say it."

Follow Dan Brown on Twitter: www.twitter.com/danbrownteacher

Comments

[JustAnotherJoe]

Here's a link to an article detailing the testing industry:

http://www.rethinkingschools.org/archive/19_02/test192.shtml

Highlights include Harold McGraw III ,CEO of McGraw-Hill with revenues of $3.84 billion and a $566 million profit, ties to the Bush family, lack of oversight, you know, the standard operating procedure of funneling public monies into the pockets of multi-millionaires, and profiles of some of the largest companies profiting from all this testing.

[MrWashy]

This is simply atrocious. While larry278 brings up a good point of the frequency of these incidents, we must also understand that many states and districts are now using the standardized tests as graduation requirements. The kind of haphazard scoring that is described is simply unacceptable at any frequency or regularity. They're not making ham sandwiches, they are determining futures and funding.

Given that the law requires that by 2014 all schools are to have 100% proficiency on these tests scoring errors, and incompetence, are unacceptable. This is particularly true with small schools. Five errors in a graduating class of 25 kids is a 20% impact. (Yes there are many schools that are this small.) If those tests that were graded while drinking were a single 100 student class, even then the error rate can put them below the requirements for annual yearly progress.

As teachers, we are forced to pound the importance of these test into the heads of our students. We take them seriously because we understand the ramifications should scores drop. It would be nice to know that the companies, who are being very well paid for their tests, take them as seriously as we do. Apparently they don't.

[Gma11]

I remember reading of psychologist Wayne Dyer, among others, taking a typical standardized test. I can't remember the reason they were doing so. To his amazement, one of the reading comprehension items was from one of his own books. There were a few paragraphs to read followed by five comprehension questions. He got one of the five questions wrong. That question dealt with the author's intentions. And he was the author!!

[blindjester]

"I don't care if the scores are right," the manager snaps. "The state wants lower scores and we'll give them lower scores."

This are the real stakes. The incompetence hurts, but the blatant rigging of the system is what is really dangerous to students and the country.

Corporations want to prove schools are failing so that they can completely privatize them.

They think we'll greet them as liberators.

[larry278]

The short anonymous review doesn't say how common these ugly incidents are in scoring standardized tests. Does the book being reviewed say how often these incidents occurred? Does the book say these ugly incidents among those who score standardized tests? Maybe the new site which is devoted to investigative reporting will choose to look into this. I think the site calls itself PROPUBLICA but I'm not sure. I am sure that the site hired Eric Umansky. You could check Eric's site to get the correct name for the new investigative reporting site. Since Eric is an experienced investigative reporter/w­riter/edit­or who did SLATE's TP-the new site is serious about investigative reporting. End of publicity & praise for the new site. I'm going off thread.
Back to thread: How common are these incidents?
larry lynch