I am highly critical of the current use of standardized tests in American public schools. While testing certainly deserves a place in the world, I believe that it has been pumped up to such a high-pressure level for students, teachers, and administrators, that its counterproductive, unintended consequences outweigh its benefits.
And that's assuming that the tests are created and scored fairly according to quality standards.
I was recently contacted by an author who has worked for some of the country's largest testing companies. This person has written an important and revealing manuscript about fifteen years spent behind the scenes of testing. The just-finished insider memoir is titled Off-Task: The Strange & True Story of the Drunks, Dingbats, and Dilettantes Who Write & Score America's Standardized Tests.
I've read part of it, and it's a jaw-on-the-table page-turner that couldn't be more relevant at this moment. Below, check out an exclusive excerpt.
I did not write the following vignettes; they are excerpts from a nonfiction manuscript by an anonymous author.
Random Scenes from a Career In Standardized Testing
Hired to score standardized tests, a temporary employee stares at her computer screen while sucking on a bulbous lollipop. Her supervisor approaches. "Charlotte," he says. "What score are you giving when a student answers 'because he studied for six years'?"
The woman thinks deeply, licking her lollipop. "I think that's a 2," she says.
"What's your scoring rubric say?" the supervisor asks.
The woman puts the lollipop in her mouth and picks up her rubric. She scans it until a look of wonder flashes across her face.
Talking with the lollipop still in her mouth, a big bulge sticking out of one cheek, she mumbles "it says here that 'because he studied for six years' gets a 3".
"Yes," the supervisor agrees. "So do you think you can go ahead and score it that way? Do you think you can at least score the responses correctly that are actually written on the rubric?"
The woman is smart enough to know she is being reprimanded. She takes the lollipop out of her mouth, a string of spittle trailing the candy from her lips. "Whoops," she says. "Sorry."
A supervisor is reviewing the work his team has done and he calls one of his scorers up to his computer. "Look at this response," the supervisor says. "Why'd you give it a 2?"
"Let me see," the scorer says, perusing the student answer on-screen. "Yes, I remember this one. I wasn't sure what that word in the middle was so I decided to give it a 2."
The supervisor raises an eyebrow. "You didn't know what the response says but you thought you'd score it anyways?"
The scorer blushes. "Yes," he says. "2 seemed like a good default score."
"Great," the supervisor says sarcastically. "Keep up the good work."
Two professional scorers are huddled over a computer, arguing about a word written on-screen. Both of the scorers are male, but English is not the native tongue of either man. The scorer whose first language is Korean keeps repeating the word "joining, joining, joining." He says the disputed word in the student response is "joining." The other scorer, whose first language is Arabic, scoffs at that. "Coming," he says. "The word is coming."
"Joining," the Korean-speaker says, "this response gets 1 point."
"Coming," the Arabic-speaker says, "this response gets 2 points."
The group's supervisor sidles over to the screen and looks over the two men's shoulders. He is from the Midwest and speaks only English. He rolls his eyes, disagreeing with both of his scorers.
"The word is 'moving,'" the supervisor says. "The response gets 3 points."
"3?" the Korean-speaker says. "Oh, sorry."
"Very sorry," the Arabic-speaker says.
A supervisor stands pensively in front of his team, who are eagerly awaiting his decision about how to deal with an unusual student response. As the supervisor thinks, alternating looks of bemusement and annoyance flash across his face.
"What the hell," he says. "I guess we do have to accept an answer that shows such understanding."
As his scorers write notes on their rubrics, the supervisor continues. "So for this item only we will accept thoughtful responses."
A supervisor stands in front of his team, bemoaning their ability to score consistently. "What's the problem?" the supervisor asks. "This item isn't very hard."
One scorer raises his hand. "I'm having a little trouble with the training," he says. "I have 25% hearing loss in both ears."
Another scorer chimes in. "And I have short-term memory loss," he adds. "Every time you tweak the scoring rules, I can't remember what you said."
The supervisor stares at the scorers. Neither is kidding.
At a table beside a hotel pool, two men and a woman drink celebratory beers in honor of the successful completion of a four-week scoring project. They are tipsy from the drinks and desperate to put an end to their month on the road. They hope to never again score another student response.
The hotel manager brings a phone out to their table. "Call for you," he says to the woman, whose face blanches as she is told by the home office that they've found another hundred tests.
"Fine," the woman at the table says. "Just read them to me over the phone." The woman turns the phone's speaker on, and she and the two men listen as the student responses are read in a squeaky voice by a secretary two-thousand miles away.
One of the men waves to the bartender to silently bring another round, and with drinks but not rubrics in their hands the two men and the woman score each student response. When the voice on the phone goes silent after reading each answer, the woman looks at the number of fingers the men hold in the air before announcing her decision.
"Three," the woman says into the phone. "Give that a three."
For one hour and two rounds of drinks this goes on. The tests all get scored.
A scoring director is reviewing the statistics of a recently completed project. She stares intently at the information for an item that did poorly. The notes provided by the supervisor in charge of scoring the item conclude as such: "Poor reliability statistics result from employee idiocy."
A manager for a test-scoring company addresses the supervisors in charge of the scoring of a project. The project is not producing the results expected, to the dismay of the test-scoring company and its client, a state department of education. The manager has been trying to calm the concerned employees, but she is losing patience.
"I don't care if the scores are right," the manager snaps. "The state wants lower scores and we'll give them lower scores."
A psychometrician, newly-hired, addresses his colleagues for the first time. "Tell me what you want the statistics to say," he explains, "and I'll make them say it."