iPhone app iPad app Android phone app Android tablet app More

Featuring fresh takes and real-time analysis from HuffPost's signature lineup of contributors
John Thompson

GET UPDATES FROM John Thompson

Why Test-Driven Accountability Is Grasping at Straws

Posted: 08/01/11 12:20 PM ET

Bill Gates says that his $5 billion experiment in education has not failed. Gates also claims that he trusts in science. That prompted me to reread the National Academy of Sciences' analysis of the failure of test-driven accountability.

A funny thing happened as I tried to find a bifocal-friendly PDF. I accidentally found the Brookings Institute's "Volatility in School Test Scores." Its preface cited John Dryden's poetry: "Errors, like straws, upon the surface flow; He who would search for pearls must dive below."

The Brookings report explained the damage caused by those errors, concluding that test-driven accountability could make teachers behave like dogs who were trained by electrical shocks. When the tests changed, dogs just laid there and waited for the pain to stop. The report then warned of similar "perverse incentives that may actually harm rather than help."

These metaphors perfectly describe the Gates Foundation's "teacher quality" research. Gates is empowering districts to either dive deep and seek pearls, or skim the surface, and condition educators to cower and endure the indignities of test-driven schooling. As the toxicity dumped on teachers pours down on students, a generation or more of them might be robbed of a respectful education.

I then found the NAS's new blue ribbon panel's report, "Incentives and Test-Based Accountability in Education," and reread that masterpiece. It showed the failure of NCLB-type accountability to improve student performance, and explained why those "reforms" had damaged students. For instance, it showed that graduation examinations had decreased graduation rates by 2%, while not increasing student performance. It also summarized the huge body of literature on the unintended, destructive consequences of the accountability movement.

The problem is that for every effort to use testing as a part of a dive-for-pearls experiment in helping students, there is one (or more) effort(s) to grab at straws. The NAS argued,

Test-based incentives for students may cause some students to achieve more and others to drop out, even with extra support and remediation. Test-based incentives for teachers may cause some teachers to become more effective and others to leave the profession. Test-based incentives for schools may cause some to focus on the full curriculum and others to focus on test preparation.

The report also issued a not-so-ringing endorsement of the hypothesis that Gates-funded teacher quality research might pan out, "The extent to which value-added models can realize their promise has not yet been determined," because "it is not yet clear how fully these models can account for student differences to provide accurate measures of teacher effectiveness."

I am cautiously optimistic that the Duncan Administration might heed the experts' warning, but I am more hopeful that the report will change the course of Gates' research. After all, their work is a product of brilliant academics. Those scholars do not want to look in the mirror and see an alchemist peering back at them. Or worse, they do not want to land in the History of Science's Hall of Shame of scientists whose discoveries produced great harm. The jury is not in, completely, but surely it is time for social scientists like Paul Hill and Tom Kane to read the NAS report and ask whether they want to invest their considerable talents in a project which is likely to produce more destruction than good.

And another funny thing happened when I found a PDF of "Incentives and Test-Based Accountability in Education." Among the noted scholars who produced the NAS report were Paul Hill and Tom Kane! In fact, it also was Kane who had written "Volatility in School Test Scores," which cited Dryden to warn that schools grasping at straws were likely to produce teachers and students who learned that there was little they could do to avoid the abuses of test-driven accountability, and who thus tried to wait out the abuse. So maybe it is time for Kane, the head of Gates' "Measures of Effective Teacher Project," to heed his own words, and choose science over political spin.

 

Follow John Thompson on Twitter: www.twitter.com/drjohnthompson

 
 
  • Comments
  • 26
  • Pending Comments
  • 0
  • View FAQ
Comments are closed for this entry
View All
Favorites
Recency  | 
Popularity
09:47 PM on 08/07/2011
I find this amazing. My thesis was on Grades as Predictors For Clinical Performance. I assessed academic grades vs grades on clinical performance for a masters program. Grades do not predict how well a student did on their clinical rotation. Learning is the key. Students learn in a variety of ways, listening, reading, tactile, and by doing. Some students learn rather quickly while other students need to practice over and over again. The focus in this country should be on the learning process. Business needs to measure its product for its worth , our students are worthy of an appropriate education and keep the private sector vampires out of the kiddy.
11:18 AM on 08/07/2011
I once heard someone say they admired Bill Gates for his efforts to improve education and that always amazes me. Bill Gates has done nothing to improve education. As a teacher I have taught just over a thousand kids to read, express themselves through writing and how find their way around a math problem, yet there are no accolades of admiration for me or other teachers. We are not consulted for ideas on change or reform, but a very wealthy man that has expressed his distaste for public schools and is a college dropout speaks of education and he is taken seriously. I have done more for education than Mr. Gates could ever hope to. My only wish is that Mr. Gates' pay could be based on how well his operating system performed. He would not be paid well.
02:35 PM on 08/08/2011
Yikes! So well stated and so true, as well. Add me to your list of having more influence on education than Bill Gates. I teach HS Spanish, but I also teach my students about the many cultures and histories of Hispanic countries. I show them pictures of my varied travels over the past 30 years to open their eyes to the fact that not everyone else in the world lives like we do here in the US---although, I'm finding I have less time to go off curriculum to do so because we're so pressed to follow the curriculum set forth by the District. I expand my kids' English vocabulary by showing them the links among Spanish, English and the Latin roots, from which both languages were inluenced. I bring in articles about study tips, college scholarships, job interview tips, nutrition & sleep habits to help them improve their focus in school. Again, I'll have to stop doing these things once the Merit Pay based tests kick in next year so I can be judged on how much I'll be paid based on my students' test scores.
It's just maddening to see what's going on in this country regarding education.
HUFFPOST SUPER USER
francisco cortes
07:20 PM on 08/05/2011
If Bill Gates believed in science he must believed in biology, if he believed in biology he must believed in ethology(the field of biology that study he evolution of behavior), if he believed in ethology he must believed in human ethology( the study of the evolution of human behavior and human animal instincts).
I wonder then how Bill gates beleived education reform will deal with student ethology in order to avoid evolutionary behavior that goes against school norms and motivate students to be good test takers that until now have no evolutionary value?
HUFFPOST SUPER USER
johnthompson
01:06 PM on 08/06/2011
"Reformers" are in the tradition of denying that a "culture of poverty" exists, so hard work will overcome the findings of etnology.

Schools, as you also imply, also are ecological systems. It blows my mind that "reformers" who also are environmentalists believe in social engineering and technological fixes.

I think Gates wants a Coperican revolution with himself as the hero. He wants to repudiate generations of social science and replace them with meta-analyses. He admits that much of his data in garbage in, but he thinks with enough data out that it will balance out.
foresure
Brash and Harsh
06:15 PM on 08/06/2011
francisco cortes

1. Has Mr. Gates ever spent a whole day in a classroom rowdier than a Harvard Seminar?.

Grand theories of more pay, less accoutability and more money abound. Computers for All.

Their number, as Someone Else once said, "is Legion".

I am sure if you went to Europe you could find libraries stuffed with treatises on how to turn "Base Metal into Gold". Or how to undo the effect of sorcery.

Like the best thinkers of the Enlightenment thought 300 years ago, Mr. Gates believes
everything can be solved and improved by thinking really hard about it with enough effort.

Except for ONE instance reported some 2,000 years ago, no one has yet reversed physical death, or for that matter avoided it.

For all the hubis of modern medicine, the life expetancy of an adolescent has not increased more than 20-25 years. And of course it has declined in many societies.

Don't believe, do your own research. The dramatic increase has been the survival children past infacy.

What Mr. Gates doesn't know, and doesn't care to know is that some families are

FUBAR. Yes FUBAR. No spelling error.

For the meaning of that word, search; www.acronymlist.com/acronym.

The solution is full time, VOLUNTARY, neighborhood boarding school, which provide for all the childs education, medical, social, and medical needs.

To be paid for by diverting support away from able bodied adults 25-55 whose only claim to support is that they are dysfunctional parents.
06:46 PM on 08/05/2011
I actually accept Gates's good intentions. It is all too easy to see what we want to see in data, which is usually rather noisy. Data driven approaches work - but they can take a prohibitive amount of time and data to eliminate noise and other effects. And the problem is that data driven approaches are particularly difficult with people - who tend to respond to the attention alone - placebo effect anyone?

We learn better from our failures than our successes. We should be learning a lot now. It doesn't appear we are at this point. The refusal to accept the rather poor results is keeping us from learning anything. If anything, we are in a replay of the efficiency expert attempt to reform education early in the 1900's. Another notable failure. In the words of my kids, "epic fail".
HUFFPOST SUPER USER
goldstone
09:24 AM on 08/06/2011
Terribly naive to assume that a billionaire has "good intentions": His whole life has been about amassing wealth; there's no reason to believe that he has turned on a dime.

Instead, it makes more sense to analyze Gates's philanthropy as an extension of the character he has demonstrated over his whole life.

To wit, recall the business drama of his life: competing with Steve Jobs, whose Apple is now the biggest company in the world (or close), with more cash on hand than the US government.

Jobs's early business strategy was education-focused, with Apples in every classroom. Consequently, for three decades now Apples have dominated the education market, and Gates has to reflect on those consequences every day: young Apple users grow up and buy more Apples.

Gates is all over schools now. But he famously dropped out of Harvard, valuing his business (money) over a degree from the finest institution in the world (his education). This is held up as narrative for meditation, supposedly showing how smart he was to know what he was doing at such a young age.

Bullshit. It shows his priorities. And if we believe that the leopard has changed his spots...

Well, I won't believe it. Gates knows that "virtual learning" and online testing will finally crack open the education market for him, so that he can train the next generation of Windows users.

And he will finally beat Steve Jobs -- and get silly people to lionize Bill Gates. FTW!
HUFFPOST SUPER USER
johnthompson
01:16 PM on 08/06/2011
Our prosperity since the New Deal has been based on checks and balances.There would be more than enough profits for "virtual learning" if used to improve student learning not to defeat teachers. Win Win makes more money, in the long run, than beggar thy neighbor. These "reformers" don't know their economic history any better than their educational and social history.
foresure
Brash and Harsh
06:22 PM on 08/06/2011
J R M

There is one institution that has been in the edcuation business for at least 2,000 years.

And they sure have made a lot of mistakes in their time.

And they tend to be compulsive about keeping a long record of what they have been doing.

Their biggest library is somewhere in Europe, but they have them all over the United States.

How about asking the American Catholic Educational Hierarchy to show the rest of us how it is done.

P.S. I am not Catholic, and never been to a Catholic School.

I also know that most Catholic school graduates hated every minute of their stay.

I also know, that Catholic school graduates display a superior education, and there is nothing wrong with the "self image" or creativity.
foresure
Brash and Harsh
06:38 PM on 08/05/2011
Conclusions of the Best Minds:

The only solution to the chronic failure of the educational system is to

1. Greatly increase teacher salaries.

2. Greatly decrease any responsibility a teacher has for student learning.

3. Increase the number of "educational theory" courses required to teach.

4. Make sure that a teacher has absolute tenure, unless found to be having sex with the students.

5. Ban the word "accountability" as it relates to teachers, except to use it as a highly perjoritive term.

6. Greatly increase the number and value of grants to study "educational innovation".

7. a) Never ask students to demonstrate any competency in any area.

b) To do so might damage their self esteem.

8. Eliminate any dress codes for either teachers, After all, isn't that what freedom is about?

9. Have a complete understanding that anarchy is just another word for "freedom'

10. Ignore the fact that some students come to school:

a) Without sleep

b) Hungry

c) Drugged

d) Armed

e) Ill or in pain or both

To be concerned with those things would be to invade the privacy of the student, and that of the student's family.

11. Never, ever mention sex, and never ever mention sex education.

12. Make sure you have a "comfortable acceptance" of 14 and 15 year old girls who become pregnant. After all, they really didn't know how it happened.

And never, never address the issue of the "Culture of Poverty". To do so would be racist, or worse.
HUFFPOST SUPER USER
johnthompson
01:12 PM on 08/06/2011
We don't want absolute tenure, and we need accountability, but the word "accountability" should be retired in favor of something less mindless and judgmental. We need to talk openly, but carefully about the "culture of poverty."
photo
HUFFPOST COMMUNITY MODERATOR
broui
No d#%& cat. No d#%& cradle.
01:03 AM on 08/09/2011
I concur with johnthompson's response.

In addition, students of course need to demonstrate competency. I and I believe most teachers would argue that high stakes, one-size-fits-all, standardized tests are simply the worst way to accomplish this goal. They tend to do more harm than good.

A student's perception of themselves is fairly engrained when they walk in the door. We (meaning teachers and the educational system in general) can have an effect. We can set them up for failure or success - what that means is far too complicated for this space.
foresure
Brash and Harsh
04:17 PM on 08/09/2011
broui

You are so right the evaluation of anyone working in any job should be eliminated in favor of image boosting seminars.

Other things are too complicated.
05:42 PM on 08/01/2011
I love how you (and Tom Kane) use poetry to explain science. Exactly the kind of translation/instruction that those of us not trained education professionals need to understand something as complex as testing. VIVA Project teachers, www.vivateachers.org share the skepticism about the wrong tests being used for the wrong purpose. BUT, they also embrace data-informed instruction. As part of an information-rich learning environment, VIVA teachers know that public education needs a common language, that includes SOME measure, of what successful learning looks like and how we know if a group of students have "gotten it". VIVA is here to make it easier for teachers-the resident experts--to share their expertise with policy makers who pay for "the tests", for parents whose partnership and support is critical to any school and for neighbors and taxpayers, who invest in our schools. We need to find a way to describe the pearls, and make sure there are an ample supply in every classroom.
HUFFPOST SUPER USER
johnthompson
06:10 PM on 08/01/2011
Thanks. Think of how much our educational civil war comes down to data-INFORMED versus data-DRIVEN. And how do you teach science or anything else without using metaphors?
photo
HUFFPOST SUPER USER
tultican
Thomas Ultican, MEd. BS Mecahnical Engineering
03:37 PM on 08/01/2011
Great post as usual. It seems that billionaire philanthropists like Bill Gates and Eli Broad jumped into saving education without a proper respect for the complexity involved. Now, they are trapped in a failing accountability paradigm and because of their incredible wealth they are causing serious damage to America. How ironic is it that wealthy men who want to do good works are damaging the education of America’s children? This is another lesson in the danger of concentrating wealth and power into just a few hands.
HUFFPOST SUPER USER
johnthompson
06:08 PM on 08/01/2011
Power still corrupts and absolute power still corrupts absolutely.
11:19 AM on 08/02/2011
tultican writes: "How ironic is it that wealthy men who want to do good works are damaging the education of America’s children?" I would say it's only ironic that there are people who still believe that there are "wealthy men who want to do good." In case you didn't kinda notice lately, pretty well all of the super rich are in the game to make more money and have more control. The Koch brothers don't sponsor tea partiers to bust up rallies because they want "good". If there's anyone out there who still believes Gates wants to do anything but bust up public education to expand microsoft's power base, they had best look at how much control they have now. Gates runs the dept of ed. 5 billion very well spent, considering public ed is worth 800 billion a year and he runs it.
photo
HUFFPOST SUPER USER
tultican
Thomas Ultican, MEd. BS Mecahnical Engineering
03:40 PM on 08/02/2011
I guess I am just a little slow, but I think these wealthy people have at least convinced themselves that they are trying to improve things. Gates is already the richest man in the world. I think he is looking for legacy now and is actually giving away billions of dollars. All that money creates distortion so even his misguided ideas are supported.