Good Evaluation Requires Data -- And Sound Judgment

The superintendents and school district human resource officers I spoke with this week said they are struggling with the political and technical aspects of linking student achievement data to teacher evaluation. The policy analysts and labor economists that discussed incorporating data into teacher evaluation at PIE-Net this week admitted that it's a challenge.

Teacher effectiveness is the dominant edreform frame these days that makes teacher evaluation a timely topic. States applying for giant Race to the Top grants were required to outline how they'd overhaul evaluation. And, most recently, states applying for waivers from the widely despised No Child Left Behind must install new evaluation systems that use test data to measure teacher effectiveness by the 2014-15 school year.

The NYTimes said in today's lead editorial, "It seems imprudent to rush the states into bringing these complex new evaluations systems and high-quality tests on line by 2014, given that they will also be expected to adopt new core curriculums."

EdWeek's Michele McNeil explains that peer reviewers will attempt to verify state commitment to using data to improve teacher evaluations by asking questions including:

Is student growth a significant enough part of the new evaluation system to differentiate among teachers who have made "significantly different contributions" (emphasis added) to student growth or closing achievement gaps? Will evaluations be frequent enough? Is there a plan for differentiated professional development based on evaluations? Will the state's plan ensure that local school districts will actually be able to put these new evaluation systems into place by 2013-14 (as a pilot), and 2014-15 (full implementation)?

There is little guidance from the feds to reviewers or states about what it means to have a new evaluation system and how it will impact personnel decisions. To the challenging political and technical aspects of using year-end standardized tests to measure the effectiveness of individual teachers add uncertainty.

I have a different concern; I'm afraid that most of these efforts intend to apply outdated psychometric tools to obsolete school models. The problem is that the sector is still approaching this topic from a data poverty mindset. About the time the labor economists optimize approaches for using cheap multiple choice tests for evaluating individual teachers there will be a flood of data informing teams of educators working in environments that blend online and onsite learning.

Blended learning environments will often leverage the leadership of lead teachers across hundreds of students and several junior staff members. Their work will be informed by rich student profiles. It will be much easier to track student growth in each subject area but more difficult to attribute causality to individual contributors.

It will remain necessary to apply judgment using a variety of observations and outcomes in order to evaluate educator effectiveness. Because staffing patterns and assessment techniques will evolve over the next few years, states and districts should plan on a series of temporary agreements incorporating all available data.

It's obviously a good thing to consider all forms of evidence when providing performance feedback to employees. But teaching is not like golf, you can't reduce it to a single score for one contributor. That's been true and will increasingly be the case as learning environments blend a variety of strategies and technologies. Good evaluations will always apply sound judgment.

 

Follow Tom Vander Ark on Twitter: www.twitter.com/tvanderark

Comments

[Chembug602]

06:39 PM on 10/03/2011

It continues to amaze me how we all think so briefly about incredibly complex topics. We strangely buy that government mandates will quickly fix everything. The recent debate on evaluation is so narrow. We and the media agonize over this issue because it lacks logic. Standardized tests are no doubt the cheapest form of evaluation possible and lack what teachers really need... deep research in the pedagogy for the subject(s) they teach. Universities should be partnering with Intermediate School Districts and administrators to evaluate research in each discipline and bring it to teachers. Our ISD never seems to be do much research beyond the latest "guy who wrote a book". I, as a teacher, have been presented so often locally made, politician initiated, half baked proposals on data driven educational re-design. They lacked any "meat" and if they could work, it would take years to gain any insight from the data. How interesting that I accidentally discovered this year that Arizona State had 30 years of data driven research on teaching my subject properly! Hmmm... perhaps we all need to take a good look at our system, not just teachers. And we as teachers need to be out there finding this stuff NOW because at this point I don't think anyone else is going to help us. Let's figure it out, so those who follow us will not have to suffer as we are.

[nypoet22]

05:58 PM on 10/01/2011

interesting point about online learning, but teacher and societal input into test scores are already incredibly diffuse. is it sound judgment to use standardized test scores for up to 50% of an individual educator's evaluations, when value-added models, even if they were valid (which they're not), would account for less than 15% of student variance?

i don't know if the psychometric tools are outdated, but they sure are lousy at evaluating teachers.

[Tom Vander Ark]

11:44 AM on 10/05/2011

My point is that a simple end of year multiple choice test is ridiculously inadequate to value learning gains. That is changing quickly as more kids spend more time learning online and more teacher benefit from a rich and daily dose of data organized in a useful way.

[changing]

05:19 PM on 10/01/2011

More data driven drivel about this silly evaluation mania. It is not about teachers or education; it is all about money and taking over education for the money. Please no more evaluation products.

[Gwenhevare]

04:44 PM on 10/01/2011

How are the teachers who don't give standardized tests to be evaluated? Is it fair that because I teach 4th grade and another teacher teaches Art that my evaluation is worse? What about the effect of class size and student populations on the test scores? Our economically disadvantaged students did 20% worse on the MCAS last year than the non-economically disadvantaged. Boys too did worse than girls. What if one year you get a class heavy in boys and sped, ELL students? What then?

[nypoet22]

06:03 PM on 10/01/2011

then you give up hope, cheat, get caught, lose your license and get replaced by a 22-year old at a charter school, for 1/3 the salary.

[Gwenhevare]

08:01 PM on 10/01/2011

That's not an acceptable option.

[dickensonr]

12:29 AM on 10/02/2011

Oh, they "fixed" that problem, at least in my state. If you don't teach a tested subject or grade (and actually, the MAJORITY of certified teachers don't), 50% of your evaluation is the school-wide test scores. If you're a Special education teacher, or an ELL teacher. . . that's mostly kids you never see. The PE teacher. All K-2 teachers. We're all thrilled. :)

[Joel Hollingsworth]

01:05 PM on 10/01/2011

This article gets to some of the underlying challenges of linking student performance to pay. The golf analogy is strong -- education is much more a team effort, like medicine, than an individual sport like golf. It will take more than vague exhortations from the feds to accelerate school change.

Principals can not provide satisfactory evaluations of dozens and dozens of staff each year -- maybe a dozen, but not the 40-60 that more standard for a principal to supervise.

[P Alan Greene]

08:05 AM on 10/02/2011

Vague exhortations will be more than enough if the change that we're trying to facilitate is, "Hey, you're failing. We'd better bring in this for-profit group over here to take over."