Six months ago, the Los Angeles Times published a series of articles on teacher effectiveness that relied upon economist Richard Buddin's study of the impact of elementary school teachers on their student test scores. Using seven years of data from the Los Angeles Unified School District, Buddin's analysis looked at how much students' test scores in math and English language arts improved while they were enrolled in particular teachers' classrooms. That change is sometimes referred to as the "value added" by the teacher. The Times decided that Buddin's study was sufficiently valid and reliable that it published a website identifying about 6,000 individual teachers by name on a five-point scale, from "least effective" to "most effective."
At the time, many social scientists raised concerns about the methodology adopted by the Times. They worried, among other things, that the newspaper did not account for all of the factors that lead to higher or lower test scores and that this failure meant that the Times' teacher rankings could not be trusted. But we didn't know for sure, as no one had conducted an independent analysis of the same data used by the Times. Until now. The National Education Policy Center at the University of Colorado has just released a reanalysis of the data. And its conclusions stand in stark contrast to those drawn in the Times reporting.
The goal of the Colorado researchers was to see if they could replicate Buddin's analysis and to test some of the assumptions behind it. They concluded that both the Buddin study and the uses to which the Times had put it were seriously flawed. Yet, on Feb. 7, 2011, the Times published an article written by Jason Felch, one of the authors of the original story with the headline, "Conclusions on Teachers Confirmed." The headline is consistent with Felch's story. Both mischaracterize the facts.
The assertion that the Colorado report confirms that the Times' analysis is based on the finding in both studies that teacher effectiveness varies across teachers. That is, students of some teachers showed more improvement on standardized tests than students of other teachers. The Times' claim is akin to finding agreement between two medical diagnostic procedures because both suggest that some people are healthier than others, even though the diagnostics disagree on who's healthy and who isn't. The Colorado researchers concluded that the Los Angeles Times' method does not reliably identify the teachers who add value to student test scores.
Here are a few of the facts reported in the study but studiously ignored by the Times:
First, the University of Colorado researchers conclude that Buddin's analysis must have left out some important variables that explain the relationship between teachers and their students' test scores. They reach this conclusion because with the variables and methods Buddin used, student test scores are influenced by teachers the students have yet to meet. A statistical model that predicts such impossible results is generally suspect. It gives strong indication that other important variables correlated with student achievement have been omitted and that these variables also influence how students are assigned to teachers in schools.
Second, the Colorado researchers conclude that when one does include those variables (like differences between schools), the rankings the Times used change dramatically, such that about half of the teachers would be assigned a different "effectiveness" rating.
Third, the Colorado researchers find that ratings are subject to significant random error, such that about half of the teachers in the full database cannot be distinguished from "average."
Finally, they contradict the conclusion by Buddin that teacher experience or credentialing do not matter to student achievement, finding instead that inexperienced teachers have significantly less positive impact on student test scores, particularly in reading.
Evaluating teacher effectiveness is a complicated matter, in which changes in test scores in math and English Language Arts may play some appropriate role. There are significant areas of disagreement among experts, policymakers and practitioners, and many opinions about how to proceed. Accordingly, we're all entitled to our own opinion. But not our own facts.
Follow John Rogers on Twitter: www.twitter.com/UCLA_IDEA
Charles Kerchner: New Analysis Shows Serious Flaws in LA Times Evaluation of Teacher Effectiveness
Approaches to Evaluating Teacher Effectiveness: A Research Synthesis
Stakes and mistakes in assessing teacher effectiveness - Blog ...
Methods of Evaluating Teacher Effectiveness - Research to ...
Is Teacher Effectiveness a Myth? - Education - GOOD
The Real Value of Teachers: Using New Information about Teacher ...
YouTube - The Value of a Teacher
Assessing a Teacher's Value - Room for Debate - NYTimes.com
Rating Teachers: The Trouble with Value-Added Analysis - TIME
1-Loopholes. For instance, in my state, the most unfair one is the one where the special ed kid can't go to my school (we don't have his/her program), so they're bussed to School B down the street. We never, ever see that kid, and School B is totally responsible for how s/he is taught. Yet at the end of the year, whatever that kid scores on the test is counted for or against my school. Make sense? Neither does the law.
2-Children see and learn from a variety of different teachers throughout their educational careers. This can be detrimental (if they get a bad one), but sometimes it's simply a matter of...how Teacher A teaches isn't how Teacher B teaches. This is neither bad nor good. I may learn great from Teacher A, you may not. You may learn great from Teacher B, I may not. But we'll both learn, and whatever Teacher B didn't give me, I'll get from Teacher A on down the road. This is true for private, public, charter schools and continues all the way through graduate school (if a student goes that far); it is a fact of becoming and educated person.
My brain doesn't work in ways to figure out how to measure this to find out who's effective, who's not. But I do know a standardized test isn't the way. That's elementary.
More to the point: Are we concerned about a teacher’s effectiveness in producing higher student test scores or a teacher’s effectiveness in providing a learning experience whereby students come away with an enhanced desire and an improved ability to learn at a higher level?
Great teachers CAN be overshadowed by cr@p management, apathetic parents, socio-economic pressures, and culture-based investment in learning.
A "great" teacher on one side of the tracks would completely FAIL if teaching on the other side of the tracks.
Districts in tough neighborhoods are in constant need of teachers. There was a time in the 90s, one near us took teachers on "emergency credential" status - they hadn't even finished their credential. These unqualified teachers were teaching in this rough district, in which pay and spending was directly tied to real estate taxes, which means basement pay and spending. Once these teachers got their full credential they didn't bat an eye at skipping off across town to make an instant increase of $10k per year (who'd blame them), leaving a continuous vacuum sucking sound in the poor district. At one point, in one school in the poor district, the SENIOR teaching in the entire school had only been teaching for six years. Is any of this in any study?
2. Tests lack content validity and reliability
All of the above are arguments that have been made for three decades. I got news for all ... there has since been much research done over that time period which now makes it possible to evaluate teachers and administer effective tests. Is it perfect? Probably not. Is it better than the status quo? Perhaps the real question is why would anyone want to keep the status quo?
It is the height of irony that in order to defend their failure, teachers -- led by their unions -- have now taken the bizarre position that testing itself is a flawed procedure and has no place in schools. Even as they are handing the students quizzes, mid-terms and exams -- that they refuse to teach to -- teachers are screaming and howling about how unfair it is to be tested.
Complaints and excuses are not solutions. Fair or not -- this isn't a workers' rights issue about teachers. The failure of our schools is a civil rights issue about students.
In many urban communities, the dropout rate is near 50%. We can't even keep the children in school . . . let alone teach them to read, write and balance a check book.
"teachers -- led by their unions -- have now taken the bizarre position that testing itself is a flawed procedure and has no place in schools."
especially with a culture of individualism and a culture with a of lack of understanding of the systems influence on human behavior.
if statistics are used be very suspect. I have worked with some of the so called best in value added and statistics and some are out and out liars to prove their points, some fudge the data and consider it harmless, and with most the paradigm effect overrides their findings, and some really care but fail to understand the influence of all the variables and they think they can account for these random variables with thier plugged in numbers. usually they find these numbers and plug in these numbers from you know where. :-)
this nation has little understanding of a system's influence on human behavior, if they did they would be able to see that our profits over people economic system we call capitalism is having a profound negative impact on the american culture.
if anyone wants to take the time to check my past comments about this study it will reveal that I stated it would prove to be flawed for the very reasons named here. 25 years of organizational consulting and statistics taught me that.
when researchers show up at your school know they have an agenda and a very strong paradigm effect working on their beliefs.
So if 5 students are performing at or above grade level, then it is probably only 2 or 3 students who are doing so as a direct result of the teacher. Teaching 2 or 3 students effectively? Hardly a score to be proud of. This means the teacher is not differentiating material, not catering for multiple learning styles, not meeting the needs of the students.
Granted, there are some students who do not come to class with the right attitude, but assuming that attitudes can't be changed, and that as long as there are a few kids keeping up with your class then you are doing a "fine job" does not make a successful teacher.
That is supposed to mean something? What are those tests testing for? Double-entry accounting is SEVEN HUNDRED YEARS OLD and economists don't say it should be mandatory in our schools. There have been over 200,000,000 cars in the US since 1995. Do our economists talk about how much Americans have lost on the depreciation of all of those machines?
Is the economy of the US and the World in the current state because our economists have FLUNKED THEIR TEST..
How about some self-reflection, teacher?
Maybe you could find a way to incorporate texting into your lesson -- since you're obviously aware that it's something the students are interested in. Or how about just dropping the sour attitude?
Stop blaming students for not learning from you, and start figuring out how to teach to them.
We looked into buying phone killers (devices that make it impossible for people to dial in or out), but were told by admin that they are illegal in the state and cannot be used.
Short of wrapping the entire building in aluminum foil there's not much we can do.
The misplace focus on the teacher is a fundamental attribution error.
Teachers are important to the extent that they facilitate a learning environment in which students deeply engage in a conversation about subject matter. However, the school culture, the parenting, and the socio economic climate of the community are all of equal or greater importance to the achievement of the child, which in any case should be measured by test scores, but rather multiple measures that are not numerical.
I've often wondered why it is that conservatives talk about the immorality of liberals, while overlooking the drug use, crimes and sexual indiscretions of famous conservatives. It seems to me to be another example of the same flawed thinking. Liberals are bad people; conservatives merely make forgivable mistakes.
Rational people recognize they're primarily responsible for their own learning; everybody else, they assume, succeeds or fails entirely according to the teachers' abilities.
F and F.