The Hewlett Foundation-funded Automated Student Assessment Prize (ASAP) released about 2300 essays to the 140 contestants this week. They have until the end of the month to score the essays with the highest possible agreement with expert graders. The top three competitors will share a $100,000 prize purse to be awarded May 9 in Washington, D.C.
In a recent private vendor demonstration, nine testing companies showed that "machine scoring engines did as well or better than the human graders," as reported by Dr. Mark Shermis, author of the study summarizing the demonstration, Contrasting State-of-the-Art Automated Scoring of Essays. Shermis is the dean of education at the University of Akron, leading authority on online assessment, and author of Classroom Assessment in Action.
Shermis did a KPCC radio interview yesterday where he said the study "demonstrates to the two Race-to-the Top consortia, PARCC and Smarter Balanced, that machines have a roll in future assessments."
Dr. Shermis was on the radio because Michael Winerup wrote a ridiculous New York Times piece quoting a critic of automated scoring. Les Perelman, MIT, has tried one essay grader and thinks he can game the system. He worries about dumbing down American education. The fact that so little writing is assigned in American schools would suggest this is already happening.
Open Education Solutions is managing ASAP because we share Hewlett's interest in seeing students write more in American classrooms. Automated essay scoring will allow teachers to assign 1500 words a week instead of 1500 words a semester. With the help of an essay grader, students get frequent standards-based feedback on key writing traits while teachers are able to focus on voice, narrative, character development, or the logic of an argument.
Perelman said scoring engines just measure sentence and essay length. Shermis said many engine did not use length as a predictor and in fact all of the scoring engines did a much better job than simply using length as a predictor. The public leaderboard shows that more than 100 of the competitors beat a simple word count metric.
On gaming the system, Shermis added that "these engines are very sophisticated -- 40 or 50 variables are operationalized. If a student had mastered 50 variables, they would be good writers."
After Shermis destroyed each of the critic's arguments, Katie, a trained essay grader for several states, called in to the show. While she admitted that she was initially skeptical, she is "totally convinced they are comparable." She continued, "I was so impressed with [computer scoring]. It scored vocabulary and sentence structure. It is a tool that can be used use efficiently and well."
Shermis got to the point, "To be a better writer, you need a lot of practice, you simply have to write more. " We think if students have to write more on state tests, they will write more in classrooms. When every teacher has an automated scoring engine and every student has an access device, we'll see a dramatic increase in writing across the curriculum in American schools
Laura, a university instructor, called in and said, "I use a TA to pre-read essays. A computer system could be used in the same way -- as a prescreening tool." Dr. Shermis agreed that automated scoring is designed to be a "teacher's helper" as Laura suggested and would allow a teacher to make more writing assignments.
Perelman continues to claim that Microsoft Word is better than essay scoring engines -- a ridiculous claim. The nine algorithms tested are several generations beyond Word grammar check capabilities. However, it is likely that powerful scoring engines will soon be as common as spellcheckers.
Next up in the ASAP sequence is short answer questions -- a more difficult challenge than longer essays. The result will be better tests, more writing and deeper learning for American students.
Follow Tom Vander Ark on Twitter: www.twitter.com/tvanderark
I also agree with the comment about the importance of reading. As a homeschool parent, the best thing I ever did was insist my kids read a minimum of one hour per day. I usually read with them - and I found that my own reading skills improved dramatically. I was an English major a couple of decades ago, but my ability to access literature had atrophied from disuse.
Any skill that is practiced for an hour every single day is going to improve. That is why writing frequently is also important.
Rather than teachers submitting student writing for automated scoring, I'd rather see an automated system in which students get immediate feedback on basics like grammar, vocabulary and sentence structure as they write.
Making automated scoring available to students as they write would shift more responsibility to students themselves. The teacher could require students to keep rewriting until they meet a minimum standard - before they are allowed to submit the assignment for teacher evaluation and a grade. The teacher should be able to customize those standards by student as well as for different lessons.
Automated scoring sounds like a powerful tool with tremendous potential to improve learning through teacher-led instruction.