The William and Flora Hewlett Foundation awarded $100,000 today to the top five teams in a competition to accurately score short answer test questions.
The first phase of the Automated Student Assessment Prize addressed essay scoring. The second phase of ASAP, awarded today, took on the more difficult challenge of scoring short answers.
Hundreds of data scientists from around the world took on the challenge. The first place winner, Luis Tandalla is from Ecuador and is a student at the University of New Orleans. Other winners are from Singapore and Solvenia -- you couldn't make this stuff up.
The Hewlett Foundation sponsored the first two phases of ASAP in an effort to improve the quality of state tests. "Giving school systems the tools to challenge students to develop critical reasoning skills is crucial to making those students competitive in the new century," said Barbara Chow, Education Program Director at the Hewlett Foundation. "And critical reasoning is one of the capabilities, along with communicating clearly, working cooperatively, and learning independently, that we call Deeper Learning would like to see broadly embraced throughout the country."
Jaison Mogan and I direct ASAP. Lynn Van Deventer managed the competition. ASAP was hosted on Kaggle (www.kaggle.com), the leading platform for predictive modeling competitions. Dr. Mark Shermis is the academic advisor.
Highlights from the release follow:
Participants in the competition had access to more than 27,000 hand-scored short-answer responses that varied in length, type and grading protocols. They were challenged to develop software designed to faithfully replicate the assessments of trained expert educators using multiple rubrics. The systems do not independently assess the merits of a response; instead, they predict how a person would have scored the response under optimal conditions.
187 participants across 150 teams tackled the incredibly difficult challenge of developing new software that can score short-answer responses to questions on state standardized tests. Competing teams developed their systems over three months and shared their technical approaches through an active discussion board. Documentation of the winning submissions will be released, under an open license, to enable others to build on this competition's success and advance the field of automated assessment.
The competition drew more than 1,800 entries, including those from two commercial vendors. Since the advent of ASAP nearly a year ago, it has inspired participants to develop innovative and accurate ways to improve on currently available scoring technologies. For this competition, Measurement, Inc., a company that provides achievement tests and scoring services, partnered with the third place team from the first competition, allowing them to outperform all other teams.
CONTEST WINNERS
The 187 participants in the competition reside in countries from around the world and work in diverse occupations. Competitors scored more than 22,000 responses to ten prompts from three different states. On average, each answer was approximately 50 words in length. Some responses were more dependent upon source materials than others, and the answers cover a broad range of disciplines (from English Language Arts to Science). The range of answer types was provided to develop a better understand of the strengths of specific solutions. Technical methods papers, outlining the winners' specific approach along with any known limitations were created and will be released to the public.
•Luis Tandalla, 1st place - Originally, from Quito, Ecuador, Luis is currently a college student at the University of New Orleans, Louisiana, majoring in mechanical engineering. A newcomer to data science, Luis's first experience was one year ago when he took a Machine Learning Course from Dr. Andrew Ng. Luis also participated as part of a team in phase 1 of ASAP, placing 13th.
•Jure Zbontar, 2nd place - Jure lives and works in Ljubljana, Slovenia, where he is a teaching assistant at the Faculty of Computer and Information Science. He's pursuing a PhD in computer science in the field of machine learning. Besides spending time behind his computer, he also enjoys rock climbing and curling.
•Xavier Conort, 3rd place - A French-born actuary, Xavier runs a consultancy in Singapore. Before becoming a data scientist enthusiast, Xavier held different roles (actuary, CFO, risk manager) in the life and non-life insurance industry in France, Brazil and China. Xavier holds two masters' degrees and is a Chartered Enterprise Risk Analyst.
•James Jesensky, 4th place - With more than 20 years' experience as a software developer, James current works in the field of financial services near Pittsburgh, PA. He enjoys these competitions because they allow him to combine his computer science expertise with his life-long love of recreational mathematics.
The 5th place team is an international duo of data experts. Members include:
•Jonathan Peters, 5th place - Based in the United Kingdom, Jonathan works for the National Health Service as a public health analyst. He spends most of his time modeling death and disease; Kaggle competitions offer some light relief.
•Paweł Jankiewicz, 5th place - Paweł lives in Poland and works as a banking reporting specialist. His machine learning experience began when he attended Dr. Andrew Ng's online Machine Learning class in 2011. Apart from Kaggle, he enjoys English audiobooks, especially the "Wheel of Time" series.
Disclosure: Tom Vander Ark is CEO of Open Education Solutions and a partner at Learn Capital, a venture capital firm that invests in educational technology.