Wouldn't it be cool if there was a simple trick to score better on college entrance exams like the SAT and other tests?
There is a reputable claim that such a trick exists. Unfortunately, the trick does not appear to be real.
This is the story of an academic paper where I am a co-author with possible lessons for life both inside and outside the Academy.
In the spring of 2012, I was reading Nobel Laureate Daniel Kahneman's book, Thinking, Fast and Slow.
Professor Kahneman discussed an intriguing finding that people score higher on a test if the questions are hard to read. The particular test used in the study is the CRT or cognitive reflection task invented by Shane Frederick of Yale. The CRT itself is interesting, but what Professor Kahneman wrote was amazing to me,
90% of the students who saw the CRT in normal font made at least one mistake in the test, but the proportion dropped to 35% when the font was barely legible. You read this correctly: Performance was better with the bad font.
I thought this was so cool. The idea is simple, powerful and easy to grasp. An oyster makes a pearl by reacting to the irritation of a grain of sand. Body builders become huge by lifting more weight. Can we kick our brains into a higher gear by making the problem harder?
Malcolm Gladwell also thought the result was cool. Here is his description his book, David and Goliath:
The CRT is really hard. But here's the strange thing. Do you know the easiest way to raise people's scores on the test? Make it just a little bit harder. The psychologists Adam Alter and Daniel Oppenheimer tried this a few years ago with a group of undergraduates at Princeton University.
First they gave the CRT the normal way, and the students averaged 1.9 correct answers out of three. That's pretty good, though it is well short of the 2.18 that MIT students averaged. Then Alter and Oppenheimer printed out the test questions in a font that was really hard to read...
The average score this time around? 2.45. Suddenly, the students were doing much better than their counterparts at MIT.
As I read Professor Kahneman's description, I looked at the clock and realized I was teaching a class in about an hour, and the class topic for the day was related to this study. I immediately created two versions of the CRT and had my students take the test -- half with an easy to read presentation and half with a hard to read version.
Within three hours of reading about the idea in Professor Kahneman's book, I had my own data in the form of the scores from 20 students. Unlike the study described by Professor Kahneman, however, my students did not perform any better statistically with the hard-to-read version. I emailed Shane Frederick at Yale with my story and data, and he responded that he was doing further research on the topic.
Roughly three years later, Andrew Meyer, Shane Frederick and eight other authors (including me) have published a paper that argues the hard-to-read presentation does not lead to higher performance.
The original paper reached its conclusions based on the test scores of 40 people. In our paper, we analyze a total of over 7,000 people by looking at the original study and 16 additional studies. Our summary:
Easy-to-read average score: 1.43/3 (17 studies, 3,657 people)
Hard-to-read average score: 1.42/3 (17 studies, 3,710 people)
Malcolm Gladwell wrote, "Do you know the easiest way to raise people's scores on the test? Make it just a little bit harder." The data suggest that Malcolm Gladwell's statement is false.
I take two lessons from this story.
1. Beware simple stories.
"The price of metaphor is eternal vigilance." Richard Lewontin attributes this quote to Arturo Rosenblueth and Norbert Wiener.
The story told by Professor Kahneman and by Malcolm Gladwell is very good. In most cases, however, reality is messier than the summary story.
2. Ideas have considerable Meme-mentum.
Richard Dawkins created the word meme to describe ideas and other cultural creations. Ideas are slow to change; I call this "meme-mentum."
"And yet it moves," This quote is attributed to Galileo when forced to retract his statement that the earth moves around the sun. The earth stayed at the center of the universe for many people for decades or even centuries after Copernicus.
One of my favorite examples of meme-mentum concerns stomach ulcers. Barry Marshall and Robin Warren faced skepticism to their view that many stomach ulcers are caused by bacteria (helicobacter pylori). Professor Marshall describes the scientific response to his idea as ridicule; in response, he gave himself an ulcer drinking the bacteria. Marshall gives a personal account of his self-infection in his Nobel Prize acceptance video (the self-infection portion starts at around 25:00).
I expect that the story, as presented by Professor Kahneman and Malcolm Gladwell, will persist for decades. Millions of people have read these false accounts. The message is simple, powerful and important.
When you hear this at a cocktail party, however, now you know that the story is not true.