Microsoft's search engine, Bing, wants you to believe users prefer its service to Google's in a blind test by a 2-to-1 margin. Yale University law professor Ian Ayres doesn't buy it.
Microsoft bases its claim on a survey of 1,000 people conducted by independent research company Answers Research. Bing repeats the claim in a large TV commercial campaign, promoting the "Bing It On" challenge.
The company boasts that 5 million people have gone to BingItOn.com to take the challenge, but has not released the results. (This is because they don't track that data for ethical and methodological reasons, Matt Wallaert, behavioral psychologist at Bing explained in a blog post.)
But Bing only commands about 18 percent of search engine use, compared to Google's 67 percent, according to comScore, with Yahoo carrying 11.3 percent.
So Ayres worked with four law students at Yale to conduct a similar test of 1,000 people, and found a majority of 53 percent chose Google, while 41 percent preferred Bing.
Ayres wrote last week on the Freakonomics blog that Bing did better when using search terms suggested by Microsoft, which could indicate a possible flaw in the company's methods:
We also interjected a bit of randomness into our study to test whether the type of search term impacts the likelihood that Bing is preferred. We randomly assigned participants to search for one of three kinds of keywords: Bing's suggested search terms, popular search terms, and self-suggested search terms. When Bing-suggested search terms were used the two engines statistically tied (47% preferring Bing vs. 48% preferring Google). But when the subjects in the study suggested their own searches or used the web's most popular searches, a sizable gap appeared: 55-57% preferred Google while only 35-39% preferred Bing. These secondary tests indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results. You can read our full paper here.
Bing's Wallaert struck back in his own blog post at Bing, referring to Ayres as "someone unfamiliar with the actual Bing It On studies." But Wallaert doesn't dispute Ayres' results, he only defends Bing's research methods.
Wallaert was a little more defensive in a statement to Slate. "The professor's analysis is flawed and based on an incomplete understanding of both the claims and the Challenge," Wallaert said. "The Bing It On claim is 100 percent accurate."
Still, Ayres believes Bing is using deceptive advertising.
"Several of Microsoft’s claims are a little fishy ... It could be worth a lot of money on lost ad revenue if the claims misled people into thinking that a substantial majority of people prefer Bing over Google," the professor wrote.
Related
Before You Go
