If restaurateurs didn't already worry about Yelp, they're about to. A new study published in the Economic Journal claims that as little as a half-star difference in a restaurant's average rating on Yelp greatly impacts its reservations each night.
The study, which examined San Francisco area restaurants, used empirical data to back up the widespread belief that internet review forums influence consumers to an incredible degree:
We implement a regression discontinuity design to estimate the effect of positive Yelp.com ratings on restaurant reservation availability. An extra half-star rating causes restaurants to sell out 19 percentage points (49%) more frequently, with larger impacts when alternate information is more scarce. These returns suggest that restaurateurs face incentives to leave fake reviews but a rich set of robustness checks confirm that restaurants do not manipulate ratings in a confounding, discontinuous manner.
The study contradicts earlier findings from a retail research firm that said that only 14 percent of a diners' motivation could be linked to online reviews and that fine dining restaurants suffered minimally from bad Yelp reviews.
UPDATE: Jeremy Magruder, one of the study's authors, provided the following description of how the study was performed. It explains what led the researchers to determine that Yelp ratings had a causal effect -- and not just a correlative one -- on restaurant reservations:
People who submit reviews to Yelp submit something which is 1, 2, 3, 4, or 5 stars. Yelp takes these together and aggregates them to create an average rating for each restaurant. Then, it rounds that average rating to the half-star level. So restaurants are displayed with 3, 3.5, or 4 stars.
The true average rating, of course, is continuous and could be any number between 1 and 5 - so for example there are some restaurants have true ratings of 3.74, while others have a true rating of 3.76. We might think these restaurants are pretty similar in terms of food quality, service, etc. - whatever people take into consideration when making a review, they've ultimately reviewed these two restaurants almost exactly the same. But because Yelp rounds its ratings, the 3.74 restaurant will be displayed as 3.5 stars, while the 3.76 restaurant will be displayed as 4 stars. By making comparisons like that, we can see where there are differences in reservation availability between two restaurants which are almost exactly the same in terms of how customers review them, but which look very different on Yelp. It's these types of comparisons which let us say that the only "real" difference between these two restaurants is Yelp - and when we compare restaurants like these, we see that the restaurants which just barely get 4 stars sell out about 19% more frequently than restaurants which almost get 4 stars. This is what lets us infer causality.
Correction: An earlier version of this article claimed that the study was unable to demonstrate causality based on a quote from the researchers taken out of context in the Guardian. The post has been updated with a statement from one of the study's authors describing the process and how it allowed them to infer a causal relationship.
Also on HuffPost:
Yelp Calls Lying "Personal Opinion"
A blog post on the SFWeekly website this week brought <a href="http://blogs.sfweekly.com/foodie/2011/09/yelp_lying_about_working_for_s.php" target="_hplink">a troubling Yelp-related incident</a> to light. A Yelper claimed, falsely, to be a writer for SFWeekly in a restaurant review. <em>SF Weekly</em>'s Food editor caught the lie and contacted the reviewer; she admitted that she actually wrote for <em>SF Weekly Voice</em>, and said she'd ask Yelp to change the review. But the website refused to amend the review -- a representative told SF Weekly that the lie in question was "personal opinion."
Yelp Is Known To Be An Outlet For Shilling
So many restaurateurs and publicists post glowing reviews of their own restaurants that Eater has a whole column dedicated to <a href="http://ny.eater.com/tags/adventures-in-shilling" target="_hplink">sniffing out shilly reviews</a>.
Yelpers Are Totally Anonymous
People do all sorts of weird things when they know they're unidentifiable -- which can throw off the average on sites like Yelp, which rely on the forthrightness and honesty of strangers.
Yelpers Can Rate Restaurants They Haven't Visited
Because of said anonymity, there's not even any guarantee that a Yelper has visited the restaurant they're reviewing. One infamous case of this sort of fraud took place in Graham Elliott's <a href="http://www.grahamwich.com/" target="_hplink">Grahamwich</a> restaurant in Chicago -- one Yelper gave it <a href="http://eater.com/archives/2010/09/01/graham-elliots-grahamwich-gets-negative-yelp-review.php" target="_hplink">a vicious one-star review before it even opened</a>.
Yelp Sorts Its Reviews In Mysterious Ways
Yelp has been <a href="http://techcrunch.com/2010/02/24/yelp-class-action-lawsuit/" target="_hplink">criticized in the past for the byzantine methods</a> it uses to sort reviews on a given restaurant's page. According to the site's FAQ, <blockquote>"Yelp's default sort order takes a number of factors into account and reflects our own attempt to present reviews in a meaningful order. For example, we'll favor reviews from your friends and the users you follow. The sort algorithm does not take into account whether the business is an advertiser or not."</blockquote> Thanks, Yelp. That clarifies literally nothing.
Yelp Bullies Restaurants
According to some reports, the "Yelp sort" has an insidious monetary element. Yelp salespeople have been known to<a href="http://www.eastbayexpress.com/ebx/yelp-and-the-business-of-extortion-20/Content?oid=1176635" target="_hplink"> call restaurants offering to push bad reviews far down in the list</a>, if they'll agree to pay to advertise.
Yelp Doesn't Give Guidelines About Stars
There are no guidelines on Yelp for what different star ratings mean -- one person's five star experience could be a jaded gastronaut's three star. Yelp does release the distribution of stars actually given out -- and they're definitely slanted toward the positive.
Only A Tiny Minority Of Yelp Users Rate Restaurants
Many more people <a href="http://bits.blogs.nytimes.com/2008/05/12/why-yelp-works/#?wtoeid=growl1_r1_v5" target="_hplink">read Yelp than actually write reviews</a>. Most reviewers fall into two categories: people who review every restaurant they visit (see the next slide for more on the "Yelp Elite") and those who have an extreme reaction. People probably won't review a restaurant if they thought it was just OK -- but they are likely to review it if they have very good or very bad experiences, making it hard to trust any given review.
Restaurants Throw Special Parties For The "Yelp Elite"
If you're one of those people who writes a Yelp review of every nail salon and yakitori joint you visit, you can become part of the "<a href="http://www.yelp.com/elite" target="_hplink">Yelp Elite</a>." Once you're part of this creme-de-la-creme, you can get invited to <a href="http://www.msnbc.msn.com/id/43344769/ns/business-local_business/t/yelps-elite-epicurean-force-totally-free-labor/" target="_hplink">special "Elite-only" parties that restaurants</a> throw to attract good reviews. Sounds nice, right? It may be, for the "Yelp Elite" -- but it's bad for the general public, because these events often translate into glowing, misleading reviews for the party-throwing restaurants.