02/07/2013 01:12 pm ET Updated Dec 06, 2017

A New and Better Way to Rate Super Bowl Ads

One of the traditions on Super Bowl Sunday is to judge and assess the value of TV ads that appear during the struggle between the NFC and AFC champions. Companies spend millions of dollars advertising their products and the media respond by offering creative ways to rate these ads. A Google search on "best and worst" and "Super Bowl" and "ads" and "2013" on Feb. 5 yielded 108 million hits. Or consider the attention Jesse Heiman has received for being kissed by Bar Refaeli. His name alone draws over 15 million hits on Google! We love our Super Bowl, but we also love our ads.

With all this attention, we need to think a bit more carefully about how to rate these much dissected Super Bowl ads. Many of the efforts to assess these ads are "seat of the pants" undertakings designed mostly for fun. That is, of course, entertaining and makes for enjoyable reading. Other efforts attempt to be a bit more systematic. The New York Times, for example, had its readers' rate ads following the Super Bowl, as did Hulu. USA Today made use of 7619 people who signed up on their website to rate the Super Bowl ads on a 1 to 10 point scale (10 being the highest rating). With ratings of more than 50 ads, USA Today declared that the Clydesdale ad by Budweiser won "by a nose." The readers of The New York Times found that ad to be the most memorable. Hulu's readers rated the Kia's "babies" ad as its top commercial.

All these efforts, while entertaining, do not provide much reliable insight into the public's reactions to these commercials. Vanderbilt/YouGov (VUYG) Ad Rating project offers a better way to get more systematic and more accurate data about the public's reactions to these ads. VUYG shows a representative sample of Americans (n=500) the actual ad and then secures their detailed reactions to the spot just watched. We ask whether Americans, for example, liked the ad, thought it was memorable, did it make them laugh, did they learn something, and would they buy this product? We have done this for the Kia "Babies" ad, Hyundai "fast" ad, Volkswagen's "Happy" ad, and Audi's "Prom" ad, providing another way to think about the "best" ad.

We collected our survey data prior to the Super Bowl so these judgments by consumers are uncontaminated by the media's coverage of these ads, which commenced in earnest following the conclusion of the game. Now to be fair, there was some coverage of these spots prior to their airing, such as the controversy over whether the VW spot mentioned above was "racist." We did ask our respondents if they had seen these ads in the media or on the Internet. About 10 percent of the public claimed they had previously seen the ad and those respondents did not report significantly different reactions to the spots.

Let's get to the results.


For starters, 58 percent of the public viewed the Audi and Kia ads as "very" or "somewhat" memorable. The Hyundai ad scored much lower on this dimension, with just 38 percent indicating the ad was memorable. Save for the Hyundai ad, a bit over 50 percent of the public "liked" each of the other three car ads. Just 36 percent liked the "fast" ad from Hyundai. Both the Kia and Audi ads made the public laugh the most, with about 40 percent finding those spots funny. The "happy" ad from VW got 31 percent of the public to claim the spot funny, which might be a bit disappointing for VW since they probably wanted it to hit the funny bone a bit more. And again, Hyundai scored very low on this dimension, with only 11 percent finding the ad funny. But that spot was not funny, so the results should not only come as no surprise but suggest that we are in fact measuring things with some accuracy.


Should we conclude that the Hyundai ad was the least effective ad? No, although that is in fact the general claim made by the other rating systems. We have some other data that give us reason to think the Hyundai may be a sleeper. For starters, the public gave the Hyundai ad marks for learning something from the spot, with nearly a third of Americans saying they had learned something from the "fast" ad. By contrast, the Audi, VW, and KIA ads scored somewhere between 10 and 15 percent. If you want consumers to learn things about your product, which is a reasonable goal, Hyundai looks pretty strong. The final piece of data comes from asking the public the obvious question: Are you more or less likely to buy the product? Hyundai draws the strongest reaction: 15 percent are more willing to buy the car and 20 percent are less willing to buy the car. The VW ad, by contrast, scores lower on both dimensions. At first blush, you might think the 20 percent who are less willing to buy a Hyundai is a bad thing. Maybe, but really what Hyundai wants is to increase market share and name recognition. If so, perhaps this less funny, less memorable, but more informative ad accomplished that.


There are surely other ways to slice these data. And this is just a first cut -- and only for a handful of ads. But there are two broader lessons: First, we may want to think more carefully about what constitutes an effective -- or the "best" -- ad. Second, we now have the technology to develop better ways to measure and assess the effectiveness of advertising with the public. VUYG does just that and in so doing gives a slight different spin on some of the ads that have been drawing so much attention since the Ravens held on to beat the 49ers.