Online publishers and advertisers for some time have measured success almost entirely with a single metric: clicks. It’s created a perverse system that rewards publishers and their content producers for goading users into landing on a page with splashy or misleading headlines and content that's attractive and easily digestible, but without much value. Measuring page views = clickbait, in other words.
Brian Boyer, the head of National Public Radio’s visuals team, wants to change that online measure of success for news organizations. Boyer and a team of three others were recently awarded a $35,000 grant from the John S. and James L. Knight Foundation to come up with a tool that measures the impact of media -- whether it’s a story, slideshow, video or interactive feature -- beyond someone having landed on a webpage. The project, called Carebot, is still in early development -- the project kicks off in February and the team will launch something by April. But the goal is to produce a prototype that measures impact in a more meaningful, substantive way. The Huffington Post spoke with Boyer about the project.
Tell me a little bit about the problem you set out to solve. How did the idea first come about?
The question that all of us eventually ask ourselves is: Are we doing our jobs, right? Do we have an impact? When I was at ProPublica, at the morning meeting we would shout, “Impact!” when Congress passed a bill related to our reporting. But at NPR, our bread and butter isn't really accountability journalism; it’s about introducing the audience to people and ideas they're not familiar with. So we started thinking about different alternative ways to look at our analytics, and other data we could be gathering that might give us more of a window into if a story made somebody give a shit.
At the end of the day, empathy is our impact.
How do you measure empathy?
My joke about it is we can't put our audience in an MRI machine every day, and we don't have a window into their souls. So, what can we use as a proxy for those things? A raw page view is just a proxy for: Did we provoke you to click? It's not a proxy for if you read it. It's not a proxy for if you gave a shit. It's a proxy for: Were we successful at provoking you? Google Analytics provides us a “time on site” number, but that only roughly allows you to estimate how much time someone spent on a story.
The important number we've come up with to measure impact so far is completion rate -- what proportion of people who landed on a page cared enough to read it through? We’re still brainstorming and experimenting with ways to measure how much someone cared about a story. We’re testing the use of buttons at the end of the story that ask the user if they loved the content. We’re looking at the number of shares on social media per page view -- the percentage of readers who were moved to tell their friends about a story. But using page views as the denominator and not the numerator seems like a smart thing to do.
How would this make a difference in which stories are celebrated?
One of my favorite pieces is a piece we did about the civil war in Yemen. That's about as “eat your broccoli” as a news story gets, right? And the number of raw page views is something like 50-60,000 people, which is not a barn burner, but not a total failure. But of those 60,000 people, something like 65 percent completed the piece. That's a perfect example of, hey, the page view number was okay, but the people who saw it gave a shit.
What has been the project’s most surprising finding thus far?
One experiment we ran earlier today asked people at the end of a story a very simple question: “Did you love this story?” Instead of having a proxy for measuring impact, why don't we just ask them? They can click on a button that says yes or no. We A/B tested it and the results were fascinating.
In one test, we put a “Support Public Radio” button at the end of a story, and only a tiny fraction of a percentage of people clicked it. But when we asked them if they loved the story before asking them to donate, they were 10 times more likely to click the donate button, which blew our minds.
I know that you're still in the beginning stages of development, but are you thinking about bringing different metrics into a single number and having something like a Klout score, which measures online social influence?
That's one of the big questions. We certainly thought about it, and the wall you run into is, “Well, if I'm going to combine these numbers, then I need to give them some kind of weighting.” And at this moment, we haven't gathered enough data about enough stories to really decide for ourselves.
What are you envisioning in terms of a product? Something like an analytics dashboard?
We're going to have to spend some time designing it, but the general feeling is to be pretty lightweight about the visuals; if there's even anything on the Web, it'll be pretty simple.
Instead of having an elaborate analytics dashboard you log into, we’re thinking of a system that is notification-based. It would be along the lines of an email that says, “Hey, Sarah, your story really killed it in making people care yesterday.” Or I can imagine us writing a Slackbot because the messaging app is the place where the team hangs out. Otherwise, you create a dashboard, people just have to go to it, and it’s always an uphill battle getting people to add anything to their routine. You've got to get in people's inboxes.
Say the entire industry starts tracking itself in terms of impact rather than clicks. What changes?
News organizations are probably more like churches than regular businesses, and NPR is one of the prime examples of an organization that actually acts like a church. We ask people to regularly tithe, and so, for me, if we can create more stories that make people care, then they will, hopefully, become giving members of their local radio station, and it reinforces our business model. We have fewer perverse incentives to write clickbait.
Watching The Washington Post staff up their digital organization and then get heavy in the clickbait business, and then celebrate surpassing The New York Times in page views bums me out. There are great journalists there, and the work that they're publishing that surpasses -- that they're celebrating -- it's not their best work!
Speaking of impact, what do you hope the impact of your project will be?
It's really important what you celebrate. If we celebrate and highlight stories that have impact as opposed to just generating page views, does our behavior change? Does their output of stories that make people care increase? I don't know the answer. That's our hypothesis.
