Counting the Countable

My last post described how Charity Navigator's decision to incorporate outcomes data into its rating system is an important step towards creating a culture in which nonprofits strive for impact and donors reward it. But the task of developing meaningful and appropriate outcome measures is complex. It will require cooperation across the nonprofit sector.

Once they have the incentive, I believe most nonprofits of a certain size can track meaningful data that bears on their impact. The Hewlett Foundation and McKinsey recently partnered in a study on the nonprofit marketplace that confirmed that such data is surprisingly knowable if nonprofits invest modest time and money. The Working Group for Effective Social Investing is also hoping to develop new tools to determine which nonprofits generate the most social value.

The whole nonprofit sector will need to participate in developing outcome measures that are cost-effective for nonprofits to track, and useful to organizations as well as to donors. Ideally, measures should be tailored to different kinds of nonprofits and allow refinement over time. Larger organizations can invest in more complex tracking systems that get closer to actual impact. But small or growing nonprofits might do best tracking interim measures of success while waiting for data on outcomes. As Jacob Harold mentioned in his guest post, when evidence-based research on outcomes is impossible, information from the organization about its goals, strategies, capacity, and progress--combined with the views of stakeholders--can provide good proxies for impact. While not all organizations that measure outcomes necessarily have great impact, donors should have serious doubts about nonprofits that can't provide any information about progress towards accomplishing their goals.

It's also important to make sure that donors understand that numbers don't tell the full story. We don't want donors or nonprofits to prioritize short-term, measurable outcomes at the expense of important long-term--but less measurable--impact. It's easy to count beds filled in a homeless shelter or acres of land purchased for conservation, but harder to measure long-term changes in attitudes or behavior.

Even for two nonprofits doing the same kind of work, outcome measures or benchmarks might differ depending on factors such as the population served, time horizon, political or cultural context, geography, etc. For example, let's take two nonprofits that house homeless individuals: One helps the newly homeless attain transitional housing. Another provides the same service to chronically homeless individuals suffering from mental illness and drug addiction. Based on a measure of how many clients remained housed a year later, the first nonprofit might seem the natural choice for a donation. But its better outcomes would actually result from serving a less challenging set of clients.

Given all this complexity, how exactly could Charity Navigator cooperate with others in building an outcomes-driven culture? Here are several options:

1. Gather better programmatic data from nonprofits, as a short-term substitute for the long-term goal of data on outcomes (DonorEdge is doing this)
2. Gather data from experts (Nonprofit Knowledge Network is working on this)
3. Gather data from other constituents (GreatNonprofits is working on this)
4. Rate nonprofits on how transparently they report outcomes (IntelligentGiving in the UK does this)

Charity Navigator's best option may be to shift its ratings from financial ratios to measures of transparency (#4) while supporting other organizations that gather and aggregate the first three types of information. Charity Navigator's CEO has identified accountability and transparency as one of the three essential components of charity evaluation. While transparency ratings don't measure outcomes directly--and transparency may have its costs--they are a step above Charity Navigator's current financial ratios. Nonprofit Knowledge Network and GreatNonprofits are already developing the systems necessary to gather the opinions of experts (#2) and other constituents (#3). GuideStar will likely be the best aggregator of all this information and outcome data from nonprofits (#1) so that nonprofits are not forced to upload information to multiple sites.

While measures of these sorts are crucial, donors will still need to combine them with other--perhaps less quantifiable--information to identify the best nonprofits. As a sign in Albert Einstein's office read, "Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted." The challenge for Charity Navigator and its partners going forward will be to create demand for impact, while giving donors immediate access to multiple kinds and sources of meaningful information when making charitable decisions.

Comments

[DebraWhatWorks]

Paul,

We applaud you for issuing a call to develop outcome measures that will enable nonprofits to track and assess their program results. We agree that such measures are necessary, but we believe that a critical element is missing – namely the need to help nonprofits identify and regularly collect and report, outcome information. Such information is critically important for nonprofits seeking to improve services and client outcomes. Good information about program results is also at the heart of what donors want, but to make outcome measurement viable, there needs to be investment in developing the organizational capacity of nonprofits to create, collect, and use outcome measures. Simply creating the measures is not enough.

The Urban Institute’s 40-year history of creating outcome measures and working with federal, state, and local governments and nonprofits, demonstrates the importance of creating an ongoing dialogue between donors and nonprofits from the start.

In our current work, The Urban Institute and The Center for What Works are putting this bottom-up, hands-on process into practice. We are also participating in several working groups that are trying to create a sound process for advancing outcome measures for the whole sector, nonprofits and donors of all types. We welcome your commitment and ideas and those of everyone working in this field.

Elizabeth T. Boris Debra B. Natenshon
Director, Center on Nonprofits and Philanthropy CEO
The Urban Institute The Center for What Works

[Ken Berger]

Accountability measures (including transparency) are a component that we intend to integrate into our system. However, (as noted earlier) it is one of three dimensions we intend to include.

The biggest challenge is outcome measurement. There is no publicly available or standardized data for this and many charities are either reticent to provide the information or are not gathering it. We are researching whether there are proxies that could get us as close to this as possible, while minimally intruding upon the charity or duplicating the efforts of others. We also agree that collaboration between evaluators of charities is vitally important.

Further, we believe we have a unique role to play in all of this. We know from feedback of many users of our site that the public requires and demands more than just information. They want experts to help them choose who is doing the best work. Many information providers are reluctant to do that because of their desire not to offend and to maximize their funding. I believe strongly that our willingness to judge one charity as performing better than another will remain a role we and only a handful of others will take. We may disagree on the particulars that go into the choosing, but the fundamental importance of the role should be our common ground.

I hope those of us who make these judgments can collaborate toward our mutual goal of seeing high-performing non-profits get more money!

Ken Berger, President & CEO
Charity Navigator