Maddeningly, the anti-poverty impact evaluation craze is precariously close to inflicting an unrealistic hegemony over social change.
No better example exists than the book More Than Good Intentions from the oft-lionized Innovations for Poverty Action institute at Yale University. The book is a genial, readable, entertaining demand for us all to become truth-seekers and truth-tellers -- to get real about the scourge of grinding poverty.
The authors are crusaders for hard-nosed research and evaluation (in particular, randomized controlled trials, the social science gold standard) of anti-poverty programs to inform an idealized donor, foundation program officer or social investor of the future, namely, the perfectly rational decision-maker.
The profession's conceit is that, until an academic evaluator evaluates it, every anti-poverty program is under suspicion. In the closed world of evaluation, what cannot be measured is invisible. What cannot be validated by an evaluator should not be funded.
Responding to the United Nations definition of poverty ("Fundamentally, poverty is a denial of choices and opportunities, a violation of human dignity. It means lack of basic capacity to participate effectively in society."), the authors ask, "This may be entirely true and accurate. But is it useful?"
Perhaps I am living on a planet populated with people, but I love the U.N. poverty definition. All advocates for economic justice do.
Of course, what the authors mean by "useful" is that human dignity is hard to measure. Social impact measurement is inherently limited to reporting on what it can conceive and quantify.
Consider a neighborhood newspaper anywhere in the world. Computing the impact of a newspaper based on its circulation and advertising revenue produces its valuation, not its value. A paper is a social asset advancing free speech and fostering community cohesion. Our support for a free press is grounded in much more than an economist's research evaluation.
To the book's credit, it acknowledges the tension between personal commitment to social justice and the impersonal process of evaluation. But, after paying lip service to our complex and very human do-gooder impulses, in a classic example of escalating commitment, the authors persist in adamantly plugging their brand of academic research evaluation for deciding what works and doesn't in economic development.
At conferences, in research publications and in the offices of funders, the troubling trend is a forming hierarchy of hubris. Soon, in-the-trenches anti-poverty practitioners with long experience, community-based organizations close to their clients, market-based programs with real revenues and real customers, and experimental, innovative initiatives with great promise may be written off as woolly-headed, undisciplined or unscalable simply because they are un-evaluated.
To my way of thinking, this is upside down. Art critics more valued than artists? Sports writers more admired than athletes? Political pundits more powerful than elected leaders?
If foundations stopped funding social impact studies, would it remain a "hot" fad? Would anyone outside the halls of academia really care?
We need evaluators and critics. The rougher and tougher, the better. What we don't need is academic hegemony over activism.
(Next week: Part Two on microfinance evaluation.)
Follow Jonathan Lewis on Twitter: www.twitter.com/IonPoverty
Craig and Marc Kielburger: Playing Video Games for Social Change
Ann Cotton: Financial Literacy More Than The Sum Of Its Parts For At-Risk Women
I've posted a blog responding to your blog here: http://wp.me/pU1pQ-zg
Whilst I disagree with much of what you say, I think you raise some interesting issues. Also your comments reflect a scepticism about impact measurement of many in the charitable/ social enterprise sector, so its useful that you aired them.
Many thanks for your contribution to the debate.
Benedict
New Philanthropy Capital
In their totality, the blogs are not arguing against evaluation. However, I am against outsourcing judgment and life experience to the evaluation community whose members often present their particular methodology as "the way" – a singular lack of humility and realism, again as my later blogs attempt to point out. Medicine is a science, but doctors practice the art of medicine – perhaps a more apt analogy.
Thanks.
http://www.p-ced.com/1/about/background/
When a few months ago, the Royal Bank of Scotland requested a social report as part of my entry for their SE 100 index, I reflected on what had changed since we began and what could reasonably be attributed to this work, irrespective of the financial cost.
http://socialbusiness.socialgo.com/magazine/read/people-centered-economic-development--social-impact-report---march-2011_25.html
Consider one of these, the case argued for more than 15 years, that business could and should have both a social and financial return in an enlightened approach to capitalism.
At See Change, we’re developing an approach called “story science,” which puts tacit knowledge and intuitive understanding (the things we actually use to make decisions) at the center of an impact inquiry. Rather than use “stories” to accessorize a quantitative report, or tug at the heart strings of donors, we see the “story” of an implementation process as told by the people most involved in it – including, but not limited to a reporting of any results – as a source of tremendous, philanthropocentric learning.
Thanks, Jonathan, for opening the floodgates of this important debate!
Whether to evaluate social change work is an important cost/benefit question. What is the value of the knowledge potentially gained from evaluation? Will it actually change a program’s practices? Alter a grant or investment decision? Research shows that despite what we like to believe, most evaluation is NOT used in these ways. Why spend money on research if you’re not going to use it?
But the most critical question is not whether, but HOW to evaluate social change work. As a field, we are attempting to do a neuroscientist’s work with a phrenologist’s tools. Innovation in evaluation methodology is essential. We must design research practices that embrace the complexity and context of the messy, noisy human situations we work in – practices that factor variables in, rather than isolate them. The good news is that when we use both sides of it, our human brain has an untapped capacity to do just this. There are some things we know intuitively through our own repeated experiences. [cont]
“…in lieu of having our own framework for evaluation, we will continue to import and use approaches borrowed from other fields. This works to some degree, but it also leaves us, and our grantees, unduly influenced and sometimes whiplashed by new models and approaches as they enter our field of vision. It would be good to develop a more “philanthropocentric” framework for our field…”
She cautions that such an approach must evolve beyond the paradigms and research practices we have used – by default – to date. Borrowing the concept of “ROI” from the business sector, or the controlled methods of the laboratory sciences will not necessarily advance our understanding of the tremendously complex human dynamics encountered in the field of social change. The causal framework that surrounds our current inquiries is the hammer in our hands that makes every NGO intervention look like a nail. [cont]
However, when there are big dollars at stake (largely looking at foundations or gov't/quasi-governmental donors) it is imperative that we strive for accountability. RCTs or similar impact evaluations shouldn't be the only measure, but I'd love to hear your thoughts on alternatives.
In fact, one of the reasons that microfinance became such a big sensation is that it offers an alternative metric. With solid risk-assessment wholesale microfinance lending allows donors (or investors) to preserve (or grow) capital while still reaching end clients. Jonathan Lewis knows this as well as anyone, which is why I'm looking forward to the next installment in this series...
As it turns out, I am looking forward to my next blog too…as soon as I sort out my thoughts. Hahaha. Your comments, and everyone else’s here, the blogosphere, private emails, etc. are causing me to think harder about this topic than I normally do. For that, thanks!!!!
However, it is hard for me to envision the millions of traditional donors, innovative social investors, foundation program officers, government aid agencies, etc. moving to some norm about best practice in this area. The human factors which drive support for international development, I think, will trump in the end the more rational evaluation process and findings. Hell, we cannot even put together a national consensus on climate change! In short, I think social impact evaluation may turn out to be diversion – good to do, but not central to long-term, scalable anti-poverty work.
Please read my extended reply to Dannno. Also, I will try to address your questions in the future blogs.
I'd be interested in your response to a thought experiment. With limited funds available, if two applications for funding both suggest they have local expertise; understanding of the context and a clear rationale for what will make a useful difference in their area - what process would you suggest to determine which to fund? Another way of looking at this - if you had, say, $1m to address a particular social challenge, how would you go about deciding how to use that money (not what would you use it for, but what process would you use to decide how to use it)?
I am not an evaluation expert, not even close. Nor am I a lawyer, accountant or field practitioner. Experts are needed, but their advice, findings, wisdom and expertise are part of the decision-making grist, not a replacement for it.
My blog, which was sparked by More Than Good Intentions (I recommend it to you), is written at a time when the anti-poverty sector is actively, almost obsessively, in search of social metrics, impact evaluation techniques, etc. It is the policy conference flavor of the moment.
The drivers: a) Foundations funding research to determine if their money is being wisely spent (a version of your thought experiment) which, for numerous reasons, feels threatening to many. b) A lack of practitioner confidence from years of being sneered at for running “soft” nonprofits (undisciplined by the market or other such claptrap), etc. c) The unproven dream that buckets of private sector money will flood into the social sector if only viable social metrics could/would demonstrate the good that is being accomplished.
(continued in next comment)
Understandably, evaluators “pitch” a narrative: Our work will determine what works best and why -- which, in turn, will both improve programs AND, precisely as your thought experiment showcases, give funders/investors the information to effectively direct their money. Also understandably, in the complex business of alleviating poverty while struggling to secure scarce resources, external validation is good to have. The synthesis endpoint, which I reject, is the evolving, de facto and mostly unsaid notion that evaluators become gatekeepers.
To answer your question, I and most social investors use a variety of information points. Certainly, external evaluation (whatever the methodology, a separate matter entirely) should be one of them. Also important is an organization’s financial sustainability and/or fiscal track record; a subjective sense of the management team’s chops and cohesiveness (i.e., hoped-for ability to handle the next challenge or crisis); the enterprise’s core concept (i.e., based on my life experience, do I understand/agree with the mission statement); program’s alignment with my personal interests (policy area, geographic focus, population profile, etc.); and a zillion other factors – some objective, some subjective; some intelligent; most less so.
My blog series is a caution to the social enterprise saector which seems enamored with outsourcing judgment. I hope I have not implied that I have at my finger tips a ready answer to your thought experiment. I don’t appreciate most of what drives my decision-making. Do any of us?
I’m not convinced by one premise of your argument:
"The profession's conceit is that, until an academic evaluator evaluates it, every anti-poverty program is under suspicion. In the closed world of evaluation, what cannot be measured is invisible. What cannot be validated by an evaluator should not be funded"
I would argue that isn't the case. I was once presented with the slur that 'sure it works in practice, but it doesn't work in theory' as a response to my question about what evaluation took place. (A side note, but for me something working in practice that doesn't make sense theoretically is the most exciting as it gives rise to the opportunity to develop our understanding further).
The essence of your challenge is that academic evaluation of 'what works' isn't as useful as the tacit evaluation provided by practitioners who have experienced what works within their own context. In that line of argument, evaluation is not a bad thing, but the person/expertise/approach that decides what the value of a given approach could be. [cont]