09/18/2013 08:30 am ET Updated Nov 18, 2013

We Need to Rethink How We Measure 'Improved Sanitation'

Building on the first debate to accelerate progress towards the MDGs, the Skoll World Forum partnered with Johnson & Johnson, the United Nations Foundation, the Bill and Melinda Gates Foundation and the Huffington Post to produce another online debate--this time focused on critical issues that do not have enough of a spotlight in the discussions on how to achieve the MDGs or what should be in the next global development framework. As part of that discussion, we asked some of the world's leading experts what's not being discussed during UN Week this year about the post-2015 development framework, but should be? View the full debate here.

Almost 15 years ago, a private contractor wrote some orbital navigation software for a Mars orbiter using American measurements, instead of the metric system NASA uses. The $125 million craft missed its orbit and became another hunk of space junk, currently orbiting the sun. As the UN discusses sanitation aims post-MDG, I'm afraid we're making a similar error: we're measuring the wrong thing, and then coming up with unreliable numbers. We may miss our real target too--and it's not improved sanitation.

The UN measures progress along the "sanitation ladder." Open defecation is at the bottom and each step up is defined by access to better facilities. The top rung is "improved sanitation," which protects users from contact with the waste. This can be anything from a ventilated & slabbed pit latrine to a flush toilet. According to the UN's own estimates, 2.5 billion people lack access to improved sanitation, including over a billion who practice open defecation.

Yet those numbers, bad as they are, may understate the problem. When researchers took a look at the UN Joint Monitoring Programme's reported sanitation gains in East Africa, they found that the results just weren't plausible: too much political upheaval, too little funding available, too much growth in crowded, impoverished, hard-to-serve urban areas. Other sources may be no more reliable. One sanitation access veteran recently wrote about the problem of "wildly, embarrassingly, inflated" numbers used by governments and funders alike, under pressure for measurable success now. He described it as the "elephant under the radar" at the recently concluded World Water Week.

Worse, "improved sanitation" may offer relatively little improvement beyond the individual level. A recent study by the Water Institute looked at improved sanitation systems to find out what happens to the waste itself. They found that sixty percent of the world's population--4.1 billion people--use sanitation systems that simply dump the untreated waste back into the environment. If we're serious about improving infant mortality, reducing fecal-borne illnesses, and using our water resources wisely, we can't stop with improved sanitation.

The first and easiest step to improve things: Interrogate the data. The researchers who studied the JMP reports recommend cross-checking the numbers with other, more reliable figures such as population and income stats. All of us--NGOs, funders, governments, private sector partners--must insist on good data and rigorous analysis. Otherwise, we have no way of knowing what's actually working.

Then, streamline the data-gathering. We've learned that community-based involvement creates sanitation systems that are better accepted and more functional than those that result from top-down planning. In the same way, community monitoring can bypass organizational filters, creating more accurate data and greater transparency. Apps, integrated GPS, camera phones, and SMS surveys put reporting power directly in the hands of users.

In 2010, Water for People and AKVO rolled out FLOW, the Field Level Operations Watch. This online system gathers and analyzes user data on tens of thousands of water points in 17 countries, and those numbers are reviewed and used by over thirty funders and NGOs. John Sauer of Water For People told me, "FLOW has been instrumental for us in adjusting our programming to meet real needs as they develop."

We need a monitoring system like this for sanitation. The 2012 Sanitation Hackathon (for which Toilet Hackers was an implementing partner) brought us a few steps closer to that goal, producing useful innovations such as Taarifa and mWater's sanitation-specific app. Last year's SanHack was just our first volley of innovation in ICT and mobile technology. There is still a great need for breakthrough tools in technology (in both software and hardware), market approaches, and behavior change--all of which requires a fearless commitment to collaboration, to improving existing tactics, and experimenting with new ones.

Finally, donors, funders, and implementers need to integrate waste treatment needs into our sanitation goals and measurements. Public support is growing. In discussing a sustainable development agenda post-2015, citizens, scholars, government workers, and business people alike "reconsidered the use, reuse, value and even meaning of 'waste' water. ... Rewards could go beyond resilience to boost growth, jobs and business certainty." And the Joint Monitoring Programme's own proposed targets (PDF at link) for ongoing sanitation development now include sewage treatment.

Most promising researchers are developing new methods of measuring our progress, moving away from specific technologies toward desired outcomes, including sanitation treatment. This allows for local flexibility, spurs innovation, and helps us all focus less on process and more on results. Truly improving sanitation requires working within, responding to, and shaping complicated social, technological, and environmental systems. That's not a linear process, and a ladder isn't going to get us to our ultimate goal.

Yes, improved sanitation has improved the lives of billions of people all over the world, and it's important to note that success. But we need to make some course corrections. The stakes are just too high. Losing a multi-million-dollar spacecraft is a shame. Leaving billions of people to live with "improved sanitation" that includes untreated sewage--that's an economic, environmental, and humanitarian catastrophe.