World Health Organization Reports Conundrum for eHealth Evaluation

05/03/2012 01:15 pm ET | Updated Jul 03, 2012

A health worker in India scanning the fingerprint of a patient to confirm they have taken their tuberculosis medication? A nurse in Kenya using a patient registration system on a cell phone at mobile maternity clinic?


This is no longer a public health wish-list: it is reality (check out examples of each at Operation ASHA and Jacaranda Health). And yesterday, the World Health Organization (WHO) released their May 2012 Bulletin chronicling many such innovations in global health, under the theme of "eHealth".

While there is celebration in order for the many eHealth initiatives designed and deployed everyday across the globe, this WHO Bulletin prompts one to wonder as well: just because impressive technologies exist, are they making a difference? Do they immediately lead to improved care quality, cost savings, or lives saved?

There is no doubt that new technologies, and the hardware and infrastructure required to make them run, are proliferating across the developing world. Just two weeks ago I attended the eHealth Africa Conference in Nairobi, Kenya, a country in which 75 percent of people who access the Internet do so through their mobile phones. Representatives from the Ghana Health Service spoke of how their national Information Technology (IT) agency has helped not only in crafting a national eHealth strategy, but also in retaining Ghana's top IT talent. And farther south in Angola -- a country in which 72% of citizens live on less than $2 a day -- a high-speed 4G network is currently being constructed.

Yet the eHealth Conference attendees -- an array of African policymakers, health practitioners, and technology developers -- collectively acknowledged that simply having technology in hand was not sufficient for producing tangible health benefits. Policy makers in particular explained their challenge in adopting new technologies: if a health system is fractured, then technology may only exacerbate existing weaknesses, or if patients cannot easily and inexpensively access technology, then its potential to promote change cannot be realized. There was not yet enough evidence, from their perspective, to suggest which kinds of eHealth interventions were wise investments.

One particular report in the WHO Bulletin would be of great interest to the concerns of these decision-makers, namely an analysis by Center for Health Market Innovations (CHMI), who documented and summarized main findings to date from eHealth programs in low- and middle-income countries. They aimed to answer the questions that policy makers have by showing evidence of programs that are already working.

As the most comprehensive, global survey in peer-reviewed literature of eHealth initiatives, the report documents 176 technology-enabled programs that aim to improve quality, affordability, and accessibility of health care in developing countries. "By identifying emerging global trends in eHealth, the study provides guidance about how ICT can help solve common health systems challenges in developing countries," said Gina Lagomarsino, a managing director at the Results for Development Institute, of which CHMI is a unit.

CHMI authors identified six primary ways in which most eHealth programs are already functioning: extending geographical access to overcome distance between physicians and patients, facilitating patient communications between health workers and patients, improving diagnosis and treatment for health workers, improving data management, streamlining financial transactions, and mitigating fraud and abuse.

But an interesting and important finding in the CHMI report is that only 16 of the 176 programs documented responded to CHMI's request for self-reported, clear, and quantifiable impacts. The report further reveals that almost no programs had independent evaluations conducted, the quality of available evaluations was low, and the generalizability of available evaluations was not sufficiently broad to make a case for scale. In some cases this may mean that programs have simply not existed for long enough to be able to show any impact. Yet for those who have, it also hints at the challenge of evaluating new technologies -- and reveals that not enough innovators are rising to this challenge.

The challenge of measuring a moving target is a conundrum -- with such rapid changes in hardware, software, and networks on which they operate, it is understandable that measuring impact is an intimidating task. Yet it is one that we must attempt to solve. If innovators cannot rigorously prove that a technology will save costs or save lives, then policy makers cannot be expected to commit effort and resources to applying it to strengthen health systems.

Closing a panel on Health Information Systems at the eHealth Africa Conference, moderator Dr. Khama Rogo lamented to the audience,"I think more Africans die from misinformation than lack of medicines."

Thankfully we now have a wide array of eHealth initiatives that can increase the availability of data for patients, providers, and policymakers alike -- what we need now is fresh, honest thinking on data management and greater effort towards evaluation if we want to see the benefits of eHealth interventions be brought to scale.