Data Collection in Developing Countries… Stories from the Field

Consider the following unrelated facts:

  • About 1 in every 19 Kenyan children dies before his or her fifth birthday.
  • Only 9 percent of health facilities in Bangladesh offer diagnostic services for tuberculosis.
  • The percentage of households in Uganda owning at least one insecticide-treated anti-malarial net increased from 16 percent in 2006 to 90 percent in 2014-2015.

Our first reaction to numbers like these is to imagine what they imply for the lives of people in those countries. But there’s another dimension to the statistics that’s easy to overlook: someone had to go to the field and collect the underlying data. Behind each of these neatly summarized findings (from the U.S. Agency for International Development’s Demographic and Health Surveys (DHS) program) lie stories of personal hardship, risks, and logistical challenges faced by data gatherers—the unsung heroes of international health and poverty research.

This is from a recent article that made the rounds in ‘development Twitter’, which concludes…

Most charities and aid agencies don’t publicize these behind-the-scenes stories, mainly because they see the challenges of working in the field to be an everyday part of the job. But these aren’t everyday jobs. Next time you see health or poverty statistics, or data used to evaluate the effectiveness of charities, take a moment to reflect on the hard work and dedication that must have gone into collecting that information.

I’ll add two short stories. I hope others add more.

The first is from Kenya. I was working on an impact evaluation for an NGO. A local pastor and I were driving around Kitale Kenya and had to drive about an hour out of town to catch up with an individual in our sample. Just as we arrived at the farm the individual owned, a heavy rain began. After performing the survey the individual, refusing to let us leave in the middle of the rain, fead us tea and mandazi. After the rain stopped we returned to our car, which we found had sunken into the mud. We spent roughly the next hour digging our car out of the mud. The road (actually pictured in the header of this blog) leading back to town was long, narrow, and now sufficiently muddy. As we attempted to drive, the car inevitably slid off the side of the road and into the surrounding bushes and ditch. Eventually a fairly large group of kids came and helped guide our car on the center of the road and away from the deep and muddy puddles on each side until we reached a semi-paved road. All told, we spent an entire afternoon just recording one individual’s survey.

The second is from Myanmar. I was working with the Myanmar Development Resource Institute implementing a rural household survey in Mon State, a coastal region close to the boarder with Thailand. The survey was designed to provide statistics that were representative of Mon State. Representative statistics are so essential to modern policy-making that the effort required to collect such data is often glossed over. To collect this data our wonderful research team and enumerators traveled throughout rural Mon State – in cars, but also by boat across water and on foot along rice paddy bunds. Once at the survey location, enumerators tirelessly walked through a 2-3 hour survey covering topics such as assets, agricultural activities, and household consumption. Below are a few pictures from the Mon State Rural Livelihoods Household Survey.



IMG_3425 IMG_2300


[Film Review] Poverty, Inc.

poverty-inchPoverty, Inc. is a new documentary* that draws attention to the flaws in the modern global aid and development industry. The film itself is quite well-made and is high in production value. For a documentary about failed attempts to aid the poor and the development of societies it is actually a remarkably engaging film.

The documentary begins by calling for reform in the US global food aid system. The film demonstrates how food aid is distributed around the world in a way that largely contains the benefits within the United States. We package up our surplus (and subsidized) agricultural products, ship them across the world on US ships, and giving them out for free in countries that are primarily agricultural based economies. As I’ve blogged before, the evidence is clear: the most effective way to help hungry people around the world is by providing cash, electronic transfers, or by purchasing food locally.

The film then moves on to the most common critique of foreign aid: pointing out that aid that goes directly to corrupt governments does very little in the lives of the poor and (perhaps) makes the whole situation worse. By providing governments with revenue, the need for the government to be accountable to its electorate is diminished – causing a weakening of “the social contract” between governments and their people. This may be called the Angus Deaton critique of aid

Next, the film tells the story of the perverse incentives orphanages create for poor families in poor countries. Orphanages are numerous in countries like Haiti and they may be actually pulling families apart. By offering to care for children in poor areas, it may be in the best interest of a poor family to give their child up to the orphanage. This effectively exacerbates the problem those who run orphanages are explicitly trying to help solve. Although this reality is well documented, orphanages continue to be popular among rich “do-gooders”.

Finally, the film stresses that real and meaningful change occurs with the change of existing institutions (i.e. change in the “rules of the game” as Doug North would say). Most who work in development, when they are being most honest, would agree that the primary reason why some countries are rich and others are poor is due to political institutions: i.e. rule of law, authoritarianism vs. democracy, judicial systems, ect. 

By raising awareness of these flaws in the global aid system, I greatly appreciate this documentary, as these issues are presented in a clear and meaningful fashion. I do, however, have two key critiques: First, none of these flaws are all that novel to those who are actually working in the aid and development industry, and they actually fail to mention the flaw that (perhaps) underscores everything that is mentioned above. Second, the film focused only on the flaws of the current system and fails to mention what the aid and development industry (perhaps miraculously) has gotten right over the years.

TOMS Shoes: When Theory Doesn’t Hold in Reality

The film makes the case that aid doesn’t work, and may be hurting those it should be helping. This is a real concern, but the film doesn’t share much in terms of real evidence or rigorous analysis. This lack of empirical evidence allows for mistakes to slip into the film.

For example, the film takes quite a bit of time explaining why TOMS Shoes, while well-intended, actually damages the local economies where the shoes are donated. This explanation uses solid and well-developed theoretical economic logic: local shoe businesses can’t compete with free, so when TOMS Shoes gives away free shoes the local shoe market suffers. Although this theory is solid, it is worth actually performing a study in order to understand if this theory holds up in reality.

Three economists from the University of San Francisco performed such a study and recently published it in the peer-reviewed Journal of Development EffectivenessThe findings of the study are surprising, given the sound logic in the paragraph above. They find no statistically significant effects on the local labor market, in rural El Salvador, due to the donations of TOMS Shoes. There is a small effect, about one fewer pair of shoes demanded and sold locally due to 20 pairs of TOMS Shoes donated, but this effect is so small (and not statistically significant) that it hardly warrants the mass prevention of shoe donations.

The key take-away from this is that shoe donations, specifically, and aid, in general, ought to be targeted to those who actually need it. In the case of shoes, those who donate shoes should be careful not to give shoes to people who would otherwise pay the going market price for shoes. In many communities, however, there are lots of people who both need shoes and can’t pay the market price for shoes that would benefit greatly from the donation of good quality shoe donations.

Foreign Aid: A Simple Cost-Benefits Analysis

Although it may be easy to point out aid and development spending that may have not generated genuine economic growth and poverty reduction, the benefits from aid and development can be so huge, when it is spent well, that the benefits may still outweigh the costs using typical cost-benefit practices.

For example, the eradication of smallpox around the world. (An example brought to my attention by William MacAskill on a recent EconTalk podcast.) Suppose, just for the time being, that all forms of aid had no benefits except for aiding in the eradication of smallpox. Before smallpox was eradicated it killed a recorded 300 million people globally. Since its eradication, in 1973, roughly 100 million lives have been saved. To put that number in perspective, that’s more lives saved than would have been saved if we achieved world peace in 1973. Now, if you crunch the numbers, counting how many lives were saved in terms of how much money has been spent on foreign aid (remember, we are assuming there have been no other benefits of aid since 1973), you’d concluded that it cost roughly $70,000 per life saved. To put this calculation in perspective: standard cost-benefit analysis within the United States government rests on the assumption that a life is “worth” saving if it can be done with a price tag of $7 million or less.

The key take-away from this is there are huge benefits to foreign aid, particularly in the sphere of global health. To make the case against spending foreign aid, it is not a sufficient argument to simply point to projects or policies that seem to have not produced benefits. Like most topics in public policy or business administration, the benefits must be weighed against the costs. In the case of human development and poverty alleviation the potential benefits are so large that the seemingly high costs are often justified.

Innovation in the Aid Industry: Cash Transfers

When the film presents the current reality of the global aid and development industry (pictured below), the newest, and perhaps, most exciting innovation in the last decade is omitted: direct cash transfers.

Screen Shot 2016-03-04 at 4.45.12 PM

Inserted into the figure above, direct cash transfers create a line of little green arrows from the yellow group of people (presumably rich people) directly to the red and orange people (the “people in poverty”). No taxes, no donor nation governments, no developing nation governments, no egotistical NGO development project. Through organizations like Give Directly you can send money from your bank account directly to the mobile money (i.e. MPESA) account of the extreme poor all around the world.

When thinking about the power of direct cash transfers consider the idea of diminishing returns. It seems wellbeing and income are related to each other, approximately, by the logarithmic function. Whereas no matter how much money you have, doubling your income will provide the same amount of gain in welfare. This means that for someone who makes $2 per day, giving this person an extra $2 per day will do approximately as much in terms of wellbeing as giving someone who makes $200 per day an extra $200 every day. 

The most remarkable detail about this omission is that (at least) the director of the film seemed unfamiliar with direct cash transfers when asked about them by Russ Roberts while a guest on the Econ Talk podcast. While this critique borders on ad hominem, this is not my intention. It is rather stunning and relatively disappointing that the director of a documentary about the modern aid and development industry, “hasn’t spent a lot of time thinking [about cash transfers]”. Particularly because talk about cash transfers has been so common in recent years. A couple years ago a large study was done on unconditional cash transfers in Western Kenya – paper here, media coverage here, here, here, here, and here. Additionally, the most famous cash transfer program is Mexico’s school attendance conditional cash transfer program, Progresa/Oportunidades  – one of the many papers here, more information here, here, here, here, and here.

The key take-away is that the aid and development industry is still evolving and improving. In fact, the increased use of direct cash transfers could help improve the flaws in the global food aid system, mentioned above. Some have suggested that direct cash transfers should serve as “index funds” of sorts for the aid and development industry. If your program is not doing as good or better than just giving all the money needed to run the program to the poor then, maybe, it’s time to make some changes to the program.

Conclusion: The Root of the Problem with Aid

To conclude this review, I’d like to highlight what may be the largest and most fundamental flaw in the aid and development industry as it was largely left out of the film: the extreme lack of evidence and rigorous feedback in decision making, program design, and management of NGOs.

The world is a complicated place. As is demonstrated above through the case of TOMS Shoes, well-developed theory and sound logic can only provide so much clarity about what will work and what won’t in various contexts. In order for aid to actually be effective data need to be collected, feedback needs to be heard, and impacts need to be measured. The problem with most NGOs, and the key reason why Haiti remains poor despite the overwhelming number of NGOs, is because these organizations often don’t do anything as far as collecting data, gathering feedback, or rigorously measuring impact.

I understand why very few NGOs spend time and money measuring impacts and collecting evidence. Doing so in any sort of rigorous and honest manor requires a lot of humility and courage. It is perhaps natural for us to want to live a (fictional) world where we know we know how things work. Admitting our own faults is often one of the most difficult challenges we face, but learning from failure and iterating through trial and error is simply the most effective problem solving strategy at our disposal. 

The Poverty, Inc. documentary tried to make the case for so-called market-based or business solutions to poverty around the world. This recommendation glosses over the primary reasoning behind the preference for the private sector over the public sector. The main reason why the private sector drives innovation and progress is because it is constantly receiving feedback vis-a-vis the market. The public sector, on the other hand, has a very slow (and sometimes nonexistent) mechanism for feedback and therefore lacks innovation and progress. Aid in and of itself is not bad or ineffective, it’s the lack of feedback caused by the lack of commitment to evidenced based management that makes much of global aid to be bad and ineffective. The challenge of poverty is that it often rests outside the scope of markets and private sector businesses. The poor can’t “entrepreneur” themselves around bad leadership or bad policies, rather it’s good leadership and politics that needs to create an environment for entrepreneurship to flourish.

In recent years, the idea that aid given to foreign countries is “bad” always seems to gain a lot of traction. I’ve never quite figured out why so many people are set on painting with such broad strokes when it comes to solving one of the world’s most persistent puzzles. The aid and development industry needs more evidence not less financial support. The last thing that is needed would be for public support of foreign aid to wane in years to come. In light of the flaws highlighted by Poverty, Inc. and the key points of this review, perhaps the most concrete task to be done would be to call your member of congress to voice support for the Foreign Aid Transparency and Accountability Act.

* A disclaimer: I personally know many of the people who were interviewed in and who were involved in making this film. While in college and for a year after, my work was affiliated with Partners Worldwide, who assisted film-makers for this project. During this time, I assisted with some preliminary research for a book about Haiti with two administrators, worked for a few months with several of the folks in Ghana (I’ve actually been to the pineapple plant shown in the film), and spent a year in Kenya working with the organization full time. Although I remain friendly with many of my former colleagues, I’ve tried to keep this review as objective as possible.

Data Collection (and Analysis) as Development

Here’s a paragraph from a boarder-line scathing short essay on “Data Driven Development Decisions” posted by The Springfield Centre. A self-proclaimed “leader in the market systems approach to development in low and middle-income economies – also referred to as making markets work for the poor”. The central point of the essay is this: setting the monitoring and evaluation function of your program apart from the rest is antithetical to the purpose of the development project.

Essentially, the job of a development programme aiming to stimulate systemic change, is to get one over on the system. Systems are pretty knowledgeable. Of all the things there are to know, they know most of them. You have to find out (or try to find out) something that it doesn’t know in a way that benefits the poor. That might be a business model that the system didn’t know was profitable, a function that the system didn’t know was needed to improve efficiency, or an input that the system didn’t know was more cost effective. One thing that systems – particularly in developing countries – aren’t that great at is producing and aggregating information into nice little digestible packages. So, in order to find the little nuggets of information that might allow you to change the system, you need to put in the effort – and that effort is in data collection.

A few things:

First, yes. I happen to agree that the key effort of a development project (for those who aren’t poor at least) is to collect and analyze data. However, it’s not just any data collection. The necessary qualifier is the collection of good data that actually informs about the things we think the data is telling us. A huge problem is collecting data that is of poor quality and doesn’t actually inform about any pertinent questions.

Second, what is data collection? Data collection need not always be quantitative. Lots of really good work has been done with careful qualitative data collection and analysis. I cut my teeth in development studies in the IDS program at Calvin College. It was (and still is) a great program and has been integral in my education and career, but most folks in the program seemed to have a bias against quantitative data collection and analysis. This might be because reading and understanding quantitative evaluations and studies is more difficult (when anyone is first learning) than reading and understanding qualitative evaluations and studies. However, once I actually got out into the world and started collecting my own data I realized that careful qualitative data collection is much more difficult and time intensive than careful quantitative data collection.

Third, It seems to me that many who are either (a) averse to working in monitoring and evaluation or (b) who resist making it the key function of their program (and not just a function to appease donors), hold these feelings because they believe that monitoring and evaluation is boring. They were first interested in development work because they wanted to help people directly and that (somehow) certainly doesn’t include sitting behind a spreadsheet all day.

I have two responses to this: (a) Well, good development is actually (strictly speaking) boring, not very glamorous, and doesn’t photograph well. (b) Although, if one can get past the mundane tasks (that are actually part of every job) the task of development is actually super exciting. Being at the forefront and being able to witness (perhaps) the most dramatic reduction in human suffering of all time is certainly exciting and anything but boring.