What is “Impact”?

Over the last several months many non-profits and NGOs I keep tabs on and follow have been releasing their annual reports. Due to the growing empirical rigor required by donor agencies, many of these organizations have been using the word “impact” in these reports. Or maybe it’s just because “impact” is a buzz word at the moment. (I don’t know.) Unfortunately, most of these organizations are misusing the word “impact” in their annual reports. So, I’d like to take a moment to provide a clear definition of the word “impact”. To do this, I’ll use some figures I made several years ago for a presentation I was never able to give.

Let’s say that you have some primary outcome – say, jobs or household income or calories consumed or maize yield – at some point in time your program or NGO institutes some sort of intervention – say, a training program or subsidized input coupon or asset/cash transfer. Notice that for all the primary outcomes I suggest here it is “good” when these things rise. Outcome variables – like child mortality or malaria incidence – could also be considered where it is “good” when these things fall.

You collect data on your primary outcome and, let’s say, you find that after the intervention your primary outcome variable began to increase (or increased in the rate at which it was increasing before the intervention). What is the impact of the intervention of your program or NGO?

Screen shot 2014-06-06 at 3.12.23 AMNow let’s say you find that after the intervention your primary outcome variable began to decrease (or decreased the rate at which it was moving before the intervention). What is the impact of the intervention of your program or NGO?

Screen shot 2014-06-06 at 3.12.51 AM

In both these cases, we actually don’t know the real impact of the program. This is because “impact” (correctly defined) = “outcomes of participants” – “outcomes of participants in a world where the intervention does not exist”. Now, in the real world (compared to a lab setting) it is impossible to find out what would have happened to participants if the intervention did not exist. In the real world a true “counterfactual” does not exist.  In the real world, an approximate of this counterfactual must be created by constructing a comparison group – a group of individuals who mimic those individuals who are participating in the program as closely as possible.

So let’s see how collecting data on this proxy counterfactual would allow us to understand the estimated impact of your program or NGO’s intervention. To demonstrate the need for a counterfactual (a comparison group or sometimes called a control group), consider the following four cases:

Case 1: Here the impact is positive because in the absence of the intervention individuals would have remained on the same trajectory.

Screen shot 2014-06-06 at 4.13.59 AM

Case 2: Here the impact is negative because in the absence of the intervention individuals experienced a large boost in the primary outcome. In this case the intervention was instituted at roughly the same time as some other major change in the lives of individuals in the area of interest and those participating in the intervention have been effectively held back.

Screen shot 2014-06-06 at 4.15.32 AM

Case 3: In this case the impact is positive because in the absence of the intervention individuals actually ended up falling farther behind than those who participated in the intervention. For this case think of an intervention that was instituted at the same time as some negative and wide-ranging economic shock, like a drought or some other harsh weather event.

Screen shot 2014-06-06 at 4.18.07 AM

Case 4: In this case the impact is negative because in the absence of the intervention individuals continued on their upward trajectory in the primary outcome.

Screen shot 2014-06-06 at 4.19.28 AM

For matters of completeness, there are other possibilities to consider. For example, it could be that participants of your program and the counterfactual follow the same exact trend in the primary outcome. In this case, the program has zero impact.

Measuring impact is important (and likely increasingly so in the world of development funding), but it is potentially misleading to simply report program “outcomes” and claim them to be “impacts”. To claim something to be an “impact” you must have subtracted away the counterfactual. If a counterfactual has not been subtracted you simply have an “outcome”.

The Economics of “It’s a Wonderful Life”

That’s an okay explanation, but I think there is an even more profound lesson about economics woven into this classic movie. It’s a topic I interact with regularly and one that is often misunderstood.

In the movie, “It’s a Wonderful Life” George Bailey gets the outrageous opportunity to see what the world would be like if he had never been born. He’s able to see the impact he’s made on the people around him. What a gift!

Notice how I use the word “impact” in the last paragraph. It is used correctly under the definition that impact is the outcomes of something minus the outcomes if that something did not happen.

Economists try to measure the impact of all sorts of things: from government policies and social programs to business innovations and market forces. When undertaking this endeavor we are presented with the same problem Clarence, George’s guardian angel, faces. We need to show impact. In Clarence’s case he needed to prove to George that he had a meaningful and positive impact on the people around him and that the world was a better place because he was born. Economists generally need to measure the effectiveness (or non-effectiveness) of a specific policy, program, or innovation.

The problem economists have is we aren’t angels and can’t go back in time and re-do the world in the absence of the thing we want to evaluate. Economists encounter what is called the “problem of the counterfactual”. We don’t have an alternate universe in which to compare hypothetical outcomes.

We have to come up with work-arounds – creating proxies of the counterfactual – to solve this problem.

One way is to randomly assign receipt from the program, policy, or innovation. With a large enough sample we would expect that the group of people assigned to receive the program, policy, or innovation would be exactly equal to the group of people who did not receive the program, policy, or innovation. Impact would then be measured by subtracting the outcomes from the group that didn’t receive the program, policy, or innovation from the group that had. This is the randomized experimental method and if it is done correctly it allows economists to show the exact same thing that Clarence showed George. Pause and reflect on that for a moment.

0010_2A_10_Futo_Peter_Evaluation_and_Impact_Assessment

 

Sometimes (most of the time) a randomized experiment is not feasible or ethical, in which case the experimental ideal (described above) needs to be approximated. There are many (in fact, seemingly infinite) ways to do this, each has it’s own strength and is best suited for a particular situation. Going into the specifics of these is beyond the scope of this blog. (Much more can be learned by reading the excellent Mastering Metrics: The Path From Cause to Effect)

Suffice to say, “It’s a Wonderful Life” has a lot to teach us about economics, particularly the problem with the counterfactual and the power of the randomized experiment.