Last week, I came across this paper by Fiona Burlig, Louis Preonas, and Matt Woerman recently published in the Journal of Development Economics. It is a paper that seems broadly applicable, so I’ll highlight the key points.
Students with the Centre for the Study of African Economies (CSAE) at the University of Oxford are creating a wonderful public good. The Coders’ Corner is a collection of tips and tricks for implementing useful statistical techniques in common statistical software (e.g., mostly Stata). This product represents a tremendous service to the broader research community. Almost anyone reading this blog should check out previous posts.
Here is an excerpt from a recent paper published in Applied Economic Perspectives and Policy by John Gibson; entitled, “Are You Estimating the Right Thing? An Editor Reflects.”
If you follow Marc Bellemare’s blog or specifically his ‘Metrics Monday series, you will already be aware of our new working paper. The paper is titled: “The Paper of How: Estimating Treatment Effects Using the Front-Door Criterion.” The number of people who are reading this post and who do not already read Marc’s blog is probably very small. So, with that in mind, I will offer a few additional thoughts based on the preliminary work writing this paper.
Back in 2015, I read a book by Morten Jerven, in which the author makes the point that over 145 variables have been found to be statistically significant explanatory variables for long-run economic growth. Morten’s point is more nuanced than this, but this suggests that when interpreting regression results we need to not only consider statistical significance, but also economic significance.
A long standing belief, held by many, is that winning the lottery actually makes people miserable. This belief is backed up by existing research in psychology finding that lottery winners were no more satisfied with their life than people who did not win the lottery. New research suggests this belief might be wrong.
Concepts such as subjective well-being, satisfaction, happiness, trust, measures of quality, and even standardized test scores are all measured using an ordinal variable. This means that we know the rank of the response categories (e.g. a respondent reporting being “very satisfied” indicates they are more satisfied than if they had reported being “satisfied”), but we do not know the interval between response categories (e.g. we don’t know how much more satisfied “very satisfied” is compared to “satisfied”). This is contrasted with cardinal variables, such as earnings, where we know $10 is more than $5 and represents twice as much money.
Last weekend, I came across a paper published in the European Economic Review by Carsten Schroder and Shlomo Yitzhaki entitled “Revising the Evidence for Cardinal Treatment of Ordinal Variables” (2017). I found the paper to be well-written, intuitive, and important. Although there are no direct policy implications of this paper, papers like these need to be publicized, for the sake of all research that does aim to provide meaningful implications for the world.
That’s an okay explanation, but I think there is an even more profound lesson about economics woven into this classic movie. It’s a topic I interact with regularly and one that is often misunderstood.
In the movie, “It’s a Wonderful Life” George Bailey gets the outrageous opportunity to see what the world would be like if he had never been born. He’s able to see the impact he’s made on the people around him. What a gift!
Notice how I use the word “impact” in the last paragraph. It is used correctly under the definition that impact is the outcomes of something minus the outcomes if that something did not happen.
Economists try to measure the impact of all sorts of things: from government policies and social programs to business innovations and market forces. When undertaking this endeavor we are presented with the same problem Clarence, George’s guardian angel, faces. We need to show impact. In Clarence’s case he needed to prove to George that he had a meaningful and positive impact on the people around him and that the world was a better place because he was born. Economists generally need to measure the effectiveness (or non-effectiveness) of a specific policy, program, or innovation.
The problem economists have is we aren’t angels and can’t go back in time and re-do the world in the absence of the thing we want to evaluate. Economists encounter what is called the “problem of the counterfactual”. We don’t have an alternate universe in which to compare hypothetical outcomes.
We have to come up with work-arounds – creating proxies of the counterfactual – to solve this problem.
One way is to randomly assign receipt from the program, policy, or innovation. With a large enough sample we would expect that the group of people assigned to receive the program, policy, or innovation would be exactly equal to the group of people who did not receive the program, policy, or innovation. Impact would then be measured by subtracting the outcomes from the group that didn’t receive the program, policy, or innovation from the group that had. This is the randomized experimental method and if it is done correctly it allows economists to show the exact same thing that Clarence showed George. Pause and reflect on that for a moment.
Sometimes (most of the time) a randomized experiment is not feasible or ethical, in which case the experimental ideal (described above) needs to be approximated. There are many (in fact, seemingly infinite) ways to do this, each has it’s own strength and is best suited for a particular situation. Going into the specifics of these is beyond the scope of this blog. (Much more can be learned by reading the excellent Mastering Metrics: The Path From Cause to Effect)
Suffice to say, “It’s a Wonderful Life” has a lot to teach us about economics, particularly the problem with the counterfactual and the power of the randomized experiment.