There is a story that floats around my department about an invited seminar speaker a few years back. Someone from the audience asked the speaker how they originally thought of the research idea. The speaker answered, “Well, my co-authors and I found some exogenous variation in our data and so we began to look for a relevant outcome.”
It is this sort of behavior that is the subject of an NBER working paper (by Christopher Ruhm) that made the rounds last week.* The paper addresses the trade-off between time spent on research projects with clean causal identification and time spent on research with important questions. The paper makes several interesting points and prompts (what I think) is a useful discussion. The main point of the paper, in the author’s own words, is as follows:
My emphasis here […] is less about potential shortcomings of experimental and quasiexperimental techniques per se – all methods have limitations – than to the possibility that excessive reliance on them leads to an overemphasis on confidence in the results obtained at the cost of failing to investigate important questions ill-suited to the use of these techniques. To some extent, this may reflect the aforementioned distinction between internal and external validity. For instance, successful IV analyses may provide an unbiased and hopefully precise measure of a local average treatment effect but only limited information on overall policy effects of broader interest. Consider Angrist and Krueger’s (1991) use of birthdates as a source of exogenous variation in the ages at which compulsory schooling ends, in their IV analysis of the effects of education on earnings. The resulting LATE indicates the returns to completing approximately the 10th or 11th grade of schooling, versus not doing so, but provides little information on the gains from higher levels of education that are probably of greater current interest.
My sense is that most applied microeconomists would agree that all else equal better designed and more credible research methods are strictly preferred to less credible methods. Ruhm’s main concern, however, is that all else is not equal. Specifically, the push for crystal clear causal identification has lead to research papers with less important questions that can be answered with crystal clear causal identification.
This sort of research is often rewarded. Going back to the previously mentioned anecdotal story from my department: Although the primary motivation for the research did not focus around an interesting and useful question (the authors stumbled upon exogenous variation and then looked for a question), I’ve been told the paper is now published in a top general-interest economics journal.
After reflecting on Ruhm’s article for a few days and reading other people’s thoughts on the topic, here is my take on the supposed identification-importance trade-off.
First, the identification-importance trade-off is (I think) real. On the margin, there are certainly instances when more important research questions are not pursued because there is a lack of plausibly exogenous variation in any available data. With that said, I don’t think that the so-called credibility revolution or the identification police have had a net negative impact on the economics profession. To the contrary, the increased understanding of the core assumptions necessary to make causal statements with real-world data has (in my view) lead to better and more informative research.
Second, as much as research with crystal clear identification strategies is rewarded, I think many implicitly understand that there is a trade-off and that answering important questions matters too. The 2018 New England Universities Development Consortium (NEUDC) conference is a testament to this idea. As John Hoddinott and Chris Barrett note in their pre-cap to the conference, although many papers use RCTs or natural experiments, many papers use credible applications of difference-in-differences or regression discontinuity designs. If this sort of thing is occurring in development economics, I expect other sub-fields are similar.
Third, despite the early marketing of the methods promoted in the credibility revolution, the current understanding of most economists is that there is no such thing as “assumption-free” causal inference. Every quantitative method—from comparing means, to matching, to randomized experiments—require untestable assumptions for causal identification. This implies that, although many claim the contrary, there is no real hierarchy of empirical methods and RCTs are not necessarily the gold standard. Some methods are best applied in specific contexts for specific questions and other methods are best applied in other contexts for other questions.
Finally, my feeling is that the best research (read: most rewarded) at the present time is work that explicitly notes the identifying assumptions necessary for causal inference. These papers then test the sensitivity of the core results to violations of these assumptions as best they can and may have even pushed the empirical model to its limits with a simulation analysis. It is not necessary to have a crystal clear identification strategy as long as one can demonstrate that the necessary identifying assumptions are palatable.
All of this points to my view on how researchers should orient themselves when they perform research: Set out to write the best paper about a topic that you can convincingly claim is important. Some questions (i.e. contract farming) are really difficult to answer with a crystal clear identification strategy. This does not necessarily mean that the topic is a worthless area of research. Now, since the trade-off is real, this also does not give anyone license to use heroic identifying assumptions (i.e. an implausibly exogenous instrumental variable). There is a minimum threshold of believable identification, of which below it is difficult to learn anything regardless of the importance of the question.
* Actually, to call this paper a working paper may be a bit inaccurate as the paper is a representation of Christopher Ruhm’s Presentential Address at the Southern Economic Meetings this fall.