From 2016: RCTs are a truly excellent tool for determining the internal validity of a study—does it accurately model the data it has. But external validity—is the data at hand representative of the broader world in any reasonable sense?—is as important. There needs to be a balance here: Angus Deaton and Nancy Cartwright (2016): The Limitations of Randomised Controlled Trials https://voxeu.org/article/limitations-randomised-controlled-trials: "The use of randomised controlled trials has spread... perhaps most prominently in development and health economics.... Some of the popularity of such trials rests on misunderstandings about what they are capable of accomplishing, and cautions against simple extrapolations.... Well-conducted RCTs... provide unbiased estimates of the average treatment effect (ATE) in the study population, provided no relevant differences between treatment and control are introduced post randomisation, which blinding of subjects, investigators, data collectors, and analysts serves to diminish.... Randomisation, while doing nothing to guarantee balance on omitted factors, gives us a method for assessing their importance. Yet even here there are pitfalls. The t-statistics for estimated ATEs from RCTs do not in general follow the t-distribution.... In healthcare experiments, where one or two individuals can account for a large share of spending (this was true in the Rand Health Experiment); or in microfinance, where a few subjects make money and most do not (where the t-distribution again breaks down). Once again, inferences are likely to be wrong, but here there is no clear fix.... The ‘credibility’ of RCTs comes from their ability to get answers without the use of potentially contentious prior information about structure.... Cumulative science happens when new results are built on top of old ones–or undermine them–and RCTs, with their refusal to use prior science, make this very difficult.... Demonstrating that a treatment works in one situation is exceedingly weak evidence that it will work in the same way elsewhere; this is the ‘transportation’ problem: what does it take to allow us to use the results in new contexts, whether policy contexts or in the development of theory? It can only be addressed by using previous knowledge and understanding, i.e. by interpreting the RCT within some structure, the structure that, somewhat paradoxically, the RCT gets its credibility from refusing to use...
#noted #2020-01-24
