Experimental design in the Social Sciences
August 19, 2011 § Leave a comment
Here’s a pretty cool piece from the City Journal by Jim Manzi from Summer 2010 talking about causal claims in the social sciences. He has a concise history of experiment design and methodology from the natural and biological sciences that plays well into the kinds of things the social sciences should be worrying about. I have always been wary of modeling the social sciences exactly after the natural sciences because there are obviously differences between the two that require attention, but for now, I think the natural science model of inquiry is a good one to aspire to.
None the less, Manzi talks about things like randomization:
In 1884, the brilliant but erratic American polymath C. S. Peirce hit upon a solution when he randomly assigned participants to the test and control groups. Random assignment permits a medical experimentalist to conclude reliably that differences in outcome are caused by differences in treatment. That’s because even causal differences among individuals of which the experimentalist is unaware—say, that genetic predisposition—should be roughly equally distributed between the test and control groups, and therefore not bias the result.
Generalizability, or external validity:
Even in classical therapeutic experiments, the assumption of uniform biological response is often a tolerable approximation that permits researchers to assert, say, that the polio vaccine that worked for a test population will also work for human beings beyond the test population. But we cannot safely assume that a literacy program that works in one school will work in all schools. Just as high causal densities in biology created the need for randomization, even higher causal densities in the social sciences create the need for even greater rigor when we try to generalize the results of an experiment.
At the end he has three take-aways that aren’t exhaustive but worth keeping in mind:
First, few programs can be shown to work in properly randomized and replicated trials. Despite complex and impressive-sounding empirical arguments by advocates and analysts, we should be very skeptical of claims for the effectiveness of new, counterintuitive programs and policies, and we should be reluctant to trump the trial-and-error process of social evolution in matters of economics or social policy.
Second, within this universe of programs that are far more likely to fail than succeed, programs that try to change people are even more likely to fail than those that try to change incentives. A litany of program ideas designed to push welfare recipients into the workforce failed when tested in those randomized experiments of the welfare-reform era; only adding mandatory work requirements succeeded in moving people from welfare to work in a humane fashion. And mandatory work-requirement programs that emphasize just getting a job are far more effective than those that emphasize skills-building. Similarly, the list of failed attempts to change people to make them less likely to commit crimes is almost endless—prisoner counseling, transitional aid to prisoners, intensive probation, juvenile boot camps—but the only program concept that tentatively demonstrated reductions in crime rates in replicated RFTs was nuisance abatement, which changes the environment in which criminals operate. (This isn’t to say that direct behavior-improvement programs can never work; one well-known program that sends nurses to visit new or expectant mothers seems to have succeeded in improving various social outcomes in replicated independent RFTs.)
And third, there is no magic. Those rare programs that do work usually lead to improvements that are quite modest, compared with the size of the problems they are meant to address or the dreams of advocates.
Also, in terms of randomized field trials, I would be remiss to not mention Banerjee and Duflo’s Poor Economics, one of the best popular titles I’ve read coming out of economics and development literature. This Foreign Policy article is a great introduction if you don’t have time to read the book.