This past week, wrangling over the Bush-era tax cuts has riveted Washington. The spectacle is only the latest round in an endless debate, one that has launched innumerable op-eds, cacophonous talk-show segments, and dinner-table quarrels. As conservatives see it, higher tax rates hurt job creation as well as undercut the incentive for entrepreneurship and hard work. Many liberals cast these downsides as modest, while stressing the value of tax revenue. Will tax cuts bring a bloom of free enterprise — or exploding deficits? Economists have studied the issue ad nauseam, but firm conclusions are elusive. It is difficult to tease out cause and effect, because at any given time, economic conditions other than taxation also shape behavior.
If only there were a scientific way to determine the real impact of taxation on industriousness, labor supply, and innovation.
According to some scholars, there is. Randomly assign a representative sample of the population — say, 10,000 taxpayers — a lower tax rate, and see what happens. Did these Americans, on average, behave any differently than their counterparts? Did they work longer hours or more jobs, start more businesses, hire more employees?
In other words, test government policies using the same technique — randomized controlled trials — used to test new drugs. A growing chorus of legal scholars, economists, and political scientists believes that such trials should be conducted to evaluate a wide range of laws: gun control, safety and environmental regulations, election reforms, securities rules, and many others. And some believe that we are ethically obligated to do this, because laws affect our lives so pervasively. Understanding the true costs and benefits of legislation, they say, is essential to making good policy — and we may know much less about our own laws than we think.
“The randomized experiment is kind of the gold standard in medicine and social science,” says Ian Ayres, a Yale law professor and economist who advances the idea of the tax experiment in a forthcoming paper. “We should use that same tool to inform us whether laws work.”
Already, randomized trials have migrated beyond medicine. In recent years, experiments to test development programs in poor countries have grown common. Even in the United States, such trials have been used for several decades, in a scattered way, to evaluate social services such as job training and welfare reform, as well as criminal justice and education policies. In one controversial experiment described last week in The New York Times, New York City is testing a homelessness-prevention program by randomly denying services to some households. But now, proponents argue that this instrument should be used in other areas of law — that a culture of experimentation should take hold, and randomized trials should become the norm in lawmaking rather than the exception.
There are certainly potential problems with this vision. First is the question of effectiveness: In some cases, it may prove too difficult to run an accurate test. The full repercussions of laws often take years to manifest themselves, and small-scale experiments do not always translate well to larger settings. Also at issue is fairness. Americans expect to be treated equally under the law, and this approach, by definition, entails disparate treatment.
“The problem is, we’re dealing with laws that have a huge impact on people’s lives,” says Barry Friedman, a law professor at New York University. “These aren’t casual tests. It’s not, you try Tide or you try laundry detergent X….Here we’re talking about basic benefits and fundamental rights.” Though Friedman is sympathetic to the goal of gaining better empirical knowledge, he says, “My guess is some of it’s doable in some contexts, and a lot of it’s not doable in other contexts.”
But others are more sanguine, and they make the opposite argument: That precisely because the stakes are so high, the laws that we enact on a large-scale, long-term basis must be more rigorously tested. This wave of thinking is part of a broader trend in fields from health care to education: Our practices should be “evidence-based,” rather than deriving from theories and unproven assumptions. The question is whether this kind of scientific approach can successfully take on a project as unruly as our society — and our politics.
…
The concept of randomized trials is usually traced back to the 1920s, when Ronald Fisher, a British geneticist and statistician, used the technique in agricultural trials. Fisher randomly assigned plots of land different fertilizers or crop varieties, and compared the results.
Before long, scientists began to apply the method to people. The virtue of randomization is that, provided the numbers are large enough, you can create two groups that are close to statistically identical — the same distribution of gender, age, height, educational background, and other characteristics. Any change in the treatment group can then be confidently attributed to the intervention, rather than to other factors. (In a variant of the concept, trials can randomly assign different interventions to multiple groups.) In observational studies, by contrast, it can be difficult to distinguish correlation from causation.
“The typical experiment involves very, very elementary statistical analysis,” says Donald Green, a Yale political scientist. “When things are done properly, you take the treatment group average and subtract the control group average. It could not be more simple.”
Randomized trials of drugs were first conducted in the late 1940s, and by the 1970s, they were widely used in the United States to evaluate new drugs. Now, of course, it’s unthinkable that the FDA would approve a new drug without clinical trials to show that it’s safe and effective.
The use of randomized trials outside medicine actually dates back even further, to a local experiment. In 1930s, the Cambridge/Somerville Youth Study, designed to reduce delinquency, randomly assigned over 500 boys to either a treatment group, which received visits from a counselor and other services, or a control group, which received neither. A 30-year follow-up found that the intervention had not diminished criminality and, strangely, seemed to have slightly exacerbated it. Since then, a number of comparable experiments have ensued. In the 1980s, police departments in Minneapolis and Milwaukee randomly assigned mandatory arrest for domestic violence offenders to assess its effect on recidivism. They initially found that arrest deterred future offenses, but over time a more complex picture emerged; particularly in areas with high unemployment, arrests appeared to provoke further battery. Similar state-run experiments have shown the effectiveness of unemployment programs that offer job-search assistance.
Though government has proved receptive to the idea in some areas, we are still far from the ideal envisioned by the most ardent proponents. They hope for a time when such experiments are the default. In the world they foresee, any politician who opposed a trial would be suspect, as resistance would appear to indicate lack of confidence in one’s position.
One promising area for expanding the use of trials is regulation. Michael Greenstone, an MIT economist, laid out this case in a chapter in the 2009 anthology New Perspectives on Regulation. While regulations profoundly affect everyday life, he argues, our system for assessing them is dreadfully inadequate. Regulations determine, he notes, the loans we can get, the kinds of materials we can use to build homes, the velocities of our vehicles, and the quality of our air and water. But they are typically evaluated, if at all, prospectively — when analysis amounts to educated guesswork.
We study regulations only at “the very moment when we know least about the consequences,” Greenstone says. “There is no culture of trying to understand ex post what the consequences are.”
As an alternative, he recommends introducing regulations on a small scale prior to rolling them out. Safety regulations, such as new rules for cars or cigarette lighters, could be randomly tried in some areas and evaluated after a designated period. If the benefits exceed the costs, they should be expanded; if not, they should be scrapped. Greenstone suggests that industrial plants could be randomly subjected to different environmental regulations. After a given amount of time, the air and water quality in the vicinity would be measured, along with health outcomes for people in the area and the effects on plants and animals. Various versions of regulations could be tried in different states or municipalities.
In a paper to be published in the University of Pennsylvania Law Review this spring, Yale professors Ayres and Yair Listokin and George Washington University law professor Michael Abramowicz advocate the systematic introduction of randomized trials throughout government — in legislatures and administrative agencies, at the state and federal level. They suggest that trials be “self-executing,” in that policies would be automatically enacted based on their results (though lawmakers would be able to overrule this default).
As proponents of this idea frequently note, Justice Louis Brandeis famously called the states the “laboratories of democracy,” but state-level innovations do not meet the standards of real, controlled experiments. Ayres and his coauthors propose that the federal government coordinate trials, randomly assigning states (provided they consent) to adopt certain laws; as Ayres puts it, “In the laboratory, you don’t let the rats design the experiments.” Another proposal, perhaps more realistic, is for states themselves to randomize across municipalities or counties.
There are concerns, of course, about how practical widespread tests would be. Some domains are more susceptible than others. Education policy, election laws, and criminology seem particularly ripe for randomized trials (and many have already been conducted, both by academics and government). In these realms, interventions are not likely to be obviously experiments. A kid in a class doesn’t know or care if his class size or teacher’s technique is part of an experiment. The same is probably true of a criminal who receives a particular sentence, or a voter who receives certain literature in the mail. Moreover, the outcomes are relatively conducive to measurement (test scores, recidivism rates, voter turnout). But in other cases, participating knowingly in an experiment could distort the result (this phenomenon even has a name, the Hawthorne effect). This problem could be especially acute for businesses that desire a particular outcome — such as lenient securities laws or safety regulations — and alter their behavior for the duration of the trial. And some experiments — such as giving individuals different tax rates — would surely be controversial. Finally, as even landmark trials like the domestic violence study underscore, the results do not always furnish clear-cut policy prescriptions.
…
The essential goal here is to take the politics out of policymaking, to replace dogma with data. It’s a noble ideal. But society will always have to contend with clashing values and priorities. Gun control law, for example, may be quite conducive to experimentation. Virginia, say, could randomly assign counties to different restrictions, and measure crime rates over the next few years. But even if the trial demonstrated that gun control reduced crime, many people believe in the liberty to bear arms as a nonnegotiable right. If data showed that air and water pollution had no effect on human health, some would oppose the regulations on industry, while environmentalists would still support them. Good data can only get us so far.
The other obstacle involves the current state of our political discourse. Opponents of a law could still claim that the trial proved nothing. While academics consider it the gold standard, the public may not place more stock in it than in other kinds of research that are more easily manipulated. Already, suspicion of government is rampant, and opponents accuse the Obama administration of technocratic overreach. People don’t like to feel like guinea pigs, and states may chafe at playing the role of lab rats. It is easy to imagine that a substantial segment of the population would view randomized trials as nothing more than another elitist scheme.
That said, proponents of this idea don’t claim it’s a panacea. They do believe randomized controlled trials are, if imperfect, the best way we have of generating empirical data. In his recent essay, Greenstone argued for a “new era in regulatory reform.” The first era, he writes, was that of the New Deal and the Great Society, when the focus was on well-meaning efforts to remedy social problems. In Greenstone’s view, the effort seemed to count more than the results, with little emphasis on follow-up and evaluation. Then, in a backlash, came the second era, under the Reagan administration, when government was recast as the problem. Now, Greenstone believes, we must instigate a third era, in which government is neither demonized nor valorized, when results are measured meticulously and count for more than good intentions.
As Donald Green of Yale says, “We test pharmaceuticals because there are billions of dollars at stake, and lives.” The same, he argues, is true of our laws, yet we don’t subject them to the same scrutiny. “In some ways the question is, how badly do we want to know?”
Source: The Boston Globe