Campaigns & Elections: The evolution of experimentation
This issue’s shoptalkers: Michael Beach, partner at the Republican firm Targeted Victory; James Gimpel, professor of political science at the University of Maryland; Don Green, professor of political science at Columbia University; and Democratic strategist Hal Malchow.
C&E: Let’s start with some history. Where does the story of experimentation and campaigns begin?
Hal Malchow: The history begins with many rejections of the idea of measuring what we do. I remember talking to the Democratic National Committee in 1992, and again in 1996, and then 2000, 2004, and in 2008, all about the importance of actually measuring what we’re doing. Almost every time the answer was no. They would say, “If we pull 10,000 voters out in a control group, it could cost us the election.”
But the real reason we were losing was that 80 percent of what we were doing was totally worthless. It’s almost a scandal that Don Green and Alan Gerber in the academic community had to come along and show us how this work can be accountable. Their work finally started to get attention, and that’s when the tide began to turn. I know the Republicans got there earlier. They were actually doing experiments and measuring in 2001 and 2002.
Michael Beach: That was obviously a huge part of the 2004 election. It was probably what helped drive the Democratic side to do a similar thing. I think it was the close call in 2000 in Florida that drove the need for experimentation, but it’s definitely been a wholesale change. The question now is can we continue to evolve and do it again?
Don Green: How did things change from 2000 to 2004?
Beach: The whole 72-hour victory program for the Republican National Committee was a big part of that. The weekend before the 2000 election you were getting ready for the election night party and not thinking about voter mobilization efforts. It seems crazy, but GOTV was basically about getting on television. Having a field program is no longer enough now. You need to look at new techniques and talk to voters in different ways.
James Gimpel: In 2002 Karl Rove and Ken Mehlman called in some Republican inclined academics to provide advice. That’s where academic involvement got started on the Republican side. It was really about two things. First, what academics have to say to consultants as far as voter file augmentation goes. The other part of it was this evolution of experimentation. Blaise Hazelwood was the political director going into 2004, and she did some experimentation on GOTV. I’m not quite sure there was all the strict balance and methodological perfection that we would demand on the academic side, but they were at least thinking in terms of doing the right comparisons.
Green: Had Karl Rove and others been doing randomized experiments prior to 2000?
Gimpel: I don’t know of anything prior to 2000. I think your early work certainly got their attention. I think there’s some suspicion when it comes to academics, so the Republican instinct is to kind of hold them at arm’s length and then try to replicate what they’re doing within the campaign. They wanted to do it on their own.
C&E: What’s your sense of the dynamic between academics and practitioners?
Green: There is this deep suspicion about two aspects. One is the one that Hal pointed out, which is the concern about having academics meddle in a campaign by withdrawing a control group. That is often a show-stopper for consultants. But then there’s the part that Jim alluded to, which is just the general concern about having findings disclosed to the public. People would be embarrassed if their tactics don’t work, so for any person who aspires to do an experiment it’s a delicate dance. One of the things that has changed in the wake of a lot of randomized experiments is the sense of how catastrophic a 10,000-person control group would be. People have seen other campaigns withdraw a control group and still live and learn to move forward with other techniques. Another change is that you now have people in this experimental enterprise who have been trained and are propagating this research method in other circles.
Gimpel: I know one of the concerns the consulting community has had is whether we can generate these results in time for them to be actionable. I think the experience with the Rick Perry gubernatorial campaign indicated that can happen. But with GOTV, if we have to wait until the voter file comes out with the completed vote history in December, how can that be useful to campaigns now?
Malchow: I think it’s worth pointing out that you still don’t see experiments inside the actual campaigns. Part of that is just the short term interest of the enterprise. But now you’re seeing an enormous number among the nonprofits and advocacy groups. Things have really come a long way.
Green: Hal, in your days as a consultant you actually conducted your own internal experiments despite the same short term pressures. What’s preventing others from doing what you did?
Malchow: We did a few experiments within campaigns, but generally we had to cut them some sort of deal. We’d give them more mail than they were going to get to make up for the control group, for example. But working with the nonprofits and getting experiments done was just the result of persistence. The initial reaction usually wasn’t, “Hey, this is a great idea.” One of the largest barriers is the consultants themselves, who in many cases are afraid of knowledge. It is an industry that has lived off hype and hyperbole for decades. All of a sudden people are measuring, and that makes folks nervous. On the Democratic side, I think that resistance has pretty much been worn down by now.
Beach: I think even when you get consultants to the point where they’re willing to make a change things tend to tighten up right when it comes to decision time. Even if you have someone willing to be proven wrong, they still end up going with what’s safe in their mind. We’ve seen that a lot on the consultant side. You actually succeed in convincing them it’s something they should do, everyone’s agreed and then right when the rubber meets the road, you go right back to the safe old way. That’s probably harming the operation more than helping.
Malchow: The organizations that are open to innovation are the ones that are highly accountable, where results are easily measured. Think of it along a big continuum. The most innovative is direct marketing because every single thing you do is measured in some way. No one ever goes into the conference room to argue about whether something worked or not. Political campaigns are at the other end of the spectrum. You could run the best campaign in the world and lose. You could run the worst campaign in the world and win. The manager is sitting there and he knows he’s going to be praised if they win and blamed if they lose. But if he does something different, then he’s really going to be blamed. I think that was always a barrier with campaigns. Less so today, but we still operate in an innovation-averse business.
