Joe Klein Is a Journalist; John McCain Is Dishonest and Dishonorable
Princeton Election Consortium

Sam Wang: Why the Polling Industry Sucks

Matthew Yglesias sends us to Brendan Nyhan who sends us to Sam Wang:

The economics of reporting polls: TThe only thing happening in the Meta-Analysis is a slight, slow widening of Obama’s lead. Some of you want to know about individual polls, such as a recent Gallup national poll showing Obama ahead by only +2% (standard likely-voter model) or +6% (high-turnout model). I confess that I tend to ignore individual polls because of the statistical variability. So it didn’t occur to me to care about this particular data point. Obama is still crushing McCain, period.

But there is a lesson to be learned here: It is not in the interest of individual pollsters or media organizations for you to have the most accurate possible picture of the horserace.

Here is why.

Uncertainties such as the margin of error can be reduced by taking more samples. An individual pollster can halve the margin of error by surveying 4 times as many people. It’s a square-root relationship: N samples lead to a sqrt(N)-fold reduction in uncertainty. The same is true for combining polls, with the added advantage of reducing the effects of methodological variation. Thus the value of poll-aggregation sites like this one. Meta-Analysis worked extremely well in 2004 and 2006, and is likely to do so again this year.

So why don’t more pollsters or media organizations aggregate polls? The CNN Poll of Polls is a start, but it’s an exception. Two forces encourage bad horserace reporting:

  • Competition among pollsters. It’s not in the interest of individual pollsters to say “average my results with the others.” It’s also not advantageous to collect a larger sample once the margin of error meets industry standards.

  • The hungry media beast. With news budgets on the decline, it’s costly to report real news. Why pay for investigative reporting when you can buy a poll and report the horserace? Within the area of poll reporting, market forces discourage high accuracy. For example, commissioning a survey of 4 times as many people would reduce uncertainty by a factor of two. But why pay 4 times as much for data that generate a lower likelihood of an apparent - and reportable - swing?

For these reasons, media organizations aren’t motivated to report polling results with the maximum possible statistical power. The Meta-Analysis of State Polls is pure data reduction, basically a more general version of averaging. As a result, the top-line result is very steady. This a case where a blogger-hobbyist can add value. We use the polling/media system to provide added value - for cheap.

Which brings us back to costs...