Nate Silver vs. the Gallup Poll
Nate Silver:
Gallup vs. the World: The Gallup national tracking poll now shows a very strong lead for Mitt Romney… seven points…. The Gallup poll is accounted for in the forecast model…. However, its results are deeply inconsistent with the results that other polling firms are showing in the presidential race, and the Gallup poll has a history of performing very poorly when that is the case.
Other national polls show a race that is roughly tied on average, while state polls continue to indicate a narrow advantage of about two points for President Obama in tipping-point states….
Our database contains records from 136… 53… have issued at least one survey since Oct. 1. With so much data… it will usually be… counterproductive… to get overly attached to… any one particular poll…. [T]he Gallup national tracking poll constitutes a relatively small part of the polling landscape….
Because national tracking polls like Gallup are published every day, they are useful for the trend part of the calculation…. Each poll receives a weight…. [T]here are quite a few interviews conducted by a tracking poll over the course of a week — about 3,000 per week in the Gallup national tracking poll, for instance…. Of the daily tracking polls, the Gallup survey receives the largest weight in the trend-line calculation…. On the other hand, the pollster ratings are also based in part on past accuracy, and Gallup’s performance is middling…. [S]ince Oct. 1, the Gallup national tracking poll has accounted for 12 percent of the information that the model uses to calculate the trend line. The other daily tracking polls, collectively, have accounted for 24 percent of the data, and the occasionally published national polls for 19 percent. Finally, the state polls account for about 45 percent….
As I mentioned, however, this [trend calculation] is only half the battle. Once “old” polls are brought up to date by adjusting them to reflect the current trend in the race, we still need to take some kind of average of them…. [T]he Gallup daily tracking poll accounts for only about 3 percent of the weight in this stage of the calculation….
Perhaps the Gallup poll accounts for 5 or 10 percent of the information that an election analyst should evaluate on a given day….
Usually, when a poll is an outlier relative to the consensus, its results turn out badly. You do not need to look any further than Gallup’s track record over the past two election cycles to find a demonstration of this. In 2008, the Gallup poll put Mr. Obama 11 points ahead of John McCain on the eve of that November’s election… Mr. Obama’s largest projected margin of victory among any of the 15 or so national polls…. The average of polls put Mr. Obama up by about seven points. The average did a good job…. In 2010, Gallup put Republicans ahead by 15 points on the national Congressional ballot, higher than other polling firms…. In fact, Republicans won the popular vote for the United States House by about seven percentage points — fairly close to the average of polls, but representing another big miss for Gallup.
Apart from Gallup’s final poll not having been especially accurate in recent years, it has often been a wild ride to get there. Their polls, for whatever reason, have often found implausibly large swings in the race.
In 2000, for example… a 26-point swing toward Mr. Gore over the course of a month and a half… [then] a 14-point swing toward Mr. Bush over the course of a few days… him ahead by 13 points… just 10 days before an election that ended in a virtual tie. In 1996, Gallup had Bill Clinton’s margin over Bob Dole increasing to 25 points from nine points over the course of four days. After the Republican convention in 2008, Gallup had John McCain leading Mr. Obama by as many as 10 points among likely voters…. It’s not clear what causes such large swings, although Gallup’s likely voter model may have something to do with it….
Because Gallup’s polls usually take large sample sizes, statistical variance alone probably cannot account these sorts of shifts. It seems to be an endemic issue with their methodology.
To be clear, I would not recommend that you literally just disregard the Gallup poll. You should consider it — but consider it in context…
