How Gallup Got It Wrong

I've seen a lot of indignant comments about TPM's pro-Obama bias, because it didn't flog the Gallup results yesterday that showed Clinton up, 52-39. I blogged about that poll, explaining that to the extent we look to tracking polls as predictors of voter sentiment, a poll that showed a thirteen point gap in the three days ending Tuesday at the same time voters divided their votes evenly was clearly flawed.

I also asked the guru of polling, Mark Blumenthal, for an explanation. He put forward a bunch of theories, and followed up with an extensive discussion this morning. Mark is on a righteous crusade to get pollsters to abide by the professional code of conduct to which they ostensibly subscribe, and to disclose their data and not just a summary of their results. So in addition to putting forward his own theories, he pressed Gallup for answers.

Gallup apparently delayed their usual early-afternoon release to crunch the numbers. And just now, Gallup has spoken. Their explanation boils down to this. They call 1,000 adults every night. About 200 of them say they're unlikely to vote, so they screen them out. The remaining 800 describe themselves as at least a little likely to vote (or that they have voted) in either the Democratic or Republican races. In other words, their sample includes 80% of American adults. But even in this seminal year, only about 30% are actually voting. And therein lies the problem.

Gallup went back and ran the numbers for the five days between John Edwards withdrawal and Super Tuesday. Among those who characterized themselves as highly likely to vote, about 50% of the total sample (thus still too high), Obama drew 48% and Clinton 45% - a statistical tie. Now there may have been more than that at work here - Mark's other questions remain unanswered, as does my biggest (what's the gender and racial composition of the sample, and does it vary night by night?) But it goes a long way to explaining the discrepancy.

I'd argue that it provides a good reason for Gallup to use a tighter screen in the future. Gallup says "a broad sample of over 80% of American adults would not be expected to match the actual voting patterns" which to me, is a very good argument for not using such a sample. But it should also provide us all with reason for caution. National tracking polls, it's worth remembering, are useful indicators of trends. Gallup picked up, succesfully, that Clinton had stopped the Obama surge by about Sunday. (And both Gallup and Rasmussen show Obama getting a post-Feb 5 bump). But they are absolutely lousy indicators, at least until they change their voter models, of how the electorate will actually vote.