Posted 8:45 pm Tuesday, August 07, 2012
Send to Kindle
Two out of three pundits don't understand polling
And so the silly season begins. Congress has gone home, the presidential campaigns are regrouping for their fall offensives, and it seems the pundits are worried. They won't have much to write about (though Chick-fil-A last week proved a godsend, so to speak).
What can we expect in coming weeks? Lots of polls — when there's little actual news, polls can always fill some space. So once again, let's discuss the real value of polls, and how much they matter in the months leading up to an election.
The industry standard is the “Real Clear Politics Poll Average,” which takes all of the polls done nationally and computes an average. As of Monday afternoon, it showed President Barack Obama up by 3.4 percent. What that means is… absolutely nothing.
Any poll “average” is only as good as the polls used in the equation. If the polls are flawed, the flaws will be preserved in the average.
Let's take the most egregious example: the Pew Research Center's “Late July Survey,” conducted July 16-26 (which figures into the RCP Average). It shows a full 10-point lead for Obama (51 percent to 41 percent).
When the poll was released last week, even top Democrats dismissed it as fanciful. Pew even went as far as issuing a press release defending its methodology — which polled nearly twice as many self-identifying Democrats as it did self-identifying Republicans.
What can we expect in coming weeks? Lots of polls — when there's little actual news, polls can always fill some space. So once again, let's discuss the real value of polls, and how much they matter in the months leading up to an election.
The industry standard is the “Real Clear Politics Poll Average,” which takes all of the polls done nationally and computes an average. As of Monday afternoon, it showed President Barack Obama up by 3.4 percent. What that means is… absolutely nothing.
Any poll “average” is only as good as the polls used in the equation. If the polls are flawed, the flaws will be preserved in the average.
Let's take the most egregious example: the Pew Research Center's “Late July Survey,” conducted July 16-26 (which figures into the RCP Average). It shows a full 10-point lead for Obama (51 percent to 41 percent).
When the poll was released last week, even top Democrats dismissed it as fanciful. Pew even went as far as issuing a press release defending its methodology — which polled nearly twice as many self-identifying Democrats as it did self-identifying Republicans.
In addition to polling more Democrats than Republicans, the Pew survey relied on registered voters, rather than likely voters.
Pew defends this by noting, correctly, that “Most — but not all — people who are registered to vote cast a ballot in presidential election years.”
But let's define “most.” In 2008, surely an extraordinary year, the voter turnout was 57 percent. In 2004, it was just over 56 percent. The recent low was the last time a Democrat ran for re-election, in 1996 (at 49 percent). The point is that with about half of the registered voters staying home, reliable polls will focus on those who will actually show up — and they're far more likely to be the people who are dissatisfied with the status quo.
In other words, any poll of registered voters will be skewed in favor of the incumbent.
Another issue faced not just by Pew, but also by other pollsters, is the shrinking number of land-line phones. Most polls rely on catching their subjects at home by the phone in the evening. That, in itself, leads to skewed results (older Americans are much more likely to pick up a ringing land-line). But nowadays, nearly everyone relies on their cell phones, on text messages and on voice mail. Pollsters haven't found a way to compensate for the busy mom, the college graduate or the middle-class dad working a second job in the evenings.
Pew defends this by noting, correctly, that “Most — but not all — people who are registered to vote cast a ballot in presidential election years.”
But let's define “most.” In 2008, surely an extraordinary year, the voter turnout was 57 percent. In 2004, it was just over 56 percent. The recent low was the last time a Democrat ran for re-election, in 1996 (at 49 percent). The point is that with about half of the registered voters staying home, reliable polls will focus on those who will actually show up — and they're far more likely to be the people who are dissatisfied with the status quo.
In other words, any poll of registered voters will be skewed in favor of the incumbent.
Another issue faced not just by Pew, but also by other pollsters, is the shrinking number of land-line phones. Most polls rely on catching their subjects at home by the phone in the evening. That, in itself, leads to skewed results (older Americans are much more likely to pick up a ringing land-line). But nowadays, nearly everyone relies on their cell phones, on text messages and on voice mail. Pollsters haven't found a way to compensate for the busy mom, the college graduate or the middle-class dad working a second job in the evenings.
A poll is at best a snapshot of the nation's mood at a particular moment in time. It's not a particularly reliable predictor of how we'll all feel a week from now, much less 90 days from now.
And a poll isn't news. It's just a shame that polls will be presented as news, and as serious news at that, in coming weeks.
And a poll isn't news. It's just a shame that polls will be presented as news, and as serious news at that, in coming weeks.
