A Cautionary Note on the Eve of an Historic Election
(updated below--with an optimistic note)
The final national polls are in and despite diverging rather wildly at times over the last month, they've now settled into a fairly narrow band. All of the polls show Obama with a lead of between 5 and 11 points. So the question on everyone's mind is this: is it possible that the polls are all wrong? Is there any precedent for that kind of systematic polling failure?
The answer is: sort of. The most troubling precedent from the perspective of an Obama supporter is 1996. In that race, most pre-election polls showed Bill Clinton leading Bob Dole by a significantly larger margin than he actually ended up winning by. As this table shows, the final set of national polls showed Clinton with a lead of between 7 and 18 points (with an average lead of 11.75 points). Clinton ended up winning by 8.5%. Of the 8 polls put out on the eve of the election, 6 of them showed Clinton with a lead of 11 or more. The CBS/New York Times poll gave Clinton an 18 point lead. Pew had the lead at 13%. ABC/WaPo, NBC/WSJ, and Harris all had the lead at 12 points. USA Today/Gallup had the lead at 11 points. Only Zogby and Hotline were close to the actual margin (at 7 and 9 respectively).
What's most notable about the 1996 results is that 7 out of the 8 pollsters missed the mark in the same direction--in Clinton's favor--most by significant margins. As this academic paper points out, the odds of that happening due to chance are very small, indicating the high likelihood of some sort of systematic failure. Polling is definitely not an exact science, and there seems to be a real tendency for group think to infect results, i.e., for pollsters to tweak their methods and screening formulas in order to produce results more in line with what everyone else is producing. That could be happening in this race.
That said, 1996 appears to be the exception, not the rule. It was complicated by the presence of Ross Perot in the race. And even in 1996--which was a terrible year for presidential pollsters--at least two of them ended up pretty close to the mark. This year there are more pollsters in the field and none of them has the race closer than4.5 5 points [update: the IBD/TIPP tracking poll--which was showing a 4.5 point lead yesterday--has now ticked up to 7.2]. If the race ends up closer than that, this will have been the worst polling failure in history.
One other word of caution, if you've watched any television tonight, you've likely been reacquainted with the Reverend Jeremiah Wright. Contrary to earlier reports, the ad buy is clearly not limited to swing states. Whether it was the plan all along or whether a last minute wave of donations allowed it to happen, this is clearly a massive national ad buy. I've personally seen the Wright commercial at least 15 times tonight (and I don't live in a swing state). It's apparently been on all day.
I have no idea whether this kind of last minute saturation ad buy has the potential to move the polls in any significant way, but it's at least worth factoring into the equation. The effect of this ad buy, if any, will not be reflected in the polls.
I'm relatively confident about Obama's prospects tomorrow, but I'm not convinced its in the bag yet. As with all things, caution is warranted. There'll be plenty of time for celebration if things go well tomorrow.
UPDATE: One development that those (like myself) who worry about some sort of systematic error in the polling should take some comfort in is the convergence of the the various Gallup tracking poll models over the last week. Gallup, in a clear effort to hedge its bets in a difficult polling environment, began releasing several different sets of polling numbers last month. The first--the "traditional" likely voter model--used the screening formulas Gallup had utilized in prior elections to determine which registered voters would actually show up. But they also released an "expanded" likely voter model with less stringent screens. For much of the last month, there has been a significant disparity between the numbers produced by those two likely voter models. In the last week, however, the two models converged and both now show an 11 point Obama lead. Importantly, both models have also converged on the registered voter numbers (which currently shows Obama ahead by 13).
What that indicates to me is that Gallup is pretty confident now that turnout will be exceptionally high and that virtually every registered voter is a "likely voter." It's also a sign that Gallup is pretty confident in its results, that it is no longer trying to hedge its bets. I interpret that as a good sign.
The final national polls are in and despite diverging rather wildly at times over the last month, they've now settled into a fairly narrow band. All of the polls show Obama with a lead of between 5 and 11 points. So the question on everyone's mind is this: is it possible that the polls are all wrong? Is there any precedent for that kind of systematic polling failure?
The answer is: sort of. The most troubling precedent from the perspective of an Obama supporter is 1996. In that race, most pre-election polls showed Bill Clinton leading Bob Dole by a significantly larger margin than he actually ended up winning by. As this table shows, the final set of national polls showed Clinton with a lead of between 7 and 18 points (with an average lead of 11.75 points). Clinton ended up winning by 8.5%. Of the 8 polls put out on the eve of the election, 6 of them showed Clinton with a lead of 11 or more. The CBS/New York Times poll gave Clinton an 18 point lead. Pew had the lead at 13%. ABC/WaPo, NBC/WSJ, and Harris all had the lead at 12 points. USA Today/Gallup had the lead at 11 points. Only Zogby and Hotline were close to the actual margin (at 7 and 9 respectively).
What's most notable about the 1996 results is that 7 out of the 8 pollsters missed the mark in the same direction--in Clinton's favor--most by significant margins. As this academic paper points out, the odds of that happening due to chance are very small, indicating the high likelihood of some sort of systematic failure. Polling is definitely not an exact science, and there seems to be a real tendency for group think to infect results, i.e., for pollsters to tweak their methods and screening formulas in order to produce results more in line with what everyone else is producing. That could be happening in this race.
That said, 1996 appears to be the exception, not the rule. It was complicated by the presence of Ross Perot in the race. And even in 1996--which was a terrible year for presidential pollsters--at least two of them ended up pretty close to the mark. This year there are more pollsters in the field and none of them has the race closer than
One other word of caution, if you've watched any television tonight, you've likely been reacquainted with the Reverend Jeremiah Wright. Contrary to earlier reports, the ad buy is clearly not limited to swing states. Whether it was the plan all along or whether a last minute wave of donations allowed it to happen, this is clearly a massive national ad buy. I've personally seen the Wright commercial at least 15 times tonight (and I don't live in a swing state). It's apparently been on all day.
I have no idea whether this kind of last minute saturation ad buy has the potential to move the polls in any significant way, but it's at least worth factoring into the equation. The effect of this ad buy, if any, will not be reflected in the polls.
I'm relatively confident about Obama's prospects tomorrow, but I'm not convinced its in the bag yet. As with all things, caution is warranted. There'll be plenty of time for celebration if things go well tomorrow.
UPDATE: One development that those (like myself) who worry about some sort of systematic error in the polling should take some comfort in is the convergence of the the various Gallup tracking poll models over the last week. Gallup, in a clear effort to hedge its bets in a difficult polling environment, began releasing several different sets of polling numbers last month. The first--the "traditional" likely voter model--used the screening formulas Gallup had utilized in prior elections to determine which registered voters would actually show up. But they also released an "expanded" likely voter model with less stringent screens. For much of the last month, there has been a significant disparity between the numbers produced by those two likely voter models. In the last week, however, the two models converged and both now show an 11 point Obama lead. Importantly, both models have also converged on the registered voter numbers (which currently shows Obama ahead by 13).
What that indicates to me is that Gallup is pretty confident now that turnout will be exceptionally high and that virtually every registered voter is a "likely voter." It's also a sign that Gallup is pretty confident in its results, that it is no longer trying to hedge its bets. I interpret that as a good sign.
posted by A.L. at 10:20 PM
9 Comments:
A.L.,
If you're inclined to believe Nate Silver (and as a baseball fan, his word is unquestioned for me), you'll see that if anything, the polls may be indeed be wrong, but wrong in McCain's direction. The main difference between today and '96 is the cellphone problem, which he's written about extensively and which I haven't really heard talked about anywhere else. Basically, 18-30 year olds, Obama's bread and butter demographic, increasingly use cellphones exclusively, and so pollsters don't call them. So I'm not sure 1996 is a perfectly apt example here.
But don't pollsters know about the cell phone phenomenon and allow for it (weighting) in their formulas?
It's not as if they're ignoring the issue altogether. You're saying they're not giving it enough weight (maybe they're giving it too much, or maybe they're right on target). But what is your basis for thinking that?
Weighting the sample certainly does help, but then you run into the problem of small sample size: if you can only find a smallish number of 18-30 year olds to sample, and then you blow up that number to account for cellphone users you couldn't reach, you're increasing the chance that your small cohort wasn't a representative sample.
I think the polls can be a little misleading int his election because a lot of people are not admitting their racism.
The thing about the Bradley Effect I don’t get is why a person who is racist (or isn’t racist but doesn’t want to be perceived as racist) would feel it necessary to lie to an anonymous pollster.
I mean, come on: lots of ppl are voting for the other guy, so what’s wrong with just saying “I’m voting for ______ (the white guy).” That doesn’t imply you’re racist. You’re not required to tell the pollster “I’m voting for the white guy because I’m a raging racist.” Now I can see how people might lie instead of say THAT, but that’s not what’s going on here.
It’s not like you’re talking to your neighbors who you want to maintain a nice relationship with. This is a random pollster. We as a society have so little respect for random ppl who call us on the phone that we created a “do not call list.” So random people who don’t know us who call us are as low as dirt, basically. Who cares what they think of you. And you don’t have to tell them WHY you’re not voting for the black guy – just that you’re not (like thousands of other ppl).
That’s what I mean by not understanding the pyschological basis for the Bradley Effect, and therefore doubting it’s existence (based on stuff I’ve read doubting it’s existence). Racism, I understand completely.
Obvious caveat is that no one turned out for the 1996 election, which contributed to the drop relative to final polling numbers.
Enthusiasm is not an issue this year.
Random thoughts:
I was right about the Bradley Effect. There is no rational basis for its supposed psychological underpinnings, ie: it isn't real.
Also on the talk about cell-only voters getting under-polled was wrong. The pollsters apparently weighted correctly for this in their formulas. Why this was not more obvious to most people, I don't know.
The polls were essentially right on the money.
I thought Obama would win via the VA (and PA), IA, CO, NM route, but I didn't expect such a wave. FL, OH, IN, SC!? Wow.
Can't believe convicted felon Stevens is ahead in AK. Though he may deserve a new trial due to the shinanigans, and might get one on appeal. But if the Senate kicks him, Palin could win a special election. Ugh on the one hand, and I hope she becomes the new face of the party on the other - because they'll be in the wilderness that much longer if so. Funny would be if the Dem senators refused to act on kicking out Stevens, so that Palin would not have the opportunity to insert herself into the senate, but rather be left to rot in Alaska. On the other hand, if she's the new face of the party...
The thing I'm most sick about is Bachmann winning in MN. Now we've got to look at her fixedly-smiling face and listen to her lie for at least two more years. She is a first rate propagandist, but on the other hand Americans (a small majority at least) seem to have become inoculated to that to some degree.
I hope Franken pulls it out, and Chamblis gets beat in the runoff in GA - that seems possible given it'll be low turnout and awash in Dem money and activists.
Will Obama appoint anyone like Specter or Snow to the cabinet, thereby gaining a Dem senator appointed by a Dem governor? I think this must be a maneuver he's seriously considering.
How obstructionist will the Republican rump in the Senate be? There are some big senate races coming up in 2010. McCain for one will have to defend against the popular AZ governor Napolitano.
I am so happy for the Supreme Court. Talk about a close call...
A minor nit: I think you'll find it was 1992 when Perot messed everything up, not 1996. We probably have Perot to thank/curse for President Clinton instead of 2 terms of Bush 41.
Post a Comment
Links to this post:
Create a Link
<< Home