Yet still more about polls and all that Tom Smith

Well, one thing I learned for sure is that I have to pay more attention to the world of statisticians who aggregate polls if I want to follow polls at all.  This most recent time around I relied on a subjective method of looking at RCP averages and intrade and discounting them by a subjective factor based on how likely I thought it was that the state polls might be systematically biased. By the time Sandy had passed I had reached a subjective probability of 60/40 Obama/Romney, still pretty good odds for Mitt, but I think it's clear now, and not just as a matter of hindsight bias, that Romney's chances were far less than that going in. Knowing that could have saved me some disappointment.  

Since I'm hardly a statistician, I relied on the judgments of people I regarded as pretty wise in these things and they were telling me that the state poll aggregators and the prediction markets needed to be treated with great caution, that they were assuming an implausibly high turnout for the O-man.  It turns out the state polls were right on, at least in the matter of relative turn out of demographic groups, but it remains unclear, to me at least, if they turned out to be right for the reasons they thought they would be in the first place. 

I'm not sure how someone like me, a news junkie but not a math person, is supposed to have known that Gallup and Rasmussen would end up with egg on their faces, while PPP and Marist, which have pretty poor reputations, especially PPP, would end up looking good.  I still don't know, but would like to, if there was some methodological difference that explains why one succeeded and the other failed -- I know they had different turnout models, but my question is, do they have different models that produced different turnout expectations, or do they just make judgment calls on, for example, what percentage of the turnout will be white or black, based on historical data or something? A naive question probably. But not one I saw explained in political presses. Some of this information seems proprietary and pollsters reluctant to talk about it.

The big winners in my book are the modellers and prediction markets.  Of course, we can't know I guess, but I wonder who was closer to being right, the prediction markets or the modellers, over the course of the campaign.  Intrade was generally a little more bullish on Mitt than Nate Silver, for example.  Did the market know something that Silver didn't?  I like to think it did.

It's good news, in a way.  I want powerful tools to understand what is going on around me.  The GOP better have figured out that it needs to move into a new age, well beyond Karl Rove and his little whiteboard.  I watched that whole debacle for him on Fox live. It was pretty sad, though I could watch Megyn Kelly all day. You did get the sense of an era closing, with the geeks in the back room poning the wise, old insider. The GOP needs to get its geek on. Old school doesn't cut it any more.

Comments

[Thomas]

Yes, there are methodological differences: Rasmussen interviews a demographically representative sample of adults to determine party split, and then uses a stable average of party split to determine weighting for his head-to-head political polling (which interviews sample of registered voters), while also using a likely voter qualifier. (Not clear if the sample of registered voters is first weighted for partisan ID, or if he adjusts on the back-end.)

Gallup doesn't weight for party ID. They start with interviews of a demographically representative sample of registered voters (with some variation over time in what it means to be representative--they switched from being representative of those with phones to those in the underlying population this year) They then use a likely voter screen validated from prior elections to determine turn out. Because there's no partisan weighting, there's much more movement in this poll than in Rasmussen. They do not weight the results for projected turnout--voters qualify themselves through the screen, and everyone past the screen is in the pool.

PPP doesn't do partisan weighting. They do however weight the results based on projected turnout. They chose the demographics of the 2008 election as the basis for this year's. This is the weakest polling choice, methodologically speaking, that I can think of. It wouldn't have worked last time, to say the least. But it sure worked this time.