I have to admit that I had been hoping to take a closer look at polling on Maine's Question 1 on Gay Marriage over the weekend, but got distracted by the fuss over the New York 23rd District special election. The polling is difficult to evaluate partly because there has been so little of it. While I have a lot of confidence in our trend estimates in states with large numbers of polls, the small number of polls in Maine (7 total since Labor Dayl) allow for just crude linear trend lines.
Another reason why the Maine polls are difficult to evaluate is that issue referenda polling is so treacherous and prone to error. A 2004 paper by Joe Shipman, then director of election polling for SurveyUSA, showed that polling on ballot measures had triple the rate of error (9.5 average error on the margin) as polls in presidential elections (3.4) and nearly double that of contests for statewide offices (4.6). I summarized the assumed reasons for that greater error rate in a long post four years ago today, but the most relevant to Maine are a greater difficulty modeling the likely electorate and the problem of accurately conveying ballot language.
A particularly painful example followed a few days after that post, when a set of ballot initiatives in Ohio produced some of the biggest polling errors in recent memory. The combination of failing to poll late and not accurately reproducing the actual ballot language were likely culprits.
Let's start with the ballot language in Maine that voters are confronting right now:
Do you want to reject the new law that lets same-sex couples marry and allows individuals and religious groups to refuse to perform these marriages?
So, a "Yes" vote is a vote against gay marriage, and a "No" vote is for gay marriage. Confused? Imagine the uncertainty some Maine voters may be experiencing without that extra bit of explanation. As you can see in the table at the bottom of our chart, only the Pan Atlantic SMS surveys reproduce the actual ballot language -- and nothing else -- while the other pollsters provide a line of explanation to clarify the meaning of "Yes" and "No."
The final round of polling has shown a relatively close race, although results have varied. A survey conducted two weeks ago by Pan Atlantic SMS, shows the No side prevailing by an 11-point margin (53% to 42%), while a Daily Kos/Research 2000 poll conducted late last week shows a dead-heat (No 48%, Yes, 47%) Finally, the Democratic automated polling firm PPP conducted a survey over the weekend that had the Yes side ahead by a not-quite-statistically significant four points (51% to 47%) despite a very large (n=1,133) sample.
Complicating the issue further is that the final poll from PPP differed in both the age of the "likely voters" they selected and the way they interviewed (via an automated, recorded voice methodology). Less than a third of PPP's likely voters (32%) were under age 45, compared to more than half (51%) of the Research 2000/DailyKos survey. Both showed much more support for the No side from younger voters.
On the question of age, the Research 2000 sample was even younger than the Maine exit poll in November 2008 (43% age 18-45) and far younger than in November 2006 (36% age 18-45). Of course, the PPP sample was older than both, but keep in mind that exit poll estimates are sometimes too young.
The question of the automated mode is more complicated. The automated polls conducted by SurveyUSA in California 2008 may have picked up more support for ultimately successful anti-gay marriage Proposition 8 than on live interviewer surveys conducted at the same time. However, the convoluted nature of the timing of the various polls and the final result from SurveyUSA (showing Prop 8 narrowly failing) make it impossible to draw firm conclusions.
So what conclusions can we reach about tonight's outcome in Maine? I have more faith in the age composition of the PPP poll than the one from Research 2000, but given the much larger potential for error in ballot referenda and the close margins on the two final polls, your guess is probably as good as mine.