SurveyMonkey shares previously unreleased 2014 polls that outperformed Pollster's averages. Another poll shows bipartisan support for President Obama's immigration policy, if not his executive action. And the latest investigation changed few minds on Benghazi. This is HuffPollster for Thursday, December 4, 2014.
SURVEYMONKEY'S 2014 ELECTION POLLS - A set of previously unreleased polls from SurveyMonkey shared exclusively with HuffPollster helps demonstrate the potential for their internet-based survey methodology.
A few days before the election, SurveyMonkey shared with HuffPollster a spreadsheet that included horse race results from surveys they conducted over the last two weeks before the election in 45 states, covering every contested race for Senate and governor in 2014.
Since the project was both experimental and relatively cursory in its modeling of the likely electorate, HuffPollster agreed to wait to publish the results until after the election. That said, we can verify that the analysis that follows derives from data shared with us in final form on the morning of November 4, before votes were counted.
Some background on the methodology: While telephone polls typically involve true random samples of working telephone numbers, the Internet allows no way to randomly select a sample of all email addresses or those who go online. Instead, most Internet research companies, including HuffPost's polling partner YouGov, recruit large "panels" of respondents, usually via web advertising, who agree to participate in online surveys. Since the panels are not representative of larger populations, Internet researchers strive to reduce bias and make their data representative through the use of weighting, sample matching and other efforts to "model" a representative sample.
For this project, SurveyMonkey took a somewhat different approach. They did not draw participants from a pre-recruited panel. Instead, they solicited respondents from the millions of people that complete SurveyMonkey's "do it yourself" surveys every day run by their customers for companies, schools and community organizations. At the very end of these customer surveys, they asked respondents if they could answer additional questions to "help us predict the 2014 elections." That process yielded over 130,000 completed interviews across the 45 states with contested races for Senate or governor.
SurveyMonkey tabulated the results for all adult respondents in each state after weighting to match Census estimates for gender, age, education and race for adults -- a relatively simple approach analogous to the way most pollsters weight random sample telephone polls. SurveyMonkey provided HuffPollster with results for each contest tabulated among all respondents as well as among subgroups of self-identified registered voters and among "likely voters -- those who said they had either already voted or were absolutely certain or very likely to vote (full results are published here).
“We sliced the data by these traditional cuts so we could easily compare them with other surveys," explains Jon Cohen, SurveyMonkey's vice president of survey research, "but there’s growing evidence that we shouldn’t necessarily use voters’ own assessments of whether or not they'll vote." In future elections, Cohen adds, they plan "to dig in and build more sophisticated models that leverage the particular attributes of the data we collect." (In a blog post published separately on Thursday, Cohen adds more detail about how the surveys were conducted).
The results are relatively straightforward. The full SurveyMonkey samples did very well in forecasting winners, showing the ultimate victor ahead in all 36 Senate races and missing in just three contests for Governor (Connecticut, Florida and Maryland).
The more impressive finding is the way the SurveyMonkey samples outperformed the estimates produced by HuffPost Pollster's poll tracking model. Our models, which are essentially averages of public polls, were based on all available surveys and calibrated to corresponded to results from the non-partisan polls that had performed well in previous elections. SurveyMonkey's full samples in each state showed virtually no bias, on average. By comparison, the Pollster models overstated the Democrats' margins against Republican candidates by an average 4 percent. And while SurveyMonkey's margins were off in individual contests, the spread of those errors was slightly smaller than the spread of those for the Pollster averages (as indicated by the total error, the average of the absolute values of the error on the Democrat vs Republican margins).
Not surprisingly, the cuts that narrowed their full samples to self-reported registered and likely voters made the SurveyMonkey results more Republican and introduced some overstatement of Republican performance. Nonetheless, the average error for their results among these subgroups were still smaller than Pollster tracking models for both bias and total error. Given the solicitation used -- an explicit pitch to help predict the elections -- it's likely that their initial participants were already "likely voters." Interest in the survey topic has been shown to influence response, and validation studies find that voters are more likely than non-voters to participate in telephone surveys.
It is possible that the voters in this particular election, with an apparently lower turnout of key Democratic constituencies, closely matched a Republican leaning pool of respondents that completed SurveyMonkey customer surveys. "If you’re a polling firm who always shows Rs ahead," as election forecaster Drew Linzer put it on Twitter in the context of automated phone polls, "an election where lots of Rs win doesn't validate your methods."
But that said, these results suggest that SurveyMonkey's pool of respondents is diverse (or "heterogeneous") enough that only relatively simple modeling was necessary, at least in this case, to remove their bias as compared to a true random sample of voters. While SurveyMonkey plans on "more sophisticated models" in the future, this characteristic means their efforts may not need to be quite as heroic as those applied to other data sources with more inherent bias.
"This was certainly exciting for me," said Cohen, "given the last dozen years I’ve spent with high-end telephone surveys." Cohen was formerly the polling director at the Washington Post and a vice president of research at the Pew Research Center. "The data on voter preferences were consistently different (read: more Republican) from what was publicly available, even as other measures — presidential approval and issues — were closely in sync with public numbers. Having tens of thousands of cases in important sub-groups also felt like an amazing luxury. There’s lots of work ahead, but this was a gratifying start to figuring out exactly what we can do with our data.”
REPORT CHANGES FEW MINDS ON BENGHAZI - Kathy Frankovic: "After a two-year investigation, the House Intelligence Committee has found no wrongdoing related to the 2012 attack on the U.S. Benghazi consulate that resulted in the deaths of four Americans, including the U.S. Ambassador to Libya. The report has not convinced many, however, according to the latestEconomist/YouGov Poll. Perhaps part of the reason is that only 20% of the public say they have heard a lot about the report, which was released on a Friday a week before Thanksgiving….44% overall and three-quarters of Republicans do not believe the Administration did all it could to rescue those trapped in the Benghazi Consulate….There is yet another Congressional investigation of Benghazi yet to come: one by the GOP-led House Select Committee appointed in May. Many Americans see at least a little bit of politics in these continued investigations: less than one in four think they are solely serious attempts to find out what really happened." [YouGov]
SUPPORT FOR OBAMA'S IMMIGRATION PLAN - PRRI: "More than 7-in-10 (72%) Americans favor allowing illegal immigrants who are the parents of children with legal status to stay in the U.S. for three years without being subject to deportation if they pass a background check and have lived in the country at least five years. Roughly 3-in-10 (27%) Americans oppose this policy. There is bipartisan and cross-religious support for the policy, although the strength of support varies substantially. More than 8-in-10 (82%) Democrats, 7-in-10 (70%) independents, and more than 6-in-10 (62%) Republicans support allowing illegal immigrants who have children with legal status to remain in the country for three years without being deported if they pass a background check and have lived in the country for five years….Americans are divided over whether Obama should have pursued executive action on immigration. Half (50%) of Americans say Obama should have taken action on immigration given that Congress has not acted on the issue, while 45% say he should not have taken action….In the wake of the high profile announcement about the executive action on immigration, President Obama’s job approval rating among Hispanics increased significantly. Currently, more than 6-in-10 (62%) Hispanics say they approve of the job Obama is doing as President—a marked uptick from October 2014 when fewer than half (46%) of Hispanics expressed approval." [PRRI]
ERIC GARNER JURY MAY HAVE BEEN ESPECIALLY UNLIKELY TO INDICT - Harry Enten: "Staten Island residents are particularly sympathetic to the NYPD compared to New York’s other boroughs, and a majority of residents there think the officer should not have been charged. In an average of Quinnipiac University polls taken in August and November, only 41 percent of Staten Island residents supported bringing charges against Officer Daniel Pantaleo (the margin of error on these combined subsamples is 7 percentage points). In New York overall, 64 percent approved of criminal charges. Staten Island isn’t like the rest of the city….Had this trial been held in any other borough, the pool of potential grand jurors would have been less favorable to the NYPD." [538]
HUFFPOLLSTER VIA EMAIL! - You can receive this daily update every weekday morning via email! Just click here, enter your email address, and and click "sign up." That's all there is to it (and you can unsubscribe anytime).
THURSDAY'S 'OUTLIERS' - Links to the best of news at the intersection of polling, politics and political data:
-Aaron Blake reviews a history of post election party ID polling bumps. [WashPost]
-Frank Newport considers the implications of opposition to the notion of the federal government ensuring that all Americans have healthcare coverage. [Gallup]
-Anna Greenberg (D) says Democrats should double down on "women's issues." [Politico]
-Latino Decisions (D) message-tests the impact of immigration on potential support for Hillary Clinton. [Latino Decisions]
-Microsoft makes its Bing Pulse audience polling service available to anyone. [TechCrunch via @OpinionToday]
-How Chicago health officials and (D) used data mining to refine outreach for breast cancer screening. [Kaiser Health News]
CORRECTION: An earlier version of this story misstated the title of Jon Cohen, who is SurveyMonkey's vice president of survey research
Before You Go
