My post on methodological disclosure by pollsters earlier this week reproduced list of "minimal (Level 1)" information that members of the National Council on Public Polls (NCPP) are required to include in all public reports, including "fieldwork provider (if applicable)." That means that if a pollster subcontracts telephone interviewing to a call center, they are supposed to disclose the name of the company in their reports.
Yesterday, I received an email from a media pollster who asks a reasonable question:
I've never understood why a firm would be required to disclose the fieldwork provider. I cannot see how it reveals anything about the quality of the poll, unless it is done by some firm everyone knows to be unreliable. To me, it is sort of a trade secret. Why would I want to reveal my subcontractor?
I emailed NCPP President Evans Witt (who is also the CEO of Princeton Survey Research Associates International - PSRAI) for comment. This is his response:
One of the issues that we deal with at NCPP is that the question of "Who did the poll?" or "Who was the pollster?" can be answered in a surprising number of ways. It arises more clearly for NCPP since NCPP is an organization whose members are organizations.
For example, The New York Times poll is simple: they designed and conducted the poll and did the interviewing in-house.
ABC News, for example, used ICR for a recent poll, providing this disclosure: "Sampling, data collection and tabulation by Social Science Research Solutions at ICR-International Communications Research of Media, Pa."** Very good disclosure, even if it is not necessarily simple.
PSRAI does for polls for Newsweek, and (usually) the telephone interviewing is done at Braun Research in Princeton, N.J., (a company independent of PSRAI) under PSRAI supervision.
Or take a survey that is done for an interest group via a public relations firm. The public relations firm may or may not have internal survey expertise. So the PR firm may have contracted out just the telephone interviewing or the PR firm may have contracted out the design, execution and interviewing. All are possible and legitimate arrangements.
To try to cover all this complexity, the field house requirement was included in the standard. Therefore, the standard covers the circumstance where the real knowledge about how the poll was done resides with the field house. It also helps to deal with a situation where there is a basic question of whether any poll was actually done, by providing at least one more avenue for asking for confirmation of the interviewing.
The discussion also raises some practical issues for us in trying to assess this aspect of disclosure. How do we know if an organization has fully disclosed field house, "where applicable?"
It is relatively easy for us to take note of field house when disclosed, as in the examples above. But how do we know when the need to disclose the field house "applies?" Consider that even very detailed New York Times "How the Poll Was Conducted" sidebar neglects to say explicitly that Times/CBS poll conducts interviews "in-house." If we flunk organizations for failing to make clear that they did not use a field house, my guess is that we'd strike a fair number of NCPP members.
But practical issues aside, I want to come back to the question that provoked this post from the news consumers' perspective. Does knowing the name of the field work provider, or knowing that pollster did interviewing in-house, help you assess the quality of the poll? Should this information be something we insist that pollsters disclose more explicitly?
*Correction*: We modified the response from Evans Witt at his request. Although ABC News used ICR for a recent survey, they more typically work with TNS of Horsham, Pa.