If it walks and quacks like a duck, we are told, it must be
a duck. Yesterday, MSNBC posted a story describing results from
an "online survey" conducted in partnership with Elle magazine showing that "men rule" in the workplace:
Of male respondents, 41 percent
said men are more likely to be good leaders, and 33 percent of women agreed.
And three out of four women who expressed a preference said they would rather
work for a man than a woman.
Read the article and the survey cited certainly "quacks"
like the sort of projective, representative, "scientific" survey that news
organizations typically report. The article used survey results to describe how
"men" and "women" feel. Academic experts explained and reacted to the findings.
But all appearances to the contrary, it wasn't that sort of duck at all: The
"survey" was neither based on a random sample nor something even MSNBC editors
considered "nationally representative."
The description of the polls methodology consists of the
following paragraph that originally appeared as a sidebar toward end of the
story:
About the Survey
Our online survey was completed by
61,647 people, about 50 percent male and 50 percent female. The average age was
42, 94 percent said they work full-time and 44 percent said they supervise
other workers. Although the sample size is large and diverse, it is not
considered nationally representative because it was largely restricted to
MSNBC.com readers.
While this paragraph tells you what the survey is not ("nationally representative), it
manages to leave out a few critical pieces of information: How were respondents
selected? By saying that the "sample"
was "largely restricted to MSNBC.com readers," the paragraph implies that it
was not a "sample" at all, but rather the result of a straw poll open to anyone
that noticed the invitation to participate on the MSNBC. If so, respondents
were not selected, they selected themselves.**
As a pollster, that distinction is obvious to me, as it was
obvious to my colleagues on the listserv of the American Association for Public
Opinion Research (AAPOR) who were abuzz about
the survey yesterday. Late last night,
AAPOR's president elect Nancy Mathiowetz sent an open letter-to-the-editor
criticizing the article for not making it clear "from the start" that the
survey was "decidedly not a valid scientific, nationally-representative
survey." She continued:
Questions administered to a
self-selected audience are one thing; real surveys that use scientifically
determined random samples in an attempt to measure a population's attitudes and
behaviors accurately are something altogether different.Indulging in the former is not just harmless fun. Labeling self-selected,
online data collection efforts as legitimate survey research are at best
misleading and lead to a diminution of the field as a whole.The survey MSNBC conducted is a
grand way to involve readers with their website. Many news organizations use
such techniques to enrich their relationships with readers. However, these
polls cannot be construed to be a poll representative of anything more than
those who chose to take part, and certainly shouldn't be characterized as
representative of some larger population.
Two additional points worth noting: The first comes from
David Moore, a former senior analyst and vice president of the Gallup
Organization. Yesterday, he reminded his colleagues on the AAPOR listserv
that the basic contention of the MSNBC story is essentially what Gallup surveys found between 1982 and 2002
(quoted with permission):
Whatever the problems with the
"poll," the findings are not far off from a Gallup poll conducted in April 22-24, 2002 (the last time, I believe, that Gallup asked the question: "If you were taking a new job and had your
choice of a boss would you prefer to work for a man or a woman?"). Gallup, by the way, has a trend on this question that goes back to 1953.In April 2002, among males: 29% preferred a male boss, 13% a female boss, and 57% said no preference (a volunteered response); 1% no opinion Among females: 32% preferred a male boss, 23% a female boss, and 43% had no preference; 2% no opinion.
Moore had apparently not seen
an update
by Gallup
conducted in September 2006 that produced very similar results:
A majority of men, 56%, say it
makes no difference whether their boss is a man or a woman, while those who
have a preference favor a man (34%) rather than a woman (10%). Among women, 40%
say they would prefer a new boss to be a man, while 26% would prefer a woman.
Thirty-two percent of women say it makes no difference.
Let's be clear that Moore and Mathowietz both condemn the
MSNBC poll. "MSNBC should have conducted its own contemporary poll," Moore says, "rather than
rely on the admittedly unrepresentative responses it received online."
Second, although totally non-random pseudo can confuse and
mislead, they are relatively easy to condemn. Unfortunately, the more difficult
issue for the survey industry remains what to do about the theoretically
"scientific" surveys that fail to contact or interview individuals at three out
of four randomly selected households. We need to be clear that even surveys based on scientific sampling may still reflect the unique slant of those who choose to participate. Unfortunately,
the line between "scientific" and "pseudo" is not as clear as it used to be.
Interests Disclosed: I currently serve as AAPOR's associate
chair for publications and information, and in that capacity, saw a draft of
the Mathiowetz letter before its release.
Update: Eve Tahmincioglu, author of the MSNBC story, confirms by email that the survey "was based on responses to an invitation to participate on the MSNBC.com website." So the respondents were entirely self-selected, as per my speculation above.
Typo corrected