THE BLOG
12/20/2006 01:09 pm ET | Updated May 25, 2011

Does Spelling Count?

"To retain respect for sausages and laws, one must not
watch them in the making." Unfortunately,
that quotation (widely attributed
to Otto von Bismarck) may sometimes also apply to surveys. It is all too easy to put out percentages on
a press release calculated to one decimal place that add to 100.0%. More often than not, the factors that
distinguish excellence in survey research are invisible to the consumer. Of course, with an online survey the
respondent does have the ability to save the verbatim questionnaire, and that
brings us today's story about the remarkable number of spelling and grammatical
errors in two recent surveys conducted online by Zogby International.

I should say up front that yours truly is in no position to
cast the proverbial first stone when it comes to spelling and typographical
errors. In my first week of blogging,
Mickey Kaus observed that the
Mystery Pollster "also seems to be a bit of a Mystery Speller." While I have tried to have others proof my
copy before it goes online, I am painfully aware that typos probably remain in my
archives. Alas, proofreading is not my
strong suit.

However, all pollsters understand the critical importance of
proofing our work product -- be it a draft questionnaire or the final results
of a poll -- before it goes into the field or out the door. In a business where consumers expect
precision, evidence of sloppy work takes on added significance. Just one careless, trivial mistake can
undermine a client's confidence in an otherwise brilliantly executed
survey.

The importance that most pollsters place on proofreading makes
the email I received last week from a long time reader, someone with
considerable training and experience in survey research, so remarkable:

As you
may recall I'm on Zogby's Interactive panel. In the past
the surveys often contained oddly worded questions, but at least they were
copy-edited, spell-checked, proof-read, etc. Lately? Not so much. A few days
ago I took a survey that included a number of topics, including the
Israeli-Palestinian situation. I was struck by a number of misspellings. Here are the items that included simple
misspellings of the type that are caught by spell-check (some are the actual
question; some are response options) [Misspelled words in bold type]:

The U.S.
Constitution says that a president can be impeached for high crimes and misdeameanors.
Knowing this, would you favor or oppose impeaching George W. Bush?

  • Have you ever visted Bethlehem?
  • How interested are you in visting Bethlehem again or for the first time?
  • Do you support or oppose the contruction of this wall?


  • The wall cuts through a historic Christian Diocese, seperating Bethlehem from Jerusalem

  • The wall seperates Bethlehem and Jerusalem - two cities that have been histoircally interlinked and interdependent through community ties, trade and religious traditions

  • The wall seperates Bethlehem and Jerusalem - two cities that have been histoircally interlinked and interdependent through community ties, trade and religious traditions

  • The wall seperates some Bethlehem families from one another

  • The wall cuts through a historic Christian Diocese, seperating Bethlehem from Jerusalem
  • The wall ",1] ); //--> }catch(e){}The wall seperates some Bethlehem families from one another [repeats twice]

  • The wall cuts through a historic Christian Diocese, seperating Bethlehem from Jerusalem [repeats twice]
  • The wall seperates Bethlehem and Jerusalem - two cities that have been histoircally interlinked and interdependent through community ties, trade and religious traditions [repeats twice]
  • Do you think that the land that is being confiscated by Israel in order to build the wall is primarly taken from. . .?
  • Chrisitan residents of Bethlehem
  • Of those listed, which of the following would be the greatest deterent to visiting Bethlehem?
  • Peacful Coexistence
  • That's
    bad enough, but today I received [another Zogby online] survey that was
    so awful it was painful. After an annoying but not terrible section on moral choice
    situations, they started with some civil lawsuit scenarios and then started
    using the word "libel" all over the place. At first I'm trying to
    figure out what libel has to do with anything, but then I realize that they
    actually mean "liable." They do this not just once, but for several
    scenarios, each involving several questions. It's really awful!

    My reader goes on to provide examples of questions. The survey presented a series of factual
    "scenarios" involving two fictional physicians, Dr. Smith and Dr. Jones, then
    asked questions like the following:

    13. On a scale of one to five, with
    one being not libel and five being libel, how likely are you to find Dr. Smith
    libel in this case?

    Not libel

    Probably not libel

    Neutral

    Probably libel

    Libel

    Not sure

    14. On a scale of one to five, with
    one being not libel and five being libel, how likely are you to find Dr. Smith
    libel if you knew that he had never had a child suffer the damage Jane did,
    even though he has had several other children also complain of a stinging
    feeling while receiving the same vaccination?

    Not libel

    Probably not libel

    Neutral

    Probably libel

    Libel

    Not sure

    My reader continues:

    That's just the first of 21
    questions like that, all of which make the same repeated mistake. All in all,
    the word "libel" shows up 137 times in the survey! This is indeed a
    train wreck.

    But that's not all. With my jaw
    still on the floor, on the fourth or fifth scenario I noticed something else.
    They keep on asking "on a scale of one to five" but there's no scale,
    no numbers, just the response options. This error is made 30 times in the
    survey.

    I
    emailed Zogby's Director of Communications for comment on Friday of last week
    and have not received any response.
    Zogby posted results
    of the Bethlehem
    survey yesterday.

    Readers may ask whether these misspellings affect the
    results of the survey. Of course, spelling
    errors matter less when a survey is conducted by telephone (assuming that the
    interviewers pronounce misspelled words like "visted" correctly rather than phonetically). The two Zogby surveys discussed here were conducted online, yet even when online respondents notice the misspellings
    of words like "seperate" and "historically," they can presumably guess at
    the intended meaning. But the
    libel/liable mix-up raises more troubling questions. If respondents did not know the difference
    between the two words, did they understand the issues raised by the questions
    well enough to provide meaningful answers?
    If they knew the meaning of "libel" but did not guess it really meant "liable,"
    how did they interpret the question? And if they deciphered the error, as my
    friend did, how many times could they tolerate its repetition before bailing
    out of the survey altogether?

    But
    mostly, this sort of error should make consumers wonder. If a pollster neglects something as simple as
    spell checking a questionnaire before fielding it, what other mistakes is that
    pollster making that are not quite so obvious? Oh wait...