03/17/2010 10:50 am ET | Updated May 25, 2011

Re: The Literary Digest Poll

Last week, I [linked]( to an [article]( on Dr. George Gallup that included a reaction from statistical consultant Dominic Lusinchi that had been posted on the members-only listserv mailing list of the American Association for Public Opinion Research (AAPOR). Lusinchi argued that the original article perpetuated two "myths" about the infamous Literary Digest poll of 1936 -- the poll whose failure helped establish competitor Gallup as a household name. No sooner had I reproduced Lusinchi's email than Jan Werner, another knowledgeable AAPOR member took to the listserv to quarrel with Lusinchi's version of the history.

While a discussion of the shortcomings of a 74-year-old poll may seem a little out of place here, it involves issues that remain highly relevant to the contemporary debates about modern surveys, including the merits of probability sampling and the errors that can result from poor coverage or a low response rate. Since I posted Lusinchi's original comments, I also want to share the full exchange, which begins after the jump. Thanks to Dominic and Jan for allowing me to reproduce their comments here.

**Original comment from Dominic Lusinchi**:

>[This story]( perpetuates two myths:

>1) That Gallup "predicted" that the Digest poll would forecast a Landon victory and

>2) That the Digest failed because its sampling frame was "skewed ... to the wealthy".

>Myth 1: In a July 12, 1936 syndicated column "America Speaks", Gallup wrote:"If the Literary Digest were conducting its poll *at the present time* [my emphasis], following its usual procedure, Landon would be shown in the lead." (Wash. Post, Section III, p.2, col. 7, Sunday, July 12, 1936) It's one thing to say "at the present time" and another to say "when the Digest presents its final results".... It is only after the Digest poll debacle that this story morphed into a "prediction". What Gallup really predicted, at that time (7/12/1936), was that the election was going to be a close one: the title of his column "1936 Election Seen As Closest in Years".

>Myth 2: The Digest poll failed because its original sample, composed mainly of telephone and/or car owners, was irretrievably skewed against Roosevelt. A close analysis of a May 1937 Gallup (yes, Gallup!) poll, which asked its respondents if they had received and returned a Digest ballot card, shows that the principal cause of the Digest poll's failure was non-response bias. As Peverill Squire wrote in POQ (vol. 52, 1988, p.125), "if all those who were polled had responded, the magazine would have, at least, correctly predicted Roosevelt the winner." In fact, its prediction (my analysis) would have been as good if not better than Gallup's - he was off by nearly 7 points of the two-party vote.

**Jan Werner**:

>It's one thing to debunk a myth, it's something quite different to replace one myth with another.

>Myth 1

>What Gallup wrote in the July 12 1936 article was: "...if The Literary Digest were conducting a poll at the present time...the actual figures would be in the neighborhood of 44 per cent for Roosevelt and 56 per cent for Landon." Although it was presented as a throw-away comment, Gallup did not come up with those numbers lightly. As he explained to a journalist a few years later, he "was able to call [The Literary Digest's] shot by making a separate survey of telephone subscribers and automobile owners -- the class which would mail back most of the Digest's ballots." (Williston Rich, Jr., "The Human Yardstick," Saturday Evening Post, January 21, 1939)

>The statement was clearly intended to provoke a response from The Literary Digest and thus generate publicity for Gallup's fledgling
organization. In that regard, he succeeded. On July 19, 1936, the New York Times quoted an open letter from Wilfred J. Funk, editor of The Literary Digest, as follows: "I am beginning to wish ... that the esteemed Dr. Gallup would confine his political crystal-gazing to the offices of the American Institute of Public Opinion and leave our Literary Digest and its figures politely and completely alone...We've been through many poll battles...We've been buffeted by the gales of claims and counter-claims. But never before has any one foretold what our poll was going to show before it was even started."

>Thus, even if Gallup's claim might not technically be described today as a prediction, it was certainly taken as such at the time by his target, and was probably meant to be.

>Myth 2

>Peverill Squire did indeed write in POQ (vol. 52, 1988, p.125), "if all those who were polled had responded, the magazine would have, at least, correctly predicted Roosevelt the winner." However, he then goes on to say: "But more importantly, the initial sample was flawed; when compounded with the response bias, it produced the wildly erroneous forecast of the vote percentages." and then states that "a rough calculation of the bias produced by the sample is around 11%, with another 7% accounted for by problems with the responses." Unfortunately, there is no reason to trust the data from the May 1937 Gallup poll used to estimate the nonresponse rate (as Squire himself admits, before going on to do so). There certainly is nothing to justify this kind of precision in divvying up error among specific causes.

>The fact is that The Literary Digest did not define a sampling frame and did not conduct anything resembling a representative sample, random or otherwise, of the voting population, nor did they make much of an effort to keep track of what was sent out and what was returned. It's convenient to use the spectacular failure of their straw poll as a cautionary tale, but without any real documentation of what they did, or data to analyze, any attribution of specific causality is little more than speculation.

**Dominic Lusinchi:**


>Thanks for your response. Here are my comments.

>Your statement to the effect that "It's one thing to debunk a myth, it's something quite different to replace one myth with another" is an inaccurate characterization of what I am saying. I am not replacing one myth by another as you seem to suggest - if I am: what myth is it that I am promoting?

>Myth 1

>The fact that the protagonists (Funk and Gallup) believe that what Gallup said in his July 12 column is a prediction does not make it a prediction. I repeat that what Gallup was **really** predicting was a close race: so says the title of his column (which I hope you received). The Digest editors came to believe, and were encouraged to do so by all the praise they received regarding its "uncanny accuracy" (10/31/1936, 6), that their poll was a "forecasting machine" (8/22/1936, 3), when in reality they had no idea how their poll results were produced - in other words, the belief does not make it so.

>In any case, this (the so-called "prediction") was used as a great promotional story after the Digest fiasco to show and tell: show that the new "scientific" polling was the wave of the future and tell that the Digest "straw" polling was a thing of the past. For example, Crossley wrote: "The Institute [Gallup] even went so far as to forecast the DIGEST vote, which it did with remarkable accuracy by the simple means of tabulating separately that part of its ballots which followed the DIGEST's general basis." ("Straw Polls in 1936", Archibald M. Crossley, The Public Opinion Quarterly, Vol. 1, No. 1. (Jan., 1937), p.29) The moral of the story: the "scientific" polls are superior because not only can they predict correctly the election but they can also predict what the Digest would forecast.

>I agree with you when you say it was **"presented"** as a "throw-away comment"; but it really was a very cleverly worded statement. I also agree when you say that it was meant to "generate publicity for Gallup's fledgling organization." Think about it: if I (Gallup) am wrong, I can always say (plausible deniability) that's how things (the presidential race) were at that time (July); if I'm right, I can present it as a prediction (of the results the Digest will publish right before the election) and thereby trumpet the superiority of my "scientific" polling methods. The statement has to be analyzed within an overall strategy to dominate the opinion
polling domain.

>Gallup and his fellow "scientific" pollsters were not disinterested parties searching for the cause of the failure of the DIGEST poll out of some academic interest. They wanted to be top dogs and knock the Digest off its pedestal. They used the Digest incident, especially in the early years, to promote what they considered to be a superior "product": their "scientific" polling. Of course, they were to receive a nasty shock in 1948 - although not fatal to them, as 1936 was for the Digest.

>Myth 2

>I had the opportunity, thanks to the Roper Center, to have access to the raw data from the May 1937 AIPO (Gallup) poll. First the data in its raw form requires some editing because the file contains some anomalies. For example, you might have noticed that in Squire's table 2 (p.130) there are 780 respondents that claim to have received the Digest ballot, while in table 3 (p.131), which reports what they did with the ballot (returned or not or don't know), there are 829 respondents! Squire analyzed the data at face value.

>Second, Squire did not weight the data despite the fact that we have information that allows us to do so: each candidate's share of the actual vote, the response rate to the Digest (~24%), and the percent each candidate received from respondents to the poll. This takes care of anomalies, pointed out by Squire, such as over-report of support for FDR, over-sampling of Digest poll respondents, and over-report of support for FDR among respondents.

>The weighted results confirm much of Squire's conclusions: had the Digest relied solely on telephone and car owners it would have forecast a Roosevelt victory; Digest non-respondents were strongly in favor of FDR; and, last but not least, had all those who were polled by the Digest responded, the magazine would have pointed to the correct candidate. In other words, yes the Digest original sample of 10M was biased but not sufficiently to have prevented it from calling a Roosevelt victory. In fact, a less "rough
calculation" (Squire, p.131) shows that non-response bias was the main culprit.

>Ironically, the samples Gallup used throughout the 30s and 40s were also biased - in favor of Republicans. And he was called on that (e.g. Special House Committee Investigating Campaign Expenditures, 1944).

>The conclusions based on the May 1937 poll (e.g. non-response as the main cause of the 1936 Digest poll failure) are consistent with other available data. I have re-analyzed the data provided by Cahalan in his Cedar Rapids study in 1936-7 reported in his 1989 POQ (vol. 53 pp.129-133) and in 1939 Psychological Record (vol.1, no.1, pp.3-11) papers. The results show that the Digest list used for that city was not biased and that respondents and non-respondents were very different in their candidate preference. Add to that the results from Allentown, Scranton and Chicago where only registered voters were polled (Digest, 14 November, 1936, p.7) and I would say we have some pretty solid "real documentation".

>Does this amount to a definitive answer? No, absolutely not. The only way we could have resolved the question would have been to conduct, on the original sample of 10M, the same type of study that Cahalan performed in Cedar Rapids: select a random sample from the Digest list and ask respondents whether or not they sent in their Digest ballot and which candidate they favored at that time.

>Short of that the May 1937 AIPO survey, the Cahalan study and the results from Allentown, Scranton and Chicago is the best we have. Speculation you say? Well, yes, I suppose you're right, but a lot less than simply repeating the old refrain repeated by so many: "The failure of the Literary Digest's polling approach can be explained simply. The Digest's sample of voters was drawn from lists of automobile and telephones owners." (Gallup, 1972, "Opinion Polling in a Democracy", in Statistics: A Guide to the Unknown, Judith M. Tanur et al. (eds.), San Francisco, Holden-Day, p.147.)

**Jan Werner**:

>Some really good stuff here, but, as you suggested in an earlier (off-list) message, we are probably going to have to agree to disagree
-- My reaction is that, if anything, your additional material supports my previously expressed opinions.

>So, let me clarify what I mean by replacing one set of myths with another.

>Myth 1:

>Original myth: Gallup predicted the Literary Digest straw poll results.

>Revised myth: Gallup did not predict anything.

>My take: Gallup presented what he knew full well would be interpreted as a prediction, although he worded it so as to provide him with an escape if he were wrong. Given what we know now (see Myth 2) he was very lucky to come as close as he did. Whether or not that constitutes a prediction is a semantic judgment.

>Myth 2:

>Original myth: The Literary Digest poll failed because it did not use a representative sample.

>Revised myth: It failed because of nonresponse bias (as shown by the May 1937 Gallup data, and other sources).

>My take: The Literary Digest polling procedures were a complete mess. Evidence from other sources does show that an unrepresentative sample was not the sole cause, but is not strong enough to prove anything beyond that. In particular, I don't see the May 1937 Gallup data as being reliable enough to justify giving either bad sampling or nonresponse bias pride of place, so to speak, in assigning blame.

>None of this addresses what I consider to be the worst myth about the Literary Digest straw poll failure, namely that it was the reason
pollsters began using probability samples.

**Dominic Lusinchi:**

>Well, Jan, if you consider these two statements semantically equivalent:

>>"If the Literary Digest were conducting its poll at the present time, following its usual procedure, Landon would be shown in the lead." - "When all the results are in the Digest poll will show Landon in the lead."

>then I can't much argue about that - we've reached a dead-end.

>You say that you don't consider "the May 1937 Gallup data as being reliable enough" to say anything about what caused the 1936 Digest poll to fail so miserably. I agree (along with Squire) that the Gallup survey is less than perfect. But I disagree that it cannot be used for the problem at hand. Gallup polls of the 30s and 40s have been put to good use by researchers: e.g., Baum & Kernell "Economic Class and Popular Support for Franklin Roosevelt in War and Peace," Public Opinion Quarterly 65, 2001, 198-229; and Berinsky's 2006 discussion "American Public Opinion in the 1930s and 1940s: The Analysis of Quota-Controlled Sample Survey Data," Public Opinion
Quarterly 70: 499-529. The polls have their limitations and the data have to be analyzed with that in mind.

>The "conventional explanation", as it has been called, that the 1936 Digest poll failed because it was biased in favor of Landon due to the fact that it relied mainly on lists of phone owners and car owners, is pure speculation, plausible speculation perhaps, but speculation nonetheless. In contrast, the only real evidence we have, although it is limited and has flaws (the results from Allentown, Scranton, Chicago, Cedar Rapids and the Gallup data) all points to the same problem: non-response bias. My preference in describing what happened to the 1936 Digest poll leans towards the latter (warts and all). It is, of course, a tentative explanation but it has the advantage of being backed by data, however imperfect.

>As for your last comment about the belief that the Literary Digest poll failure being the reason why pollsters began using probability samples is, of course, nonsense, so I agree with you on that - but since the issue was not part of the original post, I'll leave it at that.

**Jan Werner:**

>The late John Gorman, who founded Opinion Dynamics, liked to mention a specific date in the late 1970's, which he estimated was the day on which George McGovern edged out Richard Nixon, based on trending how respondents said they voted in the 1972 presidential election in polls conducted during the years following the Watergate affair. John's estimate was also based on data, however imperfect.

>As for what is, or isn't, a prediction, I'll let it go at that. Others can make up their own minds as to what Gallup intended when he wrote what he did in 1936, and what it should be called.

>Regardless of whether or not we agree, I'd like to thank you for an enlightening discussion and much valuable information. Let me note that the image you sent me of the 7-12-1936 Washington Post article does not include the beginning of the article on page 1, and there is no way to tell whether the headline "1936 Election Seen As Closest in Years" was written by Gallup or by the Washington Post's editor.

**Dominic Lusinchi:**

>Thank you, Jan, for your insights; always glad to discuss what I see as the defining moment in the genesis of modern polling.

>Aside from our difference regarding the semantics of what constitutes a prediction, the **perception** is just as important: i.e. that the Digest editors saw it as such reveals a lot about them - specifically how convinced they were that their poll was an "uncanny" "forecasting machine".

[*Typos corrected and updated to include the last exchange of comments that I omitted from the original post*].

Subscribe to the Politics email.
How will Trump’s administration impact you?