THE BLOG

Sampling Error - What It Isn't

01/25/2007 01:43 pm 13:43:15 | Updated May 25, 2011

Back to basics for a moment, to a point we cannot make often enough. Yes, polls report a "margin of error," which derives
from the random variation that comes from drawing a sample rather than interviewing
the full population. But that is all it
is. The "margin of error" statistic is
not an overall measure of quality and can tell you nothing about the potential
for other sources of error, including as a bias from missing certain voters (such
as those who lack a landline telephone or who hang up when called) or from asking
questions in a way that produces misleading results.

A friend sent this
take on sampling error from Good Math, Bad Math, a blog by a computer
scientist:

People frequently believe that the margin of error is a measure of the quality of a statistic - that is, that a well-designed poll will have a smaller margin of error than a poorly-designed poll. It doesn't - the MoE only represents sampling errors! A great poll with a sample size of 100 will virtually always have a considerably larger MoE than a terrible poll with a sample size of 1000. If you want to know the quality of a poll, you need to know more information it than just the margin of error; If you want to gauge the relative quality of two different polls, you need to know more than just the margin of error. In either case, you really need to know the sample size, how the sample was collected, and most importantly exactly what they measure.

It's worth a click.

More:

Pollster