Carl Bialik, the "Numbers Guy" at the Wall Street Journal, devotes his column
this week (free to non-subscribers) to a change in the Nielsen television
ratings to include college students living in dorms. Not everyone realizes that
the Nielsen ratings are based on a random sample survey - a randomly selected panel
of households that agrees to let Nielsen monitor its viewing habits. According
to Bialik, while the inclusion of roughly 2.1 million college dorm residents
into the Nielsen sample has increased the audience size estimate for some shows
among 18-24 year olds by as much as 60%, it is based on a very small sample:
[T]o calculate these estimates, Nielsen
is extrapolating from the viewing habits of just 130 students around the
country who have agreed to have electronic monitors installed in their dorm
rooms. That means the decisions of a handful of those students can lead to a
huge swing in ratings -- that 163,000 jump for "Drawn Together" was
based on 12 people in Nielsen's survey group who tuned in to the show.
Bialik's piece explains the mechanics of the Nielsen ratings
and goes into depth on the question of whether that sample of 130 is large
enough. A Nielsen spokesman says the sample is "adequate, but some academic
statisticians Bialik talked to had doubts.
Statisticians I've spoken with told me
national surveys typically include at least 500 people, to limit the margin of
error. (An ongoing Princeton
University survey of college students,
examining minority students' academic progress, tracked about 4,000 students.)
Political pollsters often survey just 1,000 or so people to determine broad
national trends, but sampling error is less of a problem when respondents are
choosing among two presidential candidates. It's a different matter when asking
someone which of 100 or more channels they are watching. Mr. Holmes said
Nielsen isn't calculating a margin of error for the college group.News reports and promotional press
releases usually don't make it clear how small samples can influence ratings.
They sometimes crown networks, shows and personalities as winners based on
differences of just 0.1 rating point (representing about 110,000 households).
"The ratings just aren't accurate to that point," said Lynne S.
Gross, professor of radio, TV and film at California
State University,
Fullerton.
Nielsen's Mr. Holmes said the company supplies information to clients that
would allow them to calculate an overall margin of error that they could
publicize.
It's a fascinating column. Go read the whole
thing.