If you read polls in the news, you’re probably familiar with the term “margin of error.” What you may not know is that pollsters disagree fiercely about when it should be used.
In an actual debate last week, sponsored by the do-it-yourself sampling firm Peanut Labs, polling experts got together to argue whether a margin of error should ever be reported for surveys conducted online — which is how more and more surveys are conducted.
Most industry standards and guidelines say that surveys drawn from nonrandom samples — typically the case with online polling — should not provide a margin of error when their results are generalized to the wider population. Yet, as debate moderator Annie Petit noted, many readers expect to see the margin of error, regardless of how the poll was done. So she threw out a provocative question: “Is it really so terrible to use a statistic that everyone understands so well?”
At HuffPost Pollster, which regularly conducts online surveys with YouGov, we don’t have a perfect answer to that. But we’ve decided it’s time for even greater transparency with our readers.
–
So we’ve come up with this solution: We’ll add the following text to the methodological details we note when we report on HuffPost/YouGov surveys and link to the additional information prepared by YouGov:
“Most surveys report a margin of error that represents some, but not all, potential survey errors. YouGov’s reports include a model-based margin of error, which rests on a specific set of statistical assumptions about the selected sample, rather than the standard methodology for random probability sampling. If these assumptions are wrong, the model-based margin of error may also be inaccurate. Click here for a more detailed explanation of the model-based margin of error.”
Last week’s debate on the margin of error didn’t produce any real fireworks, partly because most of the participants conduct online panel surveys themselves. They were unified in hoping for better direction from industry standards.
