November 19, 2008

Polling Accuracy

One of the amusing sidebars in the recent U.S. election was the story on conflicting polls. Some news organizations published polls of polls, averages of several polls, while others pushed their favorites.

The classic case on inaccurate polling was the 1948 U.S. presidential election between Thomas Dewey and Harry Truman. The Gallop organization was so confident of a win by Dewey that the Chicago Daily Tribune ran with the headline of a Dewey win, only to have to retract and celebrate Truman. Ever since students of statistics are offered that as an example of what happens when a statistical sample is not truly random. In 1948 Gallop used telephones to poll, but there were still a lot of families without phones.

A brouhaha between polling firms is breaking out in British Columbia. In recent years the field was dominated by Ipsos-Reid and Mustel Research Group Ltd., but Angus Reid is back in business as Angus Reid Strategies. In June Mustel showed a 10 point lead for the BC Liberals. In November Ipsos reported a 9 point lead for the Liberals, but Angus Reid showed the NDP 5 points ahead of the BC Liberals. The Ipsos poll randomly selected a sample of 801 adult British Columbians by telephone between November 5th and November 12th; the Angus Reid poll randomly selected a sample of 802 adult British Columbians online between November 6th and November 11th.

Does the use of telephone or online technology fully account for the difference between a five point lead for the NDP and a 9 point lead for the BC Liberals? There are other variables including how questions were worded, how they were rotated, the skill of the interviewer, or in the case of online polling, what effect not having an interviewer might make. Many people have cell phones rather than land lines, many more use voice mail and don't answer if they don't know the caller. Likewise, everyone is not computer literate; even amongst those who say they have access to the Internet there are many who in practice don't use the technology. Pollsters have a difficult time reaching those who are excluded by the type of communications technology they use. It is inaccurate to assume that those who are excluded by technology are identical in their preferences to those who end up answering the pollster's questions. Low income voters and seniors are less likely to use the Internet than other voters. Young voters are more likely to use only cell phones. Pollsters may weight their samples to reflect what they know about those who are not sampled in proportion to their representation in the population, but that introduces assumptions in place of true random sampling.

Google "poll accuracy" and you'll find many firms that make claims about how close their polling results come to election outcomes. Ipsos included such comparisons with its latest BC political poll. Despite the claims, users of polls have to be skeptical about their reliability. It might make the public less interested in polls and more interested in issues and party positions. It also might make news organizations spend more time on policy and less on the horserace, but don' t count on that.

Other sources will be sought for confirmation or refutation of polling results. The NDP has the benefit of two by-election victories and municipal sweeps in Burnaby, Vancouver and Victoria. It is hard for Gordon Campbell to find that kind of evidence to indicate whether he is on the right track. NDP candidates and campaign workers who might have been deflated by results out of Mustel and Ipsos, can turn to solid victories and work with renewed enthusiasm as they approach the only poll that counts, voting day on May 12th.