This is the sort of information-based discussion that I would like to see on all blogs.

Economical ways to compensate for these and other skewing factors have been on my mind for a while. It would not surprise me if some of the larger research firms already have developed math models to compensate for the screening and other non-participation factors.

Since CNU is near the DoD Modeling and Simulation Center, I wonder if someone over there could help develop some compensation models that could adjust for the non-participant factors, for future surveys?

The primary attraction of telephone surveying is that it enables data to be collected cheaply and quickly (having said that, a state-wide survey is still very expensive). The primary potential problem, however, is with obtaining an adequately representative sample of the general population one purports to represent. This has to do with sampling, and how one decides to try to obtain a representative probability sample. There are several problems with obtaining a representative probability sample, but the two biggest are that a certain portion of the population does not have a telephone in their home (and this portion is getting larger as people go to cell phones and drop their land lines) and that a portion of the population has their numbers “blocked” from access.

How do we do our survey? We purchase for each survey a telephone lists from a national vendor who specializes in up to date list of telephone numbers drawn from various sources such as nationally published sets of telephone listings, marketing data bases, etc. These lists are entered into a Computer-Aided Telephone Interview (CATI) system. We use a random sample management module, which means that each number in the CATI system has the same chance of being drawn for the sample as any other number in the system. Regional quotas are employed to ensure adequate coverage of the various regions of Virginia. Finally, reported frequencies and crosstabs are weighted using a demographic profile (age, race, sex) to provide a sample that best represents the targeted population from which the sample is drawn. The proportions comprising the demographic profile are drawn from census estimates of the population.

Sampling error, or margin of error, simply refers to the percentage that our survey results likely differ from the actual population due to the size of the sample drawn. We seek a margin of error less than 5%, which is why we draw a sample of 700. Our margin of error would be zero if we were able to interview every person in the population (all people in Virginia). As we note, our margin of error is a statistical estimation based upon a mathematical theorem, and there are other possible sources of error in survey research such as a bad sample design.

This does not explain how the surveyors (pollsters) compensated for the fact that those at the higher income levels and highest education levels, screen-out calls from “phone polls” and have been doing so in such increasing numbers, that the validity of phone only polling must be questioned.

It is no longer adequate for a polling company to pledge that the randomized sample provides 95% or better accuracy, since the population who are answering these random calls is skewed to the lower income and lower educated segments of the population.

I have the utmost respect for CNU and especially for Dr. Kidd. All I wish to point out here, is that the Press and others who are presented with polling results, should be aware that the results may be skewed, based on a lack of compensation for the socio-economic clustering brought about by the use of phone-only polling.

For example, if the poll was conducted by phone, how does the poll designer compensate for the fact that many citizens screen out unfamiliar callers. The last poll I ran that gathered responses strictly by phone was terribly skewed by the fact that many of those in the upper income brackets would never answer the phone, or would refuse to participate, consequently our sample was heavily populated by the poor and unemployed, who at the time, were yet to join the trend of using caller ID and blocking features. Merely selecting the phone numbers at random does not correct for the lack of upper income, well educated citizens, in the sample.

I do not say that the CNU and other polls are invalid. I only wish to point out that before the results are given credibility, journalists must be able to explain how the data was gathered and how the poll designer compensated for factors that skew the results.