Views from the Hills by R. E. Stevens, GENESIS II (The Second Beginning) E-Mail views@aol.com

From Reliability and Validity to Usable and Beneficial

The holidays seem to be a time to catch up on my reading and reflections.  While reading an issue of Marketing Research,
I came upon an article where the pros and cons of Internet interviewing were being debated.  I was particularly disturbed by the way the Internet interviewing was being justified.  It was definitely not on the basis of the quality of data.  It was only on the ease and cost of the research.  Actually, when addressing the issue of data quality, I felt the presenter was intentionally trying to deceive the reader.  The justification of the protocol was that the data are BENEFICIAL and USABLE.  There was no mention of RELIABILITY and VALIDITY.

I agree that the use of the CATI interviewing through the Internet will result in a number of potential advantages such as speed, ease of use of visual stimuli, eliminating interviewer bias, eliminating skip pattern errors, cost of interviewers, and real-time data cleaning.  If these are the criteria of importance, just walk down the hallway of your office building and interview 10 people.

My concern is that we are using the data to extrapolate to the universe.  The data should accurately predict what will happen in the future if we make the changes we are evaluating.  How did they address projectability?  To quote them,

"In most cases, Web surveys when properly employed are or can be made projectable to the needed universe of inquiry.  Are they good enough to be usable in predicting behavior of the general population?  And are they good enough to be usable in predicting behavior of more limited universes?  (We refer to 'usable' as the criteria because any fair-minded person will admit that no interviewing environment is conducive to perfectly random sampling of the general population.)"


The proponents go on to cite the current pinnacle of valid survey sampling as being CATI.  While at the same time mentioning that completion rates are falling toward 30%.  The other comparison was with Mall Intercept interviewing.  to call these methods the pinnacle of valid sampling is like saying we should replace Exit Polling with Mall Interviewing or Telephone Interviews for predicting election results.  I don't think so.

They quote A. C. Nielson BASES as saying the "Surveys on the Web panel provide the same forecast accuracy as the mail surveys."  That is a very strong general statement.  I could expect comparable data if I was surveying Internet usage, appeal of computer services or equipment, or software acceptability.  But not the very general statement.  Do we really think the results of a "Demographic" or "Habits and Practices" study among Internet users and a Random sample of the population would be comparable?  Note the wiggle words in the following statement:  "Sampling on the Web can be projectionable as in any other methodology when all or most members of the population are accessible."

My position on the use of Web interviewing is simple.  It is very appropriate when the population we intend to research is computer/Web/Internet users.  Web interviewing can be used with caution in other areas when we and our clients know the risks.  I would use Web interviewing for learning and understanding research but at the present never for decision-making research unless the population of interest was as stated in the first sentence of this paragraph.  To quote Mr. Charles Hamlin, President of InsightExpress, "InsightExpress is an appropriate tool for getting a good sense of the marketplace.  If you are making decisions of magnitude, we recommend that you use the traditional market research tools."  Using Web research and projecting to the general population is like asking NRA members about gun safety and projecting to the population as a whole.

We frequently report that 40% of the population have access to the Internet.  This is a deceiving number.  It is not access that counts, it is the ability to use and actually using it which is 25% of the population (67.8 million).  Focusing on "access" is like studying spending power and saying a bank teller has access to millions.  If we think we can take results from a selective sample representing 25% of the population and then extrapolate to the total population by proportionalizing the results based on some national parameter, we do not understand sampling.

We seem to keep comparing our new protocols to the current state of the art.  To me this is like the slow degradation of a product over the years where year after year we compare cost saving initiatives with the previous year's product.  We need to have and maintain a standard of excellence for comparison purposes.  This is much like the Anheuser-Busch Company procedure where they have frozen product that they use annually to compare with the current product.  The frozen product is a 20-year standard.

We need to bring back the concept of RELIABILITY and VALIDITY. For the purists, I do realize that Web interviewing may be very reliable, much like a double headed coin.  It is the validity that I am questioning.
 

A Web Site You May Find Interesting

A very good friend, Mike Courtney, has been trying to introduce me to the computer world for about five years.  Last month he recommended that I look at an Irish company.  He knows both my wife and I love Ireland.  I guess he thought this was one way to stimulate some interest.  It did.  I think that considering the views stated above, this is a good time to say here is a web site I found very interesting.  The company is Nua.  Fortune magazine calls it "one of Europe's leading strategic internet companies."  if you are interested, check www.nua.ie

Some thoughts to consider:



[Back][Index][Forward]