Views from the Hills by R. E. Stevens, GENESIS II (The Second Beginning) E-Mail views@aol.com

Remembering the Basics: Reliability and Validity in Marketing Research

[Written by Mr. Russell Proops, PAR (Product Acceptance and Research) Evansville, Indiana]

As product categories mature and the marketplace becomes oversaturated with offerings of every imaginable variety, marketing researchers are often called upon to try new and different approaches in hopes of discovering unique insights into customers' hearts and minds. In our zeal to gain a competitive advantage through consumer understanding, it is easy to forget the basic requirements of any system of measurement reliability and validity. Consider the following story.

A major manufacturer was desperately trying to reposition a product developed for the healthcare market for mainstream consumers. After several disappointing tests of alternative selling propositions, the VP of marketing suggested switching from the established supplier to a European testing service. The marketing researcher on the brand team did not immediately reject the idea of using the new supplier when he learned that their methodology involved measuring the respondents' pulse, respiration and skin conductivity while they viewed television commercials in a group setting. These were, after all, the same measures used in lie detectors.

But how VALID a measure of consumer interest could such a test be? Given the generalized responses of the autonomic nervous system to stimuli causing fear, anger, surprise and other states of arousal, what would the test actually be measuring without eye-tracking cameras, how would anyone know what the subjects were responding to?

What PREDICTIVE VALIDITY of product trial could the measure have when the test did not involve any type of simulated or real purchases? Sample reports included a purchase intent question, but neither the client nor the supplier had norms against which to compare these scores. If Marketing was unwilling to pay for two tests of the selling propositions to allow a comparison of result of the new methodology versus the old, how would he ever determine if the new test had any CONCURRENT VALIDITY?

Given all his questions about what the test was actually measuring, he didn't care if the results were RELIABLE, though he doubted whether the same commercial had ever been tested among different samples, let alone re-tested among the same respondents.

Luckily, the brand manager who reported to the VP had, upon reflection, developed similar reservations about the new methodology. After much debate, the VP of marketing agreed to continue to use the current supplier to evaluate the new selling propositions so that consistency of measure could be maintained. Besides, he didn't need to make enemies in the Marketing Research Department.

The next day the VP reluctantly signed the proposal, but made a mental note to call the new supplier himself if the next round of testing didn't turn out right.


[Back][Index][Forward]