New Tree Header

[Views home]
[Views home]
Register for
Email Views

Limitations in Paired Comparison Testing

January 9, 2006 - by Robert E. Stevens, GENESIS II(The Second Beginning) E-Mail:

Over the years, I have developed serious concerns about the use of the Paired Comparison technique. It has been my experience that the biases in the psychological and physical areas occur all too often. As a result of being victim to these biases, I find that I only use the technique for studies where our objective is to determine if one product is a good replacement for another, especially among the users of the current product.

For example, when we ask "Which product did you like better?" the assumption is that the respondent likes both to some degree. But it is possible that the respondent prefers her regular brand to either of the two test products. In such cases, the respondent's vote is for either the product liked more or disliked less.

Frequently, we will find a product that wins by default. Consider the testing of two Italian salad dressings. One dressing is of average spice flavor while the other is heavy on spices. It is assumed that respondents that do not like either will vote a "no preference," however, it has been my experience that while some will vote "no preference", some not liking either and probably would never purchase either, will opt for the milder, less spicy dressing. This can lead to a misleading preference for the milder flavor.

A spot test of similar flavors would probably result in a significant order effect whereby in the sampling of the milder first followed by the more spicy would show a much bigger difference in flavor than when the stronger, more spicy is sampled first resulting in big differences depending on the order of presentation.

Another area of concern is when one product has technical superiority over the other, the pair test results can be misleading. Consider the testing of Tide vs. Tide with Bleach. Respondents using Tide first followed by Tide with Bleach will see much bigger differences than those respondents using Tide with Bleach first followed by Tide. In the latter case, the Tide with Bleach cleans up the old stains and improves the whiteness so that little difference is seen with the use of the Tide without bleach.

Pair Testing is extremely vulnerable to order effects. Whenever the pair test design is utilized, check for order effects. If present, it says that you have two answers, one for each order of presentation. The results also say that the pair test design is inappropriate for the testing of the two products.

The above are just a few of the many limitations of the pair test design.

These pages are proudly sponsored by:
[Sorensen Associates Inc]
"Dedicated to the relentless pursuit of WHY?"

~ ~ Oregon: 800.542.0123 ~ ~ Minnesota: 888.616.0123 ~ ~ Pennsylvania: 866.993.0123 ~ ~