Views from the Hills by R. E. Stevens, GENESIS II (The Second Beginning) E-Mail views@aol.com

Chapter 2 -- There is No Such Thing as a Free Lunch

On September 21, 1996, I wrote the first, "There is no such thing as a free lunch," paper. It involved the increased speed in which studies are put into the field and reports are issued. In the past two months, I have encountered two studies where the choice of a test protocol also fits the old saying. In each of these two cases, the researchers were conducting what we have called a Protomonadic or Sequential Single Product design. In both cases, the summarization of results presented serious difficulties. Both studies resulted in strong order effects. As a result, only the first product data could be used in the analysis of the study.

It has been my experience that when the Protomondadic design is properly utilized, you seldom encounter a significant order effect. I was always taught that a Protomonadic study involved an interview at the first recall that obtained Overall Assessments and Likes and Dislikes, and no attribute evaluations. In cases when an order effect is present, it is usually evidenced in the degree of difference between the main effects (i.e. no reversal between orders).

More serious problems occur when the researcher is attempting to maximize data return by asking direct attribute questions after the use of the first product and before the use of the second product. In this protocol, the respondents use the first product and assess it on the basis of their criteria of excellence. However, asking direct questions in the first interview, sensitizes the respondent to attributes that are important to the test designer, and may not have even been considered by the respondent during the use of the first product. Thus, the respondent uses one set of criteria for the first product, and another for the second product, producing a test design order bias.

I have heard the argument that the order effect is just a position effect or average order effect, and the true answer is the average of both orders. However, we cannot conclude that a product by order interaction does not exist. Averaging of results from both positions can be totally misleading.

If it is the researchers' intention to compare the two products in the context of specific attributes much like a Concept & Use format, the respondents should be given a list of attributes at the start of the test before the use of the first product. In this way, both products are evaluated, utilizing the same criteria. However, we should be aware that we no longer have a blind test, we are utilizing our criteria of excellence and not that of the respondent. The resulting test is now a directed interest study. 


[Back][Index][Forward]