The Liar Factor
Two weeks ago I gave a talk on Unique Testing Protocols to a group at a
major food products company. Leading into the discussion of protocols,
I introduced some of the reasons for the protocols. The basis of these
protocols was to eliminate or at least reduce problems encountered in research.
One of the four major problems discussed was that of Sampling. Sampling
as it relates to identifying the sample, contacting the sample and finally
obtaining cooperation. Within this structure, sampling is a very big
problem in today's approaches to research. But it appears that my concerns
are only the tip of the iceberg.
Since giving the talk, I have been exposed to the "Liar Factor," a problem
I did not address. Exposed may be putting it mildly. During the
past week I have been trying to catch up on my reading. The following
are just a few of the articles relating to the Sampling problem.
In the January Quirks, Mr. Michael Sack, president of Image
Engineering, writes about the "Liar Factor" encountered in an Internet Usage
study. A total of 500 respondents were recruited from 10 Nationwide
sites. Multiple agencies were used in the recruiting. There were
about five eligibility questions but basically they wanted women between the
ages of 30 and 60 who use the internet at least once a week and searched for
health related information. The respondents were screened twice, over
the phone and at a CLT. Even after dual screening, 66% gave inconsistent
responses to at least one of the eligibility questions. Some could not
even use a computer mouse, while others lacked the skills to do a computer
aided interview.
In the February issue of Quirks, Gary Mullet of Mullet Associates
writes about a food-item Internet study, where a follow-up interview was done
to verify that the respondent was actually the Female Head of Household. Somewhere
around 50% of the purported respondents claimed that they didn't even know
about the study. Seems that another member of the household, generally
a teenaged child, was using the computer with mom's internet account.
These kinds of problems are not bad enough. Now two good friends,
Mike Courney (UGA/MMR graduate) and Ms. Janet Pizzarello (Sorensen Associates)
each sent me an Internet article that says "Turn Your Opinions into Cash."
GetPaidForOpinions is a Marketing Research Service designed to put people
in contact with marketing companies who need their opinion! For just
a one-time fee of $29.95, they will put you in contact with over 100 paid
survey, focus group and market research companies. As Mike says, "amazing,
isn't it?"
Do you know who is doing your screening? Do you know the source of
the database?
There is no perfect sampling plan. There are just degrees of imperfect.
It is our job to be aware of the problems and try to reduce the effects
test biases. After all it is research and research is all about estimating.
As my son would say, "Don't tell me the problem without telling me a solution."
While I continue to use resources like internet and malls, I prefer
to recruit in supermarkets for obvious reasons. There are down sides
to in-store recruiting but when I am concerned with validity of sample, this
is where I end up.
Sponsor: Sorensen Associates Inc
Portland, OR 800.542.4321
Minneapolis, MN 888.616.0123
The In-Store Research Company