Two Firms Concede Errors in Exit
Polls
March
15, 2005 - by
Robert E. Stevens, GENESIS II
(The
Second Beginning) E-Mail: views@aol.com
The
statement, "Two Firms Concede Errors in Exit
Polls" was a front page article in the January 20, 2005 Cincinnati
Enquirer.
My first reaction was "so what, we knew that." Actually anyone
following the election day polling results knew this. My second
reaction was
more analytical, that is, what will participants and especially users
of market
research think about our profession? In this polling that lasted about
a total
of 12 hours, the researchers invested 1,400 interviewer days to come up
with a
prediction that picked the wrong man, and had an error of approximately
13%.
Unfortunately for the researchers and the clients, validation of the
results
was almost instantaneous. They knew the true results within 24 hours.
In the analysis of
what went wrong, the researchers cited a number of reasons such as:
-
The Kerry
voters were more willing to be interviewed.
- The
interviewers were too young.
- The
interviewers were too inexperienced.
- There
was a programming error that led to too many women voters being
interviewed.
The
researchers, however, only
recommended better training and more monitoring of the interviewers.
They
totally ignored their sampling model. There are solutions to these
problems.
The solutions are not easy and they definitely add to the cost of the
research.
From my perspective, additional information needs to be obtained beyond
what
the respondents had to say. There are two populations at play, first,
those
being interviewed and second those voting. The disproportionate
number of women
respondents should have been determined very early in the process so
that
adjustments could be made to the results before publication. The
willingness to
be interviewed in my estimation is the difficult problem but still
do-able
through sample control and data analysis.
I am amazed at how
infrequently we verify the assumptions we make when designing the
research. It
seems to me that if it is important enough to build in a research
parameter, it
should be important that we verify the assumption is actually in-play.
Build in
checks and balances to verify your assumptions.
Early in my career, while auditing a research
project
for another corporate division, I asked the designer why some specific
test
parameters were not being collected and recorded. His response was that
he did
not want to know and he definitely did not want the client to know if
some of
the test effects were significantly affecting the results. I did not
consider
this individual to be a researcher because of his willingness to ignore
accepted standards, in my mind, he became a clerk collecting numbers
and not
data.
I do have sympathy
for
pollsters. Their results, unlike those I had to deal with, are
validated very
quickly. Personally, considering my history, I do not think I would
have liked
being involved in a system whereby validation is almost immediate. I
prefer
spending time getting to know and understand my data before drawing
conclusions.
Sponsor: Sorensen Associates Inc
Portland, OR: 800.542.0123
Minneapolis, MN: 888.616.0123
the
in-store research company™ -- Dedicated to the
relentless pursuit of WHY?