A Representative Sample

A couple of weeks ago, I wrote about an experience with a law firm dealing with a market research project.  In the Views, I wrote about how the research company spent a lot of time and money doing questionable research.  Looking at both how the study was conducted and the results obtained, you have to question the validity of the entire study.  What amazed me was how the study was defended.  Because of ethical concerns, I cannot go into the specifics of the study.  I can, however, raise some issues that I believe are universal in some of our research.  I will be vague on some points so as not to identify those involved.

The researcher and the sponsoring company stated that they had no idea about the profile of those they wished to interview except for the case of age which was people within the approximate ages of 15 to 35 years old.  The researcher stated that this was the reason they felt it necessary to start with a nationally representative sample.  To achieve the nationally representative sample, they selected approximately 15 malls which they stated made up a representative sample.  There is no way I believe you can achieve a Valid representative sample from mall shoppers.  I do not believe there is any way of screening mall shoppers to obtain a valid sample.  Consider that about half the population does not shop malls.  How are the non-mall shoppers different from the mall shoppers?  Within the mall we will have about 90% of the shoppers refuse to be interviewed (wave-offs).  How are the refusers different from those who volunteer to be interviewed?  Even after being screened, about half the eligible respondents will refuse to leave the mall area for additional interviewing.  No, I do not believe that you can obtain a nationally representative sample using mall shoppers unless your target population is people who will participate in mall research.

I believe the researchers in the project, and others, deceive the client by stating that mall shoppers are representative of the national population.  Even if they were, it is NOT the demographic profile of the mall shoppers that is paramount to a valid sample, it is the profile of those that you interview.  While the researcher did, to her credit, have a very explicit set of instructions for the mall interviewers such as "you will intercept every nth shopper," I doubt that the instructions were even followed.  I base my suspicions on my own observational experiences through the Researching Research Program.  The researcher stated, however, that they insured that the interviewers followed instructions by having on-site supervision.  The supervisors were not from the researcher's company but they were supervisors from the mall facilities themselves.  In an effort to lend credibility to these supervisors, the researcher stated that all the mall supervisors in this study were trained psychologists.  what is the likelihood that ny mall would employ a Ph.D. in psychology or that a psychologist would even accept such a position?

One of my recommendations to the law firm was that we sponsor some Confirmatory Research.  We have the names of the mall facilities, the supervisors and the test participants along with their phone numbers.  We could confirm the training of the supervisors, obtain a demographic profile of the population in the vicinity of the mall (i.e.  5 or 10 mile radius) and we could visit some of the homes of the respondents to determine if they had in their homes what they said in the survey they had purchased.

When we design a study for our clients, how often do we discuss the strengths and weaknesses of the research proposal and the available options as well as the risks/rewards vs. the investment/return?  Always, I hope.