Screening Respondents for Market Research Surveys, Best Practices 1

Updated: November 22, 2010

Market research survey questions should never be leading, and this rule applies to the screener portion of the survey. The respondent should not be able to guess easily at what answer will allow them to proceed. A common mistake is present screener questions that are simply yes-no, or questions with only two answer choices. This can make it obvious as to which answer will allow you to continue and which will terminate the survey. Even by guessing, someone would have a 50/50 chance on passing with a two answer choice question.

A better approach is to build a short list where each answer option is a check-box style answer choice, or in other words a multi-select construct. The respondents check which items on the list apply to themselves. Further, to eliminate those who might check all on the list just to get into the survey, you can construct the list such that is impossible, or nearly so, to that all items apply. Finally, the order of the items on the list should be randomized the same order is not presented to every respondent.

Example 1

Consider this example, where we are screening to find out if the respondent owns a dog.

Poor question design:

Do you own a dog?

  • Yes
  • No [TERMINATE]

Best practice question design:

Do you own any of these types of pets?

Check all that apply.

[RANDOMIZE LIST]

  • Dog [IF DOG NOT SELECTED, THEN TERMINATE]
  • Cat
  • Bird
  • Reptile
  • Rabbit
  • Rodent
  • Horse
  • Fish
  • Pig
  • Other [ANCHOR, not rotated always in 10th position]
  • I do not have any pets. [ANCHOR. CANNOT BE CHECKED IF OTHERS ARE CHECKED]

In the above dog owner-screener example, the survey programmer notes are in square brackets [programming notes]. The best practice design makes it harder for the person who is guessing to select "Dog" and proceed with the survey. In a randomized list, it would not be obvious that we are looking for dog owners. Further, we can terminate anyone who selects all items 1 to 9. Although it is remotely possible that someone owns all nine pet types, it is very improbable. We accept the very small chance of mistakenly screening out a truthful many-pet-owner respondent. Better to eliminate a more likely scenario that someone simply checked all pet types just to into the survey. Contrast this with the poorly designed yes-no question, where we have no indication of foul play and are giving a would-be guesser a coin-flip chance of passing.

Example 2

Consider another example, where we are screening to find out if the respondent has shopped at Rite Aid in the past four weeks.

Poor question design:

Have you shopped at Rite Aid in the past four weeks?

  • š Yes
  • š No [TERMINATE]

Best practice question design:

Which of the following stores have you shopped at in the past 4 weeks, if any?

Check all that apply

[RANDOMIZE LIST]

  • Rite Aid [1. IF NOT SELECTED, THEN TERMINATE]
  • Wal-Mart [2]
  • Target [3]
  • Sears [4]
  • CVS [5]
  • Safeway [6]
  • Crest Foods [7. survey design note: Oklahoma City, OK, area]
  • Mayfair Markets [8. survey design note: supermarket only in Hollywood, California]
  • Roth's [9. survey design note: supermarket only in Oregon]
  • None of the above [10. ANCHOR, EXCLUSIVE]

[IF ANY TWO OF 7, 8, AND 9 ARE SELECTED THEN TERMINATE.]

On this best practice questions design, we are again showing a list which is randomized per respondent. Not only are we terminating if Rite Aid is not selected, but also we are eliminating anyone who selects any two the regional stores, 7 Crest Foods, 8 Mayfair Markets, and 9 Roth's. Because of the dispersed geographic locations of those stores, it is unreasonable that someone would have shopped in all three in the past four weeks.

Featured Research
  • Why Q4 is the Perfect Time to Invest in Video Conferencing

    If you’re currently relying on an outdated video conferencing solution - or if you don’t have one at all - you’re in luck. While any time is a good time to invest in video conferencing, Q4 may be the best. One big reason is that Q4 is when you can get the best deals, but there are also other important factors like increased access to seasonal employees. more

  • 8 Ways to Get More From Your VoIP System

    Many businesses adopt VoIP to take advantage of the cost savings without spending enough time reviewing the features and benefits made available by different solutions. If this is true for your business, there’s a good chance you could be getting more from your VoIP system in the form of even lower costs or improved employee productivity. You may even find that your current software offers features that you aren’t taking advantage of! more

  • Best ERP Features and Benefits for Your Business

    Are you considering investing in ERP software for the first time? Or maybe you already have an ERP solution but you’re worried it’s becoming dated. If either of the above apply to you, read our latest guide on the top ERP features and benefits based on the size of your business. You may be surprised at how versatile and cost-effective it is becoming - regardless of if you own a small business or run a large enterprise. more

  • 9 Spooky Signs You Need a Contact Center Upgrade

    When was the last time you evaluated the performance of your current contact center and the software you are using? The results may be frightening! If it’s been awhile since you invested in contact center software, there is a good chance that your needs have changed or that there are better options available now. Fortunately, it’s relatively easy to determine if you need an upgrade or not. more

  • 7 Ways the Wrong Phone System Can Haunt Your Business

    The wrong phone system could be haunting your business - and we’re talking about problems more serious than ghosts and ghouls. From increased costs to issues with scaling, we’ve identified seven important ways that a less than ideal phone system could be holding you back. You’ll be surprised at how much of a difference this can make to your bottom line too. more