Developing a questionnaire seems easy, right? Decide what you need to know, write a bunch of questions and get people to answer them. What you need to know, is that the responses people give are affected by the way you word each item or question. Many problems arise and much information is rendered useless because of inattention to the way items are worded during questionnaire design.
Item writing is affected by item wording (the term item is used because not all items on a questionnaire are questions). Questionnaire designers must consider the following: Does the item elicit information related to the topic of interest? Is wording clear and unambiguous? Will the item mean the same thing to everyone?
Does the item demand knowledge and information the respondent has (or not)? Does the item contain personal or potentially threatening issues the respondent might resist? Are items loaded with social (un)desirability? Often the approach to items and questions will differ depending on how the survey is administered.
Open-ended items allow respondents to create their own answer; it is a great way to gather stories and anecdotes. Closed-ended items offer responses from which to choose, a list which should be exhaustive and mutually exclusive. Research suggests a two-step sequence to obtain a mutually exclusive list – pose open-ended items and solicit answers that will then be used as a set of closed responses alternatives, for example a checklist.
Researchers advocate using open-ended questions on threatening or sensitive topics and as follow-up “why” questions to reveal respondent insight. Closed-ended items can bias respondents by showing them alternatives they might not have thought of. Consider whether it’s important for the respondents to recall their response or be able to recognize it from a list.
Avoid double-barreled items. Responses to “The United States should abandon its space program and spend money on domestic programs” are misleading because you don’t know which phrase respondents are agreeing or disagreeing with. Better divide the sentence into two.
Item length seems to be arbitrary. Some researchers say that questions should be kept short and to the point, so that responses may be made in a time efficient manner. Others say that longer questions are advisable, to allow the respondents more time to think and form an answer.
Questions need clear context and an explicit frame of reference, unless the purpose of the study needs to elicit different frames of reference! Remember that the survey content is new to the respondent. Questions should be precise.
If few respondents have thought about or care about the topic they may express attitudes without giving thought to the issue and mislead results. Contingency or filter questions can be used when questions are clearly relevant to some respondents and not to others. Whether they are asked is contingent upon their response to the first question in the series. Filter questions can reduce pressure on respondents as they answer socially undesirable questions.
The use of negatives can be confusing and lead to misinterpretation. Often the negative word is missed when reading the item. Biased items and terms should be avoided. Care must be taken not to bias respondents or give leading questions; simply identifying an attitude or behavior with a prestigious person can bias a response.
Respondents wish to provide useful information and wish to be socially desirable, looking good to those asking the questions; as a result, respondents tend to over-report or under-report to save face.
We’ve all responded to items in a matrix format, with categories across the top from which we select. Try to use behaviors as quantifiers (e.g., often, sometimes and always are adverbs, not behaviors; strongly agree, somewhat agree are behaviors).
Should we use an even or odd number of quantifiers; should we have a midpoint in our survey scale? Research suggests it is better to offer an even number of scaled response options. And, whether you offer a ‘neutral’ or ‘not applicable’ category depends on the topic of study. If it is necessary to include, then do so. Otherwise do not, because you are conducting the survey to gather opinions, so force people to make a choice and not wimp out.
Once items are worded, pretest the questionnaire with people seated in the same room. Discuss it afterword and revise the questionnaire accordingly. Then pretest it again – online if that’s the method of dissemination. Remember, there is no one right way to word an item. It depends on your intended outcomes and questionnaire context.
At a loss for just the right words? Need help formatting questionnaire items? Let the experts at NBRI help. Contact us now at 1-800-756-6168.
Terrie Nolinske, Ph.D.
Research Associate
National Business Research Institute
Addison, Texas
The State of Casino Gaming
Employee Engagement: Leading by Example
Surveys: Do You Need an Expert?